Microservices Data Consistency: Sagas, Outbox and Observability

A customer buys a sofa. The stock service reserves one unit, the payment service charges the card, and then the delivery service reports that no slots are left for six weeks. The card is charged, the sofa is held, and no order exists that anyone can fulfil. In a monolith with one database, a rollback would have erased all of it. Across services there is no such button, and designing what happens instead is the heart of this lesson.

Why there is no distributed rollback

Each service owns its own database, so a single transaction cannot cover them. Protocols such as two-phase commit exist, but they hold locks across services, depend on every participant being available at the same moment and are rarely available across HTTP and message brokers. The practical approach is to accept that the system is temporarily inconsistent and to make sure it converges to a correct state. This is the same trade-off explored in CAP and consistency models.

Sagas: a chain of local transactions with undo steps

A saga replaces one big transaction with a sequence of local transactions. Each step has a compensating action that semantically undoes it (refund the card, release the stock). If a step fails, you run the compensations of the completed steps in reverse order. Compensation is business logic, not a database rollback: a refund is a new payment going the other way, and some effects (an email already sent) cannot be undone and need a follow-up message instead.

class StepFailed(Exception):
    pass


def reserve_stock(order):
    print(f"[stock]    reserved {order['qty']} x {order['sku']}")


def release_stock(order):
    print(f"[stock]    released {order['qty']} x {order['sku']}")


def charge_card(order):
    print(f"[payment]  charged {order['amount']}")


def refund_card(order):
    print(f"[payment]  refunded {order['amount']}")


def book_delivery(order):
    raise StepFailed("no delivery slots left")


# (name, action, compensation); the last step has nothing to undo yet.
SAGA = [
    ("reserve_stock", reserve_stock, release_stock),
    ("charge_card", charge_card, refund_card),
    ("book_delivery", book_delivery, None),
]


def run_saga(order):
    done = []
    for name, action, compensate in SAGA:
        try:
            action(order)
            done.append((name, compensate))
        except StepFailed as err:
            print(f"[saga]     {name} failed: {err}; compensating")
            for _, undo in reversed(done):
                if undo:
                    undo(order)
            return "CANCELLED"
    return "COMPLETED"


print(run_saga({"sku": "SOFA-2S", "qty": 1, "amount": "499.00"}))

[stock]    reserved 1 x SOFA-2S
[payment]  charged 499.00
[saga]     book_delivery failed: no delivery slots left; compensating
[payment]  refunded 499.00
[stock]    released 1 x SOFA-2S
CANCELLED

This demo runs in one process. In production, the saga's state (which steps completed) must itself be persisted, and compensations must be retried until they succeed and be idempotent. Two styles exist for coordinating the steps:

The dual-write problem and the outbox

A saga relies on services publishing events. But "save the order, then publish an event" is two separate writes to two systems. If the process dies between them, the order exists and nobody hears about it, or an event is announced for an order that was never saved. The transactional outbox fixes this: write the event into an outbox table in the same local transaction as the business row, and let a separate relay publish unsent rows to the broker.

import sqlite3, json

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE orders (id INTEGER PRIMARY KEY, sku TEXT NOT NULL, status TEXT NOT NULL);
CREATE TABLE outbox (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  event_type TEXT NOT NULL,
  payload TEXT NOT NULL,
  published_at TEXT
);
""")

# One local transaction writes the business row AND the event.
with db:
    db.execute("INSERT INTO orders (id, sku, status) VALUES (1, 'SOFA-2S', 'PLACED')")
    db.execute("INSERT INTO outbox (event_type, payload) VALUES (?, ?)",
               ("OrderPlaced", json.dumps({"orderId": 1, "sku": "SOFA-2S"})))

# A separate relay polls unpublished rows and sends them to the broker.
def relay_once(send):
    rows = db.execute(
        "SELECT id, event_type, payload FROM outbox WHERE published_at IS NULL ORDER BY id"
    ).fetchall()
    for row_id, event_type, payload in rows:
        send(event_type, payload)           # may fail -> row stays unpublished -> retried later
        with db:
            db.execute("UPDATE outbox SET published_at = datetime('now') WHERE id = ?", (row_id,))

relay_once(lambda t, p: print("published", t, p))
relay_once(lambda t, p: print("published", t, p))   # second pass finds nothing
print(db.execute("SELECT COUNT(*) FROM outbox WHERE published_at IS NULL").fetchone()[0], "unpublished left")

published OrderPlaced {"orderId": 1, "sku": "SOFA-2S"}
0 unpublished left

Notice that the relay can crash after sending but before marking the row, so the event may be sent twice. The outbox gives you at-least-once publishing, not exactly-once. That is acceptable only if the consumer is idempotent: it records the ids of events it has handled in the same transaction as its own effect.

import sqlite3

db = sqlite3.connect(":memory:")
db.executescript("""
CREATE TABLE processed_events (event_id TEXT PRIMARY KEY);
CREATE TABLE stock_holds (sku TEXT, qty INTEGER);
""")


def handle(event_id, sku, qty):
    with db:  # one transaction: the dedupe marker and the effect commit together
        cur = db.execute("INSERT OR IGNORE INTO processed_events (event_id) VALUES (?)", (event_id,))
        if cur.rowcount == 0:
            print(f"skip duplicate {event_id}")
            return
        db.execute("INSERT INTO stock_holds (sku, qty) VALUES (?, ?)", (sku, qty))
        print(f"applied {event_id}")


handle("evt-101", "SOFA-2S", 1)
handle("evt-101", "SOFA-2S", 1)  # broker redelivered the same message
print(db.execute("SELECT COUNT(*) FROM stock_holds").fetchone()[0], "hold row(s)")

applied evt-101
skip duplicate evt-101
1 hold row(s)

Seeing the whole request: observability

Once a single order spans five services, "check the logs" stops working. Three signals work together. Logs record what happened; make them structured (JSON) and include a correlation id generated at the gateway and passed to every downstream call and event, so one search returns the whole story. Metrics show aggregates over time; for each service start with request rate, error rate and duration (the RED method), plus queue depth for consumers. Traces record one request as a tree of timed spans across services, so you can see that the slow part of checkout was the stock lookup rather than the code you were about to optimise. OpenTelemetry is a widely used, vendor-neutral way to produce all three.

Common mistakes and the failure modes teams hit

• Publishing the event outside the transaction. Symptom: orders that exist but never trigger shipping, found weeks later by a customer. Use the outbox.
• Non-idempotent consumers. Symptom: duplicate stock holds or double emails after a broker redelivery or consumer restart.
• Compensations that can fail and are never retried. Symptom: money refunded in the ledger but not at the payment provider, discovered during reconciliation. Retry compensations and alert on stuck sagas.
• Cascading failure from missing timeouts and bulkheads. Symptom: one slow dependency makes unrelated endpoints fail across the system.
• The distributed monolith. Symptom: services share a database or must be deployed together. You pay the network cost and keep the coupling.
• Poison messages and unbounded queues. Symptom: one malformed message is retried forever and blocks a partition, or a queue grows silently until memory runs out. Use a dead-letter queue and alert on depth.

How this shows up in real projects and interviews

"How do you handle a transaction across services?" is a staple question, and a complete answer contains four ideas: no distributed rollback, a saga with compensations, an outbox so state and events cannot diverge, and idempotent consumers because delivery is at least once. In real projects, the hardest part is rarely the code in this lesson but deciding what the business wants when compensation is impossible, such as an email already sent. For delivery pipelines that make these services deployable, see CI/CD and the container tooling in Docker.