A user clicks “Pay.” The money leaves their card. Then the network hiccups and the client times out. The user, seeing no confirmation, clicks “Pay” again.

Now you’ve charged them twice.

This is the most common failure I see in payment, order, and booking systems, and it is almost never a payments bug. It is a system design bug. The charge happened, the confirmation got lost, and the retry had no idea the first attempt already succeeded.

The instinct is to “just stop the user from double-clicking.” That doesn’t work. Real retries don’t come from angry fingers. They come from:

• Network timeouts that the client can’t tell apart from real failures.
• Server crashes mid-request, before the response is sent.
• Message brokers redelivering the same event because the consumer acked late.
• Background workers retrying a job after a deadlock.
• Webhooks from Stripe or a bank arriving more than once.
• Two requests racing on the same key at the same instant.

You can’t make any of those go away. What you can do is design the workflow so that the same operation arriving many times produces the business result exactly once. That’s what this article is about.

What “retry-safe” actually means

The goal is small enough to fit on a sticky note:

The same operation may be received many times, but the business result happens once.

Notice I didn’t say “the request runs once.” In a distributed system you will absolutely process the same intent more than once. The trick is to make the effect happen once. That’s idempotency, and it’s a property of your design, not a setting you flip on.

There are four layers that get you there, and each one covers a different gap:

1. Idempotency keys at the API so a retried request returns the first result.
2. Transactional outbox so publishing an event doesn’t get lost when the database write succeeds but the broker call fails.
3. Inbox table so a consumer ignores a message it already handled.
4. Saga so a multi-step flow can undo earlier steps when a later one fails.

Skip any one of them and you get a hole. Idempotency keys don’t help you if the event was never published. The outbox doesn’t help you if the consumer applies the same event twice.

Idempotency keys at the API layer

This is the layer closest to the client, so it’s where most double-charges are actually stopped. The client generates a key before it sends the request and stamps it on every retry.

POST /api/orders
Idempotency-Key: pay-req-user-42-cart-7

On the server side the rule is simple:

1. The client sends an Idempotency-Key header.
2. The server stores the key the first time it sees it, linked to a hash of the request, the response status, and the response body.
3. Any later request with the same key returns the stored response, without doing the work again.

Stripe’s API docs describe exactly this behavior, and it’s worth copying. For a given idempotency key, Stripe saves the status code and body of the first request and replays that same result on later requests with the same key. They also reject the key if you reuse it with a different request body, because that almost always means a client bug.

Here is a handler that does it. The important part is the atomic insert of the key and the early return when the key is already settled:

def handle(conn, *, idempotency_key, method, path, body, work):
    r_hash = request_hash(method, path, body)

    # Key already seen? Replay and verify it's the same request.
    row = conn.execute(
        "SELECT status, response_status, response_body, request_hash "
        "FROM idempotency_keys WHERE idempotency_key = ?",
        (idempotency_key,),
    ).fetchone()

    if row is not None:
        if row["request_hash"] != r_hash:
            return 422, {"error": "idempotency key reused with a different request"}
        if row["status"] == "processing":
            return 409, {"error": "request in progress"}
        return row["response_status"], json.loads(row["response_body"])

    # New key. Reserve it atomically; the UNIQUE constraint is our race guard.
    try:
        with transaction(conn):
            conn.execute(
                "INSERT INTO idempotency_keys "
                "(idempotency_key, request_hash, status, created_at, expires_at) "
                "VALUES (?, ?, 'processing', ?, ?)",
                (idempotency_key, r_hash, now, expiry),
            )
    except sqlite3.IntegrityError:
        return 409, {"error": "request in progress"}

    # Do the real work once, then store the result under the key.
    status_code, response = work()
    with transaction(conn):
        conn.execute(
            "UPDATE idempotency_keys SET status=?, response_status=?, response_body=? "
            "WHERE idempotency_key = ?",
            (new_status, status_code, json.dumps(response), idempotency_key),
        )
    return status_code, response

Two details matter. First, storing the request hash lets you catch key reuse with a changed body and reject it with a 422 instead of silently returning the wrong thing. Second, the processing state is what protects you from two in-flight requests with the same key both running. If one is already running, the other gets a 409 and retries later.

One honest caveat: idempotency keys only protect the calls that carry them. A downstream cron job or a manual admin action that bypasses the API gets no protection from this layer. That’s fine. It just means the key is one layer, not the whole story.

The dual-write problem and the transactional outbox

Now imagine the order was created and you need to tell the rest of the system: “an order exists.” You write the order row and you publish an event. This is the dual-write problem, and it bites almost everyone.

Your service does two writes that are not atomic with each other:

1. Insert the order into the database.
2. Publish OrderCreated to the broker.

One of them will fail first, eventually. If the database write commits and the broker call throws, you have an order nobody knows about. If the broker publishes and the database write rolls back, you have an event for an order that doesn’t exist. Both are silent data corruption, and both show up at 2 a.m.

AWS’s Prescriptive Guidance describes the transactional outbox as the fix for exactly this. You stop publishing directly from the request path. Instead, you write the business row and the event in the same database transaction, into an outbox_events table.

CREATE TABLE outbox_events (
    event_id     TEXT PRIMARY KEY,
    event_type   TEXT NOT NULL,   -- OrderCreated
    aggregate_id TEXT NOT NULL,   -- order_id
    payload      TEXT NOT NULL,   -- JSON
    status       TEXT NOT NULL DEFAULT 'pending', -- pending | published
    created_at   TEXT NOT NULL,
    published_at TEXT
);

The order service now does this in one transaction:

def create_order(conn, *, idempotency_key, user_id, amount_cents, items):
    def work():
        order_id = str(uuid.uuid4())
        with transaction(conn):
            conn.execute(
                "INSERT INTO orders (order_id, user_id, amount_cents, status, idempotency_key) "
                "VALUES (?, ?, ?, 'created', ?)",
                (order_id, user_id, amount_cents, idempotency_key),
            )
            conn.execute(
                "INSERT INTO outbox_events (event_id, event_type, aggregate_id, payload) "
                "VALUES (?, 'OrderCreated', ?, ?)",
                (str(uuid.uuid4()), order_id, json.dumps({...})),
            )
        return 201, {"order_id": order_id, "status": "created"}
    return idempotent_handle(conn, idempotency_key=idempotency_key, ...)

Because both inserts share one transaction, they either both happen or neither does. No orphan orders, no phantom events.

Then a separate relay worker sweeps the pending rows and publishes them:

def sweep_once(conn):
    pending = conn.execute(
        "SELECT * FROM outbox_events WHERE status='pending' ORDER BY created_at"
    ).fetchall()
    for row in pending:
        publish(row)  # send to Kafka/RabbitMQ using row["event_id"] as the key
        with transaction(conn):
            conn.execute(
                "UPDATE outbox_events SET status='published', published_at=? WHERE event_id=?",
                (now, row["event_id"]),
            )

If the relay crashes after publishing but before the UPDATE, the next sweep sees the event as still pending and publishes it again. That’s why the broker should dedupe on event_id, or at least tolerate a duplicate. The outbox makes publishing reliable. It does not make it happen exactly once. We’ll handle the “apply once” side next.

The inbox table for safe consumption

Here’s the part people miss. The outbox makes sure the event leaves. It does not make sure it arrives once. microservices.io is blunt about this: a message relay may publish a message more than once, so consumers must be idempotent, usually by tracking message IDs.

So the payment service, when it gets OrderCreated, needs its own memory of what it has processed. That’s the inbox table:

CREATE TABLE inbox_messages (
    message_id   TEXT PRIMARY KEY,
    consumer     TEXT NOT NULL,   -- which service processed it
    status       TEXT NOT NULL DEFAULT 'done', -- done | poison
    processed_at TEXT NOT NULL
);

The consumer checks the inbox first. If the message id is already there, it skips the work but still acks, so the broker stops redelivering:

def consume_order_created(conn, *, consumer, message_id, payload):
    seen = conn.execute(
        "SELECT status FROM inbox_messages WHERE message_id=? AND consumer=?",
        (message_id, consumer),
    ).fetchone()
    if seen is not None:
        return {"handled": False, "reason": f"duplicate ({seen['status']})"}

    with transaction(conn):
        # ... apply business change (charge the card) ...
        conn.execute("INSERT INTO inbox_messages (message_id, consumer, status) VALUES (?, ?, 'done')",
                     (message_id, consumer))
    return {"handled": True}

Two things to call out. The check and the business write must be in the same transaction, or a crash between them leaves you exposed. And you should record poison messages separately: events that fail every time for a real reason, not a transient one. Ack them too, but mark them so an operator can find them instead of watching the consumer retry forever.

Outbox and inbox are two different jobs. The outbox answers “did we tell anyone?” The inbox answers “did we already act?” You need both because the failure modes are different.

Sagas and compensation, only where you need them

So far every step stood alone. Orders get created, events get published, payments get captured — each is its own local transaction. But real workflows chain steps: reserve stock, charge the card, send the receipt. If the card charge fails after stock was reserved, you don’t want stock stuck reserved forever.

A saga is the pattern for that. It’s a sequence of local transactions. Each step commits on its own database. If a later step fails a business rule, compensating actions undo the earlier steps, in reverse. microservices.io defines it the same way: local transactions where each step updates its database and triggers the next, and failed business rules lead to compensating transactions.

You do not need a saga for a single write. You need it the moment you have two or more steps that each have side effects and can’t be rolled back by one database transaction, because they live in different services or different databases.

The bookkeeping is a small state machine:

CREATE TABLE saga_instances (
    saga_id      TEXT PRIMARY KEY,
    order_id     TEXT NOT NULL,
    status       TEXT NOT NULL,  -- in_progress | completed | compensated
    current_step TEXT NOT NULL,
    updated_at   TEXT NOT NULL
);

CREATE TABLE saga_steps (
    saga_id      TEXT NOT NULL,
    step_index   INTEGER NOT NULL,
    step_name    TEXT NOT NULL,  -- reserve_stock | charge_card
    status       TEXT NOT NULL,  -- done | compensated
    compensated_at TEXT,
    PRIMARY KEY (saga_id, step_index)
);

When charge_card fails, you run the compensations for the steps that already succeeded:

def compensate(conn, *, saga_id, compensations):
    steps = conn.execute(
        "SELECT step_index, step_name FROM saga_steps "
        "WHERE saga_id=? AND status='done' ORDER BY step_index DESC",
        (saga_id,),
    ).fetchall()
    for step in steps:
        fn = compensations.get(step["step_name"])
        if fn:
            fn()  # e.g. release the reserved stock
        conn.execute(
            "UPDATE saga_steps SET status='compensated', compensated_at=? "
            "WHERE saga_id=? AND step_index=?",
            (now, saga_id, step["step_index"]),
        )

Compensation is not a rollback. You can’t undo a sent email, and you usually can’t unsend a bank transfer the same way you started it. Compensation is a new business action: issue a refund, release the hold, mark the order cancelled. Design those reverse operations on purpose. The teams that treat sagas as “distributed transactions” are the ones that get surprised at 2 a.m. when a compensation itself fails and nobody planned for that.

The full reference design, end to end

Put the four layers together and an order flow looks like this:

1. The client submits the order with an idempotency key.
2. The API stores the key (and a hash of the request) before doing anything.
3. The order service creates the order and writes OrderCreated to the outbox in one transaction.
4. The relay publishes OrderCreated to the broker, then marks it published.
5. The payment service consumes the event, but only after checking its inbox by message id.
6. The payment result is stored and a PaymentCaptured outbox event is emitted.
7. The order status is updated safely, in its own transaction.
8. If the client retried the original POST, it gets the same order back, not a second one.

The piece that ties it together in the sample repo is a short demo that fires the request twice, runs the relay, redelivers the event once, and triggers a saga compensation. Every layer is exercised:

# Retry with the same key -> same order, no second charge.
s1, b1 = create_order(conn, idempotency_key=key, ...)
s2, b2 = create_order(conn, idempotency_key=key, ...)
assert b1["order_id"] == b2["order_id"]

# Relay publishes the OrderCreated event.
run_relay(conn)

# Broker redelivers -> inbox skips it.
first  = consume_order_created(conn, consumer="payment-svc", message_id="msg-1", payload)
again  = consume_order_created(conn, consumer="payment-svc", message_id="msg-1", payload)
assert again["handled"] is False

That’s the whole point in one screen. The retry is safe because of the key. The publish is safe because of the outbox. The redelivery is safe because of the inbox. And if a later step fails, the saga cleans up.

Where teams get this wrong

A few patterns I’d avoid:

• Relying on “at most once” delivery. Brokers lie about this under load. Assume redelivery and design for it.
• Putting the idempotency check outside the transaction. Race conditions live exactly at the boundary you didn’t protect.
• Using the outbox without an inbox. You moved the duplication problem downstream instead of solving it.
• Keeping idempotency keys forever. They’re not an audit log. Expire them (24 hours is a common choice) and run a cleanup job.
• Treating sagas like ACID transactions. There is no global rollback. There are only compensations, and they can fail too.

The full, runnable version of every snippet here — schema, handlers, relay, consumer, saga, and the demo — is in the linked repo. It runs on plain SQLite, so you can step through it without standing up a broker.

Retry safety isn’t one trick. It’s a habit of asking, for every write, “what happens when this arrives again?” Answer that question at each layer, and the double-charges, duplicate orders, and 2 a.m. pages mostly stop showing up.