Delivered Twice — the Idempotent Consumer

At 14:02:11, OrderPaid — event evt_9f3k — arrives at the warehouse service. The handler flips the order to PICKING, creates a pick task, and a box heads for the door. At 14:02:43 the same event arrives again: same id, same payload, same everything. The handler runs again. Two boxes, one order, one very confused customer.

Nothing malfunctioned. The outbox post ended by signing a contract — at-least-once: never a lost event, occasionally a duplicate — and this is what that contract looks like from the consumer’s side of the table. The broker kept every promise it made. The promise just never included “exactly once.”

Where the second delivery comes from#

The outbox relay’s publish-then-stamp gap is one birthplace of duplicates, but the bigger one lives inside the consumer itself. Every broker runs the same loop with you: it hands you a message, you do your work, and then you acknowledge — a tiny “done, don’t send that again” reply. The gap is that the work and the acknowledgment are two steps. Crash after the work, before the ack, and the broker sees only silence — so it does the only safe thing it can: it hands the message to someone again.

Every messaging system has this gap, wearing its local costume:

The pattern in the last column is the pattern of the whole problem: an acknowledgment is itself a message, and messages can be lost. A broker that waited for certainty before redelivering would sometimes wait forever, so every serious system picks the recoverable failure — deliver again — over the unrecoverable one. Exactly-once delivery isn’t something a broker withholds from you; it’s something the physics of two machines and a wire doesn’t offer. Exactly-once processing is buildable — but it gets built at your end.

And the duplicate is chosen, not suffered. A consumer could ack first and work second — then nothing is ever delivered twice. But crash between the ack and the work, and the message is gone forever: the broker was told “done” about work that never happened. That ordering has a name too — at-most-once — and nobody picks it for anything that matters, because a duplicate can be detected and neutralized, while a loss is invisible and permanent. Work-first is the deliberate trade: duplicates are the fee for never losing anything.

The problem post already sketched the consumer’s move — the elevator-button check, if (processed.contains(event.id())) return; — and as a sketch it’s exactly right. As running code it has a hole you could ship two boxes through: processed is a Set in memory, and a set in memory is a bouncer memorizing faces. Works great all evening — worthless the moment the shift changes. The crash that causes the redelivery is the shift change: the service restarts with a new bouncer who has seen nobody, evt_9f3k walks up looking brand new, and in it goes. Two pods break it without any crash at all: two doors, two bouncers, and each door’s regulars are strangers at the other.

Clubs solved this before software did: don’t trust the bouncer’s memory — put the guest list on a clipboard and tick names off as people enter. The tick and the admission happen in one motion, by the same hand; the clipboard outlives every shift change; and both doors work off the same list. Everything below is that clipboard, built out of the one thing this series keeps reaching for because it survives crashes: your own database.

Writes that don’t count how often they run#

Before adding any machinery, check whether the work is already safe to repeat. Some writes simply don’t accumulate:

// runs twice → same end state; the second run changes nothing
jdbc.update("UPDATE orders SET status = 'PICKING' WHERE id = ?", event.orderId());

// insert-or-do-nothing: the second run hits the primary key and shrugs
jdbc.update("""
    INSERT INTO pick_tasks (order_id, station) VALUES (?, ?)
    ON CONFLICT (order_id) DO NOTHING
    """, event.orderId(), station);

Setting a value to an absolute (status = 'PICKING') is idempotent by shape. Adding to a value is not — and that’s the boundary to memorize:

jdbc.update("UPDATE accounts SET points = points + 50 WHERE id = ?", ...);   // twice = 100 points

Increments, appends, sends, charges — anything where the second run stacks on the first — can’t be rescued by careful phrasing. For those, the handler needs an actual memory of what it has seen.

The inbox — the outbox’s reflection#

A note on names, because this idea wears two. Idempotent consumer names the property being delivered; the inbox pattern names the shape it’s built in — a table of already-seen ids, mirroring the outbox. Same thing. And it’s the idea you’ve already met from the other direction if you’ve ever sent an Idempotency-Key header to an API: someone at every unreliable hop keeps a durable list of “seen it” ids. At that hop, the API keeps the list for you. At this hop, you’re the one keeping it.

The outbox made save the fact and announce the fact one atomic write on the producer’s side. The inbox is the same move reflected to the consumer’s side: make record that I handled this event and do the handling one atomic write.

CREATE TABLE processed_messages (
    handler   TEXT NOT NULL,          -- 'warehouse' — each handler dedupes for itself
    event_id  TEXT NOT NULL,          -- 'evt_9f3k'
    seen_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (handler, event_id)
);

@Transactional
void onOrderPaid(OrderPaid event) {
    int fresh = jdbc.update("""
        INSERT INTO processed_messages (handler, event_id) VALUES ('warehouse', ?)
        ON CONFLICT DO NOTHING
        """, event.id());
    if (fresh == 0) return;   // seen it — ack and walk away

    jdbc.update("UPDATE orders SET status = 'PICKING' WHERE id = ?", event.orderId());
    jdbc.update("INSERT INTO pick_tasks (order_id) VALUES (?)", event.orderId());
}

Walk the two deliveries through it. First delivery: the INSERT lands, fresh == 1, the real work runs, and — because of @Transactional — the inbox row and the order update commit or vanish together. If the pod dies mid-handler, both roll back, the redelivery finds no inbox row, and the work runs cleanly the second time. Second delivery: the INSERT collides with the primary key, fresh == 0, the handler returns, the message gets acked, nothing else happens. The duplicate didn’t get rejected — it got completed, instantly, because its work was already done.

Two details in that handler are load-bearing, and both are easy to get backwards:

• Insert first, don’t check first. The tempting version — SELECT to see if the id exists, then proceed — has a race: two pods receive the duplicate at the same moment, both SELECT nothing, both proceed. The INSERT with a primary-key collision is the lock; the database serializes the two attempts and exactly one wins.
• The handler column matters. The same OrderPaid fans out to warehouse, email, and loyalty. Each keeps its own inbox rows — otherwise the first service to touch the event would steal it from the other two.

All of this leans on one assumption: the event carries a stable id. Brokers and serious webhook providers do (Stripe’s evt_… survives every retry; GitHub sends an X-GitHub-Delivery header). When an upstream doesn’t, derive the key — a SHA-256 of the raw request bytes works — but hash the bytes you received, never a re-serialized copy: JSON libraries reorder keys, and a reordered body is a different hash pretending to be a different event.

The chore that comes with it: the table grows by one row per message, forever, unless purged. The purge window just has to outlive the longest redelivery horizon of anything upstream — Stripe retries for 72 hours, so keeping a week of rows means a retry can never outlive your memory of it. A nightly DELETE WHERE seen_at < now() - interval '7 days' closes the loop. (Redis SET NX with an expiry is the popular shortcut for this job — dedup with free cleanup — but it lives outside your database transaction, which quietly reopens the exact gap the inbox closed: the Redis write and the DB commit become two steps again. Fine for cheap, repeat-tolerant work; for anything that counts, the clipboard belongs in the same database as the work it guards.)

Testing it is pleasantly brutal: replay the same event five times and assert five acks, one pick task. Then change the id and assert a second task — new event, new work.

Past the edge of the transaction#

Everything above works because the handler’s side effects were rows in the same database as the inbox — one transaction could hold them all. The moment the handler sends an email or calls the payment provider, that umbrella is gone: no transaction on earth covers “my Postgres commit and Stripe’s servers.” Crash between the external call and the commit, and the redelivery will find no inbox row and make the call again.

The move at this edge is to hand the deduplication to the other side: pass the event id along as an idempotency key, and let the downstream system keep the inbox. Stripe’s API takes an Idempotency-Key header, remembers the first response for any given key for 24 hours, and replays that stored response — instead of charging again — when a retry shows up with the same key. Email APIs offer the same under different names. The handler stays dumb about retries; it just never invents a fresh key for a repeat attempt:

stripe.charges().create(
    chargeParamsFor(event),
    RequestOptions.builder().setIdempotencyKey(event.id()).build()   // the event id travels
);

sequenceDiagram
  participant W as warehouse handler
  participant S as Stripe
  W->>S: POST /charges, Idempotency-Key: evt_9f3k
  S-->>W: 201 created, charge ch_88
  Note over W: crash before the local commit
  W->>S: POST /charges, Idempotency-Key: evt_9f3k
  Note over S: key seen 4 min ago:<br/>replay the stored response
  S-->>W: 201 created, the same charge ch_88

Being the Stripe of the story#

Flip the last section around: your own API has endpoints that clients retry — a mobile app resubmits a payment because the response died in a tunnel — and now you hold the clipboard. Accepting an Idempotency-Key header is the inbox again, with two duties the message consumer never had.

Duty one: store the answer, not just the tick. A duplicate event just needs swallowing — the broker only wants an ack. But an HTTP retry arrives because the client never got the response, so a bare “already seen” is useless to it. The row has to keep what the first attempt said, and the retry gets that exact answer replayed:

CREATE TABLE idempotency_keys (
    client_id  TEXT NOT NULL,
    idem_key   TEXT NOT NULL,
    status     INT,             -- NULL while the first attempt is still running
    body       JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (client_id, idem_key)
);

The handler walks the same insert-first path: INSERT … ON CONFLICT DO NOTHING. Fresh row → do the work and write status + body into the row in the same transaction, then respond. Collision → read the row and replay its stored response — the retry can’t tell it wasn’t the original.

Duty two: the retry that races the original. The collision row can exist with status still NULL — the first attempt is mid-flight right now. There’s nothing to replay yet, and running the work a second time is the exact double-charge this whole post exists to prevent. Stripe’s answer is the honest one: return 409 with “a request with this key is already in progress,” and let the client try again in a moment. (Message consumers rarely meet this race — the broker’s visibility timeout spaces redeliveries out; HTTP clients retry in seconds.)

Scope the key per client so two customers can’t collide, and give the rows the same lifetime rule as the inbox: longer than any client’s retry horizon — Stripe keeps them 24 hours.

Idempotency is a relay race: the guarantee survives only while each hop carries the key forward, and it dies at the first hop that drops it. That’s also the honest reading of Kafka’s celebrated “exactly-once semantics” — real, but scoped: it covers Kafka-in, Kafka-out processing, where offsets and output records commit in one Kafka transaction. Touch Postgres or an HTTP API inside that flow and you’re back in this post.

The unsettled thing worth sitting with: the inbox makes this handler safe, and the outbox made that producer safe, but there is still no general, buy-it-off-the-shelf “exactly once across arbitrary systems” — every guarantee in this post was assembled by hand from a database transaction, a primary key, and a carried token. Whether that assembly can ever be packaged the way the outbox was is, as far as I can tell, still an open question.