The Dual-Write Problem — Two Writes, No Safety Net

A note on what this is. First of my distributed-systems notes. The series starts here, with the most common reliability bug in event-driven systems — because half of distributed-systems design exists to work around it, and every fix worth knowing makes more sense once this bug is felt properly.

Picture the incident this bug produces, because it always looks the same. Flash-sale afternoon. Every checkout returns 200 OK. The dashboards are green, the database shows orders piling up beautifully — and the warehouse has picked nothing for three hours. No errors anywhere, because nothing errored: the service crashed and restarted a handful of times under load, and a few of those crashes landed in the half-millisecond between two innocent-looking lines. Support finds out when the customers do.

Here are the two lines:

public void markPaid(Order order) {
    order.pay();
    orders.save(order);                          // write #1 — our database
    broker.publish(new OrderPaid(order.id()));   // write #2 — everyone else's view of the world
}

Write #1 goes to our database. Write #2 is a network call to the message broker (a separate server whose whole job is delivering announcements — Kafka, RabbitMQ, SQS) where the warehouse, email, and loyalty services are listening. Two writes, two different machines, and no transaction in existence covers both. That gap has a name — the dual-write problem — and it’s the most common reliability bug in event-driven systems.

The problem, stated plainly: one business action needs two writes — save the fact, announce the fact — and no transaction covers both systems, so a crash between the writes leaves your database and the rest of the company believing different things.

The everyday version: you finish your part of a group assignment at 2 a.m., hit save, and your laptop dies before the “done — your turn” message sends. The file is safe. Nobody knows. The teammate sleeps through the deadline. The work didn’t fail and the message app didn’t fail — the gap between them did.

Here it is happening — watch the trapdoor open:

The cruel part is where this bug lives. Not in either line — each line is correct. Not in any test — tests don’t crash mid-method. Not in code review — reviewers read lines, and the bug is the space between two lines. It only exists under a crash, a deploy, an OOM kill, a pod reschedule — and then it fails silently, because from each system’s point of view, nothing went wrong.

All four ways a run can end, in one look:

write #1 (DB)	write #2 (broker)	You get	Anyone notified?
✓	✓	the happy path	—
✓	✗ (or crash between)	the ghost order — paid, real, invisible to every downstream system	silence
✗	✓ (if you flip the lines)	the phantom event — the warehouse ships an order that doesn’t exist	silence
✗	✗	a clean failure — the caller sees an error and retries	at least it’s honest

Flipping the order of the two lines doesn’t fix anything — it just moves you from row two to row three, trading the ghost order for the phantom event. Whichever line goes first, the crash between them leaves the two systems telling different stories.

Why every obvious fix fails#

Everyone meets this bug and reaches for the same four fixes, in the same order. Each has a hole, and the holes are the actual lesson.

Fix attempt 1: try/catch and retry later.

orders.save(order);
try {
    broker.publish(new OrderPaid(order.id()));
} catch (Exception e) {
    retryQueue.add(order.id());   // in-memory list, retried every 30s
}

This handles the polite failure — the broker says no, the catch fires, the retry list saves the day. Watch what happens to that plan against the failure that actually causes incidents:

Name the real flaw: the intention to publish lived only on the call stack and in process memory — the program’s short-term memory — and that memory is wiped on crash. A retry plan that lives in the same process as the crash is no plan.

Fix attempt 2: undo the save when the publish fails.

orders.save(order);
try {
    broker.publish(new OrderPaid(order.id()));
} catch (Exception e) {
    orders.delete(order);   // pretend it never happened
}

Same hole — a hard crash skips the catch — plus a new one: the order was committed for a moment, and committed data gets seen. A report, a cache, the customer’s own orders page may have already read it. Un-saving a fact that others may have observed isn’t an undo; it’s a second lie.

Fix attempt 3: put both in one transaction.

@Transactional
public void markPaid(Order order) {
    orders.save(order);
    broker.publish(new OrderPaid(order.id()));   // feels covered. Isn't.
}

This one feels airtight, which is what makes it dangerous — it ships. A transaction is a database feature: the both-or-nothing fence is built by the DB engine around its own writes. The broker never joined the transaction — it can’t. If the transaction rolls back after the publish went out: phantom event. If the process dies after commit, before publish: ghost order. The annotation didn’t close the gap; it moved it.

Fix attempt 4: fine — give me a transaction that spans both systems. That exists. It’s called two-phase commit (2PC) — the “official” distributed transaction. A coordinator asks every participant to prepare (get ready and promise you can commit), and only when all say yes does everyone commit:

It’s worth knowing why the industry walked away from 2PC for this job. While prepared, every participant sits blocked, holding locks, waiting for a coordinator that might have just died. The coordinator is a new single point of failure attached to every single write. And the practical killer: most of what you’d want in the transaction — Kafka as normally used, SQS, plain HTTP APIs, most cloud services — simply never implemented the protocol. You can’t enroll a participant that never agreed to the rules.

So the score: retry plans die with the process, undo lies to observers, one transaction can’t cover two systems, and the official two-system transaction is one nobody can join. The gap is real, and no amount of care inside markPaid closes it.

The tell: it’s not just Kafka#

The shape to recognize is one method, two systems that can’t share a transaction. The broker version is the famous one, but the same trapdoor is under:

• Save to the DB + call a payment API — charge the card, crash before saving the charge: money moved, no record of it. (Payment providers’ docs are obsessed with this exact gap; it’s what their idempotency keys exist for.)
• Write to two databases — the orders DB and the search index, the cache, the analytics store. Every “why is Elasticsearch out of sync with Postgres” ticket is this bug wearing a different shirt.
• Save + send an email — the confirmation that goes out for an order whose transaction then rolled back.

Grep your codebase for methods that write to the database and talk to anything else over the network before returning. Each one is standing on the trapdoor. The only questions are how often it opens, and whether anything notices when it does.

The shape of every real exit#

Here’s the honest headline, and it’s one sentence: every real fix stops doing two writes.

Back to the 2 a.m. group project, because the fix lives there too. The mistake was trusting your own memory to send the message after saving. The fix: in the same motion as hitting save, you slap a sticky note on your desk — “send the group link.” Now if you pass out, the note doesn’t. Your roommate wanders in at 9, sees it, sends the text, bins the note. Two things make that work: the note is created in the same motion as the save — you can’t end up with one and not the other — and it gets delivered by someone who wasn’t in the crash.

In system terms: do one write, to one system, atomically — the order and the sticky note (the event, stored as a row) in the same database transaction — and let a separate, relentless process deliver the note, retrying until it succeeds:

And here’s the opening snippet, fixed — same method, same annotation, one crucial eviction:

@Transactional
public void markPaid(Order order) {
    order.pay();
    orders.save(order);                                      // write #1 — the fact
    outbox.save(new OutboxEvent("OrderPaid", order.id()));   // write #2 — the sticky note
}   // no broker.publish() anywhere in sight — that's the relay's job now

This time @Transactional is telling the truth: both saves are rows in the same database, so the fence genuinely covers both — they commit together or not at all. The broker call didn’t get wrapped in more care; it got evicted from the request entirely and handed to the relay.

The named versions of that shape — three competing strategies, each with its own notes:

• Transactional outbox — the sticky note as a table: the event row goes into an outbox table in the same transaction as the order, and a relay polls the table and publishes. The reach-for-it-first fix: one table, one loop, no new infrastructure.
• Change data capture (CDC) — no polling, no extra table needed: the relay tails the database’s own crash-recovery journal (the write-ahead log), so every committed change becomes an event automatically. Debezium is the famous tool — and the same stream also syncs search indexes, caches, and warehouses, so this one outgrows the bug.
• Event sourcing — the radical one: the event log is the database, so there was only ever one write to begin with — the bug becomes unrepresentable. Comes with an audit trail and time travel; costs a whole different data model.

The first two differ only in how the relay reads, and the trade is clean:

Relay style	How it reads	Wins	Costs
Polling	queries the outbox table every second or so	zero new infrastructure — it’s just a loop	query load on your DB, a little added latency
CDC (Debezium)	tails the DB’s write-ahead log	near-zero latency, zero query load	a whole Kafka Connect setup to run and monitor

And the tax the naive code never paid: the announcement now arrives a little later (eventual consistency), and sometimes more than once — the relay that crashes after publishing but before marking the row sent will publish again. Which is why idempotency stops being a vocabulary word here and becomes load-bearing: an operation is idempotent when running it twice is safe — the way pressing the elevator button five times summons one elevator, not five. The relay will eventually deliver something twice; every consumer has to be an elevator button about it:

public void onOrderPaid(OrderPaidEvent event) {
    if (processed.contains(event.id())) return;   // seen this one — one elevator is enough
    processed.add(event.id());
    warehouse.ship(event.orderId());
}

(In production, processed is a table written in the same transaction as the shipping side effect — the mirror image of the outbox, called the inbox, and it earns its own notes.) That thread runs through everything here.