Md Nayeem Hossain | Software Engineer

The Transactional Outbox Pattern: Reliable Events Without Distributed Transactions#

Here is a bug I have seen in almost every event-driven codebase: a service saves an order to the database, then publishes an OrderCreated event to a message broker. Most of the time it works. Then one day the process crashes between the two steps, and the order exists but no email is sent, no inventory is reserved and no invoice is created.

Or worse, the publish succeeds and the database commit fails, and downstream services act on an order that never existed.

This is the dual-write problem, and the transactional outbox is the standard way to solve it.

Why "just publish after commit" is not enough#

You cannot atomically commit to two different systems without a distributed transaction protocol like two-phase commit. Most brokers do not support it, and the ones that do make it slow and fragile. Any ordering of "write DB" and "publish event" leaves a crash window where one happens without the other.

The pattern#

Instead of publishing directly, write the event into an outbox table in the same database transaction as the business change. A separate relay process reads the outbox and publishes to the broker.

sql

1CREATE TABLE outbox (2  id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),3  aggregate_id uuid        NOT NULL,4  event_type   text        NOT NULL,5  payload      jsonb       NOT NULL,6  created_at   timestamptz NOT NULL DEFAULT now(),7  published_at timestamptz8);9 10CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

Now the business write and the event are one atomic unit:

1await dataSource.transaction(async (tx) => {2  const order = await tx.getRepository(Order).save({ customerId, items, total });3 4  await tx.getRepository(OutboxEvent).insert({5    aggregateId: order.id,6    eventType: "order.created",7    payload: { orderId: order.id, customerId, total },8  });9});

Either both rows exist or neither does. The crash window is gone.

The relay#

The relay polls for unpublished events, publishes them, and marks them done:

1async function relayBatch() {2  await dataSource.transaction(async (tx) => {3    const events = await tx.query(4      `SELECT * FROM outbox5       WHERE published_at IS NULL6       ORDER BY created_at7       LIMIT 1008       FOR UPDATE SKIP LOCKED`,9    );10 11    for (const event of events) {12      await broker.publish(event.event_type, event.payload, { messageId: event.id });13    }14 15    await tx.query(`UPDATE outbox SET published_at = now() WHERE id = ANY($1)`, [events.map((e) => e.id)]);16  });17}

FOR UPDATE SKIP LOCKED lets you run several relay instances without them grabbing the same rows — a simple and effective way to scale horizontally.

At-least-once, not exactly-once#

If the relay publishes an event and crashes before marking it published, the event goes out again on the next run. The outbox guarantees at-least-once delivery. That is the honest guarantee most real systems offer.

So consumers must be idempotent. The simplest approach is an inbox table keyed by the message ID:

1async function handleOrderCreated(message: Message) {2  await dataSource.transaction(async (tx) => {3    const fresh = await tx.query(4      `INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING message_id`,5      [message.id],6    );7    if (fresh.length === 0) return; // already handled8 9    await reserveInventory(tx, message.payload);10  });11}

The outbox on the producer side plus an inbox on the consumer side gives you effectively-once processing, which is what the business actually cares about.

Polling vs change data capture#

Polling is simple and good enough for most teams. If you need lower latency or very high throughput, change data capture tools such as Debezium read the database's write-ahead log and stream outbox inserts to Kafka without any polling query. It is more infrastructure, so I only reach for it when polling becomes a measurable bottleneck.

Operational details that matter#

Clean up. Delete or archive published rows after a few days, or the table and its index will grow forever.
Monitor lag. Alert on the age of the oldest unpublished event. A growing number means the relay is down or the broker is unhealthy.
Keep payloads self-contained. Include what consumers need, so they do not have to call back into your service and couple to its availability.
Version event schemas. Add an eventVersion field from day one. You will need it.

Conclusion#

The outbox pattern trades a small amount of latency for correctness: events can be delayed, but never lost and never invented. For any system where a missing or phantom event costs money or trust, that trade is always worth it.