---

Introduction

Imagine you're building an Order Service.

When a customer places an order, two things must happen:

1. Save the order into the database.
2. Publish an OrderCreated event to Kafka.

Sounds simple...

Save Order
      ↓
Publish Kafka Event

But what if this happens?

Order Saved ✅

↓

Kafka Server Down ❌

Now the order exists in the database,

but Inventory Service, Payment Service, and Notification Service never receive the event.

The system becomes inconsistent.

This problem is called Dual Write Problem.

The solution is the Transactional Outbox Pattern.

---

Learning Objectives

After completing this article, you'll understand:

• What is the Outbox Pattern?
• Why Dual Writes Fail
• Reliable Event Publishing
• Outbox Table Design
• Polling Publisher
• CDC (Change Data Capture)
• Debezium
• Kafka Integration
• Spring Boot Implementation
• Idempotent Consumers
• Retry Strategy
• Dead Letter Queue
• Best Practices
• Real-world Examples

---

The Dual Write Problem

Suppose the application performs:

saveOrder(order);

kafkaTemplate.send("orders", event);

Looks correct.

But these are two independent operations.

---

Failure Scenario 1

flowchart TD

ORDER[Save Order]

KAFKA[Publish Kafka Event]

FAIL[Kafka Failure]

ORDER --> KAFKA
KAFKA --> FAIL

Result

• Database updated ✅
• Event lost ❌

---

Failure Scenario 2

flowchart TD

EVENT[Kafka Event Published]

CRASH[Application Crash]

DATABASE[(Database)]

EVENT --> CRASH
CRASH --> DATABASE

Result

• Consumers process event
• Order does not exist

Again,

data becomes inconsistent.

---

Why Distributed Transactions Don't Help

Microservices own separate databases.

Kafka is not part of the database transaction.

Traditional ACID transactions cannot include Kafka safely.

---

What is the Outbox Pattern?

Instead of writing to Kafka immediately,

write both

• Business Data
• Event

inside one local database transaction.

Later,

publish events from the Outbox table.

---

High-Level Architecture

flowchart LR

CLIENT[Client]

ORDER[Order Service]

DB[(Database)]

OUTBOX[(Outbox Table)]

PUBLISHER[Outbox Publisher]

KAFKA[(Kafka)]

CLIENT --> ORDER

ORDER --> DB

ORDER --> OUTBOX

OUTBOX --> PUBLISHER

PUBLISHER --> KAFKA

---

Transaction Flow

sequenceDiagram

participant Client
participant Order
participant Database
participant Outbox

Client->>Order: Create Order

Order->>Database: Save Order

Order->>Outbox: Save Event

Database-->>Order: Commit

Outbox-->>Order: Commit

Order-->>Client: Success

Both records commit together.

---

Why It Works

Either

Order Saved

AND

Outbox Saved

OR

Nothing Saved

No partial success.

---

Outbox Table

Example

CREATE TABLE outbox_events(

    id UUID PRIMARY KEY,

    aggregate_id VARCHAR(100),

    aggregate_type VARCHAR(100),

    event_type VARCHAR(100),

    payload JSONB,

    status VARCHAR(20),

    created_at TIMESTAMP

);

---

Database Structure

flowchart TD

ORDERS[(Orders)]

OUTBOX[(Outbox Events)]

ORDERS --> OUTBOX

Both tables belong to the same database transaction.

---

Example Transaction

BEGIN

↓

Insert Order

↓

Insert Outbox Event

↓

COMMIT

One transaction.

Guaranteed consistency.

---

Polling Publisher

A background scheduler periodically checks the Outbox table.

flowchart LR

OUTBOX[(Outbox)]

POLLER[Polling Publisher]

KAFKA[(Kafka)]

OUTBOX --> POLLER

POLLER --> KAFKA

---

Polling Flow

sequenceDiagram

participant Scheduler
participant Outbox
participant Kafka

Scheduler->>Outbox: Read Pending Events

Outbox-->>Scheduler: Events

Scheduler->>Kafka: Publish

Kafka-->>Scheduler: Success

Scheduler->>Outbox: Mark Published

---

Event Status

Typical states

PENDING

↓

PUBLISHING

↓

PUBLISHED

↓

ARCHIVED

---

Spring Boot Scheduler

Example

@Scheduled(fixedDelay = 5000)
public void publishEvents(){

    List<OutboxEvent> events =
        repository.findPending();

}

Runs every few seconds.

---

CDC (Change Data Capture)

Instead of polling,

listen to database transaction logs.

---

CDC Architecture

flowchart LR

DATABASE[(PostgreSQL)]

BINLOG[Transaction Log]

DEBEZIUM[Debezium]

KAFKA[(Kafka)]

DATABASE --> BINLOG

BINLOG --> DEBEZIUM

DEBEZIUM --> KAFKA

No scheduler required.

---

Debezium

Debezium watches

• PostgreSQL WAL
• MySQL Binlog
• SQL Server CDC
• Oracle Redo Logs

Whenever a row is inserted,

Debezium publishes an event automatically.

---

Debezium Flow

sequenceDiagram

participant DB
participant WAL
participant Debezium
participant Kafka

DB->>WAL: Commit Transaction

WAL->>Debezium: Change

Debezium->>Kafka: Publish Event

Very popular in enterprise systems.

---

Polling vs CDC

Polling Publisher	CDC
Simple	Faster
Scheduler Required	No Scheduler
Database Queries	Reads Transaction Log
Easy Setup	More Infrastructure
Higher Latency	Near Real-Time

---

Kafka Architecture

flowchart TD

ORDER[Order Service]

OUTBOX[(Outbox)]

PUBLISHER[Publisher]

TOPIC[(Kafka)]

PAYMENT[Payment Service]

EMAIL[Notification]

INVENTORY[Inventory]

ORDER --> OUTBOX

OUTBOX --> PUBLISHER

PUBLISHER --> TOPIC

TOPIC --> PAYMENT

TOPIC --> EMAIL

TOPIC --> INVENTORY

---

Consumer Idempotency

Kafka guarantees

At Least Once Delivery

Consumers must safely ignore duplicate events.

Example

Event ID

↓

Already Processed?

↓

Ignore Duplicate

---

Retry Strategy

flowchart TD

EVENT[Publish Event]

SUCCESS{Success?}

RETRY[Retry]

DLQ[Dead Letter Queue]

EVENT --> SUCCESS

SUCCESS -->|Yes| DONE[Completed]

SUCCESS -->|No| RETRY

RETRY --> DLQ

---

Dead Letter Queue

Events that repeatedly fail are moved to a DLQ.

Benefits

• No event loss
• Easier debugging
• Safe retries

---

Event Ordering

If using Kafka,

events for the same aggregate should use the same partition key.

Example

OrderId = 1001

All order events go to one partition,

preserving order.

---

Spring Boot Architecture

flowchart TD

CLIENT[React]

ORDER[Spring Boot]

POSTGRES[(PostgreSQL)]

OUTBOX[(Outbox)]

DEBEZIUM[Debezium]

KAFKA[(Kafka)]

PAYMENT[Payment]

EMAIL[Notification]

CLIENT --> ORDER

ORDER --> POSTGRES

ORDER --> OUTBOX

OUTBOX --> DEBEZIUM

DEBEZIUM --> KAFKA

KAFKA --> PAYMENT

KAFKA --> EMAIL

---

Banking Example

Money Transfer

Debit Account

↓

Insert Outbox Event

↓

Commit

↓

Publish Event

↓

Ledger Update

The transfer is never published unless the debit transaction succeeds.

---

Amazon Example

When an order is placed,

Amazon records the order and corresponding domain event together.

Inventory, shipping, and notifications consume the event independently.

---

Uber Example

Ride Completed

↓

Save Trip

↓

Save Outbox Event

↓

Publish

↓

Generate Receipt

↓

Update Driver Earnings

---

Netflix Example

User finishes watching a movie.

↓

Watch History Saved

↓

Outbox Event Stored

↓

Recommendation Engine Updated

↓

Analytics Updated

---

Advantages

• Eliminates Dual Write Problem
• Reliable Event Delivery
• Strong Local Consistency
• Supports Event-Driven Architecture
• Easy Integration with Kafka
• Works with Saga Pattern

---

Challenges

• Additional Outbox Table
• Event Cleanup Required
• Duplicate Event Handling
• Monitoring Publisher
• More Operational Components

---

Monitoring

Monitor

• Pending Outbox Events
• Publishing Failures
• Retry Count
• DLQ Messages
• Kafka Publish Latency
• CDC Lag
• Debezium Health

Tools

• Prometheus
• Grafana
• Datadog
• Kafka UI
• Debezium UI
• CloudWatch

---

Common Mistakes

❌ Publishing directly after database commit

❌ Forgetting event cleanup

❌ Non-idempotent consumers

❌ No retry mechanism

❌ No monitoring of publisher failures

❌ Mixing multiple event types without versioning

---

Best Practices

• Store business data and outbox events in the same transaction.
• Use UUIDs for event identifiers.
• Keep event payloads immutable and versioned.
• Prefer Debezium CDC for high-throughput systems.
• Ensure consumers are idempotent.
• Archive or purge published outbox records periodically.
• Monitor outbox backlog and Kafka publish latency.

---

Outbox vs Direct Kafka Publish

Direct Publish	Outbox Pattern
Risk of dual writes	Reliable publishing
Possible event loss	No event loss after commit
Simpler	More robust
Harder to recover	Easy to retry
Not ideal for critical systems	Enterprise standard

---

Common Interview Questions

What problem does the Outbox Pattern solve?

It solves the Dual Write Problem, ensuring business data and events remain consistent.

---

Why not publish directly to Kafka?

If the database commit succeeds but Kafka publish fails (or vice versa), the system becomes inconsistent.

---

What is the difference between Polling Publisher and Debezium?

Polling	Debezium
Queries the Outbox table	Reads database transaction logs
Easier to implement	Lower latency
Uses scheduled jobs	Event-driven CDC

---

Can the Outbox Pattern guarantee exactly-once delivery?

No. The publisher or broker may retry. Instead, combine the Outbox Pattern with idempotent consumers to achieve correct business outcomes.

---

When should the Outbox Pattern be used?

Use it when:

• Publishing Kafka events
• Implementing Saga Pattern
• Building Event-Driven Microservices
• Integrating with external systems
• Ensuring reliable event delivery

---

Summary

The Transactional Outbox Pattern is the standard enterprise solution for reliably publishing events from microservices. By storing business data and events in the same database transaction, it eliminates the Dual Write Problem while supporting scalable, event-driven architectures.

In this article, we covered:

• Dual Write Problem
• Transactional Outbox Pattern
• Outbox Table
• Polling Publisher
• CDC
• Debezium
• Kafka Integration
• Retry Strategy
• Dead Letter Queue
• Idempotent Consumers
• Spring Boot Architecture
• Banking, Amazon, Uber, and Netflix examples
• Monitoring
• Best practices

When combined with the Saga Pattern, Kafka, CQRS, and Event-Driven Architecture, the Outbox Pattern provides a reliable foundation for building resilient, cloud-native enterprise applications.

---