Enterprise Event-Driven System Design

Design a scalable Enterprise Event-Driven System — covering event broker architecture, topic design, consumer group patterns, event schema evolution, exactly-once semantics, Saga orchestration, outbox pattern, dead-letter queues, and the operational practices that keep event pipelines reliable in production.

1-Hour Interview Roadmap

Time	Topic
0 – 5 min	Requirements clarification
5 – 10 min	Capacity estimation
10 – 18 min	High-level architecture + broker topology
18 – 28 min	Producer patterns — outbox, idempotency, ordering guarantees
28 – 36 min	Consumer patterns — consumer groups, exactly-once, dead-letter queues
36 – 43 min	Saga pattern — choreography vs orchestration
43 – 50 min	Schema registry + event schema evolution
50 – 56 min	Operational concerns — monitoring, replay, backpressure
56 – 60 min	Trade-offs + common interview mistakes

What Are We Building?

An enterprise Event-Driven System (EDS) that serves as the nervous system of a distributed microservices platform — reliably transporting events between producers and consumers, enforcing schema contracts, enabling loose coupling, and providing the operational foundations (monitoring, replay, dead-letter handling) that keep the platform running in production at scale.

Event-driven architecture is not a single technology — it is a style of building systems where services communicate by publishing and subscribing to events rather than calling each other directly. This decoupling is enormously powerful: producers do not need to know who consumes their events, consumers can be added without modifying producers, and events can be replayed to rebuild state or onboard new consumers.

But the power comes with complexity. Events are harder to debug than synchronous calls. Failure modes are distributed and non-obvious. Ordering, deduplication, and exactly-once delivery are genuinely hard engineering problems. And schema evolution — changing event formats without breaking existing consumers — is an ongoing operational challenge that teams routinely underestimate.

Real-world scale context:

Platform	Events/day	Topics	Latency SLA
LinkedIn (Kafka)	7 trillion/day	1,000+	< 10 ms p99
Uber (Kafka)	1 trillion/day	4,000+	< 5 ms p99
Netflix	500 B/day	700+	< 20 ms p99
Airbnb	100 B/day	300+	< 50 ms p99
Mid-size fintech	1–10 B/day	50–200	< 100 ms p99

The five challenges that make this hard:

Exactly-once semantics — network failures cause duplicate deliveries; idempotent consumers are necessary but not sufficient. True end-to-end exactly-once requires coordination between the producer, broker, and consumer transactional state.
Event ordering — Kafka guarantees ordering within a partition, not across partitions. Systems that require global ordering must funnel through a single partition (a throughput bottleneck) or accept partial ordering and design for it.
Schema evolution — event consumers may be running old code when producers deploy a new schema. Backward and forward compatibility rules must be enforced at publish time via a schema registry.
Failure isolation — a poison pill message (malformed event) can stall a consumer group indefinitely. Dead-letter queues, retry policies, and consumer circuit breakers must be designed upfront.
Distributed transaction atomicity — writing to a database and publishing an event must be atomic. Without the outbox pattern, the system will occasionally write to the DB but fail to publish (or vice versa), causing inconsistency.

Functional Requirements

Event publishing:

Accept events from producers via Kafka client API or HTTP gateway
Validate event schema against the schema registry before accepting
Guarantee at-least-once delivery to all subscribed consumer groups
Support ordered delivery within an entity's partition key

Event consumption:

Support consumer groups with independent offsets per group
Provide at-least-once delivery with idempotency infrastructure for exactly-once behavior
Route unprocessable events to dead-letter topics with full context
Support consumer lag monitoring with alerting

Schema management:

Enforce Avro/Protobuf/JSON Schema registration for all topics
Validate compatibility (BACKWARD, FORWARD, FULL) on schema updates
Version schemas with global version IDs
Provide schema catalog for discovery

Saga orchestration:

Support choreography-based sagas (event-based state machines)
Provide orchestrator service for complex multi-step distributed transactions
Emit compensation events on saga failure

Operational:

Event replay from any offset or timestamp
Consumer group offset management (seek, reset, commit)
Topic retention policies (time-based and size-based)
Real-time consumer lag dashboard

Non-Functional Requirements

Attribute	Target
Publish latency	p50 < 5 ms, p99 < 50 ms
End-to-end latency	p99 < 100 ms (producer → consumer processing complete)
Throughput	500,000 events/second sustained
Durability	Replicated to 3 brokers; no data loss on single broker failure
Availability	99.95% (< 4 hrs downtime/year)
Message ordering	Guaranteed within partition key
At-least-once delivery	Guaranteed for all consumer groups
Retention	7 days hot; configurable per topic; compacted topics permanent
Schema compatibility	Breaking changes rejected at publish time

Capacity Estimation

Baseline assumptions:

500 producer services publishing events
1,000 consumer service instances across 200 consumer groups
Average event size: 1 KB
Event volume: 100 million events/day

Throughput:

100,000,000 / 86,400 = ~1,157 events/second average
Peak (3× average): ~3,500 events/second
Design target: 10,000 events/second (3× peak headroom)

Storage:

100M events/day × 1 KB = 100 GB/day
7-day retention: 700 GB raw
Replication factor 3: 700 GB × 3 = 2.1 TB
With compression (LZ4, ~3:1): ~700 GB total broker storage

Kafka broker sizing:

10,000 events/s × 1 KB = 10 MB/s write throughput
With 200 consumer groups each reading: 10 MB/s × 200 = 2 GB/s read throughput
Kafka sequential disk I/O: ~500 MB/s per broker
Brokers needed: ceiling(2,000 / 500) = 4 brokers minimum → deploy 6 (headroom + HA)

Schema registry:

~500 schemas registered, evolving at ~20 new versions/day
Schema validation: < 1 ms (in-memory cache after first fetch)
Schema registry: 2 instances (active-active) with Kafka as backing store

High-Level Architecture

flowchart TD
    subgraph Producers
        SVC_A[Service A]
        SVC_B[Service B]
        OUTBOX[Outbox Relay]
    end

    subgraph Broker
        KAFKA[Kafka Cluster - 6 brokers]
        SR[Schema Registry]
        ZK[ZooKeeper / KRaft]
    end

    subgraph Consumers
        CG_A[Consumer Group A]
        CG_B[Consumer Group B]
        DLQ[Dead-Letter Handler]
    end

    subgraph Orchestration
        SAGA[Saga Orchestrator]
        STATE[(Saga State Store)]
    end

    subgraph Operations
        LAG[Consumer Lag Monitor]
        REPLAY[Replay Service]
        SCHEMA_CAT[Schema Catalog UI]
        CH[(ClickHouse - event analytics)]
    end

    SVC_A -->|validate schema| SR
    SVC_A -->|publish| KAFKA
    SVC_B -->|publish| KAFKA
    OUTBOX -->|transactional relay| KAFKA
    KAFKA --> ZK

    KAFKA -->|consume| CG_A
    KAFKA -->|consume| CG_B
    KAFKA -->|DLQ topic| DLQ

    SAGA --> KAFKA
    KAFKA --> SAGA
    SAGA --> STATE

    LAG --> KAFKA
    REPLAY --> KAFKA
    KAFKA --> CH
    SR --> SCHEMA_CAT

Core Component Responsibilities

Component	Responsibility
Kafka Cluster	Durable, partitioned, ordered event log with consumer group offset tracking
Schema Registry	Schema storage, versioning, and compatibility enforcement per topic
Outbox Relay	Reads outbox table in producer DB, publishes to Kafka, marks as published
Consumer Group A/B	Independent offset tracking; process events at own pace
Dead-Letter Handler	Consume DLQ topics, alert on-call, support manual reprocessing
Saga Orchestrator	Drive multi-step distributed transactions, emit compensations on failure
Consumer Lag Monitor	Track per-group, per-partition lag; alert when lag exceeds threshold
Replay Service	Allow consumers to seek to a past offset and reprocess events
ClickHouse	Event analytics — throughput, latency, error rates, topic usage dashboards

Event State Machine (Saga)

stateDiagram-v2
    [*] --> SAGA_STARTED: initiating event published

    SAGA_STARTED --> STEP_1_PENDING: saga orchestrator emits step 1 command
    STEP_1_PENDING --> STEP_1_COMPLETE: step 1 service publishes success event
    STEP_1_PENDING --> SAGA_COMPENSATING: step 1 service publishes failure event

    STEP_1_COMPLETE --> STEP_2_PENDING: orchestrator emits step 2 command
    STEP_2_PENDING --> STEP_2_COMPLETE: step 2 service publishes success event
    STEP_2_PENDING --> SAGA_COMPENSATING: step 2 service publishes failure event

    STEP_2_COMPLETE --> SAGA_COMPLETE: all steps done
    SAGA_COMPLETE --> [*]

    SAGA_COMPENSATING --> COMPENSATE_STEP_1: rollback step 1 command emitted
    COMPENSATE_STEP_1 --> SAGA_FAILED: compensation complete
    SAGA_FAILED --> [*]

Step 1: The Outbox Pattern

The most common correctness failure in event-driven systems is the dual-write problem: a service writes to its database and then tries to publish an event. If the publish fails after the DB write, the event is lost silently. If the publish succeeds but the DB write fails (and the transaction rolls back), you have an event with no corresponding state change.

The Outbox Pattern solves this by making event publishing part of the same local database transaction as the state change:

sequenceDiagram
    participant SVC as Producer Service
    participant DB as PostgreSQL
    participant RELAY as Outbox Relay
    participant KAFKA as Kafka

    SVC->>DB: BEGIN TRANSACTION
    SVC->>DB: UPDATE orders SET status = 'CONFIRMED'
    SVC->>DB: INSERT INTO outbox (event_type, payload, published=false)
    SVC->>DB: COMMIT

    RELAY->>DB: SELECT * FROM outbox WHERE published = false LIMIT 100
    RELAY->>KAFKA: publish events (batched)
    RELAY->>DB: UPDATE outbox SET published = true WHERE id IN (...)

The outbox table lives in the same database as the service's domain data. The relay runs as a background process (or uses CDC — Change Data Capture — via Debezium) to tail the outbox and publish to Kafka. If the relay crashes mid-publish, it re-reads unpublished rows and retries — Kafka producers are idempotent, so duplicate publishes are safe.

Outbox Table Schema

CREATE TABLE event_outbox (
    outbox_id      UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(50) NOT NULL,  -- ORDER, USER, PAYMENT
    aggregate_id   UUID        NOT NULL,
    event_type     VARCHAR(100) NOT NULL, -- order.confirmed, payment.processed
    event_schema   VARCHAR(50) NOT NULL,  -- schema version
    payload        JSONB       NOT NULL,
    headers        JSONB,                 -- Kafka headers (trace-id, correlation-id)
    partition_key  VARCHAR(128),          -- determines Kafka partition
    published      BOOLEAN     NOT NULL DEFAULT FALSE,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    published_at   TIMESTAMPTZ
);

CREATE INDEX idx_outbox_unpublished ON event_outbox (created_at) WHERE published = FALSE;
CREATE INDEX idx_outbox_aggregate   ON event_outbox (aggregate_type, aggregate_id, created_at DESC);

The relay uses SELECT FOR UPDATE SKIP LOCKED to allow multiple relay instances to process rows concurrently without conflicts — critical for high-throughput producers.

Step 2: Topic Design

Topic design is architecture. Getting it wrong is expensive to fix because consumers hard-code topic names and migrating consumers is painful.

Topic Naming Convention

{domain}.{aggregate}.{event-type}

Examples:
  orders.order.confirmed
  orders.order.cancelled
  payments.payment.processed
  payments.payment.failed
  inventory.stock.reserved
  inventory.stock.depleted
  users.user.registered
  users.user.kyc-approved

One topic per event type (not one topic per domain, not one massive topic). This allows:

Independent retention policies per event type
Consumers subscribing only to the events they care about
Clear schema ownership
Independent scaling of partitions per topic

Partition Strategy

The partition key determines ordering. Choose the partition key to match the ordering requirement:

order.{order_id} — all events for an order go to the same partition → events for a single order are in-order
user.{user_id} — all events for a user go to the same partition
payment.{account_id} — all payment events for an account are ordered

Never use a random partition key unless you genuinely do not need ordering — you lose the ability to reconstruct ordered history per entity.

Partition Count

Partitions = consumers. You cannot have more parallelism than partitions. Start with:

partitions = max_consumers_in_group × 2

For a consumer group that will eventually run 50 instances, create 100 partitions. Kafka allows increasing partition count but cannot decrease it, so start slightly high.

Step 3: Schema Registry and Evolution

Every event must have a registered schema. The schema registry enforces compatibility rules at publish time — a producer cannot publish an event that would break existing consumers.

Compatibility Modes

Mode	Meaning	Use When
BACKWARD	New schema can read data written by old schema	Consumers upgrade before producers
FORWARD	Old schema can read data written by new schema	Producers upgrade before consumers
FULL	Both BACKWARD and FORWARD	Rolling upgrades
NONE	No compatibility check	Development only

Production default: FULL compatibility. This allows producers and consumers to deploy in any order during a rolling upgrade.

Safe Schema Changes (FULL compatible)

✅ Adding an optional field with a default value
✅ Adding a new enum value (consumers must handle unknown values gracefully)
✅ Widening a numeric type (int → long)

Breaking Schema Changes (require versioned migration)

❌ Removing a required field
❌ Renaming a field
❌ Changing a field's type incompatibly (string → int)
❌ Adding a required field without a default

When a breaking change is necessary:

Create a new topic version (orders.order.confirmed.v2)
Produce to both topics during migration
Migrate consumers to the new topic
Deprecate and eventually retire the v1 topic

Event Envelope

Every event, regardless of schema, carries a standard envelope:

{
  "event_id": "evt_abc123",
  "event_type": "orders.order.confirmed",
  "schema_version": "1.2.0",
  "aggregate_type": "ORDER",
  "aggregate_id": "ord_xyz789",
  "correlation_id": "req_a1b2c3",
  "causation_id": "evt_prev456",
  "produced_at": "2026-07-15T10:30:00.000Z",
  "producer_service": "order-service",
  "producer_version": "3.4.1",
  "payload": { ... }
}

The correlation_id enables distributed tracing across the entire event chain. The causation_id records which event caused this event — enabling reconstruction of causal chains for debugging.

Step 4: Consumer Patterns

Consumer Groups and Offset Management

A consumer group is a set of consumer instances that collectively process all partitions of a topic. Kafka assigns each partition to exactly one consumer in the group at any time. Adding consumer instances increases parallelism up to the partition count.

sequenceDiagram
    participant KAFKA as Kafka (3 partitions)
    participant C1 as Consumer 1
    participant C2 as Consumer 2
    participant C3 as Consumer 3

    KAFKA->>C1: partition 0 (offset 0–N)
    KAFKA->>C2: partition 1 (offset 0–M)
    KAFKA->>C3: partition 2 (offset 0–P)

    C1->>C1: process event, commit offset
    C2->>C2: process event, commit offset
    C3->>C3: process event, commit offset

Offset commit strategy:

Auto-commit (every 5s): simple but risks processing an event twice if the consumer crashes between processing and auto-commit. Good for idempotent consumers.
Manual commit after processing: commit only after the event is fully processed and state is persisted. Risk: if the consumer crashes before committing, the event is reprocessed — idempotency still required.
Manual commit before processing: never do this. If processing fails, the offset is committed and the event is lost.

Idempotent Consumers

Every consumer must be idempotent — processing the same event twice must produce the same result as processing it once. Strategies:

Conditional upsert (preferred for DB consumers):

INSERT INTO processed_payments (payment_id, amount, status)
VALUES ($1, $2, 'COMPLETED')
ON CONFLICT (payment_id) DO NOTHING;

Idempotency key tracking:

CREATE TABLE consumed_events (
    event_id   UUID PRIMARY KEY,
    topic      VARCHAR(200) NOT NULL,
    consumed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- Before processing: check if event_id exists. If yes, skip.
-- After processing: insert event_id.
-- Wrap both in the same transaction as the business operation.

Natural idempotency: Many operations are naturally idempotent. Setting order.status = 'CONFIRMED' when it is already CONFIRMED is safe. The danger is transitions like balance += amount — these require the event deduplication approach.

Step 5: Dead-Letter Queues

A dead-letter queue (DLQ) is where events go when a consumer cannot process them after exhausting retries. Without DLQ handling, one poison pill message can stall an entire consumer group indefinitely.

Retry and DLQ Topology

flowchart LR
    MAIN[Main Topic] --> CONSUMER[Consumer]
    CONSUMER -->|success| COMMIT[Commit Offset]
    CONSUMER -->|failure| RETRY_1[Retry Topic - attempt 1]
    RETRY_1 -->|after 1 min delay| CONSUMER
    CONSUMER -->|failure 2nd| RETRY_2[Retry Topic - attempt 2]
    RETRY_2 -->|after 5 min delay| CONSUMER
    CONSUMER -->|failure 3rd| DLQ[Dead-Letter Topic]
    DLQ --> ALERT[Alert On-Call]
    DLQ --> REPROCESS[Manual Reprocess Tool]

Each retry topic has a configurable delay (implemented via timestamp-based consumer polling). After a configurable maximum retry count (typically 3–5), the event is moved to the DLQ topic.

DLQ Event Structure

{
  "original_event": { ... },
  "original_topic": "orders.order.confirmed",
  "original_partition": 2,
  "original_offset": 948372,
  "consumer_group": "fulfillment-service",
  "failure_reason": "NullPointerException: order.customer_id is null",
  "stack_trace": "...",
  "attempt_count": 3,
  "first_failed_at": "2026-07-15T10:30:00.000Z",
  "last_failed_at": "2026-07-15T10:45:00.000Z",
  "moved_to_dlq_at": "2026-07-15T10:46:00.000Z"
}

DLQ events are never automatically reprocessed. They require a human decision: fix the bug, then selectively reprocess via the Replay Service.

Step 6: Saga Pattern

A Saga is a sequence of local transactions coordinated across multiple services, where each step publishes an event or sends a command, and failures trigger compensating transactions.

Choreography-Based Saga

In choreography, there is no central orchestrator. Each service listens for events and publishes its own events in response. The saga progresses through event chains.

Example: Order fulfillment saga (choreography)

sequenceDiagram
    participant OS as Order Service
    participant IS as Inventory Service
    participant PS as Payment Service
    participant FS as Fulfillment Service

    OS->>OS: create order
    OS->>Kafka: orders.order.created

    IS->>Kafka: consume orders.order.created
    IS->>IS: reserve inventory
    IS->>Kafka: inventory.stock.reserved (or inventory.reservation.failed)

    PS->>Kafka: consume inventory.stock.reserved
    PS->>PS: charge payment
    PS->>Kafka: payments.payment.processed (or payments.payment.failed)

    FS->>Kafka: consume payments.payment.processed
    FS->>FS: schedule fulfillment
    FS->>Kafka: fulfillment.shipment.scheduled

Compensation (choreography): If payment fails, the Payment Service publishes payments.payment.failed. The Inventory Service subscribes and compensates by releasing the reservation.

Pros: Simple, no central bottleneck, each service owns its domain events.

Cons: Saga state is implicit — it lives in the distributed chain of events. Hard to answer "what is the current state of this saga?" or to handle complex failure scenarios with branching logic.

Orchestration-Based Saga

In orchestration, a central Saga Orchestrator drives the workflow by sending commands and receiving reply events.

sequenceDiagram
    participant ORCH as Saga Orchestrator
    participant IS as Inventory Service
    participant PS as Payment Service
    participant FS as Fulfillment Service

    ORCH->>Kafka: inventory.reserve.command
    IS->>Kafka: inventory.reserved.reply (success/failure)
    ORCH->>Kafka: payment.charge.command
    PS->>Kafka: payment.charged.reply (success/failure)
    ORCH->>Kafka: fulfillment.schedule.command
    FS->>Kafka: fulfillment.scheduled.reply
    ORCH->>ORCH: saga complete — update saga state

On failure:

ORCH receives payment.charge.reply (FAILED)
ORCH->>Kafka: inventory.release.compensation.command
IS->>Kafka: inventory.released.reply
ORCH->>ORCH: saga failed — update saga state, notify

Saga State Store (PostgreSQL):

CREATE TABLE sagas (
    saga_id         UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    saga_type       VARCHAR(50) NOT NULL,  -- ORDER_FULFILLMENT
    correlation_id  VARCHAR(128) NOT NULL UNIQUE,
    status          VARCHAR(30) NOT NULL,  -- STARTED, STEP_1_PENDING, ...COMPLETE, FAILED
    current_step    INT NOT NULL DEFAULT 0,
    context         JSONB NOT NULL,        -- business data needed for compensation
    started_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at    TIMESTAMPTZ,
    failed_at       TIMESTAMPTZ,
    failure_reason  TEXT
);

CREATE INDEX idx_sagas_correlation ON sagas (correlation_id);
CREATE INDEX idx_sagas_status      ON sagas (status, updated_at DESC);

Decision: Use orchestration for sagas with more than 3 steps or complex compensation logic. Choreography is simpler for short, linear flows. Orchestration becomes necessary when you need visibility into saga state, handle branching failures, or implement timeouts.

Step 7: Event Sourcing and CQRS

Event Sourcing is a persistence pattern where, instead of storing the current state of an entity, you store the full history of events that led to that state. Current state is derived by replaying events from the beginning (or from a snapshot).

Event Store Schema

CREATE TABLE event_store (
    event_id        UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type  VARCHAR(50) NOT NULL,
    aggregate_id    UUID        NOT NULL,
    event_type      VARCHAR(100) NOT NULL,
    event_version   BIGINT      NOT NULL,  -- monotonic sequence per aggregate
    payload         JSONB       NOT NULL,
    metadata        JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (aggregate_id, event_version)   -- optimistic concurrency
);

CREATE INDEX idx_event_store_aggregate ON event_store (aggregate_type, aggregate_id, event_version ASC);

-- Snapshots to avoid replaying full history
CREATE TABLE event_snapshots (
    snapshot_id     UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_id    UUID        NOT NULL,
    aggregate_type  VARCHAR(50) NOT NULL,
    at_version      BIGINT      NOT NULL,
    state           JSONB       NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (aggregate_id, at_version)
);

CQRS (Command Query Responsibility Segregation)

CQRS separates write operations (commands) from read operations (queries). In an event-driven system:

Write side: Commands mutate state by appending events to the event store
Read side: Projections (read models) are built by consuming events and maintaining denormalized query views

flowchart LR
    CMD[Command: ConfirmOrder] --> WS[Write Service]
    WS --> ES[(Event Store)]
    ES --> KAFKA[Kafka: order.confirmed]
    KAFKA --> PROJ[Projection Builder]
    PROJ --> RDBMS[(Read DB: order_summary view)]
    CLIENT[Query Client] --> RDBMS

The read model can be rebuilt at any time by replaying events from the event store through the projection builder — making CQRS systems naturally resilient to read-side failures.

Step 8: Operational Concerns

Consumer Lag Monitoring

Consumer lag is the number of unconsumed messages in a partition. Lag = latest offset − committed offset.

Alert thresholds (example):
  lag > 10,000 messages → WARNING
  lag > 100,000 messages → CRITICAL
  lag growing for 5 minutes → CRITICAL (consumer is falling behind)
  lag = 0 for a consumer that should be consuming → WARNING (consumer stopped)

Lag can be measured per consumer group, per topic, per partition. The combination (high lag + growing rate) indicates a consumer is overwhelmed. Remediation: add consumer instances (up to partition count), or find and fix the slow processing path.

Event Replay

The Replay Service allows consumers to reprocess historical events:

Scenario 1: A bug was found in a consumer. Fix deployed; need to reprocess the last 24 hours of events.
Scenario 2: A new consumer service is onboarded and needs to build its initial state from historical events.
Scenario 3: DLQ events are fixed and need to be reinjected into the main topic.

sequenceDiagram
    participant OPS as Ops Engineer
    participant REPLAY as Replay Service
    participant KAFKA as Kafka
    participant CG as Consumer Group

    OPS->>REPLAY: replay(topic, group, from_timestamp)
    REPLAY->>KAFKA: SeekToTimestamp(partition, timestamp)
    REPLAY->>KAFKA: consume from that offset
    REPLAY->>KAFKA: republish to replay topic
    CG->>KAFKA: consume replay topic
    CG->>CG: reprocess events

Important: Replay must not cause double side effects. The consumer must be able to distinguish a replay event from a live event (via a header flag) and skip irreversible operations (e.g., re-sending an email) while still updating query-side state.

Backpressure Handling

When consumers are slower than producers, backpressure builds. Strategies:

Strategy	When to Use
Scale out consumer group	Processing is CPU/IO bound but horizontally scalable
Increase consumer batch size	Many small events; batching reduces per-event overhead
Prioritize partitions	Partition by priority tier; process high-priority first
Rate-limit producers	Producers can buffer or slow down without data loss
Increase retention	Buy time to catch up; do not drop events

API Design

Event Publishing API (HTTP Gateway for non-Kafka producers)

POST /v1/events

{
  "topic": "orders.order.confirmed",
  "partition_key": "ord_xyz789",
  "event_type": "orders.order.confirmed",
  "schema_version": "1.2.0",
  "payload": {
    "order_id": "ord_xyz789",
    "customer_id": "usr_abc123",
    "total_cents": 4999,
    "items": [
      { "sku": "WIDGET-1", "quantity": 2, "unit_price_cents": 2499 }
    ],
    "confirmed_at": "2026-07-15T10:30:00.000Z"
  },
  "headers": {
    "correlation_id": "req_a1b2c3d4",
    "producer_service": "order-service"
  }
}

Response:

{
  "event_id": "evt_d4e5f6a7b8c9",
  "topic": "orders.order.confirmed",
  "partition": 7,
  "offset": 1029481,
  "accepted_at": "2026-07-15T10:30:00.123Z"
}

Error (schema violation):

{
  "error": "SCHEMA_VIOLATION",
  "message": "Field 'items[0].unit_price_cents' is required but missing",
  "schema_id": 42,
  "schema_version": "1.2.0"
}

Consumer Group Management API

GET /v1/consumer-groups/{group_id}/lag

{
  "consumer_group": "fulfillment-service",
  "total_lag": 1284,
  "partitions": [
    { "topic": "orders.order.confirmed", "partition": 0, "lag": 423, "latest_offset": 102948, "committed_offset": 102525 },
    { "topic": "orders.order.confirmed", "partition": 1, "lag": 861, "latest_offset": 98723, "committed_offset": 97862 }
  ],
  "measured_at": "2026-07-15T10:30:05.000Z"
}

POST /v1/consumer-groups/{group_id}/seek

{
  "topic": "orders.order.confirmed",
  "seek_type": "TIMESTAMP",
  "timestamp": "2026-07-14T00:00:00.000Z"
}

Kafka Event Architecture (Topic Catalog)

Topic	Producers	Consumers	Partitions	Retention
`orders.order.created`	Order Service	Inventory, Fraud, Analytics	50	7 days
`orders.order.confirmed`	Order Service	Fulfillment, Notification, BI	50	7 days
`orders.order.cancelled`	Order Service	Inventory, Payment, Notification	20	7 days
`payments.payment.processed`	Payment Service	Order, Fulfillment, Finance	50	30 days
`inventory.stock.reserved`	Inventory Service	Order, Analytics	30	7 days
`users.user.registered`	User Service	Onboarding, Notification, CRM	20	90 days
`sagas.commands`	Saga Orchestrator	Inventory, Payment, Fulfillment	20	7 days
`sagas.replies`	Domain Services	Saga Orchestrator	20	7 days
`*.dlq`	Consumer DLQ	DLQ Handler, Alerting	10	30 days

Observability

Key Metrics

Metric	Alert Threshold
`kafka.producer.publish_latency_p99`	> 100 ms
`kafka.consumer.lag`	> 10,000 (WARNING), > 100K (CRIT)
`kafka.consumer.lag_growth_rate`	Positive for > 5 min
`kafka.consumer.rebalance_rate`	> 2/minute (indicates instability)
`kafka.dlq.events_per_minute`	> 10 (alert on-call)
`kafka.broker.under_replicated_partitions`	> 0
`kafka.broker.disk_usage_percent`	> 75%
`schema_registry.compatibility_failures`	Any occurrence
`saga.stuck_sagas_count`	> 0 for > 10 minutes

Distributed Tracing

Every event carries a correlation_id (injected at the originating API request). All downstream services extract and propagate this ID in their spans. This enables tracing the complete causal chain:

HTTP Request (correlation_id=abc123)
  → Order Service creates order (publishes with correlation_id=abc123)
  → Inventory Service reserves stock (correlation_id=abc123 in span)
  → Payment Service charges card (correlation_id=abc123)
  → Fulfillment Service schedules shipment (correlation_id=abc123)

Design Trade-offs

1. Choreography vs Orchestration

Option A (Choreography): Each service emits events; others react. No central coordinator.

Option B (Orchestration): A saga orchestrator drives the workflow via commands and replies.

Decision: Choreography for simple two-service interactions; orchestration for sagas with 3+ steps or complex compensation. Choreography distributes responsibility but makes saga state invisible. Orchestration centralizes control, makes state queryable, and makes timeouts and retries manageable. Most production systems use both — choreography at the boundaries, orchestration for cross-domain workflows.

2. Kafka vs Traditional Message Queue (RabbitMQ / SQS)

Option A (Kafka): Partitioned, ordered, persistent log. Consumers control their offset. Replay is free.

Option B (SQS / RabbitMQ): Message deleted on successful acknowledgement. No replay. Simpler to operate.

Decision: Kafka for high-throughput event streaming with replay requirements. If events need to be replayed to rebuild state or onboard new consumers, a deletion-on-ack queue is fundamentally incompatible. SQS/RabbitMQ are appropriate for simple task queues (e.g., background jobs) where replay is not needed. For a platform-level event bus, Kafka's log semantics are the right foundation.

3. Exactly-Once vs At-Least-Once

Option A (Exactly-once): Kafka's transactional API + idempotent producers + consumer read-process-commit in a Kafka transaction. True exactly-once within the Kafka ecosystem.

Option B (At-least-once + idempotent consumers): Simpler configuration. Consumers may receive duplicates; idempotency prevents double processing.

Decision: At-least-once delivery + idempotent consumers for most use cases. Kafka's exactly-once has performance overhead (~30% throughput reduction) and only covers the Kafka-to-Kafka path. As soon as the consumer writes to a database outside Kafka, exactly-once requires a distributed transaction that is rarely worth the complexity. Design idempotent consumers consistently and you achieve the same correctness property at lower cost.

4. Event Schema in JSON vs Avro/Protobuf

Option A (JSON): Human-readable, schema-optional, easy to debug.

Option B (Avro/Protobuf): Binary encoding (3–5× smaller), schema enforcement at compile/publish time, better tooling for schema evolution.

Decision: Avro with schema registry for high-throughput topics; JSON for low-volume operational events. At 500K events/second, the difference between 1 KB JSON and 200-byte Avro is significant storage and network cost. Avro's schema registry integration also catches schema incompatibilities before they reach production. Use JSON for DLQ events and operational metadata where human readability matters more than efficiency.

Common Interview Mistakes

Proposing a message queue for event sourcing. If you need event replay (which event sourcing requires), a deletion-on-ack queue like SQS cannot replay events. Kafka's persistent log is the required primitive.
Ignoring the dual-write problem. "The service updates the database and then publishes to Kafka" — this is the most common data consistency bug in event-driven systems. The outbox pattern must be mentioned whenever a service has both DB writes and event publishes.
Designing consumer groups without idempotency. At-least-once delivery means duplicates will occur (during consumer restarts, rebalances, or offset reset). Every consumer must handle duplicate events gracefully.
Using a single topic for all events. One monolithic topic creates coupling, makes retention policy management impossible, and turns every consumer into a subscriber to irrelevant events. Topic design is architecture.
Forgetting that partitions are the parallelism unit. If you create 3 partitions, you can have at most 3 parallel consumers. A consumer group with 10 instances on a 3-partition topic leaves 7 instances idle. Size partitions to your future parallelism needs.
Treating choreography-based sagas as stateless. Choreography sagas still have state — it is implicit in the distributed chain of events. When asked "what is the current state of order X?", you must either query each service or maintain a saga state projection. Pretending there is no state is incorrect.
Missing DLQ design. Saying "the consumer retries failed events" without specifying retry count, retry delay, and what happens after all retries are exhausted leaves the system in a state where one bad message can stall a consumer group permanently.
Overlooking schema evolution. Saying "events are JSON" without discussing how you prevent a producer schema change from breaking existing consumers is a major gap. The schema registry and compatibility modes must be part of the design.

Summary

flowchart LR
    A[Producer Service] -->|outbox pattern| B[Outbox Table]
    B -->|relay| C[Kafka Broker]
    C -->|at-least-once| D[Consumer Group]
    D -->|idempotent| E[Business Logic]
    E -->|success| F[Commit Offset]
    E -->|failure| G[Retry Topic]
    G -->|3 retries| H[Dead-Letter Queue]
    H --> I[Alert + Manual Fix]
    C -->|schema validated| J[Schema Registry]
    C -->|saga event| K[Saga Orchestrator]
    K -->|command| C

Design Principles:

The outbox pattern is non-negotiable — any service that writes to a DB and publishes to Kafka must use the outbox or accept silent data loss
Topic design is architecture — one topic per event type, partition key matches ordering requirement, retention policy matches consumer needs
Idempotency is a consumer responsibility — design every consumer to handle duplicate delivery; do not rely on exactly-once delivery from the broker alone
DLQ design is mandatory — a consumer without a DLQ will stall on the first unprocessable message; dead-letter handling must be designed upfront
Schema contracts must be enforced — a schema registry with compatibility checks prevents producer changes from silently breaking consumers in production
Saga state must be observable — whether using choreography or orchestration, the current state of every in-flight saga must be queryable