---

Introduction

In production systems, failures are inevitable.

Examples

• Database temporarily unavailable
• Payment service timeout
• External REST API failure
• Kafka broker latency
• Network interruption
• Invalid business data

Some failures are temporary and can succeed if retried.

Others are permanent and should never block the consumer.

Spring Kafka provides two important mechanisms:

• Retry Topics
• Dead Letter Topics (DLT)

Together they build highly resilient, fault-tolerant event-driven applications.

---

Retry and DLT Architecture

flowchart LR

KafkaTopic --> Consumer

Consumer --> Success

Consumer --> RetryTopic

RetryTopic --> Consumer

RetryTopic --> DLT

DLT --> Operations

---

Q1. What is Retry in Spring Kafka?

Answer

Retry allows a failed record to be processed again before marking it as permanently failed.

Suitable for

• Database outages
• Network failures
• HTTP 503 responses
• Temporary infrastructure issues

Retry improves system resilience.

---

Q2. What is @RetryableTopic?

@RetryableTopic is Spring Kafka's recommended annotation for topic-based retries.

Example

@RetryableTopic(
    attempts = "4"
)
@KafkaListener(
    topics = "payment-events"
)
public void process(
    PaymentEvent event
){

}

Spring automatically creates retry topics and retry consumers.

---

Q3. How does Retry Topic work?

Workflow

1. Consumer fails.
2. Message moves to Retry Topic.
3. Retry Consumer processes the message later.
4. Success → Processing completes.
5. Failure → Next retry.
6. Final failure → Dead Letter Topic.

Retry Flow

sequenceDiagram
Kafka->>Consumer: Message
Consumer->>BusinessService: Process
BusinessService-->>Consumer: Exception
Consumer->>RetryTopic: Publish
RetryTopic->>RetryConsumer: Retry
RetryConsumer-->>DLT: Final Failure

---

Q4. What is a Dead Letter Topic (DLT)?

A Dead Letter Topic stores records that cannot be processed successfully.

Purpose

• Prevent message loss
• Separate poison messages
• Support manual replay
• Improve observability

Example

payment-events

↓

payment-events-dlt

DLTs isolate permanently failed records.

---

Q5. What is DeadLetterPublishingRecoverer?

After retries are exhausted,

Spring uses

DeadLetterPublishingRecoverer

Responsibilities

• Publish failed record
• Preserve headers
• Preserve offset information
• Route to DLT

It is the standard recovery mechanism.

---

Q6. What is Exponential Backoff?

Instead of retrying immediately,

Spring gradually increases retry delay.

Example

Retry 1

1 second

Retry 2

2 seconds

Retry 3

4 seconds

Retry 4

8 seconds

Exponential Backoff

flowchart TD

Failure --> Retry1

Retry1 --> Retry2

Retry2 --> Retry3

Retry3 --> SuccessOrDLT

Benefits

• Reduce system pressure
• Prevent retry storms

---

Q7. What is a Poison Message?

A Poison Message always fails.

Examples

• Invalid JSON
• Corrupted payload
• Unsupported schema
• Business validation failure

Retrying such messages wastes resources.

They should move directly to the DLT.

---

Q8. How do you replay DLT messages?

Typical workflow

1. Fix application issue.
2. Read records from DLT.
3. Republish to original topic.
4. Process successfully.

Replay Flow

flowchart LR

DLT --> ReplayTool

ReplayTool --> OriginalTopic

OriginalTopic --> Consumer

Replay tools are commonly implemented as administrative services or batch jobs.

---

Q9. Retry vs DLT

Retry	Dead Letter Topic
Temporary failures	Permanent failures
Automatic retry	Final destination
Message eventually succeeds	Requires investigation
Processing continues	Manual or automated replay

Retry handles transient issues.

DLT handles unrecoverable records.

---

Q10. Retry and DLT Best Practices

Retry Only Recoverable Exceptions

Do not retry validation errors.

---

Use Exponential Backoff

Reduce load on downstream systems.

---

Configure Reasonable Retry Counts

Avoid excessive retries.

---

Monitor DLT Size

A growing DLT indicates production issues.

---

Build Replay Utilities

Enable safe recovery after fixes.

---

Banking Example

flowchart TD

PaymentTopic --> Consumer

Consumer --> PaymentService

PaymentService --> Success

PaymentService --> RetryTopic

RetryTopic --> RetryConsumer

RetryConsumer --> Success

RetryConsumer --> PaymentDLT

PaymentDLT --> OperationsDashboard

Temporary failures recover automatically while permanent failures are isolated.

---

Common Interview Questions

• What is Retry?
• What is @RetryableTopic?
• How do Retry Topics work?
• What is a Dead Letter Topic?
• What is DeadLetterPublishingRecoverer?
• What is Exponential Backoff?
• What is a Poison Message?
• How do you replay DLT messages?
• Retry vs DLT?
• Retry and DLT best practices?

---

Quick Revision

Topic	Summary
Retry	Reprocess temporary failures
@RetryableTopic	Topic-based retry mechanism
Retry Topic	Intermediate retry destination
DLT	Stores permanently failed records
DeadLetterPublishingRecoverer	Publishes failed records to DLT
Exponential Backoff	Increasing retry delay
Poison Message	Permanently failing record
Replay	Reprocess DLT messages
Recovery	Final failure handling
Monitoring	Observe retry and DLT metrics

---

Retry Lifecycle

sequenceDiagram
Kafka->>Consumer: Deliver Record
Consumer->>BusinessService: Process
BusinessService-->>Consumer: Exception
Consumer->>RetryTopic: Publish Retry
RetryTopic->>RetryConsumer: Retry Processing
RetryConsumer-->>BusinessService: Retry
BusinessService-->>RetryConsumer: Failed
RetryConsumer->>DLT: Publish Failed Record

---

Production Example – Banking Payment Retry Strategy

A banking platform processes fund transfer events.

Scenario

• The Payment Service calls the Core Banking API.
• The API becomes temporarily unavailable.

Workflow

1. Consumer receives a payment event.
2. Processing fails due to an HTTP 503 response.
3. Spring publishes the record to the first retry topic.
4. Retries occur with exponential backoff.
5. If the Core Banking API becomes available, processing succeeds.
6. If all retry attempts fail, the record is published to payment-events-dlt.
7. Operations teams investigate the DLT.
8. After fixing the issue, a replay utility republishes the event to the original topic.

@RetryableTopic(
    attempts = "5",
    backoff = @Backoff(
        delay = 1000,
        multiplier = 2.0
    )
)
@KafkaListener(
    topics = "payment-events"
)
public void process(
    PaymentEvent event
){

    paymentService.process(event);

}

flowchart LR

PaymentEventsTopic --> PaymentConsumer

PaymentConsumer --> PaymentService

PaymentService --> CoreBankingAPI

CoreBankingAPI --> TemporaryFailure

TemporaryFailure --> RetryTopic1

RetryTopic1 --> RetryConsumer1

RetryConsumer1 --> RetryTopic2

RetryTopic2 --> RetryConsumer2

RetryConsumer2 --> PaymentEventsDLT

PaymentEventsDLT --> ReplayService

ReplayService --> PaymentEventsTopic

Retry Timeline

Attempt	Delay
Initial Processing	Immediate
Retry 1	1 second
Retry 2	2 seconds
Retry 3	4 seconds
Retry 4	8 seconds
Retry 5	16 seconds
Final Failure	Publish to DLT

---

Key Takeaways

• Retry Topics provide a scalable mechanism for retrying transient failures without blocking Kafka consumers.
• @RetryableTopic automatically creates retry topics, retry consumers, and integrates with Spring Kafka's retry infrastructure.
• Dead Letter Topics (DLT) safely store records that cannot be processed successfully after all retry attempts.
• DeadLetterPublishingRecoverer publishes permanently failed records while preserving metadata for later investigation.
• Exponential Backoff prevents retry storms by increasing the delay between retry attempts.
• Poison Messages should bypass repeated retries and be routed to the DLT for analysis.
• DLT replay tools allow failed events to be reprocessed after the underlying issue has been resolved.
• Combining retries, DLTs, backoff strategies, and monitoring creates highly resilient event-driven systems suitable for enterprise-scale workloads.