Kafka Retention Interview Questions and Answers

Learn Kafka Retention with real-world interview questions covering retention policies, log segments, cleanup policies, compaction, deletion, event replay, and production best practices.

Kafka Retention Interview Questions and Answers

Kafka is fundamentally different from traditional messaging systems.

Most message queues remove a message immediately after it is consumed.

Kafka does not.

Kafka stores messages based on Retention Policies, allowing consumers to replay historical events whenever required.

Retention is one of the most important Kafka interview topics because it directly affects:

Storage
Event Replay
Disaster Recovery
Analytics
Compliance
Performance

Kafka Storage Model

flowchart LR

Producer --> KafkaTopic["Kafka Topic"]

KafkaTopic["Kafka Topic"] --> LogSegments["Log Segments"]

LogSegments["Log Segments"] --> Consumer

Q1. What is Kafka Retention?

Answer

Kafka Retention defines how long messages remain stored inside Kafka.

Messages are retained even after consumers read them.

Consumers track their own offsets, so Kafka does not delete messages after consumption.

Benefits

Event Replay
Fault Recovery
Audit
Analytics
Disaster Recovery

Retention Flow

flowchart LR

Producer --> KafkaLog["Kafka Log"]

KafkaLog["Kafka Log"] --> Consumer

Consumer --> OffsetCommit["Offset Commit"]

KafkaLog["Kafka Log"] --> MessageStillExists["Message Still Exists"]

Q2. Why is Kafka different from traditional message queues?

Answer

Traditional Queue

Message

↓

Consumer

↓

Deleted

Kafka

Message

↓

Consumer

↓

Offset Updated

↓

Message Retained

Kafka separates:

Message Storage
Consumer Progress

Comparison

flowchart LR

TraditionalQueue["Traditional Queue"] --> DeleteMessage["Delete Message"]

Kafka --> RetainMessage["Retain Message"]

Q3. What are Kafka Retention Policies?

Answer

Kafka supports two primary cleanup policies:

Delete
Compact

Policies

mindmap
  root((Retention))
    Delete Policy
    Compact Policy

Q4. What is Delete Retention Policy?

Answer

Delete Policy removes old messages after the configured retention period or when the topic exceeds the configured size.

Example

Retention Time = 7 Days

Messages older than seven days are deleted automatically.

Delete Policy

flowchart LR

Messages --> RetentionPeriod["Retention Period"]
RetentionPeriod["Retention Period"] --> Delete

Common Use Cases

Application Logs
Notifications
IoT Events
Metrics
Streaming Data

Q5. What is Log Compaction?

Answer

Log Compaction keeps only the latest value for each message key.

Example

Customer=100

Address=A

↓

Customer=100

Address=B

After compaction:

Customer=100

Address=B

Older versions are removed while the latest value remains.

Compaction

flowchart LR

OldRecords["Old Records"] --> LogCleaner["Log Cleaner"]
LogCleaner["Log Cleaner"] --> LatestRecord["Latest Record"]

Benefits

Saves Storage
Maintains Latest State
Faster Recovery

Q6. What is a Log Segment?

Answer

Kafka stores partition data in Log Segments.

Instead of one large file:

Partition

↓

Segment 1

↓

Segment 2

↓

Segment 3

Retention and cleanup operate on segments rather than individual messages.

Segment Storage

flowchart LR

Partition --> Segment1["Segment 1"]

Partition --> Segment2["Segment 2"]

Partition --> Segment3["Segment 3"]

Q7. How does Kafka decide when to delete data?

Answer

Kafka deletes data based on configured limits such as:

Time
Size

Common configurations:

retention.ms

retention.hours

retention.bytes

Whichever limit is reached first determines when eligible log segments can be removed.

Retention Decision

flowchart TD

RetentionTime["Retention Time"] --> DeleteSegments["Delete Segments"]

RetentionSize["Retention Size"] --> DeleteSegments["Delete Segments"]

Q8. What is Event Replay?

Answer

Because Kafka retains messages, consumers can replay historical events.

Example

Consumer Failed

↓

Restart

↓

Read Old Events Again

Event replay is one of Kafka's biggest advantages over traditional messaging systems.

Replay

flowchart LR

KafkaLog["Kafka Log"] --> Consumer

Consumer --> Restart

Restart --> ReplayEvents["Replay Events"]

Q9. How do consumers use retained messages?

Answer

Consumers maintain offsets independently.

A consumer can:

Continue from the last committed offset
Reset offsets
Replay historical events

Example

Offset 500

↓

Offset Reset

↓

Read From Offset 100

Offset Reset

flowchart LR

Consumer --> OffsetReset["Offset Reset"]
OffsetReset["Offset Reset"] --> ReplayMessages["Replay Messages"]

Q10. What are common retention configurations?

Answer

Frequently used topic properties include:

Property	Purpose
`retention.ms`	Retention duration
`retention.bytes`	Maximum storage size
`cleanup.policy`	Delete or Compact
`segment.bytes`	Segment size
`segment.ms`	Segment rollover time

Configuration

flowchart TD

Topic --> Retention

Topic --> CleanupPolicy["Cleanup Policy"]

Topic --> Segments

Q11. What are the production best practices for Kafka Retention?

Answer

Follow these recommendations:

Choose retention based on business requirements.
Use Delete Policy for event streams.
Use Log Compaction for latest-state topics.
Monitor disk utilization.
Monitor log segment growth.
Separate long-retention and short-retention topics.
Archive important events before expiration.
Test replay scenarios regularly.
Use appropriate partition sizing.
Avoid unnecessarily long retention periods.

Enterprise Architecture

flowchart TD

SpringBoot["Spring Boot"] --> KafkaTopic["Kafka Topic"]

KafkaTopic["Kafka Topic"] --> RetentionPolicy["Retention Policy"]

RetentionPolicy["Retention Policy"] --> Delete

RetentionPolicy["Retention Policy"] --> Compact

KafkaTopic["Kafka Topic"] --> Consumers

Kafka Storage Lifecycle

sequenceDiagram
participant Producer
participant Kafka
participant Consumer
Producer->>Kafka: Publish Event
Consumer->>Kafka: Read Event
Consumer->>Kafka: Commit Offset
Kafka-->>Kafka: Retain Message
Kafka-->>Kafka: Delete/Compact Later

Kafka Retention Overview

mindmap
  root((Kafka Retention))
    Delete Policy
    Log Compaction
    Segments
    Replay
    Offsets
    Storage
    Cleanup

Delete vs Log Compaction

Delete Policy	Log Compaction
Deletes Old Data	Keeps Latest Record
Time or Size Based	Key Based
Event Streams	Latest State Topics
Simple Cleanup	State Recovery
Audit Window	Current Snapshot

Real Banking Example

A banking platform processes 10 million transactions per day.

Architecture:

Mobile Banking

↓

payments-topic

↓

Retention = 30 Days

↓

Fraud Analytics

↓

Audit

↓

Compliance

↓

Replay if Required

If a fraud investigation begins after two weeks, analysts can replay transaction events because Kafka has retained them according to the configured retention policy.

Senior Interview Tips

Interviewers frequently ask:

What is Kafka Retention?
Why doesn't Kafka delete messages after consumption?
What is Delete Policy?
What is Log Compaction?
Delete vs Compact?
What is Event Replay?
What are Log Segments?
How does Kafka decide when to delete data?
What is retention.ms?
What is cleanup.policy?
How do consumers replay events?
What retention strategy would you use for banking systems?

Remember:

Kafka retains messages independently of consumer progress.
Consumers move offsets—not messages.
Delete Policy removes old data.
Log Compaction preserves the latest value for each key.

Quick Revision

Kafka Retention determines how long messages remain stored.
Messages are retained even after successful consumption.
Consumers track offsets independently of message storage.
Delete Policy removes old log segments based on time or size.
Log Compaction keeps the latest record for each key.
Kafka stores data in log segments, which are the units of cleanup.
Event Replay allows consumers to process historical events again.
Configure retention based on business, compliance, and storage requirements.
Monitor disk usage, segment growth, and cleanup activity in production.
Kafka Retention is a key feature that enables replay, recovery, auditing, and resilient event-driven architectures.