Backend interview prep

Start

Kafka Exactly Once Semantics (EOS) Interview Questions and Answers

Learn Kafka Exactly Once Semantics (EOS) with interview questions, Mermaid diagrams, Spring Boot examples, Kafka transactions, idempotent producers, transactional consumers, and enterprise production best practices.

Kafka Exactly Once Semantics (EOS) - Interview Questions & Answers

One of the biggest improvements introduced in Apache Kafka is Exactly Once Semantics (EOS).

Before Kafka 0.11, applications commonly experienced:

  • Duplicate Events
  • Lost Messages
  • Partial Processing
  • Offset Commit Problems

Kafka EOS introduced:

  • Idempotent Producers
  • Transactions
  • Transactional Consumers
  • Read Committed Isolation

These features allow Kafka applications to build highly reliable event-driven systems.

Exactly Once Semantics are heavily used in:

  • Banking
  • Payment Systems
  • Stock Trading
  • Insurance
  • Healthcare
  • Financial Applications

Kafka Exactly Once Architecture

flowchart LR

Producer --> KafkaTransaction["Kafka Transaction"]

KafkaTransaction["Kafka Transaction"] --> KafkaTopic["Kafka Topic"]

KafkaTopic["Kafka Topic"] --> TransactionalConsumer["Transactional Consumer"]

TransactionalConsumer["Transactional Consumer"] --> Database

Database --> OffsetCommit["Offset Commit"]

Q1. What is Kafka Exactly Once Semantics (EOS)?

Answer

Exactly Once Semantics (EOS) ensures that a record is:

  • Produced once
  • Stored once
  • Consumed once
  • Processed once

within Kafka's transactional boundaries.

EOS prevents duplicate records caused by producer retries and consumer failures.

Processing Flow

flowchart TD

Producer --> Kafka

Kafka --> Consumer

Consumer --> Database

Database --> CommitOffset["Commit Offset"]

Q2. Why was Exactly Once introduced in Kafka?

Answer

Without EOS:

  • Producer retries could create duplicate events.
  • Consumer crashes could reprocess records.
  • Offset commits could become inconsistent.

These issues caused duplicate business operations.

Without EOS

flowchart LR

Producer --> Kafka

Kafka --> Consumer

Consumer --> Database

Consumer --> Retry

Retry --> DuplicateProcessing["Duplicate Processing"]

With EOS

flowchart LR

Producer --> Transaction

Transaction --> Kafka

Kafka --> TransactionalConsumer["Transactional Consumer"]

TransactionalConsumer["Transactional Consumer"] --> Database

Q3. What is an Idempotent Producer?

Answer

An Idempotent Producer guarantees that retrying the same message does not create duplicate records inside Kafka.

Kafka internally assigns:

  • Producer ID (PID)
  • Sequence Numbers

Duplicate records are automatically ignored.

Producer Flow

flowchart TD

Producer --> Retry

Retry --> Kafka

Kafka --> SingleRecord["Single Record"]

Configuration

enable.idempotence=true

Interview Tip

Idempotent Producer eliminates duplicate records inside Kafka, not duplicate business processing.

Q4. What are Kafka Transactions?

Answer

Kafka Transactions allow multiple records and offset commits to be treated as one atomic unit.

Either:

  • Everything commits

or

  • Everything rolls back

Transaction Flow

flowchart LR

BeginTransaction["Begin Transaction"] --> PublishEvents["Publish Events"]

PublishEvents["Publish Events"] --> CommitTransaction["Commit Transaction"]

CommitTransaction["Commit Transaction"] --> VisibleToConsumers["Visible to Consumers"]

Benefits

  • Atomic writes
  • Atomic offset commits
  • Reliable event processing

Q5. What is a Transactional Consumer?

Answer

A Transactional Consumer processes records and commits offsets within the same transaction.

Workflow:

  1. Read records
  2. Process business logic
  3. Commit database
  4. Commit Kafka offsets

If processing fails:

  • Offsets are not committed.
  • Records are retried safely.

Consumer Workflow

flowchart TD

Kafka --> Consumer

Consumer --> BusinessLogic["Business Logic"]

BusinessLogic["Business Logic"] --> Database

Database --> CommitOffset["Commit Offset"]

Q6. What is read_committed isolation?

Answer

Kafka consumers support two isolation levels.

read_uncommitted

Reads all records.

Including uncommitted transactions.

read_committed

Reads only committed transactions.

Recommended for production.

Isolation

flowchart LR

Kafka --> CommittedRecords["Committed Records"]

Kafka --> UncommittedRecords["Uncommitted Records"]

CommittedRecords["Committed Records"] --> Consumer

UncommittedRecords["Uncommitted Records"] --> Ignored

Configuration

isolation.level=read_committed

Q7. How does Spring Boot implement Kafka EOS?

Answer

Spring Boot integrates Kafka EOS using:

  • Spring Kafka
  • KafkaTransactionManager
  • @Transactional
  • Idempotent Producers

Typical flow:

REST API

↓

Kafka Producer

↓

Kafka Transaction

↓

Consumer

↓

Database

Spring Boot Architecture

flowchart TD

RestApi["REST API"] --> SpringBoot["Spring Boot"]

SpringBoot["Spring Boot"] --> KafkaProducer["Kafka Producer"]

KafkaProducer["Kafka Producer"] --> KafkaCluster["Kafka Cluster"]

KafkaCluster["Kafka Cluster"] --> TransactionalConsumer["Transactional Consumer"]

TransactionalConsumer["Transactional Consumer"] --> Database

Q8. What are common Kafka EOS mistakes?

Answer

Common mistakes include:

  • Assuming EOS removes the need for idempotency.
  • Ignoring database duplicates.
  • Using read_uncommitted.
  • No retry strategy.
  • No DLQ.
  • Forgetting transaction IDs.
  • Not monitoring transaction failures.

Wrong Design

Kafka EOS

↓

Duplicate Payment Impossible ❌

Correct Design

Kafka EOS

+

Idempotent Consumer

+

Database Constraint

=

Reliable Processing ✅

Q9. What are the limitations of Kafka EOS?

Answer

Kafka EOS guarantees exactly-once inside Kafka transactions.

It does not automatically protect:

  • External Databases
  • REST APIs
  • Payment Gateways
  • Third-party Services

Applications still require:

  • Idempotency
  • Outbox Pattern
  • Saga Pattern

Scope

flowchart LR

KafkaTransaction["Kafka Transaction"] --> Kafka

Kafka --> Consumer

Consumer --> ExternalDatabase["External Database"]

ExternalDatabase["External Database"] --> ApplicationLogic["Application Logic"]

Best Practice

Always combine Kafka EOS with application-level duplicate protection.

Q10. What are the enterprise best practices for Kafka EOS?

Answer

Follow these recommendations:

  • Enable idempotent producers.
  • Use Kafka transactions.
  • Configure transactional consumers.
  • Use read_committed.
  • Implement idempotent consumers.
  • Store processed event IDs.
  • Configure retries and DLQs.
  • Monitor transaction failures.
  • Combine with the Outbox Pattern.
  • Test failure scenarios regularly.

Enterprise Architecture

flowchart TD

RestApi["REST API"] --> OrderService["Order Service"]

OrderService["Order Service"] --> KafkaTransaction["Kafka Transaction"]

KafkaTransaction["Kafka Transaction"] --> KafkaCluster["Kafka Cluster"]

KafkaCluster["Kafka Cluster"] --> TransactionalConsumer["Transactional Consumer"]

TransactionalConsumer["Transactional Consumer"] --> DuplicateCheck["Duplicate Check"]

DuplicateCheck["Duplicate Check"] --> Database

TransactionalConsumer["Transactional Consumer"] --> DLQ

Processing Pipeline

flowchart LR

Producer --> Transaction

Transaction --> Kafka

Kafka --> Consumer

Consumer --> Database

Database --> OffsetCommit["Offset Commit"]

Kafka EOS Overview

mindmap
  root((Kafka EOS))
    Idempotent Producer
    Transactions
    Transactional Consumer
    Read Committed
    Retry
    DLQ
    Outbox
    Monitoring

Kafka Delivery Guarantees

Feature At Most Once At Least Once Kafka EOS
Duplicate Messages No Possible Prevented in Kafka
Message Loss Possible No No
Transactions No No Yes
Producer Retry Safe No No Yes
Offset Commit Simple After Processing Transactional
Enterprise Usage Low High Critical Systems

Banking Example

A customer transfers ₹50,000.

Transfer Request

↓

Kafka Producer

↓

Kafka Transaction

↓

Consumer

↓

Database Updated

↓

Offset Committed

↓

Consumer Crash

↓

No Duplicate Processing

The transaction remains consistent because Kafka commits both the message and offset atomically.

Senior Interview Tip

Kafka Exactly Once Semantics are not a complete business solution.

EOS protects message processing inside Kafka.

Enterprise systems still require:

  • Idempotent Consumers
  • Outbox Pattern
  • Saga Pattern
  • Database Constraints
  • Replay Services
  • Dead Letter Queues

A production-ready Kafka platform typically includes:

  • Spring Boot
  • Spring Kafka
  • Kafka Transactions
  • Idempotent Producers
  • Transactional Consumers
  • Read Committed Isolation
  • Outbox Pattern
  • DLQ
  • Retry Topics
  • Replay Services
  • Schema Registry
  • Prometheus & Grafana
  • OpenTelemetry
  • Audit Logging

Remember:

  • Idempotent Producer prevents duplicate records in Kafka.
  • Kafka Transactions provide atomic writes and offset commits.
  • Transactional Consumers safely commit offsets.
  • Application-level idempotency is still required for external systems.

Quick Revision

  • Kafka EOS provides reliable message processing within Kafka.
  • Enable enable.idempotence=true for idempotent producers.
  • Use Kafka transactions for atomic writes.
  • Configure transactional consumers.
  • Set isolation.level=read_committed in production.
  • Kafka EOS does not eliminate the need for idempotent consumers.
  • Combine Kafka EOS with the Outbox Pattern and Saga Pattern.
  • Configure retries, DLQs, and replay mechanisms.
  • Monitor transaction failures and consumer lag.
  • Combine Kafka transactions, Spring Boot, idempotency, monitoring, and replay for enterprise-grade exactly-once event processing.