Backend interview prep

Start

Kafka Architecture Interview Questions and Answers

Learn Apache Kafka architecture with real-world interview questions covering brokers, clusters, topics, partitions, leaders, followers, controllers, KRaft, producers, consumers, and production architecture.

Kafka Architecture Interview Questions and Answers

Understanding Kafka Architecture is one of the most important topics for senior Java developers, backend engineers, and solution architects.

A strong Kafka understanding requires knowing:

  • Kafka Cluster
  • Broker
  • Topic
  • Partition
  • Leader and Follower
  • Controller
  • KRaft Architecture
  • Producer Flow
  • Consumer Flow
  • Replication
  • Storage Model

Kafka is not just a messaging system. It is a distributed event streaming platform designed for high throughput, scalability, and fault tolerance.

Kafka High-Level Architecture

flowchart TD

Producer --> KafkaCluster["Kafka Cluster"]

KafkaCluster["Kafka Cluster"] --> Consumer

Q1. Explain Kafka Architecture.

Answer

Kafka architecture consists of:

  • Producers
  • Brokers
  • Topics
  • Partitions
  • Replication
  • Consumers
  • Consumer Groups

A Kafka cluster contains multiple brokers.

Each broker stores partitions and serves client requests.

Complete Architecture

flowchart LR

Producer --> Broker1["Broker 1"]

Broker1["Broker 1"] --> TopicA["Topic A"]

Broker2["Broker 2"] --> TopicA["Topic A"]

Broker3["Broker 3"] --> TopicA["Topic A"]

TopicA["Topic A"] --> ConsumerGroup["Consumer Group"]

Q2. What is a Kafka Cluster?

Answer

A Kafka Cluster is a group of Kafka brokers working together.

A cluster provides:

  • Scalability
  • High Availability
  • Fault Tolerance
  • Load Distribution

Example:

Kafka Cluster

├── Broker 1
├── Broker 2
└── Broker 3

Cluster Architecture

flowchart LR

KafkaCluster["Kafka Cluster"] --> Broker1["Broker 1"]

KafkaCluster["Kafka Cluster"] --> Broker2["Broker 2"]

KafkaCluster["Kafka Cluster"] --> Broker3["Broker 3"]

Q3. What is a Kafka Broker?

Answer

A Kafka Broker is a Kafka server responsible for:

  • Storing partitions
  • Handling producer requests
  • Serving consumer requests
  • Managing replication

Each broker has a unique ID.

Example:

Broker-101

Broker-102

Broker-103

Broker Responsibility

flowchart TD

Broker --> StorePartitions["Store Partitions"]

Broker --> HandleProducers["Handle Producers"]

Broker --> HandleConsumers["Handle Consumers"]

Broker --> Replication

Q4. What is a Kafka Topic?

Answer

A Topic is a logical name where events are stored.

Examples:

customer-events

payment-events

order-events

A topic can contain multiple partitions.

Topic Architecture

flowchart LR

PaymentTopic["Payment Topic"] --> Partition0["Partition 0"]

PaymentTopic["Payment Topic"] --> Partition1["Partition 1"]

PaymentTopic["Payment Topic"] --> Partition2["Partition 2"]

Q5. What is a Kafka Partition?

Answer

A Partition is the basic unit of parallelism in Kafka.

Each partition is:

  • Ordered
  • Immutable
  • Append-only
  • Stored on disk

Example:

Payment Topic

Partition 0
Partition 1
Partition 2

Partition Storage

flowchart LR

Partition --> Event0["Event 0"]
Event0["Event 0"] --> Event1["Event 1"]

Event1["Event 1"] --> Event2["Event 2"]
Event2["Event 2"] --> Event3["Event 3"]

Interview Tip

Kafka guarantees ordering only inside a partition.

Q6. What are Leader and Follower Partitions?

Answer

Each partition has:

  • One Leader
  • Zero or more Followers

Leader handles:

  • Producer writes
  • Consumer reads

Followers replicate data from the leader.

Replication

flowchart LR

Producer --> LeaderPartition["Leader Partition"]

LeaderPartition["Leader Partition"] --> Follower1["Follower 1"]

LeaderPartition["Leader Partition"] --> Follower2["Follower 2"]

Q7. What is ISR (In-Sync Replica)?

Answer

ISR is the group of replicas that are fully synchronized with the leader.

Example:

Leader

Follower 1

Follower 2

All three are in ISR.

If a follower falls behind:

Leader

Follower 1

Follower 2 (Out of Sync)

it is removed from ISR.

ISR Architecture

flowchart LR

Leader --> ISR

ISR --> Follower1["Follower 1"]

ISR --> Follower2["Follower 2"]

Q8. What is Kafka Controller?

Answer

The Controller manages cluster-level operations.

Responsibilities:

  • Broker failure detection
  • Leader election
  • Partition assignment
  • Metadata management

Traditional Kafka Architecture

flowchart TD

Controller --> BrokerManagement["Broker Management"]

Controller --> LeaderElection["Leader Election"]

Controller --> PartitionAssignment["Partition Assignment"]

Q9. What is KRaft Architecture?

Answer

KRaft (Kafka Raft Metadata Mode) replaces ZooKeeper.

Modern Kafka versions use KRaft for metadata management.

Benefits:

  • Simpler Architecture
  • Better Scalability
  • Faster Controller Recovery
  • No ZooKeeper Dependency

ZooKeeper Architecture

flowchart LR

KafkaBrokers["Kafka Brokers"] --> ZooKeeper

KRaft Architecture

flowchart LR

KafkaBrokers["Kafka Brokers"] --> KraftController["KRaft Controller"]

Q10. How does a Kafka Producer work?

Answer

Producer flow:

  1. Application creates event.
  2. Producer selects partition.
  3. Event is sent to broker leader.
  4. Broker stores event.
  5. Acknowledgement is returned.

Producer Flow

sequenceDiagram
participant App
participant Producer
participant Broker
App->>Producer: Create Event
Producer->>Broker: Send Event
Broker-->>Producer: ACK
Producer-->>App: Success

Q11. How does a Kafka Consumer work?

Answer

Consumer flow:

  1. Consumer joins a group.
  2. Kafka assigns partitions.
  3. Consumer reads messages.
  4. Consumer commits offsets.

Consumer Flow

flowchart LR

Consumer --> Partition

Partition --> OffsetCommit["Offset Commit"]

Q12. How does Kafka store messages?

Answer

Kafka stores messages as an append-only log.

Messages are written sequentially to disk.

Benefits:

  • Fast Writes
  • Efficient Reads
  • Replay Capability

Storage Model

flowchart LR

TopicPartition["Topic Partition"] --> SegmentFile["Segment File"]
SegmentFile["Segment File"] --> Messages

Q13. How does Kafka achieve scalability?

Answer

Kafka scales using:

  • More brokers
  • More partitions
  • Consumer groups
  • Replication

Example:

Topic

↓

Partition 0 → Consumer A

Partition 1 → Consumer B

Partition 2 → Consumer C

Scaling

flowchart LR

Topic --> Partition0["Partition 0"]

Partition0["Partition 0"] --> Consumer1["Consumer 1"]

Topic --> Partition1["Partition 1"]

Partition1["Partition 1"] --> Consumer2["Consumer 2"]

Q14. What happens when a Kafka broker fails?

Answer

Kafka handles failures through replication.

Flow:

  1. Leader broker fails.
  2. Controller detects failure.
  3. New leader is elected.
  4. Producers and consumers reconnect.

Failover

flowchart LR

LeaderBroker["Leader Broker"] --> Failure

Failure --> Follower

Follower --> NewLeader["New Leader"]

Q15. What is the production Kafka architecture?

Answer

A typical enterprise Kafka architecture includes:

  • Multiple Brokers
  • Replication Factor 3
  • Schema Registry
  • Monitoring
  • Security
  • Consumer Groups

Enterprise Architecture

flowchart TD

Applications --> SpringBootServices["Spring Boot Services"]

SpringBootServices["Spring Boot Services"] --> KafkaCluster["Kafka Cluster"]

KafkaCluster["Kafka Cluster"] --> ConsumerGroups["Consumer Groups"]

ConsumerGroups["Consumer Groups"] --> BusinessServices["Business Services"]

KafkaCluster["Kafka Cluster"] --> Monitoring

KafkaCluster["Kafka Cluster"] --> Security

Kafka Architecture Components Summary

Component Purpose
Broker Kafka Server
Cluster Group of Brokers
Topic Event Category
Partition Parallel Processing Unit
Leader Handles Reads/Writes
Follower Replication
ISR Healthy Replicas
Controller Cluster Management
KRaft Metadata Management
Consumer Group Scalability

Real Banking Example

A bank processes millions of transactions.

Mobile Banking

↓

Transaction Service

↓

Kafka Cluster

↓

Transaction Topic

↓

Fraud Detection

↓

Core Banking

↓

Notification

↓

Analytics

Kafka allows multiple systems to consume transaction events independently while maintaining scalability and reliability.

Senior Interview Tips

Interviewers commonly ask:

  • Explain Kafka architecture.
  • What is a broker?
  • Topic vs Partition?
  • Why partitions?
  • Leader vs Follower?
  • What is ISR?
  • How does replication work?
  • What is KRaft?
  • ZooKeeper vs KRaft?
  • How does Kafka achieve scalability?
  • How does Kafka handle broker failures?
  • How does Kafka store messages?
  • How does producer routing work?
  • How do consumers read messages?

Remember:

  • Broker stores data.
  • Topic organizes events.
  • Partition provides scalability.
  • Leader handles traffic.
  • Followers provide redundancy.
  • KRaft manages metadata.

Quick Revision

  • Kafka architecture is based on distributed brokers and partitioned topics.
  • Topics are divided into partitions for scalability.
  • Each partition has one leader and multiple follower replicas.
  • ISR represents synchronized replicas.
  • Producers write to partition leaders.
  • Consumers read from partitions and track offsets.
  • Kafka uses replication to provide fault tolerance.
  • Modern Kafka uses KRaft instead of ZooKeeper.
  • Kafka stores events as immutable append-only logs.
  • A well-designed Kafka cluster provides high throughput, reliability, and scalability for enterprise event-driven systems.