Backend interview prep

Start

Kafka Consumer Groups Interview Questions and Answers

Learn Kafka Consumer Groups with real-world interview questions, partition assignment, parallel processing, rebalancing, Spring Boot integration, and production best practices.

Kafka Consumer Groups Interview Questions and Answers

Consumer Groups are one of the most important concepts in Apache Kafka.

Almost every Kafka interview includes questions like:

  • What is a Consumer Group?
  • Why do we need Consumer Groups?
  • How are partitions assigned?
  • Can two consumers read the same partition?
  • What happens when a new consumer joins?

Understanding Consumer Groups is essential for designing scalable and fault-tolerant Kafka applications.

Consumer Group Architecture

flowchart LR

KafkaTopic["Kafka Topic"] --> Partition0["Partition 0"]

KafkaTopic["Kafka Topic"] --> Partition1["Partition 1"]

KafkaTopic["Kafka Topic"] --> Partition2["Partition 2"]

Partition0["Partition 0"] --> ConsumerA["Consumer A"]

Partition1["Partition 1"] --> ConsumerB["Consumer B"]

Partition2["Partition 2"] --> ConsumerC["Consumer C"]

Q1. What is a Kafka Consumer Group?

Answer

A Consumer Group is a collection of consumers working together to process messages from a Kafka topic.

Kafka distributes topic partitions among consumers in the same group.

This enables:

  • Horizontal Scaling
  • Parallel Processing
  • Fault Tolerance
  • Load Balancing

Consumer Group

flowchart LR

ConsumerGroup["Consumer Group"] --> Consumer1["Consumer 1"]

ConsumerGroup["Consumer Group"] --> Consumer2["Consumer 2"]

ConsumerGroup["Consumer Group"] --> Consumer3["Consumer 3"]

Q2. Why do we need Consumer Groups?

Answer

Without Consumer Groups:

One Consumer

↓

Processes Everything

With Consumer Groups:

Multiple Consumers

↓

Process Different Partitions

Benefits:

  • Faster Processing
  • Better Resource Utilization
  • High Availability
  • Easy Scaling

Parallel Processing

flowchart LR

Topic --> Partition0["Partition 0"]

Topic --> Partition1["Partition 1"]

Topic --> Partition2["Partition 2"]

Partition0["Partition 0"] --> Consumer1["Consumer 1"]

Partition1["Partition 1"] --> Consumer2["Consumer 2"]

Partition2["Partition 2"] --> Consumer3["Consumer 3"]

Q3. How does Kafka assign partitions to consumers?

Answer

Kafka automatically distributes partitions among consumers in the same group.

Example:

Topic

3 Partitions

3 Consumers

Assignment:

Consumer A → Partition 0

Consumer B → Partition 1

Consumer C → Partition 2

Assignment

flowchart LR

Partition0["Partition 0"] --> ConsumerA["Consumer A"]

Partition1["Partition 1"] --> ConsumerB["Consumer B"]

Partition2["Partition 2"] --> ConsumerC["Consumer C"]

Q4. Can multiple consumers read the same partition?

Answer

Within the same Consumer Group:

No

Only one consumer reads a partition.

Across different Consumer Groups:

Yes

Each group has its own offsets.

Multiple Groups

flowchart LR

Topic --> GroupA["Group A"]

Topic --> GroupB["Group B"]

GroupA["Group A"] --> ConsumerA["Consumer A"]

GroupB["Group B"] --> ConsumerB["Consumer B"]

Interview Tip

Partition ownership is unique only inside one Consumer Group.

Q5. What happens if there are more consumers than partitions?

Answer

Extra consumers remain idle.

Example:

3 Partitions

5 Consumers

Result:

Consumer 1 → Partition 0

Consumer 2 → Partition 1

Consumer 3 → Partition 2

Consumer 4 → Idle

Consumer 5 → Idle

Idle Consumers

flowchart LR

Partition0["Partition 0"] --> ConsumerA["Consumer A"]

Partition1["Partition 1"] --> ConsumerB["Consumer B"]

Partition2["Partition 2"] --> ConsumerC["Consumer C"]

ConsumerD["Consumer D"] --> Idle

ConsumerE["Consumer E"] --> Idle

Q6. What happens if there are more partitions than consumers?

Answer

Consumers process multiple partitions.

Example:

6 Partitions

3 Consumers

Assignment:

Consumer A → P0, P1

Consumer B → P2, P3

Consumer C → P4, P5

Distribution

flowchart LR

Partition0["Partition 0"] --> ConsumerA["Consumer A"]

Partition1["Partition 1"] --> ConsumerA["Consumer A"]

Partition2["Partition 2"] --> ConsumerB["Consumer B"]

Partition3["Partition 3"] --> ConsumerB["Consumer B"]

Partition4["Partition 4"] --> ConsumerC["Consumer C"]

Partition5["Partition 5"] --> ConsumerC["Consumer C"]

Q7. What happens when a consumer crashes?

Answer

Kafka detects the failure.

The remaining consumers receive the abandoned partitions automatically.

This process is called Rebalancing.

Failure Recovery

flowchart TD

ConsumerBFailed["Consumer B Failed"] --> KafkaCoordinator["Kafka Coordinator"]
KafkaCoordinator["Kafka Coordinator"] --> Rebalance

Rebalance --> ConsumerA["Consumer A"]
ConsumerA["Consumer A"] --> ConsumerC["Consumer C"]

Benefits

  • Automatic Recovery
  • High Availability
  • Fault Tolerance

Q8. How do Consumer Groups support scalability?

Answer

To increase throughput:

  • Increase partitions.
  • Add more consumers.

Example:

10 Partitions

↓

10 Consumers

↓

Parallel Processing

Horizontal Scaling

flowchart LR

KafkaTopic["Kafka Topic"] --> 10Partitions["10 Partitions"]

10Partitions["10 Partitions"] --> ConsumerGroup["Consumer Group"]

Best Practice

Plan partition count based on expected future scaling.

Q9. How do Consumer Groups work in Spring Boot?

Answer

Spring Boot uses the Kafka Consumer API internally.

Typical flow:

Spring Boot

↓

@KafkaListener

↓

Consumer Group

↓

Kafka Topic

↓

Business Logic

Spring Boot Architecture

flowchart TD

RestApi["REST API"] --> KafkaProducer["Kafka Producer"]
KafkaProducer["Kafka Producer"] --> KafkaTopic["Kafka Topic"]

KafkaTopic["Kafka Topic"] --> ConsumerGroup["Consumer Group"]

ConsumerGroup["Consumer Group"] --> @KafkaListener
@KafkaListener --> BusinessService["Business Service"]

Benefits

  • Automatic Consumer Management
  • Offset Handling
  • Rebalancing Support

Q10. What are the production best practices for Consumer Groups?

Answer

Follow these recommendations:

  • Use one consumer group per business function.
  • Choose partition count carefully.
  • Keep consumers stateless.
  • Commit offsets after successful processing.
  • Design consumers to be idempotent.
  • Monitor consumer lag.
  • Tune poll intervals.
  • Avoid long-running processing inside listeners.
  • Use retries and Dead Letter Topics.
  • Test rebalancing scenarios.

Enterprise Architecture

flowchart TD

Producer --> KafkaCluster["Kafka Cluster"]

KafkaCluster["Kafka Cluster"] --> ConsumerGroup["Consumer Group"]

ConsumerGroup["Consumer Group"] --> Consumer1["Consumer 1"]

ConsumerGroup["Consumer Group"] --> Consumer2["Consumer 2"]

ConsumerGroup["Consumer Group"] --> Consumer3["Consumer 3"]

Consumer1["Consumer 1"] --> BusinessService["Business Service"]

Consumer2["Consumer 2"] --> BusinessService["Business Service"]

Consumer3["Consumer 3"] --> BusinessService["Business Service"]

Consumer Group Lifecycle

sequenceDiagram
participant Kafka
participant Consumer A
participant Consumer B
Kafka->>Consumer A: Assign Partition 0
Kafka->>Consumer B: Assign Partition 1
Consumer B-->>Kafka: Crash
Kafka->>Consumer A: Reassign Partition 1

Consumer Group Overview

mindmap
  root((Consumer Groups))
    Partitions
    Consumers
    Rebalancing
    Offsets
    Parallel Processing
    Fault Tolerance

Single Consumer vs Consumer Group

Single Consumer Consumer Group
One Consumer Multiple Consumers
Limited Throughput High Throughput
No Parallelism Parallel Processing
Limited Scalability Horizontal Scaling
Single Point of Failure Fault Tolerant

Real Banking Example

A bank receives thousands of transfer requests every second.

Transfer Service

↓

payments-topic

↓

12 Partitions

↓

Consumer Group

↓

Fraud Detection

↓

Ledger Processing

↓

Notification

Each partition is processed by only one consumer in the group, allowing transfers to be processed in parallel while preserving ordering within each partition.

Senior Interview Tips

Interviewers commonly ask:

  • What is a Consumer Group?
  • Why are Consumer Groups needed?
  • Can multiple consumers read the same partition?
  • What happens when consumers exceed partitions?
  • What happens when partitions exceed consumers?
  • What happens if a consumer crashes?
  • What is Kafka Rebalancing?
  • How does Spring Boot manage Consumer Groups?
  • How are offsets managed?
  • How do you scale Kafka consumers?

Remember:

  • One partition is assigned to only one consumer within a consumer group.
  • Different consumer groups can independently read the same topic.
  • Consumer Groups provide scalability, parallelism, and fault tolerance.

Quick Revision

  • Consumer Groups allow multiple consumers to process Kafka topics in parallel.
  • Kafka automatically assigns partitions among consumers in the same group.
  • One partition is processed by only one consumer within a group.
  • Different consumer groups maintain independent offsets.
  • Extra consumers remain idle when partitions are insufficient.
  • Consumers handle multiple partitions when partitions exceed consumer count.
  • Consumer failures trigger automatic rebalancing.
  • Spring Boot integrates Consumer Groups through @KafkaListener.
  • Monitor consumer lag, rebalance events, and processing time in production.
  • Consumer Groups are the foundation of scalable and fault-tolerant Kafka applications.