Backend interview prep

Start

Spring Kafka Consumer Groups Interview Questions and Answers

Master Spring Kafka Consumer Groups with interview questions covering consumer groups, partition assignment, rebalancing, scaling, ordering guarantees, cooperative rebalancing, static membership, and production best practices.

Introduction

A single Kafka consumer can process only a limited number of messages.

Enterprise applications such as

  • Banking
  • E-commerce
  • Insurance
  • Trading
  • Logistics

often process millions of events per day.

To increase throughput, Kafka introduces the concept of Consumer Groups.

A Consumer Group allows multiple consumer instances to work together while ensuring that each message is processed only once within the group.

Consumer Group Architecture

flowchart LR

KafkaTopic --> Partition0

KafkaTopic --> Partition1

KafkaTopic --> Partition2

Partition0 --> Consumer1

Partition1 --> Consumer2

Partition2 --> Consumer3

Q1. What is a Consumer Group?

Answer

A Consumer Group is a collection of consumers that work together to consume records from one or more Kafka topics.

Characteristics

  • One message is processed by only one consumer within the group.
  • Partitions are distributed across consumers.
  • Enables horizontal scaling.
  • Provides fault tolerance.

Each consumer belongs to a group.id.

Q2. Why do we need Consumer Groups?

Without Consumer Groups

Consumer

↓

Processes Everything

Limited throughput.

With Consumer Groups

flowchart LR

Topic --> Consumer1

Topic --> Consumer2

Topic --> Consumer3

Processing load is distributed.

Benefits

  • Scalability
  • Parallel processing
  • High availability

Q3. How do Consumer Groups work?

Workflow

  1. Consumers join the group.
  2. Kafka assigns partitions.
  3. Each partition belongs to only one consumer.
  4. Consumers process records.
  5. Offsets are committed independently.

Consumer Group Flow

sequenceDiagram
Consumer1->>GroupCoordinator: Join
Consumer2->>GroupCoordinator: Join
Coordinator->>Consumer1: Assign Partitions
Coordinator->>Consumer2: Assign Partitions
Consumers->>Kafka: Consume Records

Q4. How are partitions assigned?

Kafka automatically distributes partitions among consumers.

Example

Topic

6 Partitions

Consumer Group

3 Consumers

Assignment

Consumer1

P0

P1

Consumer2

P2

P3

Consumer3

P4

P5

Each partition belongs to only one consumer inside the group.

Q5. What happens if there are more consumers than partitions?

Example

Topic

3 Partitions

Consumer Group

5 Consumers

Assignment

Consumer1 → P0

Consumer2 → P1

Consumer3 → P2

Consumer4 → Idle

Consumer5 → Idle

Idle consumers wait until partitions become available.

Maximum parallelism equals the number of partitions.

Q6. What is Consumer Rebalancing?

When consumers join or leave a group,

Kafka redistributes partitions.

Triggers

  • New consumer joins
  • Consumer crashes
  • Partition count changes
  • Session timeout

Rebalancing

flowchart TD

ConsumerLeaves --> Rebalance

ConsumerJoins --> Rebalance

PartitionAdded --> Rebalance

Rebalance --> NewAssignments

Rebalancing ensures all partitions remain assigned.

Q7. What is Cooperative Rebalancing?

Traditional rebalancing temporarily pauses all consumers.

Cooperative rebalancing minimizes disruption.

Advantages

  • Less downtime
  • Faster recovery
  • Better availability

Configuration

partition.assignment.strategy=

org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Recommended for production systems.

Q8. What is Static Membership?

Normally,

consumer restarts trigger rebalancing.

Static membership allows Kafka to recognize restarting consumers.

Configuration

group.instance.id=

payment-service-1

Benefits

  • Fewer rebalances
  • Faster restarts
  • Better stability

Q9. How is message ordering maintained?

Kafka guarantees ordering within a partition.

Example

Customer 1001

↓

Partition 0

↓

Consumer A

Events remain ordered.

Ordering is not guaranteed across partitions.

Ordering

flowchart LR

Partition0 --> Consumer1

Partition1 --> Consumer2

Partition2 --> Consumer3

Use the same message key to preserve ordering.

Q10. Consumer Group Best Practices

Match Partitions to Consumer Count

Avoid idle consumers.

Use Stable Message Keys

Maintain ordering.

Enable Cooperative Rebalancing

Reduce downtime.

Use Static Membership

Avoid unnecessary rebalances.

Monitor Consumer Lag

Detect slow consumers quickly.

Banking Example

flowchart TD

PaymentTopic --> Partition0

PaymentTopic --> Partition1

PaymentTopic --> Partition2

Partition0 --> PaymentConsumer1

Partition1 --> PaymentConsumer2

Partition2 --> PaymentConsumer3

PaymentConsumer1 --> PostgreSQL

PaymentConsumer2 --> PostgreSQL

PaymentConsumer3 --> PostgreSQL

Each consumer processes a different partition while preserving ordering within its assigned partition.

Common Interview Questions

  • What is a Consumer Group?
  • Why use Consumer Groups?
  • How do Consumer Groups work?
  • How are partitions assigned?
  • More consumers than partitions?
  • What is Consumer Rebalancing?
  • What is Cooperative Rebalancing?
  • What is Static Membership?
  • How is ordering maintained?
  • Consumer Group best practices?

Quick Revision

Topic Summary
Consumer Group Multiple consumers working together
group.id Consumer group identifier
Partition Assignment Kafka distributes partitions
Rebalancing Redistribute partitions
Cooperative Rebalancing Incremental reassignment
Static Membership Stable consumer identity
Ordering Guaranteed per partition
Idle Consumer No assigned partition
Consumer Lag Processing delay
Horizontal Scaling Add more consumers

Consumer Group Lifecycle

sequenceDiagram
Consumer->>GroupCoordinator: Join Group
Coordinator->>Kafka: Assign Partitions
Kafka-->>Consumer: Partition Assignment
Consumer->>Kafka: Poll Records
Consumer->>BusinessService: Process
BusinessService-->>Consumer: Success
Consumer->>Kafka: Commit Offset
Consumer->>Coordinator: Heartbeat

Production Example – Banking Payment Processing Cluster

A banking platform processes 20 million payment events daily.

Architecture

  • Topic: payment-events
  • Partitions: 12
  • Consumer Group: payment-group
  • Consumer Instances: 6

Workflow

  1. Six payment service instances join the same consumer group.

  2. Kafka distributes the twelve partitions evenly.

  3. Each instance processes two partitions.

  4. Messages with the same customerId always go to the same partition.

  5. Ordering is preserved for every customer's transactions.

  6. If one instance crashes:

    • Kafka detects the missing heartbeats.
    • Remaining consumers rebalance.
    • Unprocessed partitions are reassigned automatically.
@KafkaListener(
    topics = "payment-events",
    groupId = "payment-group"
)
public void process(
    PaymentEvent event
) {

    paymentService.process(event);

}

flowchart LR

PaymentEventsTopic --> P0

PaymentEventsTopic --> P1

PaymentEventsTopic --> P2

PaymentEventsTopic --> P3

PaymentEventsTopic --> P4

PaymentEventsTopic --> P5

P0 --> Consumer1

P1 --> Consumer1

P2 --> Consumer2

P3 --> Consumer2

P4 --> Consumer3

P5 --> Consumer3

Consumer1 --> PaymentDB

Consumer2 --> PaymentDB

Consumer3 --> PaymentDB

Production Configuration Example

spring.kafka.consumer.group-id=payment-group
spring.kafka.listener.concurrency=6
spring.kafka.consumer.properties.partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
spring.kafka.consumer.properties.group.instance.id=payment-service-1

This configuration enables scalable processing, minimizes rebalance interruptions, and maintains ordering guarantees for customer-specific events.

Key Takeaways

  • Consumer Groups enable multiple consumers to process Kafka events collaboratively while ensuring each partition is consumed by only one consumer within the group.
  • Partitions are the unit of parallelism; maximum consumer concurrency is limited by the number of partitions.
  • Kafka automatically performs partition assignment and rebalancing as consumers join or leave the group.
  • Cooperative Rebalancing minimizes service interruption by incrementally reassigning partitions.
  • Static Membership reduces unnecessary rebalances during consumer restarts.
  • Kafka guarantees message ordering within a partition, making message keys critical for related events.
  • Proper partition sizing, consumer scaling, and lag monitoring are essential for production deployments.
  • Consumer Groups provide the scalability and fault tolerance required for high-volume event-driven enterprise applications.