Backend interview prep

Start

Kafka Producers vs Consumers Interview Questions and Answers

Learn Kafka Producers and Consumers with real-world interview questions, message flow, acknowledgements, consumer offsets, Spring Boot integration, and production best practices.

Kafka Producers vs Consumers Interview Questions and Answers

Every Kafka application is built around two primary components:

  • Producer
  • Consumer

A Producer publishes events into Kafka.

A Consumer reads those events and processes them.

Understanding how producers and consumers work internally is one of the most frequently asked Kafka interview topics.

Producer → Kafka → Consumer

flowchart LR

Producer --> KafkaTopic["Kafka Topic"]
KafkaTopic["Kafka Topic"] --> Consumer

Q1. What is a Kafka Producer?

Answer

A Kafka Producer is an application that publishes events to Kafka topics.

Examples:

  • Payment Service
  • Order Service
  • Mobile Banking
  • ATM Application
  • E-Commerce Checkout

Producer responsibilities:

  • Create events
  • Select partition
  • Compress messages
  • Retry failed requests
  • Receive acknowledgements

Producer Flow

flowchart LR

Application --> Producer
Producer --> KafkaBroker["Kafka Broker"]

Q2. What is a Kafka Consumer?

Answer

A Kafka Consumer reads events from Kafka topics.

Consumers process business logic after receiving messages.

Examples:

  • Fraud Detection
  • Inventory
  • Notification
  • Analytics
  • Audit

Consumer responsibilities:

  • Poll messages
  • Process events
  • Commit offsets
  • Retry failures

Consumer Flow

flowchart LR

KafkaTopic["Kafka Topic"] --> Consumer
Consumer --> BusinessService["Business Service"]

Q3. How does a Producer send messages?

Answer

Producer workflow:

  1. Create Event
  2. Serialize Event
  3. Select Partition
  4. Send to Leader Broker
  5. Receive ACK

Producer Lifecycle

sequenceDiagram
participant Application
participant Producer
participant Broker
Application->>Producer: Create Event
Producer->>Broker: Send Event
Broker-->>Producer: ACK

Q4. How does a Consumer receive messages?

Answer

Consumer workflow:

  1. Subscribe to Topic
  2. Poll Messages
  3. Process Business Logic
  4. Commit Offset

Consumer Lifecycle

sequenceDiagram
participant Consumer
participant Broker
Consumer->>Broker: Poll()
Broker-->>Consumer: Messages
Consumer->>Consumer: Process
Consumer->>Broker: Commit Offset

Q5. What is Polling in Kafka?

Answer

Kafka consumers continuously call poll() to fetch new messages.

Kafka is pull-based, meaning consumers control the rate of message consumption.

Advantages:

  • Better scalability
  • Handles slow consumers
  • Back-pressure support

Poll Flow

flowchart LR

Consumer --> Poll()
Poll() --> Broker

Broker --> Messages

Q6. How does a Producer choose a Partition?

Answer

Partition selection depends on:

  • Partition Key
  • Custom Partitioner
  • Round Robin (if no key)

Example:

customerId = 500

All events for customer 500 go to the same partition.

Partition Selection

flowchart LR

Producer --> Hash(customerId)

Hash(customerId) --> Partition2["Partition 2"]

Q7. What are Producer Acknowledgement (ACK) Modes?

Answer

Kafka supports three acknowledgement levels.

ACK Mode Description
acks=0 No acknowledgement
acks=1 Leader acknowledgement
acks=all Leader + ISR acknowledgement

ACK Flow

flowchart LR

Producer --> Leader

Leader --> Followers

Followers --> ACK

Best Practice

Use:

acks=all

for production systems.

Q8. How do Consumers track progress?

Answer

Consumers track progress using Offsets.

Example:

Offset 500

↓

Offset 501

↓

Offset 502

Consumers commit offsets after successful processing.

Offset Commit

flowchart LR

ReadMessage["Read Message"] --> BusinessLogic["Business Logic"]
BusinessLogic["Business Logic"] --> CommitOffset["Commit Offset"]

Q9. What happens if a Consumer crashes?

Answer

If offsets have been committed:

Consumer resumes from the last committed offset.

If offsets were not committed:

Kafka redelivers unprocessed messages.

Recovery

flowchart TD

ConsumerFailure["Consumer Failure"] --> Restart
Restart --> ReadLastOffset["Read Last Offset"]

ReadLastOffset["Read Last Offset"] --> ContinueProcessing["Continue Processing"]

Benefits

  • Fault Tolerance
  • Reliable Processing
  • Message Recovery

Q10. What are the production best practices for Producers and Consumers?

Answer

Producer Best Practices

  • Enable idempotence.
  • Use acks=all.
  • Configure retries.
  • Compress messages.
  • Use meaningful partition keys.
  • Batch small messages.

Consumer Best Practices

  • Commit offsets after processing.
  • Implement idempotent consumers.
  • Handle retries.
  • Configure Dead Letter Topics.
  • Monitor consumer lag.
  • Tune poll settings.

Enterprise Architecture

flowchart TD

RestApi["REST API"] --> ProducerService["Producer Service"]

ProducerService["Producer Service"] --> KafkaCluster["Kafka Cluster"]

KafkaCluster["Kafka Cluster"] --> ConsumerGroup["Consumer Group"]

ConsumerGroup["Consumer Group"] --> BusinessServices["Business Services"]

BusinessServices["Business Services"] --> Database

Complete Kafka Flow

flowchart LR

Producer --> Topic
Topic --> Partition

Partition --> Consumer
Consumer --> BusinessLogic["Business Logic"]

BusinessLogic["Business Logic"] --> CommitOffset["Commit Offset"]

Producer vs Consumer

mindmap
  root((Kafka))
    Producer
      Send Events
      Select Partition
      Receive ACK
    Consumer
      Poll Messages
      Process Events
      Commit Offset

Producer vs Consumer Comparison

Producer Consumer
Publishes Events Reads Events
Writes to Kafka Reads from Kafka
Uses ACK Uses Offsets
Sends Data Processes Data
Selects Partition Commits Offset

Real Banking Example

A customer transfers ₹3,00,000.

Mobile Banking

↓

Transfer Service (Producer)

↓

payments-topic

↓

Kafka Cluster

↓

Fraud Detection (Consumer)

↓

Ledger Service (Consumer)

↓

Notification Service (Consumer)

One producer publishes the transfer event, while multiple independent consumers process it for different business purposes.

Senior Interview Tips

Interviewers frequently ask:

  • What is a Kafka Producer?
  • What is a Kafka Consumer?
  • How does a Producer send messages?
  • How does a Consumer receive messages?
  • Why is Kafka pull-based?
  • How are partitions selected?
  • What are ACK modes?
  • How do consumers commit offsets?
  • What happens if a consumer crashes?
  • How do you prevent duplicate processing?
  • How do you tune producer performance?
  • How do you tune consumer performance?

Remember:

  • Producer writes events.
  • Consumer reads events.
  • ACK ensures reliable writes.
  • Offsets track consumer progress.
  • Consumers should commit offsets only after successful processing.

Quick Revision

  • Producers publish events to Kafka topics.
  • Consumers read events from Kafka partitions.
  • Producers choose partitions using keys, custom partitioners, or round robin.
  • Kafka consumers fetch data using the poll() API.
  • ACK modes control producer reliability.
  • Offsets track consumer progress.
  • Consumers should commit offsets after successful business processing.
  • Enable idempotence, retries, and compression for producers.
  • Monitor consumer lag and use Dead Letter Topics for failures.
  • Producers and consumers together form the core data flow of every Kafka-based event-driven system.