Apache Kafka Architecture

Learn Apache Kafka Architecture with Spring Boot. Understand Kafka Brokers, Topics, Partitions, Producers, Consumers, Consumer Groups, Replication, Leaders, Followers, ZooKeeper, KRaft, Offsets, Message Flow, and real-world enterprise use cases with architecture diagrams.

Introduction

Modern enterprise applications generate an enormous amount of data every second.

Examples include:

Banking Transactions
Payment Events
Customer Registrations
Insurance Claims
Stock Market Trades
IoT Sensor Data
Website Click Streams
Mobile Notifications
Healthcare Records

Traditional request-response communication struggles to handle millions of events efficiently.

To process high-volume event streams reliably, organizations use Apache Kafka.

Today, Kafka powers some of the world's largest technology companies because it provides:

High Throughput
Horizontal Scalability
Fault Tolerance
Event Streaming
Distributed Messaging
Durable Storage

Kafka has become one of the most important technologies in modern event-driven architectures.

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform.

It allows applications to:

Publish Events
Store Events
Process Events
Replay Events
Stream Events

Kafka is not just a message queue.

It is a distributed commit log that stores events for a configurable retention period.

Why Kafka?

Imagine an online shopping application.

Customer places an order.

Several systems need the same event:

Payment Service
Inventory Service
Notification Service
Billing Service
Analytics Platform
Fraud Detection

Without Kafka:

Every service calls every other service.

The system becomes tightly coupled.

With Kafka:

One event is published.

Many services consume it independently.

High-Level Kafka Architecture

flowchart LR

PRODUCER[Producer]

PRODUCER --> KAFKA[(Kafka Cluster)]

KAFKA --> CONSUMER1[Payment Service]

KAFKA --> CONSUMER2[Inventory Service]

KAFKA --> CONSUMER3[Notification Service]

KAFKA --> CONSUMER4[Analytics Service]

Kafka becomes the central event backbone.

Core Components

Kafka consists of:

Producer
Broker
Topic
Partition
Consumer
Consumer Group
Offset
Leader
Follower
Controller (KRaft)

Each plays a critical role.

Producer

A Producer publishes events.

Example:

Order Service


Order Created

↓

Kafka Topic

Producers do not know who consumes the event.

Broker

A Broker is a Kafka server.

Responsibilities:

Store Messages
Replicate Data
Serve Consumers
Handle Producers

A Kafka Cluster contains multiple brokers.

Kafka Cluster

flowchart LR
    PRODUCER["Producer"]

    CLUSTER["Broker Cluster"]

    B1["Broker 1"]
    B2["Broker 2"]
    B3["Broker 3"]

    CONSUMERS["Consumers"]

    PRODUCER --> B1

    B1 <--> B2
    B2 <--> B3

    B1 --> CONSUMERS
    B2 --> CONSUMERS
    B3 --> CONSUMERS

Multiple brokers provide scalability and fault tolerance.

Topic

A Topic is a logical stream of events.

Examples:

orders
payments
customers
shipments
notifications

Messages are written into topics.

Topic Architecture

flowchart LR
    PRODUCER["Producer Service"]

    ORDERS["Orders Topic (Kafka Broker)"]

    CONSUMER_A["Consumer A (Service 1)"]
    CONSUMER_B["Consumer B (Service 2)"]

    PRODUCER --> ORDERS

    ORDERS --> CONSUMER_A
    ORDERS --> CONSUMER_B

A topic can have many producers and consumers.

Partition

Topics are divided into partitions.

Example:


Orders

↓

Partition 0

Partition 1

Partition 2

Partitions enable:

Parallel Processing
Horizontal Scaling
Ordering within a partition

Partition Architecture

flowchart LR
    TOPIC["Orders Topic"]

    P0["Partition 0"]
    P1["Partition 1"]
    P2["Partition 2"]

    TOPIC --> P0
    TOPIC --> P1
    TOPIC --> P2

Messages with the same key are typically routed to the same partition.

Why Partitions?

Suppose:

1 Million Orders.

Without partitions:

One broker processes everything.

With partitions:

Three brokers process data simultaneously.

This significantly improves throughput.

Message Ordering

Kafka guarantees ordering within a partition.

Example:


Order Created

↓

Payment Completed

↓

Shipped

↓

Delivered

These events remain ordered if they are written to the same partition.

Offset

Every message has an Offset.

Example:


Partition 0

Offset 0

Offset 1

Offset 2

Offset 3

Offsets uniquely identify messages inside a partition.

Consumers use offsets to resume processing.

Consumer

Consumers read messages.

Examples:

Payment Service
Inventory Service
Analytics
Email Service

Consumers pull messages from Kafka.

Consumer Group

Multiple consumers can form a Consumer Group.

flowchart LR
    TOPIC["Topic"]

    GROUP["Consumer Group"]

    C1["Consumer 1"]
    C2["Consumer 2"]
    C3["Consumer 3"]

    TOPIC --> GROUP

    GROUP --> C1
    GROUP --> C2
    GROUP --> C3

Each partition is processed by only one consumer within the group.

Consumer Group Scaling


3 Partitions

↓

3 Consumers

↓

Parallel Processing

Adding consumers improves scalability until the number of consumers exceeds the number of partitions.

Leader and Followers

Each partition has:

Leader
Followers

flowchart LR
    LEADER["Leader Partition"]

    FOLLOWER_1["Follower Replica 1"]
    FOLLOWER_2["Follower Replica 2"]

    PRODUCER["Producer"]

    LEADER --> FOLLOWER_1
    LEADER --> FOLLOWER_2

    PRODUCER --> LEADER

The Leader handles reads and writes.

Followers replicate data.

Replication

Replication improves availability.

Example:


Partition

↓

Broker 1

↓

Broker 2

↓

Broker 3

If one broker fails,

another replica becomes the leader.

Leader Election

flowchart LR
    FAIL["Leader Failure"]

    PROMOTE["Follower Promotion"]

    NEW["New Leader Elected"]

    FAIL --> PROMOTE --> NEW

This enables fault tolerance without losing committed data.

ZooKeeper (Legacy)

Older Kafka versions used ZooKeeper for:

Metadata
Leader Election
Cluster Management

KRaft Mode (Modern Kafka)

Modern Kafka replaces ZooKeeper with KRaft (Kafka Raft Metadata Mode).

Benefits:

Simpler Architecture
Better Scalability
Faster Startup
Easier Operations

New Kafka deployments should prefer KRaft mode.

Message Flow

sequenceDiagram

participant Producer

participant Broker

participant Consumer

Producer->>Broker: Publish Event

Broker-->>Producer: ACK

Consumer->>Broker: Poll Messages

Broker-->>Consumer: Events

Kafka uses a pull model for consumers.

Message Lifecycle

flowchart LR
    CREATE["Create Event"]

    PUBLISH["Publish"]

    STORE["Store in Topic"]

    REPLICATE["Replicate"]

    CONSUME["Consume"]

    COMMIT["Commit Offset"]

    CREATE --> PUBLISH --> STORE --> REPLICATE --> CONSUME --> COMMIT

Events remain in Kafka according to the configured retention policy.

Message Retention

Unlike traditional queues,

Kafka retains events.

Example:


Store

↓

7 Days

↓

Delete

Retention can also be configured based on:

Time
Size
Log Compaction

Consumers can replay historical events.

Event Replay

flowchart LR
    RESET["Offset Reset"]

    REPLAY["Replay Engine"]

    REPROCESS["Reprocess Events"]

    RESET --> REPLAY --> REPROCESS

Replay enables:

Recovery
Analytics
New Consumers
Bug Fixes

Kafka Delivery Guarantees

Kafka supports:

At Most Once

Possible message loss.

At Least Once

Messages may be duplicated.

Most commonly used.

Exactly Once

No duplicates.

Requires producer, broker, and consumer coordination.

Useful for financial systems.

Spring Boot Integration

Spring Boot integrates with Kafka through:

Spring for Apache Kafka.

Common components:

KafkaTemplate
@KafkaListener
ProducerFactory
ConsumerFactory

Spring simplifies producing and consuming events.

Enterprise Architecture

flowchart TD

CLIENT[Users]

CLIENT --> ORDER[Order Service]

ORDER --> KAFKA[(Kafka Cluster)]

KAFKA --> PAYMENT[Payment Service]

KAFKA --> INVENTORY[Inventory Service]

KAFKA --> BILLING[Billing Service]

KAFKA --> NOTIFICATION[Notification Service]

KAFKA --> ANALYTICS[Analytics Platform]

Each service remains loosely coupled.

Banking Example

Money Transfer


Transfer Event

↓

Kafka

↓

Ledger

↓

Fraud Detection

↓

Notifications

Every service processes the event independently.

Insurance Example

Claim Created


Claim

↓

Kafka

↓

Claims

↓

Billing

↓

Analytics

Healthcare Example

Patient Registered


Patient

↓

Kafka

↓

Appointments

↓

Billing

↓

Laboratory

Retail Example

Order Created


Order

↓

Kafka

↓

Inventory

↓

Warehouse

↓

Shipping

↓

Email

Advantages

High Throughput
Horizontal Scalability
Durable Storage
Event Replay
Fault Tolerance
Loose Coupling
Distributed Architecture
Excellent Performance

Challenges

Operational complexity
Partition management
Ordering across partitions
Consumer lag
Schema evolution
Capacity planning
Monitoring

Kafka vs Traditional Queue

Feature	Kafka	Traditional Queue
Storage	Persistent Log	Usually Removed After Processing
Replay	Yes	Limited
Throughput	Extremely High	Moderate
Ordering	Per Partition	Queue Dependent
Consumer Groups	Yes	Limited
Event Streaming	Yes	No

Kafka vs RabbitMQ

Feature	Kafka	RabbitMQ
Primary Model	Event Streaming	Message Queue
Replay	Yes	Limited
Throughput	Very High	High
Persistence	Log-Based	Queue-Based
Best For	Event-Driven Systems	Task Processing

Best Practices

Design meaningful topic names.
Use partitions to improve scalability.
Keep messages immutable.
Make consumers idempotent.
Configure replication appropriately.
Monitor consumer lag.
Use schema versioning.
Avoid excessively large messages.
Use KRaft for new clusters.
Secure producers and consumers.

Common Mistakes

❌ Too few partitions.

❌ Too many partitions without planning.

❌ Ignoring consumer lag.

❌ Large message payloads.

❌ No replication.

❌ Sharing business logic between consumers.

❌ Ignoring monitoring.

Enterprise Use Cases

Banking

Payments
Ledger
Fraud Detection

Insurance

Claims
Billing
Notifications

Healthcare

Patient Events
Laboratory Updates
Billing

Retail

Orders
Inventory
Recommendations

IoT

Sensor Streams
Device Telemetry
Analytics

Interview Questions

What is Apache Kafka?
What is a Kafka Broker?
What is a Topic?
What is a Partition?
Why are partitions important?
What is an Offset?
What is a Consumer Group?
What is replication in Kafka?
What is KRaft?
How does Kafka guarantee ordering?

Summary

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, and scalable event processing.

Its architecture consists of:

Producers
Brokers
Topics
Partitions
Consumer Groups
Offsets
Replication
Leaders and Followers
KRaft-based metadata management

Kafka enables organizations to build event-driven systems that are resilient, loosely coupled, and capable of processing millions of events per second.

Combined with Spring Boot, Kafka powers modern architectures across banking, insurance, healthcare, retail, logistics, IoT, and cloud-native platforms, making it one of the most essential technologies for enterprise software engineers and solution architects.