Backend interview prep

Start

Spring Batch Partitioning Interview Questions and Answers

Master Spring Batch Partitioning with interview questions covering parallel processing, partitioning, multi-threaded steps, partition handlers, remote partitioning, remote chunking, grid size, and production best practices.

Introduction

As data volumes grow into millions or even billions of records, sequential batch processing becomes too slow. Spring Batch provides several scaling techniques to improve throughput, with Partitioning being one of the most powerful.

Partitioning divides a large workload into smaller independent partitions, allowing multiple threads or multiple machines to process data in parallel.

Spring Batch supports:

  • Multi-threaded Steps
  • Partitioning
  • Remote Partitioning
  • Remote Chunking
  • Parallel Flows

Partitioning is widely used in:

  • Banking
  • Insurance
  • Financial Settlement
  • ETL
  • Data Migration
  • Report Generation

Spring Batch Scaling Architecture

flowchart LR

LargeDataset --> MasterStep

MasterStep --> Partition1

MasterStep --> Partition2

MasterStep --> Partition3

MasterStep --> Partition4

Partition1 --> Worker1

Partition2 --> Worker2

Partition3 --> Worker3

Partition4 --> Worker4

Worker1 --> Database

Worker2 --> Database

Worker3 --> Database

Worker4 --> Database

Q1. What is Partitioning?

Answer

Partitioning is a scaling technique that divides one Step into multiple smaller partitions that execute concurrently.

Instead of processing one million records sequentially,

Spring Batch splits them into multiple independent workloads.

Example

1,000,000 Records

↓

4 Partitions

↓

250,000 Each

↓

Parallel Processing

Advantages

  • Faster execution
  • Better CPU utilization
  • Horizontal scalability
  • Reduced processing time

Q2. Why is Partitioning used?

Sequential processing underutilizes available CPU cores.

Sequential

1 Thread

↓

1,000,000 Records

↓

2 Hours

Partitioning

4 Threads

↓

250,000 Records Each

↓

30 Minutes

Architecture

flowchart TD

Master --> Thread1

Master --> Thread2

Master --> Thread3

Master --> Thread4

Q3. How does Partitioning work?

Spring Batch creates

  • Master Step
  • Worker Steps
  • Partitions

Flow

flowchart LR

MasterStep --> PartitionHandler

PartitionHandler --> WorkerStep1

PartitionHandler --> WorkerStep2

PartitionHandler --> WorkerStep3

WorkerStep1 --> Reader

WorkerStep2 --> Reader

WorkerStep3 --> Reader

Each Worker Step processes a different data range.

Q4. What is a Partitioner?

A Partitioner divides work into partitions.

Interface

public interface Partitioner {

    Map<String,
    ExecutionContext>
    partition(int gridSize);

}

Example

Partition1

Rows 1-250000

Partition2

Rows 250001-500000

Partition3

Rows 500001-750000

Partition4

Rows 750001-1000000

Q5. What is a PartitionHandler?

PartitionHandler assigns partitions to worker threads or worker nodes.

Architecture

flowchart LR

MasterStep --> PartitionHandler

PartitionHandler --> Worker1

PartitionHandler --> Worker2

PartitionHandler --> Worker3

PartitionHandler --> Worker4

Responsibilities

  • Start workers
  • Monitor execution
  • Collect results
  • Aggregate status

Q6. What is Grid Size?

Grid Size determines the number of partitions.

Example

.gridSize(8)

Meaning

Grid Size = 8

↓

8 Partitions

↓

8 Workers

Choose a grid size based on available CPU cores, workload characteristics, and infrastructure capacity.

Q7. Partitioning vs Multi-threaded Step

Partitioning Multi-threaded Step
Separate ExecutionContext Shared Step
Independent readers Shared Step configuration
Better isolation Simpler configuration
Better scalability Moderate scalability
Suitable for huge datasets Suitable for medium workloads

Partitioning is generally preferred for very large datasets.

Q8. What is Remote Partitioning?

Remote Partitioning distributes partitions across multiple JVMs or servers.

Architecture

flowchart TD

MasterNode --> Kafka

Kafka --> Worker1

Kafka --> Worker2

Kafka --> Worker3

Worker1 --> Database

Worker2 --> Database

Worker3 --> Database

Benefits

  • Horizontal scaling
  • Distributed processing
  • Better fault tolerance

Q9. What is Remote Chunking?

In Remote Chunking,

  • Reader executes on Master
  • Processing and Writing execute on Workers

Architecture

flowchart LR

Reader --> Kafka

Kafka --> Worker1

Worker1 --> Processor

Processor --> Writer

Writer --> Database

Difference

Remote Chunking Partitioning
Master reads data Workers read data
Chunks sent to workers Partitions sent to workers
Useful when reading is centralized Useful when data can be partitioned

Q10. Partitioning Best Practices

Partition Independent Data

Avoid overlapping partitions.

Tune Grid Size

Do not create more partitions than the infrastructure can efficiently process.

Use Paging Readers

Each worker should process its own dataset efficiently.

Monitor Worker Failures

Failed partitions should be restartable.

Banking Example

flowchart TD

TransactionTable --> MasterStep

MasterStep --> NorthRegion

MasterStep --> SouthRegion

MasterStep --> EastRegion

MasterStep --> WestRegion

NorthRegion --> Worker1

SouthRegion --> Worker2

EastRegion --> Worker3

WestRegion --> Worker4

Worker1 --> PostgreSQL

Worker2 --> PostgreSQL

Worker3 --> PostgreSQL

Worker4 --> PostgreSQL

Each worker processes transactions for a different region.

Common Interview Questions

  • What is Partitioning?
  • Why use Partitioning?
  • What is a Partitioner?
  • What is a PartitionHandler?
  • What is Grid Size?
  • Partitioning vs Multi-threaded Step?
  • What is Remote Partitioning?
  • What is Remote Chunking?
  • When should you use Partitioning?
  • Partitioning best practices?

Quick Revision

Topic Summary
Partitioning Divide work into parallel partitions
Master Step Coordinates partition execution
Worker Step Processes one partition
Partitioner Creates partitions
PartitionHandler Assigns work to workers
Grid Size Number of partitions
Multi-threaded Step Parallel execution within one step
Remote Partitioning Distributed worker nodes
Remote Chunking Reader on master, processing on workers
ExecutionContext Stores partition state

Partitioning Lifecycle

sequenceDiagram
Job->>Master Step: Start
Master Step->>Partitioner: Create Partitions
Partitioner-->>Master Step: ExecutionContexts
Master Step->>PartitionHandler: Assign Workers
PartitionHandler->>Worker Steps: Execute
Worker Steps->>Database: Read/Write
Worker Steps-->>PartitionHandler: Status
PartitionHandler-->>Master Step: Complete
Master Step-->>Job: Finished

Production Example – Processing 10 Million Transactions

A banking application imports 10 million transactions every night.

Configuration:

  • Total records: 10,000,000

  • Grid Size: 8

  • Eight Worker Steps process 1.25 million records each.

  • Every Worker uses:

    • JdbcPagingItemReader
    • ItemProcessor
    • JdbcBatchItemWriter

  • Chunk size = 1000

  • If Worker 5 fails, only Partition 5 is restarted, while the remaining seven partitions remain completed.

flowchart LR

10MillionRows --> MasterStep

MasterStep --> Partition1

MasterStep --> Partition2

MasterStep --> Partition3

MasterStep --> Partition4

MasterStep --> Partition5

MasterStep --> Partition6

MasterStep --> Partition7

MasterStep --> Partition8

Partition1 --> Worker1

Partition2 --> Worker2

Partition3 --> Worker3

Partition4 --> Worker4

Partition5 --> Worker5

Partition6 --> Worker6

Partition7 --> Worker7

Partition8 --> Worker8

This approach reduces execution time dramatically while maintaining restartability and transactional consistency.

Key Takeaways

  • Partitioning improves Spring Batch performance by dividing large workloads into independent parallel partitions.
  • A Master Step coordinates execution, while Worker Steps process assigned partitions concurrently.
  • The Partitioner creates execution contexts describing each partition's workload.
  • The PartitionHandler distributes partitions to worker threads or worker nodes and aggregates the results.
  • Grid Size determines the number of partitions and should be tuned according to available hardware and workload.
  • Partitioning provides better scalability and isolation than a simple multi-threaded step for very large datasets.
  • Remote Partitioning distributes work across multiple JVMs, while Remote Chunking centralizes reading and distributes processing and writing.
  • Proper partition design, independent data ranges, restartability, and monitoring are essential for reliable enterprise batch processing.