Full Stack • Java • System Design • Cloud • AI Engineering

Database Sharding in System Design

Learn Database Sharding from a System Design perspective. Understand horizontal partitioning, shard keys, consistent hashing, resharding, cross-shard queries, Spring Boot architecture, Amazon DynamoDB, MongoDB, Vitess, CockroachDB, and real-world examples from Amazon, Netflix, Uber, and Banking systems.

Introduction

Imagine your e-commerce platform has grown rapidly.

Your database now contains:

  • 2 Billion Customers
  • 15 Billion Orders
  • 100 Billion Product Views
  • 500 Million Daily API Requests

Initially, everything was stored in a single PostgreSQL database.

Users
   ↓
Spring Boot
   ↓
PostgreSQL

After a few years, problems appear:

  • CPU reaches 100%
  • Memory becomes insufficient
  • Storage exceeds server capacity
  • Query response time increases
  • Backups take hours
  • Maintenance windows become unacceptable

Even upgrading to larger servers no longer solves the problem.

The next step is Database Sharding.

Learning Objectives

After completing this article, you'll understand:

  • What is Database Sharding?
  • Why Sharding?
  • Vertical vs Horizontal Scaling
  • Horizontal Partitioning
  • Shard Keys
  • Types of Sharding
  • Consistent Hashing
  • Cross-Shard Queries
  • Resharding
  • Spring Boot Integration
  • Amazon, Netflix, Uber Examples
  • Best Practices

What is Database Sharding?

Database Sharding is the process of splitting one large database into multiple smaller databases called Shards.

Each shard stores only part of the total data.

Instead of

1 Huge Database

we get

Database 1

Database 2

Database 3

Database 4

Why Sharding?

Imagine one database server.

flowchart TD
    U[Users]
    APP[Spring Boot]
    DB[(Single Database)]

    U --> APP
    APP --> DB

Problems

  • Limited CPU
  • Limited RAM
  • Limited Disk
  • Limited Network
  • Single Failure Point

Sharded Architecture

flowchart TD
    U[Users]
    APP[Spring Boot]

    S1[(Shard 1)]
    S2[(Shard 2)]
    S3[(Shard 3)]
    S4[(Shard 4)]

    U --> APP

    APP --> S1
    APP --> S2
    APP --> S3
    APP --> S4

Each database handles only part of the workload.

Vertical Scaling

Increase server size.

flowchart TD
    DB[(Database)]

    CPU[More CPU]

    RAM[More Memory]

    SSD[Faster Storage]

    DB --> CPU
    DB --> RAM
    DB --> SSD

Advantages

  • Simple

Disadvantages

  • Expensive
  • Hardware limits

Horizontal Scaling

Instead of upgrading one server,

add more database servers.

flowchart LR
    APP[Spring Boot]

    D1[(DB 1)]
    D2[(DB 2)]
    D3[(DB 3)]
    D4[(DB 4)]

    APP --> D1
    APP --> D2
    APP --> D3
    APP --> D4

This is horizontal scaling.

Horizontal Partitioning

Suppose we have

100 Million Customers.

Split them.

Customer 1 - 25 Million

↓

Shard 1

Customer 25M - 50M

↓

Shard 2

Customer 50M - 75M

↓

Shard 3

Customer 75M - 100M

↓

Shard 4

Each database stores only a subset.

Sharding Flow

flowchart TD
    CLIENT[Client]

    API[Spring Boot]

    ROUTER[Shard Router]

    S1[(Shard 1)]

    S2[(Shard 2)]

    S3[(Shard 3)]

    CLIENT --> API
    API --> ROUTER

    ROUTER --> S1
    ROUTER --> S2
    ROUTER --> S3

The router determines which shard stores the requested data.

Shard Key

A Shard Key determines where data is stored.

Example

Customer ID

or

Region

or

Tenant ID

Choosing the correct shard key is one of the most important design decisions.

Customer ID Sharding

flowchart TD
    C1[Customer 1-999999]
    C2[Customer 1000000-1999999]
    C3[Customer 2000000-2999999]

    S1[(Shard 1)]
    S2[(Shard 2)]
    S3[(Shard 3)]

    C1 --> S1
    C2 --> S2
    C3 --> S3

Geographic Sharding

flowchart TD
    USA[USA]

    EUROPE[Europe]

    INDIA[India]

    USDB[(US Database)]
    EDB[(EU Database)]
    IDB[(India Database)]

    USA --> USDB
    EUROPE --> EDB
    INDIA --> IDB

Useful for

  • Banking
  • Healthcare
  • GDPR compliance

Hash-Based Sharding

Hash function determines the shard.

Formula

CustomerId % NumberOfShards

Example

1005 % 4

↓

Shard 1

Hash Sharding Diagram

flowchart TD
    KEY[Customer ID]

    HASH[Hash Function]

    S1[(Shard 1)]
    S2[(Shard 2)]
    S3[(Shard 3)]
    S4[(Shard 4)]

    KEY --> HASH

    HASH --> S1
    HASH --> S2
    HASH --> S3
    HASH --> S4

Range-Based Sharding

1 - 1 Million

↓

Shard 1

1M - 2M

↓

Shard 2

2M - 3M

↓

Shard 3

Simple to implement.

Problem

Uneven growth.

Directory-Based Sharding

A lookup table decides the shard.

flowchart TD
    CLIENT[Client]

    DIRECTORY[Shard Directory]

    S1[(Shard 1)]

    S2[(Shard 2)]

    CLIENT --> DIRECTORY

    DIRECTORY --> S1
    DIRECTORY --> S2

Very flexible.

Consistent Hashing

Instead of modulo,

a hash ring distributes data.

flowchart LR
    K[Customer Key]

    H[Hash Ring]

    S[(Shard)]

    K --> H
    H --> S

Advantages

  • Easy scaling
  • Minimal data movement
  • Better load balancing

Cross-Shard Query

Suppose

SELECT *
FROM orders
WHERE amount > 5000;

Orders are stored across

  • Shard 1
  • Shard 2
  • Shard 3
  • Shard 4

Application must query every shard.

Cross-Shard Flow

flowchart TD
    QUERY[SQL Query]

    S1[(Shard 1)]
    S2[(Shard 2)]
    S3[(Shard 3)]

    MERGE[Merge Results]

    QUERY --> S1
    QUERY --> S2
    QUERY --> S3

    S1 --> MERGE
    S2 --> MERGE
    S3 --> MERGE

Resharding

As data grows,

new shards are added.

flowchart LR
    BEFORE[(2 Shards)]

    MOVE[Move Data]

    AFTER[(4 Shards)]

    BEFORE --> MOVE
    MOVE --> AFTER

Resharding should minimize downtime.

Spring Boot Architecture

flowchart TD
    CLIENT[React]

    API[Spring Boot]

    ROUTER[Shard Router]

    S1[(Shard 1)]
    S2[(Shard 2)]
    S3[(Shard 3)]

    CLIENT --> API
    API --> ROUTER

    ROUTER --> S1
    ROUTER --> S2
    ROUTER --> S3

Banking Example

Banks often shard by

  • Country
  • Branch
  • Customer Region

Example

Texas Customers

↓

Texas Database

This reduces latency and supports regional regulations.

Amazon Example

Amazon uses different databases for:

  • Orders
  • Products
  • Customers
  • Inventory

Within these services, data may also be partitioned across multiple shards to support global scale.

Netflix Example

Netflix stores massive amounts of viewing history.

Sharding distributes:

  • Users
  • Viewing History
  • Recommendations
  • Metadata

across many storage nodes.

Uber Example

Uber shards by

  • City
  • Region
  • Driver ID

This reduces query latency and keeps regional traffic localized.

Popular Sharding Technologies

Technology Supports Sharding
MongoDB ✅ Yes
Amazon DynamoDB ✅ Automatic
Vitess ✅ Yes
CockroachDB ✅ Automatic
YugabyteDB ✅ Automatic
Apache Cassandra ✅ Built-in

Advantages

  • Horizontal Scaling
  • Better Performance
  • Unlimited Growth
  • Higher Throughput
  • Reduced Database Size
  • Lower Query Latency

Disadvantages

  • Complex Queries
  • Cross-Shard Joins
  • Data Migration
  • Resharding Complexity
  • Backup Complexity
  • Operational Overhead

Monitoring

Monitor

  • Shard Size
  • Query Latency
  • CPU Usage
  • Memory Usage
  • Hot Shards
  • Cross-Shard Queries
  • Replication Lag
  • Failed Requests

Tools

  • Datadog
  • Grafana
  • Prometheus
  • Amazon CloudWatch

Common Mistakes

❌ Poor shard key selection

❌ Creating hot shards

❌ Excessive cross-shard joins

❌ Ignoring future growth

❌ Large resharding operations without planning

❌ Uneven data distribution

Best Practices

  • Choose a high-cardinality shard key.
  • Avoid sequential IDs if they create hotspots.
  • Minimize cross-shard queries.
  • Keep shard sizes balanced.
  • Monitor shard utilization continuously.
  • Automate shard provisioning.
  • Combine sharding with replication for high availability.
  • Design applications to be shard-aware when necessary.

Common Interview Questions

What is Database Sharding?

Database Sharding is the process of splitting a large database into multiple smaller databases (shards), each storing a subset of the data to improve scalability and performance.

What is a Shard Key?

A Shard Key is the field used to determine which shard stores a particular record. Common choices include Customer ID, Tenant ID, Region, or User ID.

What is the difference between Sharding and Replication?

Sharding Replication
Splits data across databases Copies the same data to multiple databases
Improves write and storage scalability Improves read scalability and availability
Different data per shard Same data on replicas

Why is choosing the correct shard key important?

A poor shard key can lead to uneven data distribution, hot shards, and performance bottlenecks. A good shard key distributes workload evenly across all shards.

When should Database Sharding be used?

Sharding should be considered when a single database can no longer meet storage, throughput, or performance requirements even after indexing, caching, replication, and vertical scaling.

Summary

Database Sharding is one of the most powerful scaling techniques used in modern distributed systems. By partitioning data across multiple databases, applications can achieve horizontal scalability, higher throughput, and support billions of records.

In this article, we covered:

  • Database Sharding fundamentals
  • Horizontal vs Vertical Scaling
  • Shard Keys
  • Hash, Range, and Directory-based Sharding
  • Consistent Hashing
  • Cross-Shard Queries
  • Resharding
  • Spring Boot integration
  • Banking, Amazon, Netflix, and Uber examples
  • Monitoring
  • Best practices

Sharding is typically introduced after optimizing queries, adding indexes, implementing caching, and using read replicas. When designed correctly, it enables applications to grow from millions to billions of records while maintaining high performance and availability.

Loading likes...

Comments

Share a question, correction, or practical insight about this article.

Loading approved comments...