Sharding and partitioning

Think of it like this

Analogy

A library outgrows its building. You open more branches and split the books between them. You could split by author's last name (A–F here, G–M there), by genre, or by a numbering scheme that spreads books evenly.

Split by genre and the fiction branch is packed while the poetry branch is empty. Split alphabetically and the branch with S (Smith!) is overloaded. Whatever you choose, “find every book published in 1999” now means calling every branch.

The problem it solves

Replication (last lesson) copies all the data to every node. That helps reads and survival, but every node still stores everything and every write still goes through one leader. Eventually the data no longer fits on one machine, or the leader can't keep up with writes. Scaling up stops being possible, or affordable.

A real example

Example

Storing messages for 500 million users

A messaging app's messages table outgrows the biggest database server. The team shards it by conversation_id: a hash of the ID picks one of 256 logical shards, and those shards are spread over 16 database servers.

All messages in a conversation live on one shard, so loading a chat is a single fast query. When the servers fill up, whole logical shards move to new machines. Nothing has to be re-split.

Some things get harder. “Search all my messages for ‘passport’” would have to ask every shard, so search gets its own index, fed from the message stream. And a giant group chat with 100,000 members can turn its shard into a hotspot, so the team watches for that.

How it works

Sharding (also called partitioning) splits the data into pieces, each a partition or shard, and stores each on a different machine. Each shard is usually also replicated for safety.

The shard key decides where each record lives. A good key:

Spreads data and traffic evenly across shards.
Keeps data that's read together on the same shard, so common queries hit one shard.
Rarely changes, because changing it means moving the record.

Try three shard keys and watch for hotspots:

Strategies

Strategy	How	Good	Bad
Hash	`hash(key) % N` or a hash ring	Even spread	Range queries hit every shard; changing N moves data unless you use consistent hashing
Range	Key ranges per shard (A–F, G–M…)	Range scans on one shard	Hotspots when keys are sequential (time, auto-increment IDs)
Directory	A lookup table maps key to shard	Full control, can move single tenants	The directory is another critical system
Geographic	By region or country	Data near users, data-residency rules	Uneven region sizes

Hashing with consistent hashing (lesson 6) is the usual default, because adding a shard only moves a small share of keys.

What gets harder

Cross-shard queries. “Top posts today across all users” must ask every shard and combine the results (scatter-gather), or be served from a separate, precomputed store.
Cross-shard transactions. Moving money between users on different shards needs a distributed transaction or a saga (a series of steps with compensating undo steps).
Unique constraints and joins across shards aren't enforced for you.
Resharding. When shards fill up, data has to move while the system stays online. Many systems start with many small logical shards (say 1,024) mapped onto a few machines, so growing means moving whole shards, not re-splitting data.
Hot keys. One extremely popular key overloads its shard no matter how good the strategy is. Fixes: cache it, or split that key across several sub-keys.

Let the database do it

Many modern databases shard for you. DynamoDB and Cosmos DB partition by the key you choose, and Spanner, Bigtable and CockroachDB split and rebalance ranges automatically. You still have to choose a good key, but you don't move data by hand.

Trade-offs

What you gain

Storage and write capacity grow by adding machines.
Each shard is smaller, so indexes fit in memory and queries on one shard are fast.
A shard failing affects only part of the data.

What it costs you

Cross-shard queries, joins and transactions become hard or slow.
A poor shard key creates hotspots that are painful to fix later.
Resharding, backups and schema changes are more complex.
Application code often needs to know about shards.

When not to use it

One machine (scaled up) plus replicas and caching still handles your data and writes. That covers most apps for a long time.
Your queries constantly need data from across the whole dataset, like heavy analytics. Use a data warehouse instead.

In the cloud

Partitioned wide-column storesEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS2

Amazon DynamoDB (opens vendor docs)
Splits tables into partitions by partition key automatically; your key choice decides hotspots.
Amazon Keyspaces (for Apache Cassandra) (opens vendor docs)
Serverless, Cassandra-compatible database.

Azure2

Azure Cosmos DB (opens vendor docs)
Every container is partitioned by a partition key you choose; partitions spread automatically.
Azure Managed Instance for Apache Cassandra (opens vendor docs)
Managed open-source Cassandra clusters.

Google Cloud2

Bigtable (opens vendor docs)
Splits tables into tablets by row-key range and rebalances them across nodes.
Spanner (opens vendor docs)
Relational database that shards (splits) data automatically and keeps strong consistency.

How you’d set it up

The partition key is hashed to pick a partition; the service moves partitions between machines for you.

App
DynamoDB table
Partitions by partition key

Interview questions

Try answering out loud first, then open the model answer.

How would you choose a shard key for an e-commerce orders table?

Model answer

List the main queries: “orders for customer X” (most common), “order by ID”, and occasional “orders in date range” for reports. Sharding by customer ID keeps a customer's orders on one shard and spreads load across many customers; I'd make order IDs contain or map to the customer's shard so lookups by order ID go straight to the right place. Reports across all customers would come from a separate analytics store fed by change data capture, not scatter-gather on production shards. I'd avoid sharding by date (hot newest shard) or by merchant if a few huge merchants dominate.

What's a hotspot and how do you deal with one?

Model answer

A hotspot is one shard getting far more traffic than the others, from a skewed shard key (sequential IDs, timestamps) or a single very popular key (a celebrity, a viral product). For skewed keys, switch to hashing or add randomness to the key. For a hot key, cache it aggressively, replicate it to more readers, or split it into several sub-keys (for example key#1 through key#10) and combine results on read. For counters, shard the counter and sum the parts.

How do you add shards without downtime?

Model answer

Pre-split into many logical shards from the start and map them onto physical nodes, so adding a node means moving whole logical shards. For each move: copy a snapshot to the new node, stream ongoing changes until it's caught up, briefly pause writes for that shard (or use dual writes), switch the routing map, then clean up the old copy. Consistent hashing or a directory service keeps the mapping change small. Managed databases automate this.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Sharding: Splitting a dataset into pieces (shards) and storing each piece on a different machine.
Shard key: The field used to decide which shard a record lives on, such as user ID.
Hotspot: One shard or server getting far more traffic than the rest.
Partition: One slice of a dataset or stream. Sharding splits data into partitions spread across machines.
Resharding: Changing how data is split across shards, which means moving data while the system keeps running.
Consistent hashing: A way to map keys to servers on a ring so that adding or removing a server only moves a small share of keys.