Lesson 3.4 · 15 min
Sharding
Split 100,000 orders four ways, find the hot shard, add a shard and count what moves, and see how fan-out multiplies slow requests.
The idea in plain words
When one filing cabinet is full, you split the files across several cabinets, say A–F, G–M and so on. Sharding does this with a database: each machine holds part of the data, so you can keep growing.
The problem it solves
Replication gives you more copies of the same data, but each copy still has to hold all of it and accept every write. When the data or the write rate outgrows one machine, you have to split the data itself.
How it works
- Choose a shard key, a field present on every row, such as user_id.
- Use the key to decide which database (shard) each row lives on, by hashing it or by ranges.
- Queries that include the key go to one shard; queries without it must ask every shard.
- A good key spreads writes evenly and keeps your most common queries on one shard.
Example
Shard orders by customer_id across 8 databases. “Show this customer’s orders” touches one shard. “All orders in the last hour” touches all 8. If one huge customer places 5% of all orders, their shard becomes the hot spot.
Key terms
- Shard
- One database that holds a slice of the rows.
- Shard key
- The field that decides which shard a row goes to, such as user_id.
- Hot spot
- One shard getting far more traffic than the rest.
- Cross-shard query
- A query that has to ask many shards and combine the answers; slow and complex.
- Resharding
- Moving data to a new layout when you add shards.
In this lesson you'll learn
- How the shard key decides write balance, data balance and which queries stay on one shard.
- How much data moves when you add a shard, and how to make that small.
- Why cross-shard queries are slow at the tail, and how to choose keys that avoid them.
Sections
- Theory: pick a shard key
- Theory: adding a shard
- Theory: cross-shard queries
- Design: choose keys
- Check yourself
Cheat sheet
| Shard key | in the hot queries + spreads load |
|---|---|
| Hash | even writes, no key ranges |
| Range | range scans, hot newest shard |
| hash % N → N+1 | moves N/(N+1) of keys |
| Fan-out slow | 1 − (1 − p)^shards |
The interactive simulations and the quiz load with JavaScript.