Lesson 4.4 · 20 min
Kafka partitions
Keys, ordering, idle consumers, hot partitions, and what really happens to offsets when a consumer crashes.
The idea in plain words
Kafka stores events in logs that you can only add to, like a ledger, and splits each topic into partitions so many machines can share the work. Order is kept within a partition, so the key you pick decides what stays in order.
The problem it solves
A single ordered log of events can only be read so fast. To process millions of events per second you need many readers in parallel, without scrambling the order of events that belong together.
How it works
- A topic is split into partitions; each is its own ordered log.
- Each event has a key, and the key decides its partition, so all events with the same key stay in order.
- A consumer group shares the partitions: each partition is read by exactly one member.
- Each consumer records how far it has read (its offset), so it can resume after a crash.
- More partitions allow more parallel consumers, up to one consumer per partition.
Example
Payments keyed by account_id across 12 partitions: deposit then withdrawal for one account always arrive in order, while 12 consumers process different accounts in parallel. Without a key, the withdrawal could be processed before the deposit.
Key terms
- Topic
- A named stream of events, such as “payments”.
- Partition
- One ordered, append-only log inside a topic; topics are split into many.
- Key
- A value on each event (such as account_id) that decides its partition.
- Consumer group
- A set of consumers sharing a topic; each partition is read by one member.
- Offset
- An event’s position in its partition; consumers save it to know where they left off.
- Consumer lag
- How far behind the newest event a consumer is.
In this lesson you'll learn
- How the message key decides ordering and parallelism.
- How consumer groups, lag and crashes behave.
- How to size partitions and choose delivery guarantees in an interview.
Sections
- Live consumer group
- Ordering vs parallelism
- Crashes and offsets
- Design review
Cheat sheet
| Ordering | only within a partition |
|---|---|
| Consumers per group | ≤ partitions |
| Partitions needed | peak ÷ per-consumer rate |
| Delivery | at-least-once + idempotency |
| Poison message | retry, then dead-letter |
The interactive simulations and the scored design review load with JavaScript.