Lesson 5.1 · 20 min
Timeouts, retries and idempotency
Place a timeout on a real latency tail, watch retries keep a server down after the incident ends, then make a retried payment charge only once.
The idea in plain words
If a phone call drops, you wait a moment and call back. Software does the same, but if everyone calls back at once the line stays jammed, and if the first call actually went through you may have ordered twice.
The problem it solves
Calls between services fail for brief reasons: a dropped packet, a restart, a moment of overload. Waiting forever freezes your service; giving up at once fails requests that would have worked; retrying carelessly makes outages worse or repeats actions.
How it works
- Set a timeout so a slow call can’t hold your resources forever.
- Retry only errors that may succeed next time (timeouts, 503s), never “bad request”.
- Wait longer between attempts (backoff) and add randomness (jitter) so clients don’t retry in sync.
- Cap the number of retries so an outage isn’t multiplied.
- Make retried writes idempotent with a request key, so doing them twice has no extra effect.
Example
A payment call times out. Did it go through? If you retry without a key you might charge twice. With an idempotency key, the second attempt is recognised and returns the first result: one charge, whatever happened.
Key terms
- Timeout
- How long you wait for an answer before giving up.
- Retry
- Trying a failed call again.
- Exponential backoff
- Waiting longer after each failure: 100 ms, 200 ms, 400 ms…
- Jitter
- Adding randomness to the wait so clients don’t all retry at the same instant.
- Idempotency key
- A unique id sent with a request so the server can recognize and ignore a repeat.
In this lesson you'll learn
- How to pick a timeout from a dependency's tail latency.
- Why naive retries turn a short slowdown into an outage, and what backoff, jitter, budgets and deadlines each fix.
- How idempotency keys make a retried payment safe.
Sections
- Theory: choose a timeout
- Theory: retry storms
- Theory: idempotency keys
- Design: safe to retry?
- Check yourself
Cheat sheet
| Timeout | above dependency p99, inside your deadline |
|---|---|
| Retry | transient errors only, a few times |
| Backoff | exponential + full jitter + budget |
| Layers | attempts multiply: retry at one layer |
| Retried POST | needs an idempotency key |
The interactive simulations and the quiz load with JavaScript.