Lesson 5.3 · 20 min

Observability

See a mean that matches no request, average three p99s into a wrong answer, run four incidents through SLO burn-rate alerts, and find the slow span in a trace.

The idea in plain words

Observability is the dashboard and black box of your system: it lets you tell what is happening inside without logging into every machine. You need it to know something is wrong, where, and why.

The problem it solves

In production you can’t attach a debugger. When users say “it’s slow”, you need data that tells you that something is wrong, where, and why, across many services.

How it works

1. Metrics: counters and timings over time (requests per second, errors, latency) to spot problems.
2. Traces: the path of one request through every service, with time spent in each, to find where.
3. Logs: detailed records of individual events, to see why.
4. Use percentiles, not averages, because the slowest users hide in the tail.
5. Set targets for what users feel (SLOs) and alert when you are burning through the error budget.

Example

The average latency is 100 ms, so all looks fine, but p99 is 3 s: 1 in 100 requests is terribly slow. A trace of a slow request shows 2.8 s spent waiting on one database query that is missing an index.

Key terms

Metrics

Numbers over time, such as requests per second or error rate.

Logs

Timestamped records of individual events.

Traces

The path of one request through all the services it touched, with timings.

Percentile (p99)

The value 99% of requests are faster than; shows the slow tail that averages hide.

SLO

An internal target, such as “99.9% of requests succeed in under 300 ms”.

In this lesson you'll learn

• What metrics, logs and traces each answer, and the RED and USE checklists.
• Why averages and averaged percentiles mislead, and what to report instead.
• How SLO burn-rate alerts page on real problems and ignore noise, and how to read a trace.

Sections

1. Theory: three signals
2. Theory: averages lie
3. Theory: SLO alerts
4. Theory: read a trace
5. Design: page or not?
6. Check yourself

Cheat sheet

The interactive simulations and the quiz load with JavaScript.