Monitoring answers questions you thought of in advance: is CPU high, is the queue long, is the endpoint returning errors. Observability is whether you can answer a question you have never asked before, using data you already collected.
The three signals, and what each is for
- Metrics tell you something is wrong and roughly where. They are cheap and aggregate well.
- Traces tell you which path a slow request took through your services. Without them, distributed debugging is guesswork.
- Logs tell you what a specific execution actually did. Expensive at volume, irreplaceable at the point of diagnosis.
Cardinality is the hidden bill
Attaching a user ID to every metric feels helpful until the storage cost arrives. High-cardinality data belongs in traces and logs, sampled. Metrics should stay low-cardinality and cheap enough that nobody hesitates to add one.
Instrument the business, not just the runtime
Knowing that the payment service is up is less useful than knowing that payments per minute dropped by half. The second is what someone actually cares about, and it catches failures that leave every health check green.