Observability Lab
You are on call for a shop. At 0:05 the card check API that the Fraud service calls starts taking four seconds for one bank's cards, and a few checkouts in every hundred hang. Choose the metrics, logs and traces you had in place, then play the next two hours and see when someone notices, when the cause is found and what all that telemetry costs each month.
Background: Application Metrics, Log Aggregation, Distributed Tracing, Health Check API.
Run these in order
Wait for customers to complain
At 0:05 the card check API starts taking four seconds for one bank's cards. About 2% of checkouts hang. There are no metrics, and each service writes logs to its own servers. Nobody notices until support escalates complaints at 0:25, and after two hours of logging in to servers one by one the cause still has not been found.
Alert on the average
Add application metrics and alert when the average checkout time passes 500 ms. The average only moves from 180 ms to 260 ms, because 98 in 100 checkouts are still fast. The alert never fires. Drag the fault up to 5%: still nothing. Averages hide exactly the customers who are suffering.
Alert on the 99th percentile
Alert on p99 instead: the time that 99 in 100 checkouts beat. It jumps to over four seconds and the alert fires at 0:07. Per-service dashboards then show Gateway, Checkout, Payment and Fraud all slow, because every caller waits for the call below it. The deepest slow one is Fraud, but finding out why still takes another 40 minutes of reading logs server by server. Drag the fault to 0.5%: fewer than 1 in 100 checkouts are slow, and the p99 does not move.
Aggregate logs with a correlation ID
Ship every service's logs to one searchable store, and put the request's correlation ID on every line. Find one slow checkout in the gateway's access log, search for its ID, and every service's lines for that request appear together. Fraud's line says the card check took 4,012 ms for one bank. Untick the correlation ID: the logs are in one place, but matching lines by timestamp takes 20 minutes longer.
Trace 1% of requests
Add distributed tracing and keep 1 in 100 traces, decided when each request starts. Only 2% of checkouts are slow, so a slow trace is kept about once every four minutes, and three take over 12. The logs win. Try 100%: the cause is found at 0:12, but storing every trace costs more than everything else together. Try 0.1%: not one slow trace in two hours.
Keep the slow traces
Tail sampling: a collector waits until each trace is complete, then keeps every slow or failed one and 0.1% of the rest. The waterfall shows the four-second card check call within minutes of the alert, for a fraction of the price of keeping everything. Metrics tell you something is wrong, traces show where, and logs say why.
This lab simplifies reality on purpose. Investigation times are fixed estimates rather than simulated detective work, traffic is flat, there is one fault with one cause, and prices are rough monthly figures for a mid-sized shop. Real incidents often have several contributing causes, alerts work best on service level objectives rather than raw thresholds, and every tool here only helps if the team has practised using it before the incident.