What is the Distributed Tracing pattern?
Each service records spans, timed units of work, and passes a trace context to every service or message it calls. A tracing backend stitches the spans into one timeline of the whole request. OpenTelemetry and the W3C Trace Context headers are the standard way to do it.
What problem does the Distributed Tracing pattern solve?
When one user request crosses eight services, per-service logs cannot tell you which hop was slow or where the error began. A trace shows the full path, timings, retries and fan-out in one view.
How do you implement the Distributed Tracing pattern?
- Adopt OpenTelemetry in each service and instrument HTTP, gRPC, database and messaging clients.
- Propagate trace context in request headers and in message headers.
- Name spans consistently and add attributes such as route, status and tenant.
- Sample thoughtfully: keep all errors and slow traces, and a fraction of the rest.
- Put the trace ID into logs so you can jump between logs, metrics and traces.
When should you use it?
- Requests cross more than two or three services.
- You need to find latency, retry and error hot spots.
When should you avoid it?
- A single service where ordinary logs and metrics are enough.
What do you gain?
- Pinpoints slow and failing hops
- Shows retries, fan-out and queues
- Links logs, metrics and traces
What does it cost?
- Instrumentation effort
- Storage cost at high volume
- Context can be lost at async boundaries if you forget it
What are common mistakes with the Distributed Tracing pattern?
- Dropping the context at message brokers and background jobs.
- Sampling away the rare slow requests you wanted to investigate.
- High-cardinality attributes that blow up storage costs.
Distributed Tracing in practice
Uber
Uber went from around five hundred microservices in fall 2015 to over two thousand by early 2017. Its first tracing tool, Merckx, was designed for the monolith era and had no concept of distributed context propagation.
A dedicated tracing team turned a prototype into Jaeger, a production tracing system, and drove adoption across the company. During 2016 it was integrated into hundreds of services and was recording thousands of traces every second. Jaeger was later donated to the CNCF.
Source: Evolving Distributed Tracing at Uber Engineering (Uber, 2017)
Example decision record
- Context
- Diagnosing cross-service incidents takes hours because logs from different services cannot be correlated.
- Decision
- Instrument all HTTP, gRPC and messaging clients and servers with OpenTelemetry.
- Consequences
- Trace IDs appear in every log line and outbound request.
- Sampling keeps tracing backend costs under control.