What is the Log Aggregation pattern?
Every service writes structured logs, usually JSON to stdout. An agent on each node ships them to a central store where they can be searched, filtered and correlated across services by fields such as service name, request ID and trace ID.
What problem does the Log Aggregation pattern solve?
With many instances that come and go, logs on local disks are scattered and often gone by the time you need them. During an incident you need to search all of them at once and follow one request through several services.
How do you implement the Log Aggregation pattern?
- Log structured events with consistent field names across services.
- Include service, version, instance, request ID and trace ID in every line.
- Write to stdout and let a node agent ship logs, so the app does not depend on the log store.
- Set retention and sampling for noisy logs, and keep debug level off in production by default.
- Never log secrets or personal data. Mask them in the logging library.
When should you use it?
- You run more than a handful of instances.
- Instances are short-lived, such as containers or serverless functions.
When should you avoid it?
- Never skip it in production. Only the scale and tooling change.
What do you gain?
- One place to search during incidents
- Logs survive instance restarts
- Correlates events across services
What does it cost?
- Storage and indexing cost grows fast
- Inconsistent formats limit what you can query
- Risk of leaking sensitive data into a central store
What are common mistakes with the Log Aggregation pattern?
- Unstructured text logs that cannot be filtered by field.
- No trace or request ID, so logs from different services cannot be joined.
- Logging every request body, which explodes cost and stores personal data.
Log Aggregation in practice
Kestrel Jobs
During an outage across dozens of hosts, on-call engineers had to SSH into machines one by one to read logs, and some logs disappeared when containers restarted.
Services now log structured JSON to stdout, and an agent ships it to a central store with the trace ID attached. Engineers search the whole fleet from one place.
Example decision record
- Context
- Container restarts destroy log files before anyone can read them.
- Decision
- Log structured JSON to stdout and ship it to a central store with an agent on every node.
- Consequences
- Logs survive container restarts and node drains.
- Debug-level logs are filtered in production to control cost.