What is the Application Metrics pattern?
Each service exposes numeric time series: counters, gauges and histograms. A standard set for every request-handling service is rate, errors and duration (the RED method). Resources such as pools and queues get utilisation, saturation and errors (the USE method). A metrics system collects them for dashboards and alerts.
What problem does the Application Metrics pattern solve?
Logs and traces are detailed but expensive to scan. Metrics are cheap, fast to query and good at showing trends, so they are what tells you that something is wrong, ideally before users notice.
How do you implement the Application Metrics pattern?
- Instrument every endpoint and client with rate, error and duration metrics using a standard library.
- Use histograms for latency so you can see percentiles, not just averages.
- Add saturation metrics for pools, queues and breakers.
- Define service level objectives and alert on how fast the error budget is burning.
- Keep label cardinality low: no user IDs, order IDs or raw URLs as labels.
When should you use it?
- Always, for every service in production.
When should you avoid it?
- Never skip it. Keep the first version small rather than leaving it out.
What do you gain?
- Cheap to store and fast to query
- Shows trends and regressions early
- Drives alerts and capacity planning
What does it cost?
- Aggregates hide individual failures
- High-cardinality labels blow up cost
- Dashboards multiply without care
What are common mistakes with the Application Metrics pattern?
- Alerting on averages, which hide a slow tail.
- Alerting on every error instead of on SLO burn, which causes alert fatigue.
- Using IDs as labels, which can take down the metrics system.
Application Metrics in practice
Forgehub
A slow memory leak and rising latency went unnoticed until users started seeing errors. The team only learned about problems from support tickets.
Every service now exports request rate, errors and duration in a standard format, with dashboards and alerts on SLO burn rate. Latency regressions page the team before users notice.
Example decision record
- Context
- On-call engineers cannot see latency percentiles for services in real time.
- Decision
- Expose rate, errors and duration metrics from every service in a standard format.
- Consequences
- Alerts fire on SLO burn rate rather than individual errors.
- High-cardinality labels such as user or order IDs are not allowed.