SLO Design

Define what "working" means for a service, in terms a user would recognise, and alert only on that.

Purpose

Use this when:

• Defining reliability targets for a service.
• Alert fatigue is a problem, or an incident was missed by the alerts that existed.
• Choosing between competing reliability work and feature work.
• A new service is being onboarded to production.

Do not use this when:

• You are debugging a live problem — use incident-triage or observability-debugging.
• The metrics needed do not exist yet — get instrumentation working first with prometheus-troubleshooting.

What this skill assumes: request-level metrics with status and duration, and enough history to know current performance. Setting a target without knowing the current number produces an SLO that is either already violated or trivially met.

Operating Procedure

1. Identify the user and the journey. Who is affected, doing what? "The API" is not a journey; "a customer completing checkout" is.
2. Choose SLIs from the user's perspective. Measured as close to the user as practical.
3. Measure current performance over at least 4 weeks, including a peak period.
4. Set the SLO just below achievable reality, not at an aspirational number.
5. Derive the error budget and state what happens when it is exhausted. An SLO with no consequence is a dashboard, not a target.
6. Write burn-rate alerts, not threshold alerts.
7. Review quarterly. An SLO that has never been close to violation is too loose to be useful.

The current performance number is Observed. The target is a Recommendation with a business trade-off attached — always present it that way, never as a technical fact.

Investigation

Step 1 - pick the SLI

A good SLI is a ratio of good events to valid events. Ratios are naturally normalised, work across traffic levels, and make the error budget arithmetic straightforward.

SLI = good events / valid events

Three rules that decide whether the SLI is worth anything:

1. Measure as close to the user as you can. Server-side metrics miss the failures that never reached the server — DNS, TLS, load balancer, the client's own network. Those are real outages your dashboards will show as perfect.
2. Exclude what the user did not do. A 400 from an invalid client request is not your failure. Define valid events explicitly.
3. Combine availability and latency into one SLI where you can. A request that takes 30 seconds is a failure even with a 200 status; users do not distinguish.

# Availability-and-latency SLI in one expression
(
  sum(rate(http_request_duration_seconds_bucket{le="0.5",status!~"5.."}[5m]))
)
/
(
  sum(rate(http_request_duration_seconds_count[5m]))
)

The numerator counts requests that were both fast enough and not server errors. Note this requires a histogram bucket boundary at exactly your latency threshold — choose the SLO threshold from the buckets you have, or add the bucket first.

Step 2 - measure before you target

# 28-day achieved SLI
sum(increase(http_requests_total{status!~"5.."}[28d]))
  / sum(increase(http_requests_total[28d]))

Run this before proposing any number. Then:

Setting an SLO above current performance means starting in violation, which teaches the team to ignore it. Setting it far below means it never triggers anything.

Step 3 - the error budget

99.9% over 30 days = 43.2 minutes of budget
99.95%             = 21.6 minutes
99.99%             =  4.3 minutes

Two things make this useful, and without both the SLO is decoration:

• A policy. What changes when the budget is exhausted? The standard answer: feature deploys pause and reliability work takes priority until the budget recovers. Agree this with whoever owns the roadmap, in advance, in writing.
• Honesty about the cost. Each extra nine costs roughly an order of magnitude more. 99.99% over 30 days is 4.3 minutes — less than one unattended restart. If the team cannot respond in four minutes, the number is fiction.

1 - (
  (1 - (sum(increase(http_requests_total{status!~"5.."}[30d]))
        / sum(increase(http_requests_total[30d]))))
  / (1 - 0.999)
)

This is the fraction of budget remaining. Put it on the dashboard; it is the number that drives decisions.

Step 4 - burn-rate alerts

A threshold alert ("error rate > 1%") fires for a 30-second blip and stays quiet through a slow leak that consumes the whole month's budget. Burn-rate alerting fixes both.

Burn rate is how fast you are consuming budget relative to steady state. Burn rate 1 exhausts the budget exactly at the end of the window; burn rate 14.4 exhausts it in 2 days.

(
  sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > (14.4 * 0.001)
)
and
(
  sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
)

The two-window construction is the important part. The long window gives significance — it will not fire on a brief spike. The short window gives reset speed — the alert clears promptly once the problem stops, instead of hanging for an hour after recovery. Using either alone gives you a noisy alert or one that never clears.

0.001 here is 1 - SLO for a 99.9% target. Substitute your own.

Step 5 - check the alert would have caught real incidents

Take the last three incidents and ask: would this alert have fired, and how early? This is the only honest test of an alerting design, and it routinely reveals that the proposed page would have been silent for the incident everyone remembers — usually because the SLI was measured server-side and the outage was in front of the server.

Decision Tree

Designing an SLO
├── Can you measure the user journey end to end?
│   ├── Yes → SLI at that boundary
│   └── No  → nearest proxy, and write down what it misses
├── What does failure look like to the user?
│   ├── Errors           → availability SLI
│   ├── Slowness         → latency SLI, or a combined one
│   ├── Stale data       → freshness SLI
│   └── Wrong data       → correctness SLI (hardest; usually needs sampling)
├── What is achieved today?
│   ├── Unknown → measure for 4 weeks before setting a target
│   └── Known   → set the SLO just below it
├── Is there a policy for budget exhaustion?
│   ├── No  → the SLO will be ignored; get agreement first
│   └── Yes → implement burn-rate alerts
└── Would the alert have caught your last three incidents?
    ├── No  → the SLI is measured in the wrong place
    └── Yes → ship it, review in a quarter

Common Causes

Safety

Read-only

Everything in this skill: querying historical metrics, computing achieved performance, writing alert expressions and evaluating them. Nothing here changes a running system.

Potentially disruptive

The outputs have operational consequences even though the analysis does not:

• Deploying new alert rules — a mistuned burn-rate alert pages people at 03:00. Validate the expression against historical data first (promtool or a range query over the last month) and confirm how often it would have fired.
• Removing old alerts as part of the migration — this is the risky half. Run the new alerts in parallel for at least one full incident cycle before deleting the old ones, or you will discover the coverage gap during an outage.
• Adopting an error-budget policy — pausing feature deploys is a real organisational change. Agree it with the people it affects rather than announcing it.

Destructive

This skill proposes no destructive operations.

Verification

An SLO is verified by how it behaves, over time, against real events.

Before: 47 alerts/week, 3 of them actionable; the last outage was reported by a customer first
After:  4 alerts/week, 4 actionable; the last outage paged 6 minutes before the first report

Check these at the end of the first quarter:

1. Did the SLI move during real incidents? If it stayed flat through an outage, it measures the wrong thing. This is the single most important check.
2. Did the page fire, and early enough? Compare page time against the first customer report.
3. Was every page actionable? A page nobody acts on is noise regardless of its design.
4. Was the budget ever close to exhausted? Never close means the target is too loose to inform any decision.
5. Did the policy get applied? A budget breached with no consequence means you have a dashboard, not an SLO.

Anti-Patterns

Bad: Target 99.99% because it sounds appropriately serious. Better: That is 4.3 minutes a month. Every deploy, dependency hiccup and certificate renewal eats into it. Pick the number the team can actually defend and staff.

Bad: Alert when error rate exceeds 1%. Better: That fires on a 30-second blip and stays silent through a week-long 0.9% leak that eats the entire budget. Burn-rate alerting covers both.

Bad: An SLO on CPU utilisation. Better: Users do not experience CPU. Resource metrics are for diagnosis; SLOs are for user-visible behaviour. Keep the CPU dashboard, do not page on it.

Bad: One SLO per microservice. Better: Users experience journeys, not services. Twenty service SLOs page twenty teams for one user-visible failure and nobody owns the whole thing.

Bad: Measure availability at the application, where you have the best metrics. Better: The best-instrumented place is behind everything that commonly breaks. A DNS or load balancer failure shows as 100% availability at the application while nobody can reach it.

Bad: Set the SLO to 100%. Better: 100% means no budget for deploys, no maintenance, and any single dropped request is a breach. It also removes the entire mechanism the SLO exists to provide.

Composition

• prometheus-troubleshooting — when the metrics needed for the SLI are missing or wrong.
• observability-debugging — once an SLO alert fires and you need to find the cause.
• postmortem-analysis — error budget spend is the natural input to a postmortem.
• kubernetes-production-readiness — an SLO with an alert is a readiness requirement.

References

• Google SRE Workbook: Implementing SLOs — the source of the multi-window burn-rate table used here.
• Google SRE Workbook: Alerting on SLOs — the derivation of the burn-rate thresholds.