Day 108: Error budgets and burn-rate alerts
The error budget: permission to take risks, quantified
If your SLO is 99.9%, your error budget is the remaining 0.1% — a concrete, spendable allowance for failures, planned maintenance, or risky deploys. As long as you're within budget, you can ship faster and take more risks; once it's exhausted, the team should shift focus to reliability work over new features. This turns a vague cultural debate ('should we prioritize speed or stability?') into a number everyone can see.
Burn-rate alerts
Alerting only when the error budget is fully exhausted is too late — by definition, you've already violated the SLO. A burn-rate alert fires when you're consuming the budget faster than sustainable, e.g. "at this current error rate, we'll exhaust the entire 28-day budget within 2 hours" — giving you advance warning while there's still budget left to react.
Fast burn (short window, e.g. 1h): catches severe, sudden incidents quickly
Slow burn (long window, e.g. 6h or 24h): catches a persistent, lower-grade degradation that a short window alone would miss
Good alerting uses BOTH windows together to avoid missing either failure shape.Same starting budget, two very different burn rates — only one needs a page right now.
This is Phase 18's "alert on symptoms, not causes" made precise
A burn-rate alert is symptom-based almost by construction — it fires based on user-facing SLI violation rate, not on any specific internal cause (CPU, a particular service). This is exactly the alert design principle Phase 18 will formalize next.
The Four Questions: error budgets (or distributed tracing)
Worked example for Docker: dependency hell → consistent runtime environments → VMs too heavy → shared kernel, weaker isolation. Apply it to error budgets: what problem (endless, unresolvable speed-vs-stability arguments) did they solve, why couldn't ad hoc judgment calls solve it, and what trade-off (requires disciplined SLO-setting and buy-in) do they introduce?
Key terms
- Error budget
- The remaining allowance for failures under an SLO, spendable on risk/velocity.
- Burn-rate alert
- Fires when the error budget is being consumed faster than sustainable, before it's fully exhausted.
Phase 17 complete — you should now be able to
Why do good burn-rate alerting setups use both a short and a long time window, rather than just one?
