Unit 6.3 · about 3 h
276 / 324SLIs, SLOs, error budgets and recovery objectives
SLOs turn monitoring into decisions: ship or freeze, page or ticket. Readiness and liveness from the Programming health-containers-runbooks unit, taken as assumed background, become SLI inputs here, and the error budget prices every later choice about load, rollout and incidents.
- derive
You can now
- Derive an error budget from an SLO and window, and convert it into allowed failures and minutes.
- Combine component availabilities for series and parallel topologies, stating the independence assumption.
- Specify RPO and RTO for a store, and check backup freshness against the RPO.
Covers
- Availability, durability, SLI, SLO and error budget
- Recovery point, recovery time and backup freshness
Study, in this order
6 steps
- SLIs, SLOs, SLAs and error budgetsGoogle Site Reliability Engineering - selected core chapters
Service Level Terminology (Indicators; Objectives, with The Global Chubby Planned Outage; Agreements); Indicators in Practice (availability as yield; durability); Aggregation; Objectives in Practice (Defining Objectives with 90%, 99% and 99.9% bands; error budget paragraph)
- Choosing targets and setting expectationsGoogle Site Reliability Engineering - selected core chapters
Choosing Targets (do not pick from current performance; as few SLOs as possible); SLOs Set Expectations (safety margin; do not overachieve)
- Risk and the motivation for error budgetsGoogle Site Reliability Engineering - selected core chapters
Measuring Service Risk; Motivation for Error Budgets; Forming Your Error Budget
- SLI ratios and the worked API exampleGoogle SRE Workbook - SLO, load, incident and canary exercises
What to Measure: Using SLIs (good events over total events); A Worked Example (API 97% availability, 90% under 450 ms, 99% under 900 ms; 109,897-error budget); Choosing an Appropriate Time Window (four-week rolling)
- Error budget policy and decisionsGoogle SRE Workbook - SLO, load, incident and canary exercises
Establishing an Error Budget Policy; Documenting the SLO and Error Budget Policy; Decision Making Using SLOs and Error Budgets (single release consuming 13% of budget)
- Burn-rate alert shapesGoogle SRE Workbook - SLO, load, incident and canary exercises
Ways to Alert on Significant Events (Alert on Burn Rate; Multiwindow, Multi-Burn-Rate Alerts)
Depth · Read for alert-shape vocabulary only. Alert implementation belongs to the incidents unit.
Focus on
- SLI is good events over total events; SLO is the target; the error budget is 100% minus the SLO times the window event count.
- A 99.9% SLO over 30 days allows 43.2 minutes of full outage; over 3 million requests it allows 3,000 errors. The same budget reads both ways.
- Series availability multiplies (\(0.999 \times 0.995 = 0.99401\)) only for independent failures. Shared power, zones or deploys break the multiplication.
- RPO bounds data loss (backup freshness); RTO bounds downtime (restore plus failover). MTBF 720 h with MTTR 1 h gives 0.99861, and holding 0.999 needs MTTR at most 0.72072 h.
Reproduce closed-book
Traps
- A 100% SLO. Change itself is the top outage source, so 100% forbids deploys and leaves no room to improve.
- Setting the SLO from current performance. It locks in heroic effort or hides user pain; start loose and tighten with evidence.
- Quoting availability without the window and without independence. 99.9% over a week is 10.1 minutes, not 43.2.
Practice
AlgoMaster: Calculate System Availability
AvailabilityCalculator.calculate: series product versus parallel 1-minus-product-of-failures, rounded to 5 places
Attempt only after your own series and parallel estimates. Compare the platform answer with the independence assumption written down.
AlgoMaster: Plan Reliability Recovery Targets
availability (\(\text{MTBF}/(\text{MTBF}+\text{MTTR})\)), maximumMttr and meetsTarget on unrounded availability
Attempt only after your own MTBF and MTTR numbers. Check meetsTarget against the unrounded value, not the rounded one.
Self-test
-
3 million requests in 4 weeks at 99.9% availability: budget? One outage caused 1500 errors: share?
3,000 errors; the outage spent 50% of the budget. -
Two replicas at 0.99 each share one rack power feed. Combined availability?
Not 0.9999. Parallel math assumes independent failures; the shared feed fails both at once, so the pair is no better than the feed itself. -
RPO is 1 hour and the last verified backup is 90 minutes old. Pass?
No. Freshness exceeds RPO, so restore would lose up to 90 minutes. Take and verify a new backup. -
A burn-rate alert fires while the window SLO dashboard is green. What does that mean?
Budget is being consumed fast relative to the window. The multiwindow alert predicts exhaustion before the window average moves.
Completion criterion
Stop when
You derive all four numbers without notes and can defend one SLO choice with its window, budget and policy action.
Mark passed, then open 6.4 Only mark this when you can do it without notes.Leads to
- 6.8 Backups, runbooks, incident timelines and postmortems Spend the budget through response, timelines and postmortems.
- 6.7 Failure domains, correctness signals and safe rollout Protect the budget with canaries and failure-domain reasoning.
Comes after
- 6.1 Metrics, logs, traces, correlation and cardinalityTurn these signals into SLI metrics with windows and budgets.
- 6.8 Backups, runbooks, incident timelines and postmortemsFeed the measured budget loss back into targets.
- 6.9 Unit cost, headroom and simpler designsRevisit targets once cost bounds what reliability can be bought.
Resources for this unit
- Google Site Reliability Engineering - selected core chapters Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
- Google SRE Workbook - SLO, load, incident and canary exercises Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
Also in this section
- OpenTelemetry Python - metrics, logs and traces Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
- AlgoMaster System Design - quantitative and failure drills Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
- Python Experimental Systems Kit - measurement slice Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.
- Gil Tene - How NOT to Measure Latency (talk) Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt
wrk,heyorab; write the small Python generator in the saturation unit instead. - M/M/1 queue - model definition and stationary analysis Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.