Unit 6.7 · about 4 h
280 / 324Failure domains, correctness signals and safe rollout
Container readiness, liveness and SIGTERM drains from the Programming health-containers-runbooks unit, taken as assumed background, tell you a process runs. This unit asks whether it serves correctly, and whether one rollout or one zone loss takes everything with it.
- explain
You can now
- Map replicas to failure domains and name the single points the replica count hides.
- Walk two simultaneous failures and state preserved and violated invariants.
- Run a canary with population, duration, representative metrics and rollback rule, and read its result.
- Separate health and readiness from domain-specific correctness signals.
Covers
- Failure domains and simultaneous-failure reasoning
- Health, readiness and domain-specific correctness signals
- Canary, rollback, version and migration tests
Study, in this order
4 steps
- Canary process and metric selectionGoogle SRE Workbook - SLO, load, incident and canary exercises
Requirements of a Canary Process; Choosing a Canary Population and Duration; Selecting and Evaluating Metrics (Metrics Should Indicate Problems; Metrics Should Be Representative and Attributable; Before/After Evaluation Is Risky); Requirements on Monitoring Data (split canary from control; metric intervals within canary duration)
- Canary isolation and gradual stagesGoogle SRE Workbook - SLO, load, incident and canary exercises
Dependencies and Isolation (shared backends couple canary and control; check absolute SLOs); Use a Gradual Canary for Better Metric Selection; Related Concepts (Blue/Green Deployment)
- Reliability test vocabularyGoogle Site Reliability Engineering - selected core chapters
Stress test; Canary test; Production Probes
Depth · Read the three test definitions only. Full disaster testing returns in the incidents unit.
- What triggers cascading failureGoogle Site Reliability Engineering - selected core chapters
Triggering Conditions for Cascading Failures (Process Death; Process Updates; New Rollouts; Organic Growth); Testing for Cascading Failures (Test Until Failure and Beyond)
Focus on
- Replicas in one zone are one failure. API across two zones survives; a database with both copies in us-east-1a does not. Count zones per component, not replicas.
- Safe quorums need \(R + W > N\) and \(2 \times W > N\). \(N=5\), \(R=3\), \(W=3\) holds; \(N=4\), \(R=2\), \(W=2\) lets two disjoint halves both take writes.
- Canary metrics must indicate user-visible problems and attribute to the change: reuse SLIs, prefer crash and failure signals in the small first stage, split canary from control in monitoring, and distrust before/after comparisons across time.
- Health says the process answers; correctness signals say the answers are right. A serving-but-wrong version passes readiness and needs a domain check (output quality, freshness watermark, reconciliation count).
Reproduce closed-book
Traps
- Counting two replicas as redundancy. Same zone, same deploy, same config or same dependency is one domain.
- Promoting a canary on aggregate error rate. A 5% canary at 20% errors reads as 1% overall; split canary versus control first.
- Splitting blame evenly between canary and control. Shared backends couple them, so also check absolute SLOs, not just the A/B gap.
Practice
AlgoMaster: Audit Failure Domains
atRiskComponents (under two distinct zones) and unavailableAfter (every replica in the failed zone), sorted without duplicates
Attempt only after your own zone map. Two replicas in one zone still share that zone.
AlgoMaster: Simulate Partition Quorums
partitionStatus (READ_WRITE, READ_ONLY, WRITE_ONLY, UNAVAILABLE) and hasSafeQuorums (\(R + W > N\) and \(2W > N\))
Attempt only after your own quorum arithmetic. A reachable majority is not a safe quorum.
Self-test
-
db replicas live in a and a; cache has one copy in c. At-risk? Dead after a fails?
At-risk: db and cache (each under two zones). Dead after a fails: db only. -
\(N=4\), \(R=2\), \(W=2\): safe quorums?
No. \(R+W\) and \(2W\) equal \(N\) instead of exceeding it, so disjoint halves can both take writes. -
Canary errors are flat but p95 is up 10%. Promote?
Not on error rate alone. Latency SLIs indicate user-visible harm; hold or roll back per the metric policy, then check attribution. -
Readiness passes but answers are wrong. What is missing?
A domain correctness signal: sampled output checks, a freshness watermark or a reconciliation count, alerted like an SLI.
Completion criterion
Stop when
You map zones, quorums and canary result without notes and can state which simultaneous pair your design survives.
Mark passed, then open 6.8 Only mark this when you can do it without notes.Leads to
- 6.8 Backups, runbooks, incident timelines and postmortems Work the timeline when the canary fails.
- 5.1 Partial failure, clocks and event ordering Reason about ordering under the same partitions.
Comes after
- 5.7 Distributed histories and deterministic failure simulationUse correctness histories alongside availability signals during faults.
- 6.3 SLIs, SLOs, error budgets and recovery objectivesProtect the budget with canaries and failure-domain reasoning.
- 6.6 Retry budgets, circuit breaking and load sheddingShed by deployment slice with canary rollback.
- 7.1 Security goals, trust boundaries and threat modelsPlace residual risks into failure domains and rollout guards.
- 7.7 Compatibility, rollback and revocation during changesApply canary and rollback mechanics to the migration plan.
Resources for this unit
- Google Site Reliability Engineering - selected core chapters Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
- Google SRE Workbook - SLO, load, incident and canary exercises Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
Also in this section
- OpenTelemetry Python - metrics, logs and traces Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
- AlgoMaster System Design - quantitative and failure drills Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
- Python Experimental Systems Kit - measurement slice Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.
- Gil Tene - How NOT to Measure Latency (talk) Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt
wrk,heyorab; write the small Python generator in the saturation unit instead. - M/M/1 queue - model definition and stationary analysis Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.