Unit 6.6 · about 3 h
279 / 324Retry budgets, circuit breaking and load shedding
Assumed background: the Programming cancellation-timeouts-retries unit (deadlines, retry budgets, full jitter) and backpressure-graceful-shutdown unit (admission and drain), plus idempotent retries across RPC from the networking units. This unit prices them: amplification math sets the budget, and shedding decides who survives.
- explain
- derive
You can now
- Derive worst-case retry amplification across layers and set a retry budget that caps it.
- Explain adaptive throttling, criticality and load shedding, and place each at client, server or both.
- Propagate deadlines and cancellation so a shed request stops work at every layer.
Covers
- Deadlines, retry budgets, circuit breaking and amplification
- Admission control, load shedding and graceful degradation
Study, in this order
3 steps
- Throttling, quotas and overload errorsGoogle Site Reliability Engineering - selected core chapters
Per-Customer Limits; Client-Side Throttling (adaptive throttling, \(\text{requests} = 2 \times \text{accepts}\) cutoff, local rejection); Criticality; Utilization Signals; Handling Overload Errors (Deciding to Retry)
- Shedding, retries and deadlinesGoogle Site Reliability Engineering - selected core chapters
Queue Management; Load Shedding and Graceful Degradation; Retries; Latency and Deadlines (Picking a deadline; Deadline propagation; Cancellation propagation; Bimodal latency)
- Slow starts and cascading testsGoogle Site Reliability Engineering - selected core chapters
Slow Startup and Cold Caching; Testing for Cascading Failures (Test Until Failure and Beyond)
Depth · Read once for vocabulary. The fault harness itself runs in the incidents unit.
Focus on
- Retries multiply per layer: 2 retries at each of 3 layers turn 1 user request into at most 27 backend attempts. Budgets cap attempts per request or per window, with jitter to de-synchronize them.
- Adaptive throttling decides at the client from local accept and reject history (cutoff at \(\text{requests} = 2 \times \text{accepts}\)); per-customer quotas keep one bad customer from spending the whole backend.
- Shed early and cheaply: reject above the bound before doing work, serve degraded answers while possible, and propagate deadlines plus cancellation so shed work actually stops.
- Bimodal latency (fast rejects mixed with slow successes) is the signature of a shedding or partially overloaded path. Average it and you see neither mode.
Reproduce closed-book
Traps
- Retrying on a saturated server. The retry adds queue wait for everyone; back off with jitter and count retries against the budget.
- Shedding at the server only. Rejected requests still cost parsing, TLS and queue; client-side throttling stops them before the wire.
- A deadline without propagation. Each layer waits out its own timeout, and the user waits for the sum.
Practice
The section mini-project covers this: bounded admission, a per-customer retry budget with jitter, client-side throttle, a breaker with fallback, and propagated deadlines with cancellation in the learner's service slice, each demonstrated under injected overload.
Self-test
-
3 layers with 1 retry each: worst-case attempts per user request?
8 (\(2^3\)). Retries compose multiplicatively across layers. -
The backend spends as much CPU rejecting as serving. What changes?
Lower the adaptive multiplier toward 1.1 and throttle at the client. Server-side rejection alone still overloads the backend. -
p50 falls while p99 climbs during overload. Explain.
Bimodal path: cheap rejects pull the median down while queued successes stretch the tail. Report both modes, not the mean. -
When is a degraded answer better than an error?
When it is cheaper to compute and still useful (stale copy, partial corpus). Under extreme overload, shed to errors rather than collapse.
Completion criterion
Stop when
You derive the 27x case and defend one budget (rate, shares, breaker placement, fallback) against a retry storm you inject.
Mark passed, then open 6.7 Only mark this when you can do it without notes.Leads to
- 6.7 Failure domains, correctness signals and safe rollout Shed by deployment slice with canary rollback.
- 2.6 Bounded queues, admission and backpressure Reuse the user-space queue bounds behind the admission decision.
Comes after
- 3.6 Deadlines, retries and idempotency across RPCGeneralize retry budgets to fleet-wide overload control.
- 6.5 Queue bounds, first saturation and open-loop loadHold the line past saturation with budgets and shedding.
Resources for this unit
- Google Site Reliability Engineering - selected core chapters Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
Also in this section
- Google SRE Workbook - SLO, load, incident and canary exercises Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
- OpenTelemetry Python - metrics, logs and traces Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
- AlgoMaster System Design - quantitative and failure drills Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
- Python Experimental Systems Kit - measurement slice Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.
- Gil Tene - How NOT to Measure Latency (talk) Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt
wrk,heyorab; write the small Python generator in the saturation unit instead. - M/M/1 queue - model definition and stationary analysis Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.