Section 6 of 7 · Systems Foundations
Observability, Reliability & Quantitative Design
A system becomes operable when its requirements, load, bottlenecks and recovery behavior are measurable. This section turns the earlier service into evidence for capacity and reliability decisions.
9 units 22 topics 0/9
Units
9 in this section
- 6.1 Metrics, logs, traces, correlation and cardinality explain3 h
- 6.2 Latency distributions, throughput and saturation explain · measure3 h
- 6.3 SLIs, SLOs, error budgets and recovery objectives derive3 h
- 6.4 Little's Law, capacity and queueing approximations derive4 h
- 6.5 Queue bounds, first saturation and open-loop load measure3 h
- 6.6 Retry budgets, circuit breaking and load shedding explain · derive3 h
- 6.7 Failure domains, correctness signals and safe rollout explain4 h
- 6.8 Backups, runbooks, incident timelines and postmortems implement3 h
- 6.9 Unit cost, headroom and simpler designs derive2 h
Before moving on
Checkpoint
Predict the first saturation point, then run the service through it and explain p50, p95, p99, goodput, queue depth, utilization, errors and recovery. Diagnose one injected incident from telemetry and defend the SLO, capacity and rollback plan under a 10x workload change.
Deliberately out of scope
Leave for later
- Cloud-specific observability products
- Kubernetes, Terraform and production deployment operations
- Fleet-level capacity management and organization design
Resources for this section
Shared by every unit above
- Google Site Reliability Engineering - selected core chapters Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
- Google SRE Workbook - SLO, load, incident and canary exercises Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
- OpenTelemetry Python - metrics, logs and traces Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
- AlgoMaster System Design - quantitative and failure drills Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
- Python Experimental Systems Kit - measurement slice Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.
- Gil Tene - How NOT to Measure Latency (talk) Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt
wrk,heyorab; write the small Python generator in the saturation unit instead. - M/M/1 queue - model definition and stationary analysis Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.
Apply the section
Mini-project · Telemetry, a load sweep through the knee, overload control and one incident (12 to 16 h)
Make the service slice measurable and upright past capacity with telemetry, a predicted knee, a retained-record load sweep and overload controls. Then inject one incident, recover from the runbook and file the SLO, capacity and postmortem in the dossier.
Steps
- Instrument every request with identity, trace context, queue, handler and storage timings, status and bounded labels, export raw records with counters and fixed-bucket histograms, and enforce a cardinality budget that fails on an unbounded label.
- Predict the knee from measured service time and the queue depth that Little's Law gives at rising fractions of capacity, stating where serialization would lower the prediction.
- Sweep rising offered rates with an open-loop Poisson generator that measures from intended send time, keep every raw record across repeats, report percentiles with counts with throughput, queue depth, utilization, errors and recovery, check \(L = \lambda W\), and contrast one closed-loop run.
- Add bounded admission, client-side throttling, a per-client retry budget with jittered backoff, a breaker with a stale-marked fallback and deadline propagation, showing each control under the same injected overload.
- Write the SLO with availability and latency indicators, budget, alert and shared failure domains, derive capacity now and at higher load with what would shed instead, and gate a versioned canary with automatic rollback, tested with a deliberately slow version.
- Inject one incident chosen from a hidden list, work it from telemetry and the runbook with detection to recovery times, restore from the last verified backup with measured recovery and counted loss, and file a blameless postmortem with owned actions and one regression guard.
Deliverables
- The instrumented service with raw telemetry, the cardinality test and a script rendering percentiles and queue depth.
- The prediction with the retained-record sweep, rate table, Little's Law check and closed-loop contrast.
- Overload-control and canary comparisons under identical injected faults, and the SLO, capacity, incident and postmortem package.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Telemetry and load | Raw records carry identity and timings, the budget test fails on unbounded labels, and the sweep reports percentiles with counts at every rate. | Percentiles come from raw samples rather than averaged percentiles, and the knee is explained against the prediction. |
| Overload and rollout | Each control is shown under the same overload, and the slow canary triggers automatic rollback. | Retry and throttle evidence shows a storm contained that a plain retrying client starts. |
| Incident and planning | The timeline, runbook recovery, measured recovery, counted loss and merged guard are kept with the SLO arithmetic. | The postmortem is blameless with owned actions, and the plan names shared failure domains and SLO limits. |
Feeds into: Capstone dossier: the SLO and capacity estimate against load with the telemetry-based incident. Overload controls stay under test in the service slice.