Unit 6.2 · about 3 h
275 / 324Latency distributions, throughput and saturation
Every SLO and capacity number later in this section reads a latency distribution. Assumed background: the Programming profiling-benchmarking unit, which built benchmark harnesses; this unit makes their output honest, with distributions first and one-number summaries never.
- explain
- measure
You can now
- Define latency, throughput, goodput, utilization and saturation, and separate queue wait from service time.
- Report p50, p95 and p99 from raw samples with the sample size, and explain what the mean hides.
- Measure a load sweep on your machine and name the saturated resource with its utilization.
Covers
- Latency distributions, p50, p95, p99 and tail interpretation
- Throughput, goodput, errors, utilization and saturation
Study, in this order
4 steps
- Latency, traffic, errors and saturationGoogle Site Reliability Engineering - selected core chapters
The Four Golden Signals (latency of success versus failure; traffic; errors by policy; saturation with the 99th percentile over one minute as early signal)
- Aggregation hides tailsGoogle Site Reliability Engineering - selected core chapters
Aggregation (burst averaging example; percentile bands in Figure 4-1); A Note on Statistical Fallacies (mean versus median under skew)
- Mean versus tailGoogle Site Reliability Engineering - selected core chapters
Worrying About Your Tail: 100 ms mean at 1,000 rps with 1% of requests at 5 seconds; exponential histogram buckets
- Saturation and served-traffic metricsGoogle SRE Workbook - SLO, load, incident and canary exercises
Metrics with Purpose (Saturation: track every resource, hard limits and degradation thresholds; Status of Served Traffic: break down by status and quota denial)
Focus on
- Latency is a distribution. The SRE example (mean 100 ms, 1% at 5 s) and the Figure 4-1 bands show why one mean plus one threshold cannot describe user experience.
- Goodput counts useful completions; throughput counts all completions; errors by policy can turn slow successes into failures. Saturation is the fraction of the most constrained resource, and most systems degrade before 100%.
- Separate service time from queue wait in every measurement. Utilization explains which one dominates.
- Percentiles need raw samples and \(n\). Report \(n\) with the value, never a bare p99.
Reproduce closed-book
random.seed(7), draw 5000 samples: if random.random() < 0.02 take max(1, random.gauss(400, 80)), else max(1, random.gauss(40, 8)), in ms. With nearest-rank percentiles (rank \(\operatorname{ceil}(p \times n / 100)\)) you should get mean 48.19 ms, p50 40.16 ms, p95 55.16 ms, p99 421.05 ms and max 627.33 ms. Count the samples above 200 ms (108) and above 400 ms (60), and explain why a dashboard showing mean 48 ms and p95 55 ms shows none of them.Traps
- Averaging percentiles from shards or windows. Shards with different traffic and tails give an average of p99s that is no request population's p99; pool the samples or merge histogram counts first.
- Mixing failed-request latency into success latency, or dropping it. Fast 500s flatter the mean while slow errors are worse than fast ones; track both.
- Reading 60% average utilization as headroom. The saturated resource is the most constrained one; queue wait grows on its utilization, not the average.
Practice
The section mini-project covers this: a load sweep through saturation on the learner's service reporting p50, p95 and p99 from retained raw samples with \(n\), plus throughput, goodput, queue depth, utilization, errors and recovery.
Self-test
-
In that run, mean is 48 ms and p95 is 55 ms over 5000 requests. How many requests are slower than 200 ms?
108, about 2%, and 60 of them are above 400 ms. Only the p99 (421 ms) hints at them. Count from the samples; the mean and p95 cannot tell you. -
Clients send 100 new requests/s. 10% fail on the first try and are retried once, and every retry succeeds. Offered load, throughput and goodput?
Offered load and throughput are 110 attempts/s (100 successes plus 10 failures); goodput is 100 successful requests/s. Retries add 10% load without adding goodput. -
Utilization reads 85% but latency is flat. Saturated?
Not necessarily. Saturation is about the most constrained resource and its queue. Check queue wait, queue depth, the 99th percentile trend and per-resource utilization. -
Why exponential histogram buckets?
Latency spans orders of magnitude. Buckets growing by about 3x (0-10, 10-30, 30-100, 100-300 ms) resolve both the 40 ms body and the 400 ms tail without storing every sample.
Completion criterion
Stop when
Your script prints the five stated numbers, and you can explain the hidden 2% tail and name the saturation signal you would alert on.
Mark passed, then open 6.3 Only mark this when you can do it without notes.Leads to
- 6.4 Little's Law, capacity and queueing approximations Turn the measured wait into concurrency and capacity math.
- 6.5 Queue bounds, first saturation and open-loop load Rerun the sweep with an open-loop generator that cannot hide queueing.
Comes after
- 2.8 Deterministic schedules, stress tests and contentionThroughput and tail latency get their full treatment there.
- 3.7 Connection shutdown, pools and latency decompositionTurn the decomposition into distributions and saturation analysis.
- 6.1 Metrics, logs, traces, correlation and cardinalityRead tails and saturation from the histograms this unit instruments.
Resources for this unit
- Google Site Reliability Engineering - selected core chapters Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
- Google SRE Workbook - SLO, load, incident and canary exercises Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
Also in this section
- OpenTelemetry Python - metrics, logs and traces Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
- AlgoMaster System Design - quantitative and failure drills Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
- Python Experimental Systems Kit - measurement slice Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.
- Gil Tene - How NOT to Measure Latency (talk) Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt
wrk,heyorab; write the small Python generator in the saturation unit instead. - M/M/1 queue - model definition and stationary analysis Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.