Unit 6.5 · about 3 h
278 / 324Queue bounds, first saturation and open-loop load
Capacity math predicts; load proves. Assumed background: the Programming profiling-benchmarking unit, which measured one client at a time. This unit drives offered load on a schedule, which is the only way to see queueing before users do.
- measure
You can now
- Run an open-loop generator and a closed-loop driver against the same slow server and explain the different latency pictures.
- Find the first saturation point of your service and name the bottleneck that migrates as load rises.
- Retain raw load records and report percentiles from samples with n.
Covers
- Queue bounds, first saturation point and bottleneck migration
- Open-loop load and coordinated-omission vocabulary
Study, in this order
3 steps
- Coordinated omission and latency correctionGil Tene - How NOT to Measure Latency (talk)
Full talk: closed-loop testers that wait for each reply, missing samples under queueing, and the correction concept of recording intended send time
Depth · Watch for the vocabulary and the correction idea. Do not adopt
wrk,heyorab; write the small Python generator in Reproduce instead. - Load, autoscaling and the shedding case studyGoogle SRE Workbook - SLO, load, incident and canary exercises
Autoscaling (Handling Unhealthy Machines; Configuring Conservatively); Combining Strategies to Manage Load; Case Study 2: When Load Shedding Attacks
- Queue signals of saturationGoogle Site Reliability Engineering - selected core chapters
The Four Golden Signals (Saturation: 99th percentile over one minute as early signal); Choosing an Appropriate Resolution for Measurements
Focus on
- Open-loop sends on a schedule whether or not replies returned; closed-loop waits for each reply. Past the knee, closed-loop self-throttles and reports flat latency while the queue explodes.
- Seeded simulation (CPython 3.14.4, seed 21, 10 ms service, 500 ms stall every 200th request, 30 s): capacity is about 80.3 rps. Open loop at 60, 80 and 90 rps gives mean 96.3, 341.1 and 1843.1 ms and p99 502.8, 759.1 and 3727.5 ms, and the 90 rps numbers keep growing with run length. Closed loop with 5 clients gives mean 62.1 ms at 80.5 completed rps. Same server, different stories.
- The first saturation point is where queue wait passes service time and goodput stops rising with offered load. Past it the bottleneck migrates: server CPU, then queue memory, then the dependency.
- Keep every raw sample. Percentiles come from the retained records with \(n\); the generator log is part of the evidence.
Reproduce closed-book
Write a small open-loop generator with CPython 3.14.4 (
random.seed(21), Poisson schedule via random.expovariate, latency counted from the intended send time, one FIFO server with 10 ms service and a 500 ms stall on every 200th request, 30 s) and confirm the 80 rps row: mean 341.1 ms, p50 341.1 ms, p95 649.5 ms, p99 759.1 ms with nearest-rank percentiles. Then run the closed-loop twin (5 clients, next request on completion: mean 62.1 ms, p99 540.0 ms, 80.5 completed rps) and state which run to trust for tail latency and why.Traps
- Tuning the closed loop until latency looks good. Fewer clients or think time hides the queue; the offered load is no longer the stated load.
- Reading server-side latency only. Queue wait before the handler is user-visible latency; instrument arrival-to-start separately.
- Calling the knee at 100% CPU. The first bottleneck is whatever queue grows first: threads, connections, memory or the dependency.
Practice
The section mini-project covers this: an open-loop sweep of the learner's service through the predicted knee with retained raw records, measured p50, p95 and p99 with \(n\), queue depth, utilization, errors and recovery, reconciled against the Little's Law prediction.
Self-test
-
Open-loop p99 is 759 ms and closed-loop p99 is 540 ms on the same server. Which is honest?
Open-loop. Closed-loop skips the requests it never sent; the missing samples are the omitted coordination. -
Throughput rises with offered load, then flattens while p95 climbs. What happened?
First saturation: the bottleneck server reached \(\rho\) near 1. Extra offered load becomes queue wait and errors, not completions. -
Why record intended send time, not just start and end?
A request that should have gone out earlier was already waiting from the user's point of view. Measuring from the intended send time keeps that wait; measuring from the actual send drops it.
Completion criterion
Stop when
Both generators run from your file, their latency pictures disagree as stated, and you can point at the knee with the bottleneck named.
Mark passed, then open 6.6 Only mark this when you can do it without notes.Leads to
- 6.6 Retry budgets, circuit breaking and load shedding Hold the line past saturation with budgets and shedding.
- 6.4 Little's Law, capacity and queueing approximations Reconcile the measured knee with utilization math.
Comes after
- 3.8 Deterministic network fault injectionSeparate fault behavior from overload behavior under open-loop load.
- 4.5 Sorting, join algorithms and query plansRelate operator work and spill to service saturation.
- 6.2 Latency distributions, throughput and saturationRerun the sweep with an open-loop generator that cannot hide queueing.
- 6.4 Little's Law, capacity and queueing approximationsCheck the prediction by driving load through the knee.
Resources for this unit
- Google Site Reliability Engineering - selected core chapters Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
- Google SRE Workbook - SLO, load, incident and canary exercises Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
- Gil Tene - How NOT to Measure Latency (talk) Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt
wrk,heyorab; write the small Python generator in the saturation unit instead.
Also in this section
- OpenTelemetry Python - metrics, logs and traces Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
- AlgoMaster System Design - quantitative and failure drills Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
- Python Experimental Systems Kit - measurement slice Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.
- M/M/1 queue - model definition and stationary analysis Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.