Backpressure, load shedding and graceful shutdown

A queue turns a capacity mismatch into waiting and memory use. Bounding only the queue is not enough if unlimited producers or retry tasks accumulate outside it.

implement

You can now

Implement bounded admission with an explicit wait, reject or shed policy under overload.
Account for queued, executing and externally waiting work when setting memory and downstream budgets.
Stop admission, drain accepted work within a deadline and report unfinished work on forced shutdown.

Covers

Bounded queues, admission control, backpressure and load shedding
Signals, graceful shutdown and partial-work disposition

Study, in this order

4 steps

Bounded Buffer
AlgoMaster Concurrency

Problem Statement; Synchronization Challenges; Solution 2: Semaphore-Based Solution; Solution 3: Monitor-Based Solution (Condition Variables)

Depth · Skip the Key Insight assertion that empty.count + full.count always equals capacity. It omits permits reserved by in-flight producers and consumers. Do not copy starvation-free claims into Python.

Done
Bounded queues and drain accounting
Python concurrency documentation

Queue: maxsize, put, put_nowait, get, task_done, join and shutdown, including immediate shutdown warning and QueueShutDown

Done
Protect useful capacity during overload
Site Reliability Engineering - overload and troubleshooting

The Pitfalls of Queries per Second; Client-Side Throttling; Utilization Signals; Handling Overload Errors, Deciding to Retry; Conclusions

Depth · Use the local admission and retry-budget principles. Skip Google-specific datacenter balancing and quota infrastructure.

Done
Deliver termination to the event loop
Python concurrency documentation

Unix signals: loop.add_signal_handler and loop.remove_signal_handler; main-thread requirement

Done

Focus on

For \(W\) workers taking one item each and queue capacity \(Q\), admitted unfinished items are at most \(W + Q\) only if there are no extra prefetches, detached retries or unbounded admission waiters. Bound payload bytes as well as item count.
If arrival rate exceeds service rate, excess work must wait elsewhere or be rejected. A semaphore limits simultaneous work, not starts per second; rate limits and retry budgets protect a different constraint.
Queue.shutdown arrived in 3.13. Graceful shutdown rejects puts, wakes blocked putters and lets existing items drain. Call task_done once per successful get after recording the item's outcome; join tracks unfinished accounting, not business success.
On SIGTERM stop admission first, drain within a monotonic deadline, then cancel and await workers and report unfinished IDs. Immediate shutdown discards queued items; it is an abort policy, not a successful drain.

Reproduce closed-book

Make a two-worker pipeline with Queue(maxsize=2), event-gated consumers and put_nowait admission. Hold two items in workers and two in the queue, assert four admitted unfinished items and reject the fifth with QueueFull. Stop admission using shutdown(), release consumers, await join and assert all four accepted IDs have recorded terminal outcomes. Verify a blocked putter wakes with QueueShutDown. Repeat with shutdown(immediate=True): queued items are discarded and join can return without those items being processed. Add a drain deadline and report abandoned IDs rather than labelling that run successful.

Traps

Creating one producer task per arrival before awaiting a bounded put. The queue is bounded but waiting tasks and their payloads are not.
Calling task_done before recording success, failure or abandonment. join can then hide silent loss.
Treating immediate shutdown as proof of completed work. In-flight items still need accounting, and discarded queued items were not processed.

Practice

AlgoMaster: Bounded Buffer

Problem Statement, The Rules and The Goal; Solution 3: Monitor-Based Solution (Condition Variables)

Solve the bounded producer-consumer contract before reading the solution. Then adapt it to the async mini-project and add overload rejection, outcome accounting and shutdown tests. The semaphore-count shortcut noted above is not an invariant during operations.

Self-test

What changes when \(\lambda\) exceeds \(\mu\) for a sustained interval?
Is Queue(maxsize=0) a zero-capacity rendezvous?
Does join prove every accepted item succeeded?
Can a concurrency semaphore enforce ten calls per second?

Completion criterion

Stop when

Your event-gated overload test respects the four-item bound, rejects excess work, drains accepted IDs with explicit outcomes, and distinguishes forced abandonment from successful completion.

Mark passed, then open 6.5 Only mark this when you can do it without notes.

Leads to

6.6 Structured logs, metrics and traces Expose queue growth, rejection and recovery instead of hiding overload.

Comes after

6.2 Synchronization primitives and concurrency hazardsApply bounded coordination to admission and draining.
6.3 Task ownership, cancellation, timeouts and retriesMake retries and cancellation obey the pipeline's admission and drain policy.

Resources for this unit

AlgoMaster Concurrency
AlgoMasterprimarycourse
Study Concurrency 101; the Python units on threads, the GIL, multiprocessing, asyncio and concurrent.futures; synchronization primitives; concurrency challenges; locking strategies; and concurrency patterns. Then solve Bounded Buffer, Thread-Safe Blocking Queue, Thread-Safe Rate Limiter and Task Scheduler with Dependencies.
Python concurrency documentation
Python Software Foundationrequiredreference
Use asyncio task groups, cancellation, queues and timeouts; concurrent.futures; multiprocessing; threading; and signal documentation to verify lifecycle and failure behavior.
Site Reliability Engineering - overload and troubleshooting
Googlerequiredoperations
Read Chapter 21, The Pitfalls of Queries per Second, Client-Side Throttling, Utilization Signals, Handling Overload Errors and Conclusions. Read Chapter 12, Triage, Examine, Diagnose, Test and Treat, and Making Troubleshooting Easier for a service runbook. Skip datacenter load-balancing implementation details.

Also in this section

High Performance Python, 3rd edition
Micha Gorelick and Ian Ozsvald, O'Reillyrequiredimplementation
Read Chapter 2 Profiling to Find Bottlenecks, Chapter 9 Asynchronous I/O, Chapter 10 The multiprocessing Module and Chapter 12 Using Less RAM, and use the companion code (https://github.com/mynameisfiber/high_performance_python_3e) as experiment starting points. Change the workloads and measure the regimes where concurrency helps, loses or only moves the bottleneck.
Operating Systems: Three Easy Pieces - concurrency
Remzi and Andrea Arpaci-Dusseausupporttheory
Read Chapters 26-33, from threads through event-based concurrency, for locks, condition variables, semaphores, concurrency bugs and event loops. Do selected homework only for a weak mechanism.
Docker guides
Dockersupportimplementation
Use the Python container guide and build best practices to run as non-root, pin dependencies, handle signals and test CPU and memory limits.
OpenTelemetry Python instrumentation
OpenTelemetrysupportoperations
Manually instrument one request so its structured logs, one low-cardinality metric and one trace share a correlation identity. Use the telemetry to locate one seeded fault.
Algorithmica - Getting Accurate Results
Sergey Slotinsupportreference
Use Measuring the Right Thing and Reducing Noise for benchmark design: input distribution, latency versus throughput, warm-up, frequency scaling and pinning. The examples are C++; the checks transfer to Python.
Timeouts, retries, and backoff with jitter
AWS Builders' Library, Marc Brookerrequiredoperations
Read Timeouts, Retries and backoff, and Jitter for deadline scope, idempotency, one retry owner and bounded retry load. Interpret the five-layer example as three total attempts per layer, not three retries plus the initial attempt.
Liveness, Readiness, and Startup Probes
Kubernetesrequiredreference
Read Liveness probe, Readiness probe and Startup probe for the different health decisions and failure actions. Apply the contracts to a local service; do not install Kubernetes or configure a cluster.