Health checks, containers, supervision and runbooks

A running process is not necessarily ready to serve, and restarting it is not always a repair. Health and supervision need distinct contracts backed by an operator's recovery checks.

implement

You can now

Separate readiness, liveness and startup checks so a dependency outage does not cause a restart storm.
Package a non-root service with pinned inputs, validated runtime configuration, explicit resource limits and a tested termination path.
Write and exercise a runbook for latency, memory growth, worker death, invalid configuration and dependency recovery.

Covers

Readiness, liveness and dependency health
Container builds, non-root execution, limits and signals
Process supervision and diagnosis runbooks

Study, in this order

5 steps

Three health questions with different actions
Liveness, Readiness, and Startup Probes

Liveness probe; Readiness probe; Startup probe

Depth · Read only the contracts and failure actions. Implement local endpoints; cluster orchestration stays outside this section.

Done
Containerize a Python service
Docker guides

Containerize a Python application: Create the Docker assets, Dockerfile and .dockerignore examples; Run the application

Depth · Stop before Use containers for Python development. Adapt only the packaging to your service. The sample uses Python 3.12 and an authenticated DHI registry; use an official Python 3.14 image pinned by digest instead, with explicit USER and exec-form CMD as below. No registry login or database is needed.

Done
Reproducible inputs and a non-root entrypoint
Docker guides

Pin base image versions; Exclude with .dockerignore; Dockerfile instructions: CMD, ENV, ENTRYPOINT and USER

Done
Resource limits and supervision
Docker guides

Memory: Limit a container's access to memory, --memory and --memory-swap; CPU: --cpus; then Start containers automatically, Use a restart policy and Restart policy details

Depth · Use https://docs.docker.com/engine/containers/start-containers-automatically/ for the restart-policy reading. Do not disable the OOM killer or configure privileged scheduling.

Done
A runbook tests hypotheses and verifies recovery
Site Reliability Engineering - overload and troubleshooting

Triage; Examine; Diagnose; Test and Treat; Making Troubleshooting Easier

Depth · Skip Case Study and Negative Results Are Magic. Use the named method sections to write commands and expected observations for your small service, not an organization-wide incident process.

Done

Focus on

Readiness controls admission; liveness asks whether restarting can restore progress. Startup checks protect slow initialization. A required dependency may make readiness fail while the process remains live; do not wire every dependency failure to liveness.
Validate typed runtime configuration before becoming ready. Keep secrets outside source, build context and image layers, and redact diagnostics. Pin base-image digests and dependencies, with an explicit update policy.
Use non-root USER and exec-form CMD so the application receives termination signals. Set CPU and memory limits at runtime; CPU quota can throttle, and memory exhaustion can kill the process before it cleans up.
A plain Docker restart policy reacts to process exit, not an unhealthy flag. Choose one supervisor, cap failure restarts where appropriate and record exit/OOM evidence. A runbook names symptom, safe test, expected observation, mitigation, rollback and recovery gate.

Reproduce closed-book

Containerize a small health-enabled version of the pipeline on CPython 3.14 with a pinned base digest, non-root USER and exec-form CMD. Run with --cpus=1 --memory=128m --memory-swap=128m and inspect both configured limits and cgroup values. Validate startup configuration before readiness. Inject a required-dependency outage: readiness fails, liveness stays healthy and recovery restores readiness without a restart. Send SIGTERM with accepted work in flight: readiness drops, new work is rejected and recorded work drains before the stop deadline. In a separate run, force a nonzero process exit and verify a bounded on-failure policy restarts it. Capture exit status and restart count, then follow your runbook to confirm recovery.

Traps

Using dependency reachability as liveness. Restarting every replica during a shared outage adds load and loses useful local state.
Deleting a secret in a later Dockerfile layer. The earlier layer still contains it; prevent it from entering the image at all.
Assuming Docker HEALTHCHECK removes traffic or on-failure repairs an unhealthy but running process. Those actions need a separate consumer of the health signal.

Practice

The section mini-project covers this: health contracts, non-root bounded container execution, configuration validation, SIGTERM drain, supervised failure recovery and an exercised operational runbook.

Self-test

What does a failed readiness probe do in Kubernetes?
Why distinguish a startup check from liveness?
Does --cpus=1 reserve a dedicated physical core?
Why check OOMKilled as well as an exit code such as 137?

Completion criterion

Stop when

The container checks demonstrate non-root identity, enforced limits, dependency recovery without restarts, graceful SIGTERM handling and bounded crash recovery; another reader can execute the runbook and confirm its recovery gate.

Mark passed, then open 7.1 Only mark this when you can do it without notes.

Leads to

6.5 Profiling CPU and memory, and benchmarking honestly Re-measure the service under its actual container limits after a recovery or optimization.

Comes after

5.7 SSH trust, keys, forwarding and tmux sessionsSupervise remote services beyond a personal tmux session.
5.8 Dependency locks, run manifests and quality gatesOperate the released artifact with supervision, health checks and runbooks.
6.6 Structured logs, metrics and tracesTurn the signals into health decisions and recovery instructions.

Resources for this unit

Docker guides
Dockersupportimplementation
Use the Python container guide and build best practices to run as non-root, pin dependencies, handle signals and test CPU and memory limits.
Site Reliability Engineering - overload and troubleshooting
Googlerequiredoperations
Read Chapter 21, The Pitfalls of Queries per Second, Client-Side Throttling, Utilization Signals, Handling Overload Errors and Conclusions. Read Chapter 12, Triage, Examine, Diagnose, Test and Treat, and Making Troubleshooting Easier for a service runbook. Skip datacenter load-balancing implementation details.
Liveness, Readiness, and Startup Probes
Kubernetesrequiredreference
Read Liveness probe, Readiness probe and Startup probe for the different health decisions and failure actions. Apply the contracts to a local service; do not install Kubernetes or configure a cluster.

Also in this section

AlgoMaster Concurrency
AlgoMasterprimarycourse
Study Concurrency 101; the Python units on threads, the GIL, multiprocessing, asyncio and concurrent.futures; synchronization primitives; concurrency challenges; locking strategies; and concurrency patterns. Then solve Bounded Buffer, Thread-Safe Blocking Queue, Thread-Safe Rate Limiter and Task Scheduler with Dependencies.
Python concurrency documentation
Python Software Foundationrequiredreference
Use asyncio task groups, cancellation, queues and timeouts; concurrent.futures; multiprocessing; threading; and signal documentation to verify lifecycle and failure behavior.
High Performance Python, 3rd edition
Micha Gorelick and Ian Ozsvald, O'Reillyrequiredimplementation
Read Chapter 2 Profiling to Find Bottlenecks, Chapter 9 Asynchronous I/O, Chapter 10 The multiprocessing Module and Chapter 12 Using Less RAM, and use the companion code (https://github.com/mynameisfiber/high_performance_python_3e) as experiment starting points. Change the workloads and measure the regimes where concurrency helps, loses or only moves the bottleneck.
Operating Systems: Three Easy Pieces - concurrency
Remzi and Andrea Arpaci-Dusseausupporttheory
Read Chapters 26-33, from threads through event-based concurrency, for locks, condition variables, semaphores, concurrency bugs and event loops. Do selected homework only for a weak mechanism.
OpenTelemetry Python instrumentation
OpenTelemetrysupportoperations
Manually instrument one request so its structured logs, one low-cardinality metric and one trace share a correlation identity. Use the telemetry to locate one seeded fault.
Algorithmica - Getting Accurate Results
Sergey Slotinsupportreference
Use Measuring the Right Thing and Reducing Noise for benchmark design: input distribution, latency versus throughput, warm-up, frequency scaling and pinning. The examples are C++; the checks transfer to Python.
Timeouts, retries, and backoff with jitter
AWS Builders' Library, Marc Brookerrequiredoperations
Read Timeouts, Retries and backoff, and Jitter for deadline scope, idempotency, one retry owner and bounded retry load. Interpret the five-layer example as three total attempts per layer, not three retries plus the initial attempt.