Backups, runbooks, incident timelines and postmortems

All earlier evidence pays off here. Assumed background: hypothesis-driven debugging from the Programming testing-debugging-reproducibility units. This unit turns it into team procedure: timelines, roles, runbooks and postmortems.

implement

You can now

Work a timeline from telemetry and state detection, diagnosis, mitigation and recovery times.
Execute a runbook for process death, dependency outage, bad rollout, corrupt state and overload, and verify restore against RPO and RTO.
Write a blameless postmortem with owned action items and one regression guard.

Covers

Backup, restore, runbooks and incident timelines
Postmortems, regression guards and prevention actions

Study, in this order

5 steps

Troubleshooting method
Google Site Reliability Engineering - selected core chapters

Triage; Examine (Logging); Diagnose (Simplify and reduce; Ask what, where and why); Test and Treat; Negative Results Are Magic

Done
Incident response roles and drills
Google SRE Workbook - SLO, load, incident and canary exercises

Incident Management at Google (Incident Command System; Main Roles in Incident Response); Putting Best Practices into Practice (Prepare Beforehand; Drills)

Done
Postmortem culture
Google SRE Workbook - SLO, load, incident and canary exercises

Bad Postmortem versus Good Postmortem (satellite diskerase: context, blamelessness, concrete action items with owners); Organizational Incentives (Reward action item closeout)

Done
Disaster testing discipline
Google Site Reliability Engineering - selected core chapters

Testing Disaster

Depth · Read for restore-verification discipline only: backups count when a drill restores them inside RPO and RTO.

Done
Runbook and artifact evidence
Python Experimental Systems Kit - measurement slice

Failure-injection requirements (worker crash and hang, corrupt partial run, incompatible resume); Measurement requirements (distributions, queue occupancy, failed and retried work); Milestone F (reviewer-seeded unseen failure diagnosed from artifacts); short runbook for interrupted, hung or corrupt runs

Done

Focus on

Timelines carry times, not adjectives: detected at, diagnosed at, mitigated at, recovered at, plus measured RTO and RPO loss counted from the newest restorable record.
Runbooks are tested procedures with owners: one each for process death, dependency outage, bad rollout, corrupt state and capacity exhaustion, each ending in a verified restore.
Good postmortems are blameless, prompt and owned: context, mechanism, action items with owners and dates, and one regression guard that fails if the bug returns.
Negative results are evidence. A ruled-out cause with its test stays in the timeline so the next responder does not repeat it.

Reproduce closed-book

Seed one fault in your service (kill a worker mid-run, corrupt one record, or deploy the bad version behind your canary split), work it with your runbook, and produce a timeline with detection, diagnosis, mitigation and recovery times, measured RTO and counted RPO loss, plus one merged regression test. There are no fixed numbers: yours are the evidence.

Traps

Declaring recovery at mitigation. Recovery is verified restore plus SLO compliance over the window, not the moment the alert clears.
Writing the postmortem as verdict. Blame ends inquiry; mechanism plus owned actions prevent recurrence.
Trusting untested backups. An unverified backup is a hope; restore drills against RPO and RTO are the practice.

Practice

The section mini-project covers this: one injected incident worked end to end (timeline, runbook execution, verified restore with measured RTO and counted RPO loss), a blameless postmortem with owned actions, and one merged regression guard.

Self-test

The alert clears 10 minutes after the fix but p99 stays high for an hour. Recovered at 10 minutes?
Two responders each fixed something but the timeline shows one fix. Problem?
A restore drill recovers everything but the last 90 minutes with RPO 1 hour. Pass?
Action item reads improve monitoring with no owner. Accept?

Completion criterion

Stop when

Your timeline, runbook run, restore drill and postmortem exist as artifacts, and the regression test fails on the re-seeded fault.

Mark passed, then open 6.9 Only mark this when you can do it without notes.

Leads to

6.9 Unit cost, headroom and simpler designs Price the prevention work the postmortem asks for.
6.3 SLIs, SLOs, error budgets and recovery objectives Feed the measured budget loss back into targets.

Comes after

6.3 SLIs, SLOs, error budgets and recovery objectivesSpend the budget through response, timelines and postmortems.
6.7 Failure domains, correctness signals and safe rolloutWork the timeline when the canary fails.
7.4 Audit records, fail-open versus fail-closed and supply chainFeed audit trails into incident timelines and postmortems.
7.7 Compatibility, rollback and revocation during changesTurn the migration runbook into an incident timeline when it fails.

Resources for this unit

Google Site Reliability Engineering - selected core chapters
Google SREprimarycourse
Read Chapters 3, 4, 6, 10, 12, 17, 19-22: risk, SLOs, monitoring, alerting, troubleshooting, reliability tests, load balancing, overload and cascading failure. Apply every concept to the same bounded RPC and store rather than inventing separate examples.
Google SRE Workbook - SLO, load, incident and canary exercises
Google SRErequiredpractice
Use Chapters 2, 4, 9-11, 12, 14, 16 and 17. Define one SLO, estimate capacity, load through saturation, canary a changed version, roll it back, and write one evidence-based incident timeline.
Python Experimental Systems Kit - measurement slice
Roadmap-Finalsupportproject
Use its artifact, bounded-pipeline, failure-injection, profiling and reproducibility requirements as the evidence checklist. Reuse the current service instead of building the full project again.

Also in this section

OpenTelemetry Python - metrics, logs and traces
OpenTelemetryrequiredreference
Instrument request ID, queue wait, handler time, storage time, status and bounded labels. Export locally, retain raw load records and verify that instrumentation does not hide the bottleneck or create unbounded cardinality.
AlgoMaster System Design - quantitative and failure drills
AlgoMastersupportpractice
Use Solve Little's Law, Calculate System Availability, Plan Reliability Recovery Targets, Calculate Amdahl's Law Speedup, Analyze Capacity Bottlenecks, Simulate Partition Quorums and Audit Failure Domains only after producing your own estimates. Compare the platform answer with measured service behavior.
Gil Tene - How NOT to Measure Latency (talk)
Strange Loop Conference via YouTubesupportreference
Use for coordinated-omission vocabulary only: closed-loop testers that wait for each reply, the missing samples under queueing, and the correction concept of recording intended send time. Do not adopt wrk, hey or ab; write the small Python generator in the saturation unit instead.
M/M/1 queue - model definition and stationary analysis
Wikipediareferencereference
Read Model definition (Poisson arrivals, exponential service, single server, infinite buffer, FIFO) and Stationary analysis (stability \(\lambda < \mu\), \(\rho = \lambda/\mu\), \(L = \rho/(1 - \rho)\), response time \(1/(\mu - \lambda)\) via Little's Law). Use the closed forms as approximations with assumptions stated, checked by your own simulation.