Unit 7.7 · about 3 h
289 / 324Compatibility, rollback and revocation during changes
Assumes the Programming units on schema migrations with Alembic and idempotency-key ledgers as background. This unit adds the failure-aware half: rollback that does not revive vulnerabilities, revocation that works during outages, and restores that keep tenant boundaries.
- explain
You can now
- State compatibility rules for schema and protocol changes, and gate rollout on explicit version signals.
- Plan canary, rollback and backup-restore so a bad release or a revoked credential cannot strand the system.
- Carry tenant isolation, audit and privacy accounting across migration and restore paths.
Covers
- Schema and protocol compatibility during migration
- Canary, rollback, backup, restore and revocation under auth changes
Study, in this order
3 steps
- Change speeds and architecture for changeBuilding Secure and Reliable Systems - selected design route
Chapter 7, Design for a Changing Landscape: Types of Security Changes; Architecture Decisions to Make Changes Easier (Keep Dependencies Up to Date, Release Frequently); Different Changes: Short-Term Change (Shellshock), Medium-Term Change (FIDO keys), Long-Term Change (HTTPS)
- Rollback trade-offs and revocationBuilding Secure and Reliable Systems - selected design route
Chapter 9, Design Principles for Recovery: Rollbacks Represent a Tradeoff (Deny lists, Minimum Acceptable Security Version Numbers, Rotating signing keys); Use an Explicit Revocation Mechanism (Handling emergencies directly, Revoking credentials at scale); Know Your Intended State, Down to the Bytes
- Deployment policy and breakglassBuilding Secure and Reliable Systems - selected design route
Chapter 14, Deploying Code: Provenance-Based Deployment Policies (Implementing policy decisions); Post-Deployment Verification; Practical Advice (Take It One Step at a Time, Include a Deployment Breakglass)
Focus on
- Different changes move at different speeds: zero-days, posture improvements and external demands each get their own timeline and test budget.
- Rollback trades reliability against security: minimum version numbers, deny lists and key rotation stop a rollback from reviving holes.
- Revocation must work while hurt: cached revocation lists beat fail-open timeouts, and emergencies go through the same list mechanism, not hand-edited key files or a rarely used emergency list.
- Know the intended bytes: host, firmware and data state are declared before migration so verification has something to compare.
Reproduce closed-book
Traps
- Rolling back to a version with a known hole to fix an outage. Minimum acceptable versions exist for exactly this moment.
- Depending on accurate time during revocation. Clocks are an external dependency; cached revocation data removes it.
- Migrating privacy bounds or budgets silently. Bounds, \(\epsilon\) and \(\delta\) inputs and the spent budget are versioned configuration and state, not code constants.
Practice
The section mini-project covers this: canary and rollback drill, backup-restore verification, and a revocation check under a failed dependency.
Self-test
-
A rollback fixes availability but revives a patched injection. What control should have stopped it?
A minimum acceptable security version number. Rollback targets below it are refused even in an outage. -
Why does the book prefer cached revocation lists over fail-open on timeout?
Cached lists keep the best-known deny state without a live dependency. Fail-open revalidates revoked credentials for the whole outage. -
What does a canary buy that a fast full rollout does not?
A bounded blast radius: production signals from a small slice before full exposure. A fast rollout can be right for an actively exploited zero-day, but it gives up that check. -
A restore brings back last month's tenant table. What must the restore check?
Tenant predicates, credential validity and accounting inputs against current versions. Restored data must not widen access or budgets.
Completion criterion
Stop when
You can walk a schema change from canary through rollback or revoke without dropping isolation, audit or budget accounting.
Mark passed, then open 1.1 Only mark this when you can do it without notes.Leads to
- 6.7 Failure domains, correctness signals and safe rollout Apply canary and rollback mechanics to the migration plan.
- 6.8 Backups, runbooks, incident timelines and postmortems Turn the migration runbook into an incident timeline when it fails.
Comes after
- 4.9 Record evolution, amplification and LSM treesUse the compatibility matrix to define rollback boundaries.
- 7.4 Audit records, fail-open versus fail-closed and supply chainCarry audit and revocation behavior through rollback and migration.
- 7.6 Differential privacy: adjacency, sensitivity, noise and compositionVersion the bounds, budget and accounting inputs so migration cannot silently change them.
Resources for this unit
- Building Secure and Reliable Systems - selected design route Read Chapters 1, 2, 4-9 and 12-14: adversaries, trade-offs, least privilege, understandability, changing landscapes, resilience, recovery, code, tests and deployment. Apply the chapter questions to the same RPC, store, tenant and migration boundary.
Also in this section
- OpenDP theory - one bounded privacy mechanism Read A Framework to Understand DP and Differential Privacy with OpenDP (in the 0.16.0 user guide, Framework API), then Accuracy: Pitfalls and Edge Cases and the Differencing, Membership and Reconstruction attack notebooks. Define one privacy unit, clamp an aggregate, add calibrated noise, record \(\epsilon\) and \(\delta\) inputs, measure utility and list everything outside the guarantee.
- OWASP Threat Modeling Cheat Sheet Model trust boundaries, data flows, data stores, processes and external entities, and list the assets they carry. Write misuse cases for unauthorized mutation, cross-tenant reads, stale credentials, sensitive logs and restore or migration paths, then turn each into a test or explicit residual risk.
- OSTEP - introductory security chapters Use Chapters 53-56 for introductory security, authentication, access control and cryptography vocabulary. Connect process and container isolation to the threat model and state where those boundaries do not protect the system.
- Systems Foundations Checkpoint - security and privacy slice Use only the tenant-access, privacy aggregate, abuse, migration, backup and residual-risk requirements as the final assessment. A toy privacy mechanism is evidence for its stated aggregate, not a claim that the service or model is private.
- Data Privacy and Security (MIT Data-Centric AI) Read the threat-model examples, shadow-training and metrics-based membership attacks, extraction by sampling plus perplexity ranking, empirical-defense evaluation, and the \((\epsilon, \delta)\) definition with DP-SGD.
- A Survey of Privacy Attacks in Machine Learning Use the attack taxonomy and threat model (adversarial knowledge, assets under attack) plus the causes of leaks and the defenses overview; skip the per-paper catalog.
- Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning Read the weak threat model (no model or training-set knowledge, about 50 poisoning samples) and the reported result of above 90 percent attack success; skip the optimization details.
- OWASP Input Validation Cheat Sheet Apply allowlist validation, parse-safely-then-validate, structured-data rules, and the common pitfalls (client-only checks, trusted internal sources, denylisting, logging rejected input).
- OWASP Deserialization Cheat Sheet Apply the Python clear-box rules (
pickle, unsafe YAML loaders,jsonpickle) and the language-agnostic methods (pure-data formats, signed data).