Section 7 of 7 · Systems Foundations
Security, Privacy & Safe Evolution
Availability does not protect data, tenants or model behavior. This final section sets a concrete trust boundary, tests least privilege, and keeps privacy claims limited to the mechanism actually implemented.
7 units 23 topics 0/7
Units
7 in this section
- 7.1 Security goals, trust boundaries and threat models explain3 h
- 7.2 Identity, authorization, least privilege and secrets explain4 h
- 7.3 Tenant isolation, input validation and injection explain · implement4 h
- 7.4 Audit records, fail-open versus fail-closed and supply chain explain3 h
- 7.5 Membership inference, extraction, inversion and poisoning explain · measure4 h
- 7.6 Differential privacy: adjacency, sensitivity, noise and composition derive · implement5 h
- 7.7 Compatibility, rollback and revocation during changes explain3 h
Before moving on
Checkpoint
Reject unauthorized and cross-tenant operations across normal, retry, cache, restore and migration paths. Demonstrate one clipped and noised aggregate with explicit accounting inputs, measured utility and a correct residual-risk statement.
Deliberately out of scope
Leave for later
- Production cryptographic protocol design
- Private model training and advanced privacy accounting
- Compliance certification and enterprise identity platforms
Resources for this section
Shared by every unit above
- Building Secure and Reliable Systems - selected design route Read Chapters 1, 2, 4-9 and 12-14: adversaries, trade-offs, least privilege, understandability, changing landscapes, resilience, recovery, code, tests and deployment. Apply the chapter questions to the same RPC, store, tenant and migration boundary.
- OpenDP theory - one bounded privacy mechanism Read A Framework to Understand DP and Differential Privacy with OpenDP (in the 0.16.0 user guide, Framework API), then Accuracy: Pitfalls and Edge Cases and the Differencing, Membership and Reconstruction attack notebooks. Define one privacy unit, clamp an aggregate, add calibrated noise, record \(\epsilon\) and \(\delta\) inputs, measure utility and list everything outside the guarantee.
- OWASP Threat Modeling Cheat Sheet Model trust boundaries, data flows, data stores, processes and external entities, and list the assets they carry. Write misuse cases for unauthorized mutation, cross-tenant reads, stale credentials, sensitive logs and restore or migration paths, then turn each into a test or explicit residual risk.
- OSTEP - introductory security chapters Use Chapters 53-56 for introductory security, authentication, access control and cryptography vocabulary. Connect process and container isolation to the threat model and state where those boundaries do not protect the system.
- Systems Foundations Checkpoint - security and privacy slice Use only the tenant-access, privacy aggregate, abuse, migration, backup and residual-risk requirements as the final assessment. A toy privacy mechanism is evidence for its stated aggregate, not a claim that the service or model is private.
- Data Privacy and Security (MIT Data-Centric AI) Read the threat-model examples, shadow-training and metrics-based membership attacks, extraction by sampling plus perplexity ranking, empirical-defense evaluation, and the \((\epsilon, \delta)\) definition with DP-SGD.
- A Survey of Privacy Attacks in Machine Learning Use the attack taxonomy and threat model (adversarial knowledge, assets under attack) plus the causes of leaks and the defenses overview; skip the per-paper catalog.
- Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning Read the weak threat model (no model or training-set knowledge, about 50 poisoning samples) and the reported result of above 90 percent attack success; skip the optimization details.
- OWASP Input Validation Cheat Sheet Apply allowlist validation, parse-safely-then-validate, structured-data rules, and the common pitfalls (client-only checks, trusted internal sources, denylisting, logging rejected input).
- OWASP Deserialization Cheat Sheet Apply the Python clear-box rules (
pickle, unsafe YAML loaders,jsonpickle) and the language-agnostic methods (pure-data formats, signed data).
Apply the section
Mini-project · Threat model, least-privilege tests, a privacy-bounded aggregate and a safe migration (12 to 16 h)
Set the trust boundary of the service slice with a threat model, scoped expiring credentials, tenant isolation on every path, a hash-chained audit log and verified backups. Add one bounded private aggregate and a canary migration with rollback and revocation.
Steps
- Draw the system model with trust boundaries, data flows, stores, processes and external actors following the OWASP cheat sheet, list assets with an insider profile, and close each abuse case in a test or an explicit residual-risk sentence.
- Issue per-client credentials carrying tenant, scopes and expiry under a server key, check signature, expiry and scope on the server clock with a stated skew allowance, and reject missing, forged, expired, out-of-scope and wrong-tenant callers with a separate logged admin path.
- Scope every key and query by the authenticated tenant on the normal, retry, fallback and restore paths, keep the injection fixture that separates interpolated from parameterized queries, and never deserialize untrusted input.
- Record authentication, scope and cross-tenant denials in a hash-chained audit log with redacted secrets, verify that tampering breaks the chain, deny when the policy cannot load, and verify backups by digest, counts, predicates and chain before serving, refusing corruption.
- Release one clamped and noised aggregate with OpenDP from the usage data with stated unit, adjacency, sensitivity, \(\epsilon\) and \(\delta\) inputs and a ledgered budget, measure utility across repeated runs, and state what the guarantee leaves out.
- Migrate records behind a canary that reads both versions and rolls back with every record still readable, practise the rollback once, then revoke a credential while the issuer is down and show cached revocation still denies.
Deliverables
- The threat model with abuse cases and residual risks, plus credential, isolation, injection and audit tests with their failing cases.
- The backup and restore verification script that refuses corruption before serving.
- The private release with parameters, ledger, utility measurements and residual-risk statement, and the migration drill with rollback and revocation.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Threat and privilege | Every abuse case ends in a test or a residual-risk sentence, and forged, expired, out-of-scope and cross-tenant callers are rejected by tests. | Tenant scope comes from the credential on every path, including retry, fallback and restore. |
| Integrity and recovery | Tampering breaks the audit chain with secrets redacted, and a corrupted backup is refused before serving. | Restore checks predicates and the chain rather than digests alone. |
| Privacy and evolution | The release states unit, adjacency, sensitivity, budget and measured utility, the canary rolls back cleanly, and revocation holds during the outage. | Residual risks name cases outside the guarantee, the ledger blocks over-budget release, and rollback fires from metrics. |
Feeds into: Capstone dossier: the threat model, least-privilege tests, migration with rollback and the bounded private aggregate with its limits.