Unit 7.6 · about 5 h
288 / 324Differential privacy: adjacency, sensitivity, noise and composition
The only privacy claim in this stage with a proof. Everything outside the stated mechanism, adjacency and budget keeps no promise.
- derive
- implement
You can now
- Define adjacency, unit of privacy, sensitivity and the \((\epsilon, \delta)\) guarantee for one aggregate.
- Build a clamped Laplace release, derive its \(\epsilon\) from sensitivity over scale, and compose two releases.
- Measure utility loss against the true aggregate and list every component outside the guarantee.
Covers
- Differential privacy adjacency and unit of privacy
- Sensitivity, clipping, \(\epsilon\), \(\delta\) and calibrated noise
- Composition, subsampling intuition and accounting inputs
- Privacy utility trade-offs and unprotected system boundaries
Study, in this order
4 steps
- Adjacency, sensitivity and privacy mapsOpenDP theory - one bounded privacy mechanism
A Framework to Understand DP: Distance Between Datasets - Adjacency (symmetric distance); Definition of Privacy; Distance Between Aggregates - Sensitivity (clamped sum); Stability Maps and Privacy Maps (chained Laplace sum with \(\epsilon\) 0.48)
- Clamp, sum and Laplace chaining in codeOpenDP theory - one bounded privacy mechanism
Differential Privacy with OpenDP: The Laplace Mechanism; Transformation Example: Sum; Transformation Example: Clamp; Chaining (clamp, bounded sum and Laplace composed with the privacy map); Retrospective
Depth · Run the examples with OpenDP 0.16.0, the version these pages document.
make_sequential_compositiongroups releases under one budget. - Accuracy limits of noisy releasesOpenDP theory - one bounded privacy mechanism
Accuracy: Pitfalls and Edge Cases: Accuracy vs Confidence Intervals; What is \(D\) (preprocessing and resize effects on the guarantee target)
- The \((\epsilon, \delta)\) definition and DP-SGDData Privacy and Security (MIT Data-Centric AI)
Towards a solution: differential privacy (the \((\epsilon, \delta)\) inequality over adjacent datasets; DP-SGD as the training analogue; parameter-setting and utility challenges)
Focus on
- Adjacency is the whole promise: symmetric distance 1 means adding or removing one record, and multi-row contributors scale the bound by \(k\).
- Clamping sets sensitivity before noise means anything: with bounds
(0, 10)one record moves the sum by at most 10, so Laplace scale 10 buys \(\epsilon\) about 1. - Chaining multiplies the reasoning: the clamp map fed into the noise map gives the release map, and sequential composition adds epsilons across releases.
- Accuracy utilities bound noise error only: the guarantee covers the release minus the true aggregate, never the sample minus the population.
Reproduce closed-book
(0, 10), sum and Laplace scale 10 over float vectors under symmetric distance. map(1) returns 1.000000009313226: the sum reports sensitivity 10.00000009313226 to cover float rounding, and the Laplace map alone turns 10 into exactly 1.0. Draw 200 values uniform on [0, 10] with NumPy default_rng(7) (true clamped sum 1005.9) and make 2000 releases. OpenDP draws noise from its own unseeded generator, so the mean absolute error changes per run: three runs gave 9.79, 9.96 and 10.03 against the Laplace mean absolute deviation of 10 (standard error about 0.22). Then derive that two such releases compose to \(\epsilon\) about 2.0.Traps
- Quoting \(\epsilon\) without adjacency, bounds and \(\delta\). The number is meaningless until the unit of privacy and the clamp are stated.
- Reading accuracy output as a confidence interval for the population. It covers noise around this dataset aggregate only.
- Assuming the guarantee covers the pipeline. Ingestion, bounds choice, budget accounting and the served model sit outside unless each is modeled.
Practice
The section mini-project covers this: one clipped and noised aggregate with stated adjacency, \(\epsilon\) and \(\delta\) inputs, measured utility and a residual-risk statement.
Self-test
-
Bounds are
(0, 10)and one person contributes one row. What is the clamped-sum sensitivity?10. Removing or adding one clamped record changes the sum by at most the clamp range. -
Scale is 10 and sensitivity is 10. What \(\epsilon\), and what changes with scale 5?
About 1.0. Scale 5 doubles it to about 2.0 and halves the noise: \(\epsilon\) equals sensitivity divided by scale. -
Two \(\epsilon\)-1.0 releases run on the same data. What is the pair budget?
About 2.0 by basic sequential composition. Each release spends separately against the same adjacency. -
\(\delta\) is 0 for this Laplace release. What does \(\delta\) allow in general?
An additive slack in the guarantee, informally a small probability that the \(\epsilon\) bound does not hold, as with the Gaussian mechanism. Pure Laplace keeps \(\delta\) at 0.
Completion criterion
Stop when
You can derive \(\epsilon\) from bounds and scale, quote measured error with its setup, and name everything the guarantee excludes.
Mark passed, then open 7.7 Only mark this when you can do it without notes.Leads to
- 7.7 Compatibility, rollback and revocation during changes Version the bounds, budget and accounting inputs so migration cannot silently change them.
Comes after
- 7.5 Membership inference, extraction, inversion and poisoningSpend a measured budget against the attacks quantified here.
Resources for this unit
- OpenDP theory - one bounded privacy mechanism Read A Framework to Understand DP and Differential Privacy with OpenDP (in the 0.16.0 user guide, Framework API), then Accuracy: Pitfalls and Edge Cases and the Differencing, Membership and Reconstruction attack notebooks. Define one privacy unit, clamp an aggregate, add calibrated noise, record \(\epsilon\) and \(\delta\) inputs, measure utility and list everything outside the guarantee.
- Data Privacy and Security (MIT Data-Centric AI) Read the threat-model examples, shadow-training and metrics-based membership attacks, extraction by sampling plus perplexity ranking, empirical-defense evaluation, and the \((\epsilon, \delta)\) definition with DP-SGD.
Also in this section
- Building Secure and Reliable Systems - selected design route Read Chapters 1, 2, 4-9 and 12-14: adversaries, trade-offs, least privilege, understandability, changing landscapes, resilience, recovery, code, tests and deployment. Apply the chapter questions to the same RPC, store, tenant and migration boundary.
- OWASP Threat Modeling Cheat Sheet Model trust boundaries, data flows, data stores, processes and external entities, and list the assets they carry. Write misuse cases for unauthorized mutation, cross-tenant reads, stale credentials, sensitive logs and restore or migration paths, then turn each into a test or explicit residual risk.
- OSTEP - introductory security chapters Use Chapters 53-56 for introductory security, authentication, access control and cryptography vocabulary. Connect process and container isolation to the threat model and state where those boundaries do not protect the system.
- Systems Foundations Checkpoint - security and privacy slice Use only the tenant-access, privacy aggregate, abuse, migration, backup and residual-risk requirements as the final assessment. A toy privacy mechanism is evidence for its stated aggregate, not a claim that the service or model is private.
- A Survey of Privacy Attacks in Machine Learning Use the attack taxonomy and threat model (adversarial knowledge, assets under attack) plus the causes of leaks and the defenses overview; skip the per-paper catalog.
- Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning Read the weak threat model (no model or training-set knowledge, about 50 poisoning samples) and the reported result of above 90 percent attack success; skip the optimization details.
- OWASP Input Validation Cheat Sheet Apply allowlist validation, parse-safely-then-validate, structured-data rules, and the common pitfalls (client-only checks, trusted internal sources, denylisting, logging rejected input).
- OWASP Deserialization Cheat Sheet Apply the Python clear-box rules (
pickle, unsafe YAML loaders,jsonpickle) and the language-agnostic methods (pure-data formats, signed data).