Unit 7.5 · about 4 h
287 / 324Membership inference, extraction, inversion and poisoning
Models exposed as endpoints leak through their outputs. This unit measures how much, with a control, before the next unit spends a privacy budget to stop it.
- explain
- measure
You can now
- Threat-model membership inference, extraction, inversion and poisoning for one model endpoint, and separate privacy loss from integrity loss.
- Run a loss-threshold membership attack with a non-overfit control and report AUC with sample sizes.
- Describe backdoor poisoning under a weak threat model and state the serving-side exfiltration path it opens.
Covers
- Membership inference, model extraction and inversion
- Data poisoning, backdoors and training-serving exfiltration
Study, in this order
4 steps
- Threat models, membership inference and extractionData Privacy and Security (MIT Data-Centric AI)
Defining security: security goals and threat models (Image prediction API, Patient risk prediction model); Membership inference attacks (Shadow training Steps 1 to 3; Metrics-based attacks: correctness, loss, confidence, entropy); Data extraction attacks (sampling plus perplexity ranking); Evaluating defenses (Kerckhoffs's principle)
- Aggregate disclosure attacksOpenDP theory - one bounded privacy mechanism
Differencing Attack notebook (reconstructing one income from two means); continue to the Membership and Reconstruction notebooks from the Attacks index
Depth · Read as aggregate-query attacks that motivate noise, not as model attacks. The model attacks are the lecture's sections above.
- Privacy-attack taxonomy and threat modelA Survey of Privacy Attacks in Machine Learning
Attack taxonomy and threat model (adversarial knowledge, assets under attack); causes of privacy leaks; defenses overview and open problems
Depth · Survey, not a textbook: read the taxonomy and one attack family end to end, and skip the per-paper catalog.
- Backdoor poisoning resultTargeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
Threat model (no model or training-set knowledge, small poisoning fraction, stealthy key) and evaluation (about 50 poisoning samples, above 90 percent attack success)
Depth · Read the threat model and results; skip the optimization details. Defensive and conceptual only: study the poisoning pattern, never deploy a backdoor.
Focus on
- Fix goal, access and knowledge first: the image-API and hospital examples change what black-box versus white-box even means.
- Cheap metrics beat shadow models for the demo: correctness, loss, confidence and entropy all read the same overfitting gap.
- Extraction is sampling plus membership ranking: perplexity selects the memorized sequences out of many samples.
- Empirical defenses are claims until attacked: Kerckhoffs-style evaluation tries to break the defense before trusting it.
Reproduce closed-book
default_rng(0)); the first 60 are members, the other 440 nonmembers. Fit a StandardScaler on the members, then train an unregularized MLP (hidden layers 128 and 64, alpha 0, max_iter 5000, random_state 0) and a LogisticRegression control with C=1e-4. Score each point by minus its loss on the true label (probability clipped at 1e-15). The memorizer gives AUC 0.865 (train accuracy 1.000, mean member loss 0.003, nonmember 2.185); the control gives 0.584 (train accuracy 0.533). Across seeds 0 to 4 the ranges are 0.865 to 0.948 and 0.556 to 0.602.Traps
- Reporting attack AUC without the control. Even the near-chance control scores 0.556 to 0.602, so the gap, not the level, is the evidence.
- Reading the lecture's correctness-attack intuition as written. Its text swaps train and test: the model classifies most of its training data correctly and a smaller fraction of unseen data, and the attack exploits that gap.
- Treating a weakened attack demo as protection. Regularization and output rounding only narrow the gap until a stronger attack is tried.
Practice
MIT Data-Centric AI: Lab - Membership Inference notebook
Lab - Membership Inference notebook: black-box membership inference against a given prediction function
Do the loss-threshold variant first with a control model, then try one stronger variant. Report AUC with member and nonmember counts.
Self-test
-
What does a loss-threshold attack output, and how is the threshold picked?
Member when the point loss is below \(\tau\). \(\tau\) can come from published train-loss statistics; the demo avoids picking one by reporting threshold-free AUC. -
Why does extraction rank samples by perplexity?
Memorized sequences get high predicted probability token by token, hence low perplexity. Sampling proposes candidates; ranking selects likely members. -
A defense drops the attack AUC from 0.9 to 0.6. Is the leak fixed?
Not shown. A new attack or more queries can reopen it; only a budgeted guarantee bounds all attacks in the threat model. -
How do poisoning and inversion differ in what they break?
Poisoning breaks integrity by planting behavior through training data; inversion breaks privacy by recovering inputs through outputs.
Completion criterion
Stop when
You can run the attack with its control, quote both AUCs with sample sizes, and threat-model one endpoint across all four attack families.
Mark passed, then open 7.6 Only mark this when you can do it without notes.Leads to
- 7.6 Differential privacy: adjacency, sensitivity, noise and composition Spend a measured budget against the attacks quantified here.
Resources for this unit
- OpenDP theory - one bounded privacy mechanism Read A Framework to Understand DP and Differential Privacy with OpenDP (in the 0.16.0 user guide, Framework API), then Accuracy: Pitfalls and Edge Cases and the Differencing, Membership and Reconstruction attack notebooks. Define one privacy unit, clamp an aggregate, add calibrated noise, record \(\epsilon\) and \(\delta\) inputs, measure utility and list everything outside the guarantee.
- Data Privacy and Security (MIT Data-Centric AI) Read the threat-model examples, shadow-training and metrics-based membership attacks, extraction by sampling plus perplexity ranking, empirical-defense evaluation, and the \((\epsilon, \delta)\) definition with DP-SGD.
- A Survey of Privacy Attacks in Machine Learning Use the attack taxonomy and threat model (adversarial knowledge, assets under attack) plus the causes of leaks and the defenses overview; skip the per-paper catalog.
- Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning Read the weak threat model (no model or training-set knowledge, about 50 poisoning samples) and the reported result of above 90 percent attack success; skip the optimization details.
Also in this section
- Building Secure and Reliable Systems - selected design route Read Chapters 1, 2, 4-9 and 12-14: adversaries, trade-offs, least privilege, understandability, changing landscapes, resilience, recovery, code, tests and deployment. Apply the chapter questions to the same RPC, store, tenant and migration boundary.
- OWASP Threat Modeling Cheat Sheet Model trust boundaries, data flows, data stores, processes and external entities, and list the assets they carry. Write misuse cases for unauthorized mutation, cross-tenant reads, stale credentials, sensitive logs and restore or migration paths, then turn each into a test or explicit residual risk.
- OSTEP - introductory security chapters Use Chapters 53-56 for introductory security, authentication, access control and cryptography vocabulary. Connect process and container isolation to the threat model and state where those boundaries do not protect the system.
- Systems Foundations Checkpoint - security and privacy slice Use only the tenant-access, privacy aggregate, abuse, migration, backup and residual-risk requirements as the final assessment. A toy privacy mechanism is evidence for its stated aggregate, not a claim that the service or model is private.
- OWASP Input Validation Cheat Sheet Apply allowlist validation, parse-safely-then-validate, structured-data rules, and the common pitfalls (client-only checks, trusted internal sources, denylisting, logging rejected input).
- OWASP Deserialization Cheat Sheet Apply the Python clear-box rules (
pickle, unsafe YAML loaders,jsonpickle) and the language-agnostic methods (pure-data formats, signed data).