Unit 1.8 · about 3 h
135 / 324Snapshots, lineage, pipeline serialization and serving parity
Reproducibility is what makes every earlier claim checkable. Without snapshots and parity tests, fixes cannot be told apart from noise.
- implement
You can now
- Serialize a fitted pipeline and prove reload parity with identical predictions in a fresh process.
- Trace a train/serve feature mismatch to its data contract and add a prevention test.
- Record immutable snapshots, split identities, seeds, and raw-prediction lineage for a full run.
Covers
- Immutable snapshots, split identities, seeds and raw-prediction lineage
- Pipeline serialization and training-serving feature parity
Study, in this order
4 steps
- Stable split identities and joblib persistenceHands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2
Book Chapter 2: Create a Test Set (printed pp. 57-62) and saving the model with joblib in Launch, Monitor, and Maintain Your System (pp. 100-101). Notebook 02: Create a Test Set (
crc32id-hash split,longitude/latitudestable-id pattern,random_state42) and Model persistence using joblib (dump the final model, reload it, predict on new data, with the pickle-versus-joblib note) - `Pipeline` structure for serializationscikit-learn - Common pitfalls and pipelines
8.1.1
Pipeline: chaining estimators (one object holding transformers plus estimator, with named steps access) through 8.1.1.1 Usage - Infrastructure tests, freshness, and export gatesRules of Machine Learning - first controls
Rule 5: Test the infrastructure independently from the machine learning (same score in training and serving, fixed model at serving); Rule 8: freshness requirements; Rule 9: pre-export held-out checks
- Raw files and dictionary to snapshotUCI Bike Sharing Dataset
Dataset Files (
hour.csv,day.csv,Readme.txt) with sizes; Variables Table and Additional Variable Information as the dictionary to store beside the raw snapshot
Focus on
- Lineage is mechanical: a raw snapshot (file, hash, row count, download date) with the data dictionary, immutable split identities (the chronological cut at 2012-01-01 reproduces 8645 and 8734 rows and needs no seed), and per-row raw predictions joined back to
instant. - Serialize the fitted pipeline, not the recipe: joblib dump of the exact object with its learned state, reloaded in a fresh process. Verified probe:
StandardScalerplusLinearRegressionontemp/atemp/hum/windspeedpredicts[101.7, 99.1, 72.7, 58.1, 52.2]on the first five 2012 test rows identically before and after the round trip (scikit-learn 1.9.1). - Parity is Rule 5: the served model must score identically on the same rows as training. Test the feature path with a fixed model before trusting any learner change, and gate exports with Rule 9 checks.
- Seeds are recorded, not tuned: one integer seed threaded through every random split, resample and estimator, stored with the snapshot. Re-runs must reproduce split counts and probe predictions exactly.
Reproduce closed-book
hour.csv row count 17379, the split rule that sends dates from 2012-01-01 to test, seeds for any random step, pipeline steps with scikit-learn 1.9.1) and run the joblib round trip asserting the identical five-row probe predictions.Traps
- Dumping the recipe instead of the fitted object. Unfitted parameters reload but learned state (means, categories, coefficients) is gone; dump the fitted pipeline and verify with predictions, not file existence.
- Reimplementing transforms at serving time. A second implementation of scaling or encoding drifts silently; serve through the same serialized pipeline object and prove it with the Rule 5 same-score test.
- Mutable snapshots. A dataset page or file can change between downloads, and the page already disagrees with its own file (17389 instances stated, 17379 rows). Keep the stored snapshot with its hash, never re-derive lineage from a fresh download, and version any refresh separately.
Practice
Hands-On ML Chapter 2: model persistence with joblib
Model persistence using joblib: dump the final model, reload it in a fresh session, and predict on new rows
Do it on your Bike pipeline: assert identical predictions on five held-out rows and record the scikit-learn and Python versions with the artifact.
Self-test
-
Why must split identities be stable rather than reshuffled each run?
Otherwise test rows migrate into train across runs, voiding every earlier metric and leakage guarantee. A recorded date rule, or recorded ids plus seed for a random split, reproduces the same rows exactly. -
joblib reload predicts identically on the probe rows. What does that prove, and what does it not prove?
It proves serialization fidelity: same object, same features, same result. It does not prove serving parity: live feature values, types, and missingness can still differ, which needs the separate Rule 5 test. -
The pipeline is reimplemented at serving for speed. What is the first test to demand?
The Rule 5 same-score test: fixed model, same rows, train path versus serve path must agree exactly before any learner comparison. -
A teammate refreshes
hour.csvand the row count changes. What breaks, and what is the repair?Every lineage claim (splits, metrics, predictions) loses its referent. Restore work to the stored snapshot and record the refresh as a separate versioned snapshot, never an in-place overwrite.
Completion criterion
Stop when
A fresh process reloads your artifact, reproduces the five-row probe exactly, and your lineage record pins source, rows, split rule, seed, and library versions.
Mark passed, then open 2.1 Only mark this when you can do it without notes.Leads to
- 5.10 Ablations, negative controls, threats to validity and decision memos Use frozen lineage and parity as the base for ablations and controls.
- 2.1 Least squares: residuals, normal equations and stable solvers Deepen the label-free linear baseline into solvers and diagnostics.
Comes after
- 1.5 Schema, provenance, missingness, duplicate and availability auditsSnapshot the audited raw data and dictionary under versioned lineage.
- 1.6 Training-only imputation, scaling, encoding and learned stateSerialize the fitted pipeline and prove serving parity.
- 1.7 Baselines: constant, frequency, heuristic, last-value and simple learnedFreeze the ladder harness and model artifacts under lineage.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2 Read Chapters 1 and 2 of the book, then run notebooks 01 and 02 and use ml-project-checklist.md for framing, data inspection, test-set creation, preprocessing pipelines, cross-validation, error analysis and persistence. Rebuild the workflow on UCI Bike Sharing with a chronological split and prohibit casual and registered when predicting cnt.
- scikit-learn - Common pitfalls and pipelines Read Common Pitfalls sections on inconsistent preprocessing and leakage, then Pipeline and ColumnTransformer. Seed global scaling, a target-derived feature and a partition duplicate, and make tests reject each defect.
- Rules of Machine Learning - first controls Read Rules 1-5 and 8-9. Turn the existing heuristic into a baseline and test the feature path independently of the learner.
- UCI Bike Sharing Dataset Use hour.csv as the compact integrated dataset. Preserve a raw snapshot and data dictionary, split chronologically, and document why casual and registered are target proxies for cnt.
Also in this section
- scikit-learn - Cross-validation splitters Use the group-aware and time-aware splitter sections plus GroupKFold and TimeSeriesSplit. Choose from the dependence structure and assert that no prohibited group or future timestamp crosses the boundary.
- Google ML Problem Framing - framing an ML problem Read Identify the output you need through Proxy labels and Define the success metrics, including the video-caching threshold analysis, the proxy-issue table, and both Check Your Understanding boxes.