Section 1 of 5 · Classical Machine Learning
Formulation, Data Contracts & Baselines
A model cannot repair a target defined after the decision, a feature unavailable at prediction time, or an evaluation set that already influenced the work. This section fixes the scientific contract before estimator choice.
8 units 17 topics 0/8
Units
8 in this section
- 1.1 Unit of observation, prediction time and label horizon explain3 h
- 1.2 Targets, losses, metrics, decision rules and problem framing explain3 h
- 1.3 Split design, nested selection and test isolation explain · implement4 h
- 1.4 Leakage: target, future, entity, duplicate, preprocessing and feedback explain · measure4 h
- 1.5 Schema, provenance, missingness, duplicate and availability audits implement · measure3 h
- 1.6 Training-only imputation, scaling, encoding and learned state implement3 h
- 1.7 Baselines: constant, frequency, heuristic, last-value and simple learned implement · measure3 h
- 1.8 Snapshots, lineage, pipeline serialization and serving parity implement3 h
Before moving on
Checkpoint
Defend a one-page prediction contract, split and baseline ladder, then show a seeded leakage or availability defect being caught and changing the conclusion after repair.
Deliberately out of scope
Leave for later
- Feature-store and streaming architecture
- Production monitoring implementation
- Causal intervention and policy claims
Resources for this section
Shared by every unit above
- Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2 Read Chapters 1 and 2 of the book, then run notebooks 01 and 02 and use ml-project-checklist.md for framing, data inspection, test-set creation, preprocessing pipelines, cross-validation, error analysis and persistence. Rebuild the workflow on UCI Bike Sharing with a chronological split and prohibit casual and registered when predicting cnt.
- scikit-learn - Common pitfalls and pipelines Read Common Pitfalls sections on inconsistent preprocessing and leakage, then Pipeline and ColumnTransformer. Seed global scaling, a target-derived feature and a partition duplicate, and make tests reject each defect.
- scikit-learn - Cross-validation splitters Use the group-aware and time-aware splitter sections plus GroupKFold and TimeSeriesSplit. Choose from the dependence structure and assert that no prohibited group or future timestamp crosses the boundary.
- Rules of Machine Learning - first controls Read Rules 1-5 and 8-9. Turn the existing heuristic into a baseline and test the feature path independently of the learner.
- UCI Bike Sharing Dataset Use hour.csv as the compact integrated dataset. Preserve a raw snapshot and data dictionary, split chronologically, and document why casual and registered are target proxies for cnt.
- Google ML Problem Framing - framing an ML problem Read Identify the output you need through Proxy labels and Define the success metrics, including the video-caching threshold analysis, the proxy-issue table, and both Check Your Understanding boxes.
Apply the section
Mini-project · Bike demand contract, split, audits and baseline ladder (10 to 14 h)
Rebuild the hourly UCI Bike Sharing demand problem in your own repository as a contract-first investigation: a one-page prediction contract, an immutable snapshot with a chronological split, an audit suite, a baseline ladder under one harness, and three seeded leakage defects that failing tests must catch. No model tuning happens until the contract and ladder are frozen.
Steps
- Write the contract for next-hour
cnt: unit, prediction time, cutoff, horizon, served action, misprediction costs, and an availability table forhour.csvmarking same-hour weather as oracle-forecast andcasualandregisteredprohibited for the leak you name, with the feedback loop kept as a monitored exclusion. - Store
hour.csvwith its hash and download date. Build the chronological split, the first year to train and the second to test, asserting no test timestamp precedes train and no prohibited column reaches features. - Write the audit suite covering schema and
dtypes, missingness, keys and duplicates, time continuity, ranges and normalizations, rare categories and availability. Seed one defect per audit in a copy and show each test fails. - Build the baseline ladder in one harness scored by RMSE: train mean, train median, train-last constant, previous row and a label-free linear model, checked against scikit-learn, stating which rung a new model must beat and why.
- Seed three leakage defects: global scaling before the split, the target-derived
casualandregisteredfeatures, and cross-partition duplicates (copy 500 test rows into train). Catch each with a failing test, report the score before and after repair, and score the contaminated rows separately. - Wrap preprocessing in a
ColumnTransformerpipeline fitted on train only, write the lineage record (source URL, hash, split rule, seeds, library versions), and run a joblib round trip that asserts identical predictions on five probe rows.
Deliverables
contract.mdwith the availability table, the declared assumptions, the cost statement and the feedback note.- The data package: snapshot manifest, split builder and audit suite with its seeded-defect tests.
- The ladder report: one table of rungs with RMSE and the verdict on whether complexity is justified.
- The leakage casebook: three defects, each with its failing test, its repair and the conclusion it changed.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Contract | Every column is classified with a reason, and prediction time, cutoff and horizon are stated. | The oracle-forecast assumption is priced: the linear rung is rerun without same-hour weather and the change is reported. |
| Split and audits | The split is chronological with no prohibited column, and every audit fails on its seeded defect. | One command runs all audits, and a property test shows the split builder never places a test row before a train row. |
| Baseline ladder | All five rungs land within a few RMSE points of the reference under one harness. | The report explains why the previous row wins, and why the median loses to the mean on the later test period. |
| Leakage | Three defects are each caught by a failing test, with scores before and after repair. | The casebook shows a defect the score cannot reveal (scaler leakage leaves a least-squares RMSE unchanged) and one that only the separately scored rows expose. |
| Lineage | The lineage record is complete and the joblib round trip passes. | A fresh clone rebuilds every reported number with one command. |
Feeds into: Milestone artifact 1, the contract and leakage pack: the contract, manifest, split builder, audits, the naive baselines and the preserved leakage failure come from here.