Unit 1.7 · about 3 h
134 / 324Baselines: constant, frequency, heuristic, last-value and simple learned
Baselines decide whether complexity earns its place. Without the naive rungs, every learned score looks like progress.
- implement
- measure
You can now
- Implement at least two naive or heuristic baselines and one simple learned baseline under one fixed evaluation harness.
- Compare baseline performance across relevant slices and state whether model complexity is justified.
- Explain why trivial, heuristic, last-value, frequency, and simple statistical baselines are necessary controls.
Covers
- Constant, frequency, heuristic, last-value and simple learned baselines
Study, in this order
3 steps
- Heuristics, simple first models, and baseline behaviorRules of Machine Learning - first controls
Rule 1: launch without machine learning (a heuristic covers about half the imagined gain); Rule 3: choose machine learning over a complex heuristic; Rule 4: keep the first model simple and get the infrastructure right, with baseline metrics and neutral first launch; Rule 7: turn heuristics into features
Depth · Treat Rule 1 fractions as planning lore, not measured constants. Your ladder numbers replace them.
- Train, cross-validate, and compare patternHands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2
Book Chapter 2: Train and Evaluate on the Training Set and Better Evaluation Using Cross-Validation (printed pp. 90-94). Notebook 02: Training and Evaluating on the Training Set (linear predictions versus actuals, train RMSE) and Better Evaluation Using Cross-Validation (
cross_val_scorewithneg_root_mean_squared_errorandcv10 for tree, linear, and forest comparisons, plus the train-versus-validation overfit reading) - Ladder dataset with the target prohibitionUCI Bike Sharing Dataset
hour.csvas the ladder dataset with the chronological 2011/2012 split; Additional Variable Information for thecasual/registeredprohibition oncntprediction
Focus on
- Order the ladder by information, not cleverness: train-mean constant (RMSE 227.8), train-median (243.8), train-last constant (291.8), previous-hour (121.6), and a linear model on 12 calendar and same-hour weather fields (192.3). All numbers are RMSE on the 2012 test rows with scikit-learn 1.9.1. The weather fields assume a perfect forecast, and
yris 0 on every training row, so its coefficient is 0 and the linear model cannot see the 2012 growth. - Previous-hour beats the learned linear model (121.6 versus 192.3) because demand is strongly autocorrelated. That is the finding, not a failure: complexity must clear the naive temporal rung. Here previous-hour means the previous row; 75 rows follow a gap in the hours, so state the rule.
- The leaking rung (linear on
casualplusregistered: RMSE 0.0, coefficients[1, 1], intercept 0) is the control proving the harness catches label construction. Keep it in the report as the excluded row. - Rule 4 sets the attitude: a neutral first launch that prioritizes infrastructure and baseline behavior over fancy models. Rule 7 says what to do with a good heuristic next: keep its raw value as a feature.
Reproduce closed-book
[1, 1]. State which rung a new model must beat and why.Traps
- Comparing models across different splits or metrics. One fixed harness (same rows, same RMSE, same prohibition) or the comparison is void; retune nothing between rungs.
- Dropping the naive temporal baseline because it feels dumb. Previous-hour at 121.6 is the strongest rung; without it a model at 150 looks like real progress over the mean (227.8) while it still loses to the previous hour.
- Keeping the leaking features to see how well we could do. RMSE 0.0 measures label arithmetic, not demand; the only correct action is exclusion plus a regression test.
Practice
The section mini-project covers this: the full baseline ladder under one fixed harness with the verdict on whether complexity is justified.
Self-test
-
A new gradient booster scores 180 on the test split. Is the complexity justified?
Not yet. It beats the linear rung (192.3) but loses to previous-hour (121.6) by about 58. Give it the lagged count as an input and compare again; report both gaps, not just the win. -
Why is the train median (243.8) worse than the train mean (227.8) here?
Demand grew: the 2012 test mean is 234.7, against 143.8 in 2011. The best constant for RMSE on test is the test mean (208.9), and both train constants sit below it, the median of 109 furthest. The mean is also the squared-error minimizer on train; the median minimizes absolute error. -
What does the leak rung coefficient vector prove?
Thatcntis exactlycasualplusregistered(coefficients[1, 1], intercept 0, RMSE 0.0): the features are the label addends, so the rung is excluded and fenced by test. -
Rule 1 suggests heuristics reach half the ML gain, yet prev-hour beats linear regression. Contradiction?
No. Rule 1 is planning lore and the heuristic here exploits time structure the linear rung lacks. Measured rungs override the adage.
Completion criterion
Stop when
Your ladder script prints all six RMSE numbers matching the card within a few points, and your report states the complexity verdict against prev-hour.
Mark passed, then open 1.8 Only mark this when you can do it without notes.Leads to
- 1.8 Snapshots, lineage, pipeline serialization and serving parity Freeze the ladder harness and model artifacts under lineage.
- 5.2 Regression error and confusion-matrix metrics Carry the fixed-harness discipline into error metrics.
Comes after
- 1.6 Training-only imputation, scaling, encoding and learned stateRun every baseline through the same fitted pipeline.
- 5.10 Ablations, negative controls, threats to validity and decision memosReturn stronger baselines as standing controls.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2 Read Chapters 1 and 2 of the book, then run notebooks 01 and 02 and use ml-project-checklist.md for framing, data inspection, test-set creation, preprocessing pipelines, cross-validation, error analysis and persistence. Rebuild the workflow on UCI Bike Sharing with a chronological split and prohibit casual and registered when predicting cnt.
- Rules of Machine Learning - first controls Read Rules 1-5 and 8-9. Turn the existing heuristic into a baseline and test the feature path independently of the learner.
- UCI Bike Sharing Dataset Use hour.csv as the compact integrated dataset. Preserve a raw snapshot and data dictionary, split chronologically, and document why casual and registered are target proxies for cnt.
Also in this section
- scikit-learn - Common pitfalls and pipelines Read Common Pitfalls sections on inconsistent preprocessing and leakage, then Pipeline and ColumnTransformer. Seed global scaling, a target-derived feature and a partition duplicate, and make tests reject each defect.
- scikit-learn - Cross-validation splitters Use the group-aware and time-aware splitter sections plus GroupKFold and TimeSeriesSplit. Choose from the dependence structure and assert that no prohibited group or future timestamp crosses the boundary.
- Google ML Problem Framing - framing an ML problem Read Identify the output you need through Proxy labels and Define the success metrics, including the video-caching threshold analysis, the proxy-issue table, and both Check Your Understanding boxes.