Unit 2.4 · about 3 h
8 / 324Evaluate the model and inspect its errors
An aggregate score is a summary, not an explanation. Error analysis shows where the model is useful, where it fails and whether its apparent improvement is meaningful.
- measure
- explain
You can now
- Compute MAE and RMSE on the untouched test partition.
- Compare the model against the DummyRegressor baseline.
- Plot residuals and inspect errors by hour, weekday and weather category.
- Explain why a single aggregate metric can hide a weak subgroup or systematic failure.
Covers
- Regression metrics and error analysis
Study, in this order
3 steps
- Machine Learning Crash Course — regression and classification evaluationGoogle Machine Learning Crash Course
Use the regression, classification and datasets/generalization modules for the principles of evaluation and generalization.
- Model evaluation: regression metricsModel evaluation: regression metrics
Read the definitions of mean absolute error, mean squared error and R-squared; derive RMSE as the square root of MSE.
- Machine Learning in Python with scikit-learn (MOOC)Machine Learning in Python with scikit-learn (MOOC)
Use the sections on predictive performance and model evaluation as a deeper follow-on.
Focus on
- MAE in bike rentals per hour and RMSE sensitivity to large misses
- Baseline comparison on the same test rows
- Residual plots and stratified error slices
- Limits of the observed test period
- Post-hoc weather error slices are diagnostic only; observed weather is not a predictor under the start-of-hour contract
Reproduce closed-book
Report baseline and model MAE/RMSE, plot predicted versus observed values and residuals, and compare errors across hour-of-day, weekday and observed weather categories. Do not feed the weather category into the model. Keep the 2012 test period fixed.
Traps
- Do not report only training score.
- Do not pick the metric after seeing which one makes the model look best.
- Do not claim that feature importance explains causality.
- Do not overinterpret tiny subgroup slices.
Practice
Complete the regression and datasets/generalization modules, then explain which evaluation decisions they support in this project.
Keep the target and evaluation protocol fixed while comparing models.
Self-test
-
What is the practical difference between MAE and RMSE?
MAE averages absolute error in the target's units. RMSE also uses the target's units but penalizes large errors more strongly because it squares errors before averaging. -
Why inspect errors by hour and weather after computing an overall MAE?
The model may perform well on common conditions while failing at rush-hour peaks or rare weather conditions; slices reveal those uneven error patterns.
Completion criterion
Stop when
You can compare the model against its baseline on the fixed test set, explain MAE/RMSE and identify at least one slice where its errors differ.
Mark passed, then open 2.5 Only mark this when you can do it without notes.Leads to
- 2.5 Make the experiment reproducible and explain its limits Follow the practical starter sequence; each unit uses the artifact from the previous step.
Comes after
- 2.3 Build a preprocessing pipeline and first modelFollow the practical starter sequence; each unit uses the artifact from the previous step.
Resources for this unit
- Google Machine Learning Crash Course Use the modules on regression, numerical and categorical data, datasets/generalization/overfitting, and classification metrics as a structured conceptual companion. The course is free and includes interactive exercises.
- Model evaluation: regression metrics Use the regression metric definitions for MAE, MSE/RMSE and R-squared. Explain the units and limitations of each metric instead of reporting a score without context.
- Machine Learning in Python with scikit-learn (MOOC) Use as the free, more thorough follow-on after this guided first model. Its static course and notebooks are openly available; the materials are published under CC BY.
Also in this section
- Bike Sharing Dataset Use the hourly dataset (17,389 records) for a first regression project. Sort chronologically; use 2011 for training and 2012 as the untouched final test period.
cntis the target. Excludecasual,registered,cnt,instant,dtedayandyrfrom the model feature matrix; usedtedayonly to sort/split. - Introduction to Machine Learning Problem Framing Complete the short course on deciding whether ML fits the problem, framing the model output and choosing success metrics before touching the estimator.
- Kaggle Learn — Intro to Machine Learning Complete the lessons on how models work, data exploration, the first model, model validation, underfitting/overfitting and random forests. Reproduce the exercises in a fresh notebook.
- Pipelines and composite estimators Study
PipelineandColumnTransformer. Keep preprocessing inside the pipeline so transformations are fitted only on the training partition and applied consistently at evaluation time. - Common pitfalls and recommended practices Read the sections on inconsistent preprocessing and data leakage. Use the wrong/right examples to audit the project before trusting the reported test score.
- TimeSeriesSplit Read why ordinary cross-validation can train on future observations and evaluate on the past. The starter project uses one chronological year holdout; use TimeSeriesSplit when extending it to repeated time-aware validation.
- Model persistence Read the security and version-compatibility warnings before saving an estimator. Treat the training code, package versions, data split and metrics as the reproducible artifact, not just a serialized model file.