Unit 2.5 · about 2 h
9 / 324Make the experiment reproducible and explain its limits
A working notebook is not yet a reliable project. A reader should be able to understand the data boundary, reproduce the experiment and distinguish measured evidence from unsupported claims.
- implement
- explain
You can now
- Save the training code, feature contract and fitted pipeline with its dependency versions.
- Record the dataset source, license, chronological split, baseline and test metrics.
- Write a README that explains how to reproduce the run from a clean environment.
- State the dataset's limitations and avoid presenting the first model as a production forecast.
- Record the exact prediction moment and the calendar-only feature contract, including why observed weather fields are excluded.
Covers
- Reproducibility, model artifacts and technical reporting
Study, in this order
3 steps
- Model persistenceModel persistence
Read the persistence options, security considerations and version compatibility warnings before saving the pipeline.
- Machine Learning in Python with scikit-learn (MOOC)Machine Learning in Python with scikit-learn (MOOC)
Use as the next structured learning resource after the first project; the course is free and its material is CC BY.
- Bike Sharing DatasetBike Sharing Dataset
Include the dataset citation and CC BY 4.0 attribution in the project README.
Focus on
- Repository structure and clean-run instructions
- Dataset citation, feature contract and split boundaries
- Package versions, seeds and recorded metrics
- Limitations, drift and the difference between a study project and production service
- Prediction-time assumptions and the permitted calendar-only feature list
Reproduce closed-book
Clone your project into a clean directory, follow the README, run the training script and verify that it recreates the recorded metrics within the stated tolerance. Confirm the saved feature contract excludes target-derived fields, identifiers, split-only fields and observed weather.
Traps
- Do not rely on an untracked notebook state or a local absolute path.
- Do not save a serialized estimator without recording package versions and trust boundaries.
- Do not claim generalization to future periods beyond the single held-out year.
Practice
scikit-learn model persistence guide
Choose a persistence method only after reading its security and version-compatibility trade-offs.
Keep the training code and data contract as the primary reproducibility record.
Self-test
-
Why is a serialized model file insufficient as a reproducible ML project?
The model file does not document the data source, feature contract, split, preprocessing, package versions, evaluation procedure or limitations. -
What is the strongest claim this starter project can make?
It can report measured performance on the specified 2012 holdout under the documented split and features. It cannot establish production readiness or guaranteed future performance.
Completion criterion
Stop when
A clean checkout can rebuild the calendar-only pipeline and reproduce the documented evaluation; the README clearly states the dataset license, prediction moment, split, feature contract, metrics and limits.
Mark passed, then open 3.1 Only mark this when you can do it without notes.Leads to
- 3.1 Names, identity, mutability, copying and hashability After completing the first supervised ML project, deepen the Python object model and runtime behavior behind reliable code.
Comes after
- 2.4 Evaluate the model and inspect its errorsFollow the practical starter sequence; each unit uses the artifact from the previous step.
Resources for this unit
- Bike Sharing Dataset Use the hourly dataset (17,389 records) for a first regression project. Sort chronologically; use 2011 for training and 2012 as the untouched final test period.
cntis the target. Excludecasual,registered,cnt,instant,dtedayandyrfrom the model feature matrix; usedtedayonly to sort/split. - Machine Learning in Python with scikit-learn (MOOC) Use as the free, more thorough follow-on after this guided first model. Its static course and notebooks are openly available; the materials are published under CC BY.
- Model persistence Read the security and version-compatibility warnings before saving an estimator. Treat the training code, package versions, data split and metrics as the reproducible artifact, not just a serialized model file.
Also in this section
- Introduction to Machine Learning Problem Framing Complete the short course on deciding whether ML fits the problem, framing the model output and choosing success metrics before touching the estimator.
- Kaggle Learn — Intro to Machine Learning Complete the lessons on how models work, data exploration, the first model, model validation, underfitting/overfitting and random forests. Reproduce the exercises in a fresh notebook.
- Google Machine Learning Crash Course Use the modules on regression, numerical and categorical data, datasets/generalization/overfitting, and classification metrics as a structured conceptual companion. The course is free and includes interactive exercises.
- Pipelines and composite estimators Study
PipelineandColumnTransformer. Keep preprocessing inside the pipeline so transformations are fitted only on the training partition and applied consistently at evaluation time. - Common pitfalls and recommended practices Read the sections on inconsistent preprocessing and data leakage. Use the wrong/right examples to audit the project before trusting the reported test score.
- TimeSeriesSplit Read why ordinary cross-validation can train on future observations and evaluate on the past. The starter project uses one chronological year holdout; use TimeSeriesSplit when extending it to repeated time-aware validation.
- Model evaluation: regression metrics Use the regression metric definitions for MAE, MSE/RMSE and R-squared. Explain the units and limitations of each metric instead of reporting a score without context.