Unit 1.2 · about 3 h
129 / 324Targets, losses, metrics, decision rules and problem framing
The loss and the decision rule decide what the model learns to care about. A perfect score on the wrong target still ships the wrong behavior.
- explain
You can now
- Distinguish the scientific question, prediction target, estimand, action, outcome, loss, metric, and product decision on a worked example.
- Choose regression, classification, or an unsupervised framing from the decision thresholds, and defend a proxy label with its failure modes.
- State success metrics separately from evaluation metrics, and keep prediction language associative rather than causal.
Covers
- Target, estimand, loss, metric and decision rule
- Regression, classification, ranking and unsupervised framing
- Prediction language versus causal language
Study, in this order
4 steps
- Framing: outputs, thresholds, proxies, success metricsGoogle ML Problem Framing - framing an ML problem
Identify the output you need; Classification; Regression including the video-caching 50/30-view threshold analysis; Proxy labels including the usefulness proxy table; Define the success metrics including success versus evaluation metrics and the clickbait failure paragraph
Depth · Skip Generation. This section uses predictive ML only; generative outputs are out of scope.
- Objectives versus metricsRules of Machine Learning - first controls
Rule 2: First, design and implement metrics; Rule 12: Do not overthink which objective you choose to directly optimize; Rule 13: Choose a simple, observable and attributable metric for your first objective
- Framing checklist and task typesHands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2
Frame the problem, items 4 to 7: supervised/unsupervised and online/offline framing, performance measure, alignment with the business objective, minimum performance
- Supervised and unsupervised task vocabularyHands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2
Exercise Solutions 4, 5, and 8: regression and classification as the two most common supervised tasks, clustering/visualization/dimensionality-reduction/association as unsupervised tasks, spam detection as supervised
Focus on
- Success metrics differ from evaluation metrics. Success is product behavior (feature opened 50 percent more; 20 percent more time on site); evaluation is model arithmetic (accuracy, RMSE, AUC). A better model score that does not move success is not progress.
- Fixed thresholds call for classification, moving thresholds for regression. For a video with 30 views, predictions 28 and 32 both miss by 2 and get the same loss (squared error 4), yet the 30-view cache cutoff sends them to different caches.
- Proxy labels substitute for unmeasurable goals (shared OR liked for usefulness). Each candidate in the table carries a stated failure mode: like-prediction misses users who never click like, popularity is not personalized, click prediction maximizes clickbait, watch time favors long videos.
- Prediction language stays associative: predicts, is associated with. Acting on a prediction can change the outcome, as the clickbait paragraph shows engagement dropping after click-optimized ranking, so causal verbs need an experiment, not a model score.
Reproduce closed-book
no-cache, cheap-cache, expensive-cache labels) and when regression stays right (thresholds set and moving in app code).Traps
- Calling the objective the metric. The Terminology section separates them: the objective is the one number the algorithm optimizes; metrics are every number you care about. Rule 12 notes they rise together early on, so a lone objective gain later is not proof of system health.
- Treating a proxy as the goal. Liking, sharing, clicking, and watch time each mismeasure usefulness in a different direction; the table assigns one failure mode per proxy.
- Writing causal conclusions from predictive wins. A higher click score does not show the ranking causes satisfaction; the clickbait paragraph shows behavior diverging after the intervention.
Practice
Google ML Problem Framing: Check Your Understanding boxes
Proxy labels Check Your Understanding (health-app proxy question) and Define the success metrics Check Your Understanding (fashion-firm ideal outcome versus model goal)
Answer both cold, then map each wrong fashion option to the mismatched ideal outcome, goal, output, or metric.
Self-test
-
Truth is 30 views; model A predicts 28 and model B predicts 32. Which has lower squared error, and which caches correctly at a 30-view cutoff?
Neither has lower error: both miss by 2, so both have squared error 4. B caches (32 clears 30) while A does not, so equal loss hides opposite decisions. -
The team wants useful recommendations but logs only clicks. Name the proxy and two failure modes from the table.
Proxy: predict click or play. Failures: it maximizes clickbait, and popularity prediction is not personalized; like-prediction separately misses most users who never click like. -
What distinguishes the ML objective from a success metric, per Rules 2, 12, 13 and the weather example?
The objective is the single directly optimized number, kept simple, observable, and attributable (for exampleclicked). Success is product behavior (feature opened 50 percent more); a strong objective score need not move it. -
Why is the claim that the model proves longer videos are more useful a causal overclaim?
Watch-time prediction favors long videos by construction, per the proxy table. Usefulness needs an experiment or policy comparison; the prediction only associates length with watched time under current exposure.
Completion criterion
Stop when
You answer both Check Your Understanding boxes cold and can redo the 28/32 threshold arithmetic without notes.
Mark passed, then open 1.3 Only mark this when you can do it without notes.Leads to
- 1.3 Split design, nested selection and test isolation Evaluate the chosen metric under a split that respects the decision process.
- 5.2 Regression error and confusion-matrix metrics Deepen error and confusion-matrix metrics after framing them here.
Comes after
- 1.1 Unit of observation, prediction time and label horizonTurn the timed context into a target, loss, metric, and decision rule.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - Chapters 1 and 2 Read Chapters 1 and 2 of the book, then run notebooks 01 and 02 and use ml-project-checklist.md for framing, data inspection, test-set creation, preprocessing pipelines, cross-validation, error analysis and persistence. Rebuild the workflow on UCI Bike Sharing with a chronological split and prohibit casual and registered when predicting cnt.
- Rules of Machine Learning - first controls Read Rules 1-5 and 8-9. Turn the existing heuristic into a baseline and test the feature path independently of the learner.
- Google ML Problem Framing - framing an ML problem Read Identify the output you need through Proxy labels and Define the success metrics, including the video-caching threshold analysis, the proxy-issue table, and both Check Your Understanding boxes.
Also in this section
- scikit-learn - Common pitfalls and pipelines Read Common Pitfalls sections on inconsistent preprocessing and leakage, then Pipeline and ColumnTransformer. Seed global scaling, a target-derived feature and a partition duplicate, and make tests reject each defect.
- scikit-learn - Cross-validation splitters Use the group-aware and time-aware splitter sections plus GroupKFold and TimeSeriesSplit. Choose from the dependence structure and assert that no prohibited group or future timestamp crosses the boundary.
- UCI Bike Sharing Dataset Use hour.csv as the compact integrated dataset. Preserve a raw snapshot and data dictionary, split chronologically, and document why casual and registered are target proxies for cnt.