Unit 2.8 · about 2 h
143 / 324Probability estimates, class priors and asymmetric thresholds
A probability is not a decision. The same fitted scores need different thresholds under different costs and different priors, and 0.5 is a cost claim most projects never stated.
- derive
- measure
You can now
- Derive the cost-optimal threshold from class probabilities and misclassification costs, and show why 0.5 is only optimal for symmetric costs.
- Adjust decisions for changed class priors without refitting, through the prior-odds correction.
- Measure precision-recall and ROC tradeoffs on fixed validation data and select a threshold from stated costs.
Covers
- Probability estimates, class priors and asymmetric thresholds
Study, in this order
4 steps
- Precision-recall tradeoff and ROC thresholdsHands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models
Book Chapter 3: The Precision/Recall Trade-Off and The ROC Curve (printed pp. 115-124). Notebook 03 cells: Precision/Recall Trade-off section (
precision_recall_curveon the MNIST 5-detector, the threshold-3000 demo,idx_for_90_precision) and The ROC Curve section with the 90%-precision operating point - Predicted probabilities and dummy encodingAn Introduction to Statistical Learning with Python
Chapter 4, Section 4.3.3 (pp. 141-142): probabilities computed from fitted coefficients with the balance arithmetic, plus dummy variables for qualitative predictors
- Default threshold, weights and calibration limitsscikit-learn - linear models, discriminant analysis and naive Bayes
Section 1.1.11 introduction (predicted probability becomes a classifier at a threshold, by default 0.5), Binary Case (sample and class weights rescale the effective
CthroughS) and Solvers details (joint multinomial loss calibrates better than OvR) - Priors inside the discriminant scoresscikit-learn - linear models, discriminant analysis and naive Bayes
Section 1.2.2 (posteriors combine class conditionals with \(P(y = k)\); LDA adds \(\log P(y = k)\) to the discriminant) and 1.2.4 (shrinkage as the small-sample stabilizer for those estimates)
Focus on
- sklearn predicts the positive class at 0.5 by default. That bakes in equal costs and matched priors; any asymmetric cost matrix moves the optimal threshold to \(C_{\text{FP}}/(C_{\text{FP}} + C_{\text{FN}})\).
- Naive Bayes scores rank well but estimate badly, and OvR margins are not calibrated across classifiers. Thresholding an uncalibrated score encodes its bias as policy.
- Priors shift without refitting: multiply the posterior odds by the new-over-old prior-odds ratio. Thresholds move with prevalence even when the feature evidence stays fixed.
Reproduce closed-book
precision_recall_curve on binary scores and implement the first-index rule that finds the 90%-precision threshold.Traps
- Tuning the threshold on the test set. Thresholds are fitted parameters: select on validation, report once on test, or the reported precision is optimistic.
- Comparing thresholds across models with different score scales. A 0.3 cutoff on calibrated probabilities means something different from 3000 on raw margins; compare at matched precision or recall, not matched knob values.
- Keeping 0.5 after reweighting classes. Class and sample weights change the score scale and the effective penalty, so retune the threshold after any weighting change.
Practice
HoML 03 Precision/Recall Trade-off: the 90%-precision threshold
Precision/Recall Trade-off section: the precision-recall-versus-threshold plot, idx_for_90_precision, and the ROC point for that threshold
Rebuild the curve and both threshold picks from a blank script; then move the threshold to the cost-optimal point for a stated 1:9 cost ratio and report the precision-recall change.
NeetCode ML: full linear-model closing drill
Math Foundations six: Gradient Descent, Sigmoid and ReLU, Softmax, Cross-Entropy Loss, Linear Regression Forward, Linear Regression Training
After the section gate, solve all six cold. Repeat only failed items from a blank file with changed values.
Self-test
-
False negatives cost 9 times false positives. Which threshold, and why not 0.5?
\(t = 1/(1 + 9) = 0.1\): predict positive whenever \(p\) exceeds 0.1. The 0.5 default assumes equal costs and would miss cheap-to-catch positives. -
Deployment prevalence of positives halves with the same feature evidence. How do the optimal decisions change without refitting?
Posterior odds are the likelihood ratio times the prior odds, so every case's posterior odds are multiplied by the new-over-old prior-odds ratio, about one half when positives are rare. Fewer cases clear any fixed threshold; recompute posteriors with the new priors, or move the threshold equivalently, instead of reusing the old cutoff. -
Two detectors both report 90% precision, one at margin 3000 and one at probability 0.42. Which threshold is better?
The knobs are incomparable across scales. Compare recall at matched precision, or expected cost, on the same validation data. -
The guide warns naive Bayes is a bad estimator. What breaks if its raw scores are thresholded at 0.5?
Ranking can be good while false independence pushes absolute scores to extremes, so 0.5 lands in the wrong place. Set the threshold from measured precision and recall, or calibrate first.
Completion criterion
Stop when
You can derive the cost-optimal threshold, move it correctly under a stated prior shift, and replicate the 90%-precision procedure on new scores without touching the test set.
Mark passed, then open 3.1 Only mark this when you can do it without notes.Leads to
- 5.5 Cost-sensitive thresholds, imbalance and prevalence change Extend cost-sensitive thresholds to imbalance and prevalence change.
- 5.4 Calibration curves and Platt or isotonic recalibration Recalibrate the estimates this unit learns to distrust.
Comes after
- 2.3 Logistic regression and stable binary cross-entropyTurn the fitted probabilities into cost-aware decisions.
- 2.4 Softmax regression and one-versus-restTurn multiclass scores into decisions with costs attached.
- 2.5 Residual, leverage, separation and misspecification diagnosticsSet the decision rule only after the fit survives diagnostics.
- 2.6 LDA, QDA and generative versus discriminative classifiersWeight discriminant scores by class priors before deciding.
- 2.7 Naive Bayes: event models, smoothing and log-space scoringCorrect the overconfident scores before thresholding them.
Resources for this unit
- An Introduction to Statistical Learning with Python Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
- scikit-learn - linear models, discriminant analysis and naive Bayes Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
- Hands-On Machine Learning with Scikit-Learn and PyTorch - classification and linear models Read the Chapter 3 and 4 sections each unit names, then run notebooks 03 and 04 for classification diagnostics, linear and polynomial regression, regularization and logistic regression. Recreate the target mechanism independently before inspecting worked exercise code.
Also in this section
- Dive into Deep Learning - linear models slice Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
- NeetCode ML - linear-model closing drill After the section gate, solve Gradient Descent, Sigmoid and ReLU, Softmax, Cross-Entropy Loss, Linear Regression Forward and Linear Regression Training cold. Repeat only failed items from a blank file with changed values.
- Deep-ML - linear and probabilistic model problems Use 15 Linear Regression Using Gradient Descent, 106 Train Logistic Regression with Gradient Descent and 140 Bernoulli Naive Bayes Classifier as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin. The local implementation remains the evidence.
- Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.