Unit 3.6 · about 4 h
149 / 324Support vector machines: margins, hinge loss and slack
The margin turns classification into geometry with a price list: violations cost \(C\), dimensions cost kernels, probabilities cost calibration. Every later kernel and calibration discussion assumes this pricing.
- derive
You can now
- Derive the hard-margin width \(2/\|w\|\), the hinge loss and the slack budget, and connect \(C\) to margin width and support-vector count.
- Classify any training point as support vector or not from its margin position and read dual coefficients on a toy.
- Defend scaling, \(C\) choice and
LinearSVC-versus-SVCselection from measurements, and state whatSVCprobabilities mean in sklearn 1.9.
Covers
- Hard and soft margins, hinge loss, slack and support vectors
Study, in this order
6 steps
- ISLP Chapter 9: maximal margin and support vector classifiersISLP - tree-based methods and support vector machines
Sections 9.1 and 9.2 (pp. 367 to 376): hyperplanes, the maximal margin classifier, the non-separable case, and the
SVCoptimization with \(C\), slack variables and support vectors - ISLP Chapter 9: multiclass extensions and the hinge lossISLP - tree-based methods and support vector machines
Sections 9.4 and 9.5 (pp. 383 to 386): one-versus-one against one-versus-all, and the Loss plus Penalty form with hinge versus logistic loss
- HoML Appendix C: linear and soft-margin SVMsHands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models
Linear SVM Classification, Soft Margin Classification, Under the hood with the hinge plots, and the
LinearSVCversusSVCversusSGDClassifierexercise - sklearn SVM: classification, imbalance and probabilitiesscikit-learn - neighbors, SVMs, trees and ensembles
1.4.1 Classification with 1.4.1.2 Scores and probabilities and 1.4.1.3 Unbalanced problems
Depth · The
probability=Truepath is deprecated in sklearn 1.9 and warns withFutureWarningtoward removal in 1.11. A defaultSVChas nopredict_proba; useCalibratedClassifierCVwithSVCandensembleFalsefor probabilities. - sklearn SVM: tips, complexity and mathscikit-learn - neighbors, SVMs, trees and ensembles
1.4.5 Tips on practical use (scale features,
Cdefault 1.0,cache_size) with 1.4.4 Complexity and 1.4.7 Mathematical formulation - ISLP lab: support vector classifier cost demoISLP - tree-based methods and support vector machines
Support Vector Classifier: \(C\) 10 versus \(C\) 0.1 on the 2D example with wider margin and more support vectors at small \(C\), then grid search selecting \(C\) 1 at 0.74 CV accuracy
Focus on
- The gutter constraints fix the margin width at \(2/\|w\|\). The toy gives \(w = 1\), \(b = 0\), width 2.0, support vectors at \(x = -1\) and \(x = 1\) with dual coefficients -0.5 and 0.5. Only points on or inside the margin enter the solution.
- \(C\) is the violation budget. On negatives at -2, -1 and 2.4 and positives at 0.5, 1 and 2, \(C\) 0.05 gives width 9.756 with all 6 points as support vectors, while \(C\) 100 gives width 2.0 with 4. Train accuracy is 5 of 6 at both, because the point at 2.4 breaks linear separability.
- Hinge loss is exactly 0 past the margin, so safe points vanish from the fit. Logistic loss never reaches 0, so every point keeps pulling, which is why only support vectors matter for the
SVC. - Scale before fitting. With features on \([0, 1]\) and \([0, 100]\) and the label \(x_0 + x_1/100 > 1\), a default RBF
SVCscores 0.75 unscaled against 0.97 standardized (200 train and 100 test points, seed 2).SVCdefaults to RBF withgammascaleand \(C\) 1.0;LinearSVCdefaults tosquared_hingeand exposes no support vectors.
Reproduce closed-book
SVC and kernel linear at large \(C\).Traps
- Promising
SVCprobabilities from older tutorials now warns in sklearn 1.9 and disappears in 1.11. Never shippredict_probawithout the calibrator wrapper. - Reusing a \(C\) value between
SVCandLinearSVC.LinearSVCuses squared hinge and, through liblinear, also regularizes the intercept, so the same \(C\) gives a different model; compare decisions, not parameter values. - Skipping scaling is the classic SVM bug. \(\gamma\) and the margin both assume comparable units, and the mixed-scale example loses 22 points of accuracy without it.
Practice
Deep-ML 21: Pegasos Kernel SVM Implementation
Implement deterministic pegasos_kernel_svm; the visible tests check ([2.0, 2.0, 6.0, 1.0], 36.2027) for linear and ([1.0, 1.0, 1.0, 1.0], 0.0) for RBF with sigma 0.5
Labels are plus-minus 1. The update rule is in the problem's learn section: follow it literally, updating \(\alpha_i\) and \(b\) in place sample by sample within each iteration (a batch update fails test one), with the RBF kernel \(\exp(-\|x - y\|^2 / (2 \sigma^2))\). Note that \(\eta_t \lambda\) is 1 at \(t = 1\), so that step resets every \(\alpha\).
Self-test
-
For \(w = [1]\) and \(b = 0\), what is the margin width and which toy points are support vectors?
Width 2.0 with support vectors at \(x = -1\) and \(x = 1\), carrying dual coefficients -0.5 and 0.5. -
\(C\) 0.05 gives width 9.756 with 6 support vectors while \(C\) 100 gives width 2.0 with 4. Explain both changes.
Small \(C\) tolerates violations, so the margin grows wide and every point sits on or inside it. Large \(C\) punishes violations, so the margin narrows onto 4 points. -
A 4-class problem goes to
SVC. How many internal classifiers fit, and what doesdecision_function_shapeovrchange?Six one-versus-one pairs fit inside libsvm. Theovrshape only reshapes the output scores, not the fits. -
A far-correct point contributes what to hinge versus logistic loss, and what follows for sparsity?
Exactly 0 to hinge and a small positive amount to logistic. Hinge lets safe points drop out of the solution, leaving support vectors only.
Completion criterion
Stop when
You derive the width, reproduce the toy coefficients, predict the \(C\) effect before running, and problem 21 passes.
Mark passed, then open 3.7 Only mark this when you can do it without notes.Leads to
- 3.7 Kernels, bandwidth and kernel matrix cost Extend the margin to nonlinear boundaries with kernels.
- 3.8 Feature importance caveats and class weighting effects Trace how \(C\) and class weights move decisions.
Comes after
- 3.5 k-nearest neighbors, search cost and the curse of dimensionalityReplace votes with margins on the same geometry.
Resources for this unit
- Hands-On Machine Learning with Scikit-Learn and PyTorch - nonlinear classical models Read Chapters 3, 5 and 6 and the online Appendix C notebook as the first pass through performance measures, trees, ensembles and SVMs, running notebooks 03, 05 and 06 alongside. Vary depth, estimators, learning rate, k, C and RBF width under fixed splits.
- scikit-learn - neighbors, SVMs, trees and ensembles Check official estimator constraints, complexity, preprocessing needs, missing-value behavior and probability caveats. Do not use library calls as the mechanism implementation.
- ISLP - tree-based methods and support vector machines Use Chapters 8 and 9 with their Python labs for the statistical account of trees, bagging, random forests, boosting and SVMs that the notebooks leave implicit, and for model-choice and feature-importance arguments.
Also in this section
- Deep-ML - tree, ensemble and kernel problems Use 138 Best Gini-Based Split, 20 Decision Tree Learning, 307 Bagging Classifier, 21 Pegasos Kernel SVM and 280 RBF Kernel as unit checks after the local implementation passes. Their tests were read in review; each card states what they pin, including tie and update-order conventions.
- Probabilistic Machine Learning, Chapter 16: exemplar-based methods Read Sections 16.1 to 16.1.3 for the KNN decision rule, the curse of dimensionality with the growing-hypercube example, and the speed section on k-d trees, hashing and approximate search.
- XGBoost parameters and sklearn estimator interface Read the tree-booster parameter list for
eta, depth, sampling and regularization names and defaults, and the sklearn page for early stopping witheval_set. - LightGBM parameters Read
learning_rate,num_leaves,max_depth,min_data_in_leaf,feature_fraction,bagging_fractionwithbagging_freq,lambda_l2and early stopping defaults. - CatBoost common parameters and categorical features Read
iterations,learning_rate,l2_leaf_reg,depth,grow_policy,subsample,rsm,boosting_type,use_best_modelandone_hot_max_sizedefaults, plus native categorical handling.