Naive Bayes: event models, smoothing and log-space scoring

1. Naive Bayes: independence, smoothing and parameter counts

   Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression

   Sections 2.1-2.3 (conditional independence, the naive Bayes derivation, discrete inputs with the smoothed estimate (7) and Laplace at \(l = 1\), prior smoothing (9), \(n(J - 1)K\) plus \(K - 1\) free parameters) and 2.4 (Gaussian naive Bayes with \(2nK\) mean and variance parameters)

   Done
2. Naive Bayes in the generative family

   An Introduction to Statistical Learning with Python

   Chapter 4, Section 4.4.4 (p. 158): the naive Bayes model beside LDA and QDA

   Done
3. Naive Bayes on binary MNIST pixels

   Dive into Deep Learning - linear models slice

   Section 22.9: The Probabilistic Model for Classification, The Naive Bayes Classifier, Training on pixels binarized at 128, then Exercises 1 (XOR failure) and 2 (unseen values without smoothing)

   Done
4. Event models, smoothing and estimator limits

   scikit-learn - linear models, discriminant analysis and naive Bayes

   Section 1.9 (MAP rule, the bad-estimator warning on predict_proba), 1.9.1 (GaussianNB), 1.9.2 (MultinomialNB smoothing formula, Laplace at \(\alpha = 1\) versus Lidstone below 1) and 1.9.4 (BernoulliNB decision rule, explicit penalty on non-occurrence, binarize behavior)

   Done
5. Gaussian naive Bayes lab on Smarket

   An Introduction to Statistical Learning with Python

   Naive Bayes section: GaussianNB fit on Smarket with classes_ and class_prior_ readouts

   Done

Focus on

• Smoothing is a MAP estimate under a Dirichlet prior, not a hack. Mitchell's \((\text{count} + l)/(\text{total} + lJ)\) at \(l = 1\) is Laplace, and BernoulliNB's \(\alpha = 1.0\) is the same add-one per value; without it one unseen value zeroes the whole product.
• Bernoulli penalizes absent indicator features while multinomial ignores absent counts. That is why occurrence vectors can beat count vectors on short documents, and why the two models are not interchangeable.
• Score in log space always. A product of 1100 halves is exactly 0.0 in float64 while the log sum stays finite at -762.46; \(\operatorname{argmax}\) needs only the sums.

Reproduce closed-book

Traps

• Feeding counts to BernoulliNB or magnitudes to a binary model without thought. BernoulliNB binarizes continuous input at binarize = 0.0 by default, silently turning magnitudes into presence flags.
• Multiplying raw probabilities and comparing. Beyond underflow, one unsmoothed zero vetoes the class; sum logs and smooth first.
• Reading predict_proba as calibrated confidence. The guide calls naive Bayes a bad estimator: use the scores to rank, then threshold or calibrate downstream.

Practice

Self-test

1. A test word never appeared in training. What happens with and without smoothing?

   Show answer Got it Missed

2. When does Bernoulli beat multinomial on text, and why?

   Show answer Got it Missed

3. Write the smoothed discrete likelihood and name its prior.

   Show answer Got it Missed

4. XOR data with labels [0,1,1,0]. Does naive Bayes succeed, and which assumption fails?

   Show answer Got it Missed

Leads to

• 2.8 Probability estimates, class priors and asymmetric thresholds Correct the overconfident scores before thresholding them.
• 5.3 ROC AUC, average precision, log loss and Brier score Score the outputs with log loss and Brier rather than accuracy.

Comes after

• 3.1 Probability axioms and random variablesClass-conditional probabilities need a clean sample space first.
• 3.2 Joint, marginal and conditional distributions, and independenceNaive Bayes is this unit's conditional-independence assumption applied to features.
• 4.2 Maximum likelihood and MAP estimationApply count likelihoods and distinguish smoothing conventions.
• 2.6 LDA, QDA and generative versus discriminative classifiersShrink the Gaussian assumption to diagonal covariance and smoothed counts.

Resources for this unit

• An Introduction to Statistical Learning with Python
  James, Witten, Hastie, Tibshirani and Taylorprimarytheory
  Use Chapters 3, 4 and 6 with their Python labs as the first pass: least squares and its diagnostics, logistic regression, LDA, QDA and naive Bayes, basis expansion and regularization paths.
• Dive into Deep Learning - linear models slice
  D2L authorsrequiredimplementation
  Use 3.1, 3.4, 3.6, 3.7, 4.1, 4.4, 22.7 and 22.9 for linear regression, generalization, weight decay, softmax, maximum likelihood and naive Bayes. Implement with NumPy first and use the PyTorch versions only as a differential reference.
• scikit-learn - linear models, discriminant analysis and naive Bayes
  scikit-learnreferencereference
  Use the official Linear Models, LDA and QDA, and Naive Bayes guides to verify estimator semantics and selected edge cases after the independent implementation works.
• Tom Mitchell - Generative and Discriminative Classifiers: Naive Bayes and Logistic Regression
  Tom Mitchell, Carnegie Mellon Universityrequiredtheory
  Read Sections 2 (naive Bayes, the smoothed estimates (7) and (9), parameter counts), 3 (Gaussian naive Bayes implies the logistic form, gradient ascent, the \(L_2\) penalty, the \(K\)-class rules) and 4 (generative versus discriminative, Ng and Jordan rates). This is the CS446 reading for Lectures 6-8; rework each derivation on a hand-sized dataset.