04.08 Lesson 8 of 16
54 minNaive Bayes Classifiers
Apply Bayes' rule to classification and understand the conditional-independence assumption behind Naive Bayes.
By the end of this lesson
You will be able to
- Explain the conditional-independence assumption.
- Distinguish Gaussian, Multinomial, Bernoulli and categorical variants.
- Explain why smoothing prevents zero-probability failures.
- Recognize when probability estimates need calibration.
Bayes' rule updates a prior with evidence
Bayes' rule is P(class | features) ∝ P(features | class) P(class). The prior is the class probability before observing features; the likelihood describes how compatible the observed features are with that class; the posterior is the updated probability. Naive Bayes simplifies the likelihood by assuming features are conditionally independent given the class. This assumption is often unrealistic, but the simplification makes the model fast and can still produce useful rankings.
from sklearn.naive_bayes import GaussianNB
# GaussianNB is shown here for continuous numeric features.
model = GaussianNB()
model.fit(X_train_numeric, y_train)
probability = model.predict_proba(X_valid_numeric)[:, 1]The feature representation determines the variant
GaussianNB models each numeric feature with a class-conditional Gaussian distribution. MultinomialNB is commonly used with non-negative counts such as word counts. BernoulliNB models binary indicators. CategoricalNB is for discrete category codes encoded as non-negative integers; those integers are category identifiers, not ordered measurements. Smoothing adds a small amount to likelihood terms so an unseen event does not force a class probability to zero.
Apply what you learned
ExerciseMatch Naive Bayes to the data
Train an appropriate Naive Bayes estimator on a small dataset and document which assumptions are reasonable or doubtful.
- Choose a representation: continuous values, token counts, binary indicators or discrete category codes.
- Select GaussianNB, MultinomialNB, BernoulliNB or CategoricalNB accordingly.
- Fit the model using training data and produce class probabilities.
- Compare ranking and classification metrics with a simple baseline.
- Write down two feature relationships that may violate conditional independence.
The estimator matches the feature representation, the learner explains the independence assumption and the result is compared with a baseline.
Free references
Study resources
- 01 Naive Bayes, Clearly Explained Video
Watch the individual Naive Bayes explanation. Focus on priors, feature likelihoods and why smoothing is necessary.
- 02 Naive Bayes Article
Read the estimator overview and the GaussianNB and MultinomialNB sections. Check assumptions before choosing a variant.
Check your understanding
Self-test
-
Question 1
What is the naive assumption?
Features are treated as conditionally independent of one another once the class is known. -
Question 2
What problem does smoothing address?
It prevents an unseen feature event from assigning an exact zero probability to a class likelihood.
Completion check
Move on when
You can explain the Bayes update, select a suitable estimator for the representation and identify where conditional independence may be weak.