Section 3 of 5 · Mathematical Foundations

Probability, Information & Bayesian Reasoning

Model outputs, losses, sampling and uncertainty all depend on conditioning correctly and keeping the data-generating assumptions visible.

7 units 17 topics 0/7

Units

7 in this section

3.1 Probability axioms and random variables
explain · derive
3 h
3.2 Joint, marginal and conditional distributions, and independence
derive
3 h
3.3 Expectation, variance, covariance and conditioning laws
derive
4 h
3.4 Common distribution families and transformations
derive · implement
4 h
3.5 Bayes rule, conjugate priors and posterior predictives
derive · implement
4 h
3.6 Laws of large numbers, CLT, concentration and Monte Carlo
derive · measure
4 h
3.7 Entropy, cross-entropy, KL divergence and mutual information
derive
3 h

Before moving on

Checkpoint

Solve a new conditional-probability problem, verify it by simulation, and derive a posterior predictive calculation with its assumptions.

Deliberately out of scope

Leave for later

Measure-theoretic proofs
Advanced stochastic processes
General-purpose approximate inference

Resources for this section

Shared by every unit above

Harvard Stat 110 - Probability
Harvard Universitycourse

primary

Follow lectures 4-14, 17-23 and 25-30 with matching Strategic Practice. Use lectures 1-3 only for prerequisite repair.
Stat 110 Strategic Practice
Harvard Universitypractice

required

Solve the matching sets before marking a probability topic Practised; use solutions to audit reasoning, not to begin it.
MIT 18.05 - Bayesian bridge
MIT OpenCourseWarecourse

required

Use classes 10-16 for MLE, Bayesian updating, posterior prediction, conjugacy, prior choice and credible intervals.
Probabilistic Machine Learning: An Introduction
Kevin Murphytheory

reference

Consult targeted parts of Chapters 2, 3, 4 and 6 for ML notation, Monte Carlo, factorization, posterior prediction and information theory.
D2L - Information Theory
D2L authorsimplementation

required

Use §§22.11.1-22.11.5 for runnable entropy, cross-entropy, KL and mutual-information examples and exercises.

Apply the section

Mini-project · A Monte Carlo verifier for conditional probability, posterior predictives and information (8 to 12 h)

Build exact enumeration on finite spaces, a Monte Carlo estimator that reports its standard error, a Beta-Binomial posterior predictive and discrete information measures. Solve new conditioning problems exactly, confirm them by simulation with an error-aware rule, and show where a wrong model or a heavy tail breaks the check.

Steps

Write an exact-enumeration helper over finite sample spaces with fractions.Fraction, and a Monte Carlo estimator that returns the estimate, its standard error and the sample count. State an acceptance rule for simulation checks and justify it.
Solve Stat 110 Homework 2 Problem 6 and 18.05 Problem Set 7 Problem 1 exactly by enumeration, then by simulating each story including how the information was revealed. Encode one tempting wrong answer as well and show the simulation rejecting it.
Check the law of total variance on a two-stage model of your own, comparing the exact decomposition with a simulation of both stages.
Derive the Beta-Binomial posterior predictive for further trials, naming each assumption, and match scipy.stats.betabinom and a two-stage simulation drawing from the posterior. Then generate data whose chance drifts mid-stream and show the predictive intervals missing later batches more often than claimed.
Compare Chebyshev and Hoeffding bounds with simulated tail probabilities for a bounded variable. Then simulate a heavy-tailed variable with a mean but no variance and show standard-error intervals for its mean missing far more often than claimed.
Implement entropy, cross-entropy and KL divergence in bits and nats, handling zero probabilities and the infinite case. Match scipy.stats.entropy and check the cross-entropy identity and the asymmetry of KL on a pair of your own.

Deliverables

numlab/prob/: enumeration, the Monte Carlo estimator, the posterior predictive and information measures.
Tests with exact and simulated answers for both problems and the rejected wrong answer, plus the variance, predictive, drift, tail and information checks.
notes/probability.md with the derivations, each with its assumptions, and the simulation acceptance rule.

Checks

rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Exact answers	Both conditioning problems are solved exactly by enumeration and match the official solutions.	The notes say which event was conditioned on in each version and why the tempting answer conditions on the wrong one.
Simulation discipline	Every simulation states its sample count and standard error, and acceptance uses the stated rule.	The wrong answer is rejected by the same rule that accepts the right one.
Posterior predictive	The Beta-Binomial predictive is derived, matches `scipy.stats.betabinom` and agrees with the two-stage simulation.	The drift experiment names the failed assumption and shows how the predictive intervals miss.
Limits and information	Bounds are compared with simulated tails, the heavy-tail failure is shown, and information measures match SciPy with zeros handled.	The heavy-tail note explains the failure through the missing variance, not through too few samples.

Feeds into: Capstone Monte Carlo uncertainty: the estimator with standard errors and its heavy-tail failure, reused by the statistics checks.