Unit 3.1 · about 3 h
65 / 324Probability axioms and random variables
Every later loss, sampler and posterior conditions on an event. A vague sample space now becomes a wrong conditional later.
- explain
- derive
You can now
- Explain sample spaces, events, probability mass functions, density functions and cumulative distribution functions, and state when each one applies.
- Derive a probability in a new finite model by counting, and check it by exact arithmetic and by simulation with a stated seed.
Covers
- Sample spaces, events and probability axioms
- Random variables, PMFs, PDFs and CDFs
Study, in this order
4 steps
- Sample spaces, counting and axiomsHarvard Stat 110 - Probability
Lectures 1 to 3: naive definition, counting, axioms and inclusion-exclusion
- Random variables, PMFs, PDFs and CDFsHarvard Stat 110 - Probability
Lectures 7 and 8: Bernoulli and Binomial variables, CDFs, PMFs and Hypergeometric
- Counting and discrete-variable drillsStat 110 Strategic Practice
Strategic Practice: Naive Definition Q1-4 and Story Proofs Q5-7; Homework Q1-7
- Probability and random-variable notationProbabilistic Machine Learning: An Introduction
Chapter 2, Sections 2.1-2.2.2 (pp. 33-38): events, the conditional-probability definition, discrete and continuous variables, CDFs and quantiles
Focus on
- The naive definition needs finite and equally likely outcomes. State both assumptions before counting.
- A PMF gives probabilities that sum to 1. A density does not: only its integral over a set is a probability, and a density value can exceed 1.
- The CDF \(F(x)=P(X \le x)\) works for every variable type and determines the distribution.
- Universality: for continuous \(X\) with CDF \(F\), \(F(X)\) is \(\operatorname{Uniform}(0,1)\). This is the seed of inverse-CDF sampling in the transforms unit.
Reproduce closed-book
float64: numpy.random.default_rng(7), a (200000, 2) int64 array from integers(1, 7); the fraction of sums equal to 9 is 0.11259 with standard error 0.00071, within about 2 standard errors of 1/9.Traps
- Reading a density value as a probability. For continuous \(X\), \(P(X=x)=0\); report \(P(X \in \text{an interval})\) instead.
- Calling an outcome random without a distribution. Name the set, the probabilities on it and the sampling scheme.
- Assuming distinct-looking formulas give distinct distributions. Compare the CDF or the PMF values, which settle identity.
Practice
Stat 110 Strategic Practice 1: Naive Definition Q1-4 and Story Proofs Q5-7
Strategic Practice Q1-7 with the inline solutions; Homework Q1-7 for counting drills
Attempt each problem before reading its solution. The solutions are printed in the same file.
Self-test
-
Two dice: why is \(P(\text{sum }9)=4/36\) and not 4/11?
Sums 2 to 12 are 11 values but not equally likely, so the naive definition does not apply to them. The 36 ordered pairs are equally likely by symmetry of fair independent dice, and 4 of them sum to 9. The step that uses the assumption is choosing the 36 pairs as the counting set. -
Give a density that exceeds 1 somewhere and show it is still valid.
\(\operatorname{Uniform}(0, 0.5)\) has density 2 on its support and integrates to 1. Validity uses non-negativity plus integral 1; no step bounds the density pointwise. -
\(X\) is continuous with strictly increasing CDF \(F\). Prove \(F(X)\) is \(\operatorname{Uniform}(0,1)\), naming each assumption.
\(P(F(X) \le u)=P(X \le F^{-1}(u))\) uses strict increase for the inverse step, then equals \(F(F^{-1}(u))=u\) by continuity of \(F\). The result \(u\) on \((0,1)\) is the Uniform CDF. -
A learner simulates 2000 dice pairs, gets 0.125, and claims the exact answer is wrong. What is the error?
Simulation noise. The standard error is \(\sqrt{(1/9)(8/9)/2000}=0.0070\), so 0.125 sits 1.98 standard errors above 1/9, a gap about one run in 20 produces. A simulation reports an estimate with its standard error; it never refutes an exact count.
Completion criterion
Stop when
You derive the 1/9 by counting without notes, your rerun of the seed-7 simulation matches 0.11259 to 4 decimals, and you can state for any new finite model whether naive counting applies.
Mark passed, then open 3.2 Only mark this when you can do it without notes.Leads to
- 3.2 Joint, marginal and conditional distributions, and independence Condition on the events defined here.
- 2.7 Naive Bayes: event models, smoothing and log-space scoring Class-conditional probabilities need a clean sample space first.
Resources for this unit
- Harvard Stat 110 - Probability Follow lectures 4-14, 17-23 and 25-30 with matching Strategic Practice. Use lectures 1-3 only for prerequisite repair.
- Stat 110 Strategic Practice Solve the matching sets before marking a probability topic Practised; use solutions to audit reasoning, not to begin it.
- Probabilistic Machine Learning: An Introduction Consult targeted parts of Chapters 2, 3, 4 and 6 for ML notation, Monte Carlo, factorization, posterior prediction and information theory.
Also in this section
- MIT 18.05 - Bayesian bridge Use classes 10-16 for MLE, Bayesian updating, posterior prediction, conjugacy, prior choice and credible intervals.
- D2L - Information Theory Use §§22.11.1-22.11.5 for runnable entropy, cross-entropy, KL and mutual-information examples and exercises.