Section 5 of 7 · Programming & Data Foundations
Testing, Debugging & Reproducibility
Correctness claims need an independent oracle and failures need evidence. This section turns testing, source control and environment capture into one repeatable investigation discipline.
8 units 18 topics 0/8
Units
8 in this section
- 5.1 Test levels, oracles, boundary cases and test doubles explain · implement4 h
- 5.2 Property, metamorphic and differential testing with shrinking implement4 h
- 5.3 Regression tests, failure injection and flaky tests implement · measure3 h
- 5.4 Debuggers, logs, traces and competing hypotheses measure3 h
- 5.5 Shell streams, exit status, signals, processes and permissions explain · implement4 h
- 5.6 Git history, merges, rebases, bisect and recovery explain · implement4 h
- 5.7 SSH trust, keys, forwarding and tmux sessions implement2 h
- 5.8 Dependency locks, run manifests and quality gates implement4 h
Before moving on
Checkpoint
Resolve one unfamiliar seeded defect with a hypothesis log and regression test, then rebuild and reproduce the result from a clean checkout and recover one displaced Git commit.
Deliberately out of scope
Leave for later
- Organization-wide CI platform administration
- Formal verification systems
- Reproducible builds across every operating system
Resources for this section
Shared by every unit above
- The Missing Semester of Your CS Education Complete the 2026 lectures on the shell, command-line environment, development tools, debugging and profiling, Git, packaging and shipping code, and code quality. Apply every exercise to the same stage project.
- pytest documentation Use Get Started, fixtures, parametrization, monkeypatch, caplog and temporary paths to build behavior and failure-path tests without order dependence.
- Hypothesis documentation Follow the Quick start and property-based testing guides, then generate operation histories and retain the minimized example for one real defect.
- The Art of Debugging Use the Unix, Python and compiled-code sections during a real seeded defect. Follow the evidence and tiny-reproducer workflow rather than reading the unfinished book front to back.
- Pro Git Use Chapters 2-3 and the sections on debugging, internals and recovery while constructing a history for bisect and recovering a deliberately displaced commit.
- Stas' Python Cookbook - debugging and testing Use Part III and the packaging sections for debugger, traceback, pytest, logging, profiling and clean-environment recipes.
- AlgoMaster Python - mocking Use the Mocking lesson for
unittest.mock:MockversusMagicMock,patchtargets,autospecand the over-mocking trap. The pytest docs remain the reference for fixtures andmonkeypatch.
Apply the section
Mini-project · Test, debug and pin the taskrunner (8 to 12 h)
Make the task runner's correctness and environment reproducible with oracle-based tests and a property that finds a seeded defect. Keep an evidence-first debugging log and a locked environment whose runs write a manifest.
Steps
- Test
params_hashagainst boundary cases andplanagainst an independent edge-by-edge oracle that also rejects cyclic input, running each test alone, with the Missing Semester code-quality unit test as reference. - Seed an order-dependent hashing defect, write the Hypothesis property over same-items different-order inputs that finds it following the Hypothesis quickstart shrinking pattern, pin the shrunk case with
@example, then fix the canonical form. - Seed order-dependent output that makes a test fail across fresh processes using the Missing Semester run-until-failure script, show a fixed seed only repeats the outcome, then fix the ordering and record failure counts before and after.
- Seed an off-by-one in a CLI output path and, before touching the code, log competing hypotheses with the observation that would kill each following the Missing Semester merge-sort debugger exercise, then fix it with quoted tool output beside each rejection.
- Rebuild a regression from history on a branch with a status-coded check script, locate it with
git bisect runfollowing the Missing Semester version-control history exercises, and delete and recover a branch tip throughgit reflog. - Write a fail-fast shell entry with quoted arguments, a trapped temporary directory and separate logs checked against the Missing Semester course-shell stream and status exercises, and show it exits cleanly on
TERMwith no leftovers. - Lock the environment, gate on sync plus tests plus typecheck, write a run manifest with commit, config, inputs, seed and outputs, and pass the gate over SSH inside tmux following the Missing Semester SSH, tmux and lockfile exercises.
Deliverables
- The test suite: oracle and boundary tests, the property with its pinned case, and the flaky test's counts before and after.
- The debugging log with hypotheses and quoted evidence, and the
bisecttranscript naming the first bad commit. - The shell wrapper transcripts, the lockfile, the one-command gate, two manifests with their diff and the remote transcript.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Independent oracles | Each test names its oracle, the plan is checked edge by edge, and every test passes alone. | Each oracle stays independent of the code under test, and a deliberate mutation makes a test fail. |
| Properties and flakiness | The property finds the order defect, the shrunk case is pinned, and the flaky test is fixed with counts before and after. | The write-up explains why the shrunk case is minimal and why a pinned seed would hide the flake. |
| Evidence-first diagnosis | The log rejects at least two hypotheses with quoted tool output, and bisect names the seeded commit unaided. | Each hypothesis states its killing observation first, and the regression test fails on the found commit. |
| Locked reproducibility | A clean checkout reaches a green gate, the wrapper exits as specified with no leftovers, and every run writes a manifest. | Two manifests explain an output difference field by field, and the gate also passes on the remote host. |
Feeds into: The clean-checkout gate, the shell wrapper and the run manifest the runbook relies on, plus the debugging log as the model evidence-first report.