Unit 5.1 · about 4 h
27 / 324Test levels, oracles, boundary cases and test doubles
Most ML bugs hide at component boundaries, not inside one function. The test level decides which failures you can see, and the oracle decides whether the test checks anything at all.
- explain
- implement
You can now
- Distinguish unit, contract, integration, end-to-end and performance tests by the failure each can detect.
- Turn a requirement into an invariant, an independent oracle, representative examples and boundary cases.
- Build a pytest suite with fixtures, parametrization and test doubles that has no order dependence.
Covers
- Unit, contract, integration, end-to-end and performance tests
- Requirements, invariants, independent oracles and boundary cases
- Fixtures, fakes, stubs, mocks and integration risk
Study, in this order
6 steps
- Code Quality: Testing and code coverageThe Missing Semester of Your CS Education
Testing: unit, integration and functional tests, test-driven development, regression tests, property-based tests and mocks; Code coverage
- pytest Get Startedpytest documentation
Install pytest through Request a unique temporary directory for functional tests: first test, exceptions, classes and
approx - pytest fixtures: scope and teardownpytest documentation
Requesting fixtures; Scope: sharing fixtures across classes, modules, packages or session; Teardown/Cleanup (AKA Fixture finalization)
- pytest monkeypatch and mockspytest documentation
Monkeypatching functions; Monkeypatching returned objects: building mock classes; Monkeypatching environment variables; Monkeypatching dictionaries
- pytest and unittest recipesStas' Python Cookbook - debugging and testing
Testing with pytest and unittest: pytest basics, Fixtures with
tmp_path,capsys,monkeypatchandcaplog, and Capturing output - MockingAlgoMaster Python - mocking
MockvsMagicMock;patch: Decorator, Context Manager, and Manual; The "Where to Patch" Gotcha;autospecandcreate_autospec; Over-Mocking: The Brittleness Trap
Focus on
- Each level detects a different failure: unit tests catch logic errors in one function, contract tests catch seam drift, integration tests catch wiring errors, end-to-end tests catch requirement misses.
- An oracle must be independent of the code under test: a reimplementation, the standard library, a known value or a metamorphic relation. Asserting code output against itself proves nothing.
- Boundary cases carry the failures: empty input, single element, duplicates, limits and malformed rows. Parametrize them explicitly.
- Function scope is the default for a reason: shared mutable fixtures create order dependence. Use
yieldfixtures for teardown andmonkeypatchfor automatic undo.
Reproduce closed-book
yield fixture using tmp_path, and one monkeypatched environment test. Run pytest and get 5 passed in about 0.3 s on a 12th Gen Intel Core i9-12900K with Python 3.14.4 and pytest 9.1.1.Traps
- Mocking the unit you should be testing. Mock at the seam you do not own, such as network, clock or filesystem, and keep one contract test that touches the real dependency.
- Module- or session-scoped mutable fixtures. Tests pass alone and fail together; keep shared fixtures immutable or hand out fresh copies from a factory.
- Reading 100 percent line coverage as correctness. Coverage shows which lines ran, not which requirements were checked, so the lecture warns against over-indexing on it.
Practice
Missing Semester Code Quality: unit test plus coverage report
Exercises, item 2: write a unit test for a project, generate an HTML coverage report and inspect uncovered lines
Apply it to the parsing module from Reproduce and add one boundary test for each uncovered line the report shows.
Self-test
-
A function returns correct values but writes rows the loader cannot read. Which test level catches this?
An integration or contract test at the writer and loader seam. A unit test of either side alone passes. -
Why is asserting your sort equals your second sort implementation weak, and what is stronger?
Both can share your misunderstanding. A stronger oracle is independent: the standard librarysorted, a fixed fixture, or a property such as the output being ordered. -
Two tests share a module-scoped dict fixture and pass alone but fail together. What is wrong?
One test mutates shared state the other reads. Make the fixture function-scoped or return a fresh copy per test. -
monkeypatch.setenvin one test seems to change a later test. Where is the leak?Not inos.environ: pytest undoes everymonkeypatchchange at teardown. Either a test wrote toos.environdirectly, or code cached the value while the patch was active, such as a module-level setting or anlru_cache. -
mod.pydoesfrom json import dumps. A test patchesjson.dumps, yetmod.runstill calls the real one. Why?patchreplaces a name where it is looked up.modbound its own namedumpsat import, sopatchmod.dumpsinstead. Verified withunittest.mockon CPython 3.14.
Completion criterion
Stop when
Your five-test suite passes, every test also passes alone by node id, and you can name the oracle and the boundary behind each assertion.
Mark passed, then open 5.2 Only mark this when you can do it without notes.Leads to
- 5.2 Property, metamorphic and differential testing with shrinking Generalize example tests into properties with generated inputs.
- 5.3 Regression tests, failure injection and flaky tests Pin fixed behavior as regression tests and isolate nondeterminism.
Comes after
- 3.5 Decorators, wrappers and parameterized decorators
unittest.mock.patchworks as a decorator and as a context manager.
Resources for this unit
- The Missing Semester of Your CS Education Complete the 2026 lectures on the shell, command-line environment, development tools, debugging and profiling, Git, packaging and shipping code, and code quality. Apply every exercise to the same stage project.
- pytest documentation Use Get Started, fixtures, parametrization, monkeypatch, caplog and temporary paths to build behavior and failure-path tests without order dependence.
- Stas' Python Cookbook - debugging and testing Use Part III and the packaging sections for debugger, traceback, pytest, logging, profiling and clean-environment recipes.
- AlgoMaster Python - mocking Use the Mocking lesson for
unittest.mock:MockversusMagicMock,patchtargets,autospecand the over-mocking trap. The pytest docs remain the reference for fixtures andmonkeypatch.
Also in this section
- Hypothesis documentation Follow the Quick start and property-based testing guides, then generate operation histories and retain the minimized example for one real defect.
- The Art of Debugging Use the Unix, Python and compiled-code sections during a real seeded defect. Follow the evidence and tiny-reproducer workflow rather than reading the unfinished book front to back.
- Pro Git Use Chapters 2-3 and the sections on debugging, internals and recovery while constructing a history for bisect and recovering a deliberately displaced commit.