Unit 5.2 · about 4 h
28 / 324Property, metamorphic and differential testing with shrinking
Examples check the inputs you thought of. Properties check the inputs you did not, and shrinking hands you the smallest case that breaks your claim.
- implement
You can now
- State a property, metamorphic relation or differential oracle for a small function and encode it with Hypothesis strategies.
- Read a shrunk counterexample and turn it into a minimal permanent regression case.
- Replay persisted failures from the example database and pin them with
@exampleinstead of opaque blobs.
Covers
- Property, metamorphic and differential testing
- Generated cases, shrinking and durable failure replay
Study, in this order
4 steps
- Hypothesis quickstartHypothesis documentation
Install Hypothesis through Write your first test; Running in a test suite; the
max_examplesdefault of 100; Filtering inside a test - Hypothesis strategies referenceHypothesis documentation
Primitives; Numeric (
integers); Strings (text); Collections (lists); Combinators - Hypothesis test settingsHypothesis documentation
Changing the number of test cases; Changing settings across your test suite with profiles
- Replaying failed Hypothesis testsHypothesis documentation
The Hypothesis database; Using
@exampleto run a specific input; Replaying test cases with@reproduce_failure; Prefer@exampleover the database for correctness
Focus on
- A property relates many inputs to expected structure: ordered output, round trips, idempotence. A metamorphic relation compares outputs across related inputs; a differential test compares two implementations on the same inputs.
- Given the same drawn data, the test must behave the same way. Randomness, clocks and global state inside the test break shrinking and replay, and Hypothesis reports the test as flaky.
- Shrinking minimizes the failing case automatically. The report shows the minimal input, not the first one found.
- The database replays past failures before generating new ones. Pin durable cases with
@example; a@reproduce_failureblob is opaque and version-fragile, so convert it promptly.
Reproduce closed-book
@given(st.integers(min_value=0)) test asserting \(n < 50\) with max_examples=200: it fails and reports \(n=50\), the minimal failure, in well under a second on a 12th Gen Intel Core i9-12900K with Python 3.14.4 and Hypothesis 6.169.0. Run it again and it replays \(n=50\) from the .hypothesis database. Add derandomize=True, delete .hypothesis, rerun, and see the database stay empty. Then assert sorted(sorted(xs)) == sorted(xs) over st.lists(st.integers()) and count 200 examples.Traps
- Skipping unwanted inputs with a plain
ifandreturninside the test. Those examples count as passes, so coverage shrinks silently; constrain the strategy or callassume, and expect thefilter_too_muchhealth check when most inputs are rejected. - Keeping a
@reproduce_failureblob as the regression test. It is opaque and may stop replaying after upgrades, so rewrite it as@example. - Calling a nondeterministic property suite green. A flaky failure poisons the database, so fix determinism first.
Practice
Hypothesis quickstart failing test
Running in a test suite: the assert \(n < 50\) example
Run it, read the shrunk \(n=50\) report, then convert the case to @example.
Hypothesis rule-based state machines: DatabaseComparison
Rule-based state machines: the DatabaseComparison example with add_key, save, delete and values_agree
Rebuild it against a dict-backed store of your own and run at least 50 examples.
Self-test
-
Hypothesis tried a large integer first but reports \(n=50\). Why?
Shrinking. After finding a failure it searches for the smallest input that still fails, so the report shows the minimal case. -
When should a failing case live in @example rather than the database?
For every durable regression. The database is a local directory that can go stale across versions, andderandomize=True, which the built-inciprofile sets, turns it off, while@exampleruns on every machine in the explicit phase. -
Give a round-trip property for a serializer plus parser, and a metamorphic relation for a sort.
Round trip:parse(serialize(tree))equalstreefor every generated tree. Metamorphic:sorted(xs + ys)equalssorted(ys + xs), relating two runs without knowing the right answer. -
A property test fails only on some runs of identical code. What must you check?
Determinism of draws plus outcome: unmanaged randomness, time, filesystem or global state. Only that sequence needs to be deterministic.
Completion criterion
Stop when
Your property suite fails on a seeded bug with a one-line minimal case, and every durable failure exists as @example with the database replay passing.
Leads to
- 5.3 Regression tests, failure injection and flaky tests Treat nondeterministic property failures as flakiness to isolate, not to rerun.
- 5.4 Debuggers, logs, traces and competing hypotheses Feed shrunk counterexamples into a hypothesis log as the first observation.
Comes after
- 3.4 Iterators, generators and lazy pipelinesState iterator invariants as properties: exhaustion, repeatability, input rejection.
- 5.1 Test levels, oracles, boundary cases and test doublesGeneralize example tests into properties with generated inputs.
Resources for this unit
- Hypothesis documentation Follow the Quick start and property-based testing guides, then generate operation histories and retain the minimized example for one real defect.
Also in this section
- The Missing Semester of Your CS Education Complete the 2026 lectures on the shell, command-line environment, development tools, debugging and profiling, Git, packaging and shipping code, and code quality. Apply every exercise to the same stage project.
- pytest documentation Use Get Started, fixtures, parametrization, monkeypatch, caplog and temporary paths to build behavior and failure-path tests without order dependence.
- The Art of Debugging Use the Unix, Python and compiled-code sections during a real seeded defect. Follow the evidence and tiny-reproducer workflow rather than reading the unfinished book front to back.
- Pro Git Use Chapters 2-3 and the sections on debugging, internals and recovery while constructing a history for bisect and recovering a deliberately displaced commit.
- Stas' Python Cookbook - debugging and testing Use Part III and the packaging sections for debugger, traceback, pytest, logging, profiling and clean-environment recipes.
- AlgoMaster Python - mocking Use the Mocking lesson for
unittest.mock:MockversusMagicMock,patchtargets,autospecand the over-mocking trap. The pytest docs remain the reference for fixtures andmonkeypatch.