Unit 5.3 · about 3 h
29 / 324Regression tests, failure injection and flaky tests
A fixed bug returns unless a test pins it. A flaky test teaches the team to ignore red builds, which hides the next real failure.
- implement
- measure
You can now
- Write a regression test that fails on the old code and passes on the fix, and mark known-bad behavior with
strictxfail. - Inject a targeted fault, such as a full disk or a failing dependency, and show the suite catching it.
- Measure a flake rate, isolate its randomness, time, concurrency or state source, and quarantine deliberately instead of rerunning blindly.
Covers
- Regression tests and targeted failure injection
- Randomness, time, concurrency, external state and flaky tests
Study, in this order
4 steps
- pytest skip and xfailpytest documentation
Skipping test functions and
skipif;XFailwithcondition,reason,raises,runandstrictparameters; Skip/xfailwithparametrize - pytest flaky tests explanationpytest documentation
Why flaky tests are a problem; Potential root causes: system state, overly strict assertion and thread safety; Other general strategies: split up test suites and quarantine
- Hypothesis flaky failuresHypothesis documentation
Common sources of flakiness; Types of flakiness: Flaky failure and Flaky strategy definition
- Reproducing resource issuesThe Art of Debugging
Watching and reproducing resource issues: Running out of resources, Emulating an almost full disk partition and Emulating running out of CPU memory
Focus on
- A regression test names the defect: it fails before the fix and passes after. Write it before the fix when you can.
xfailwithstrict=Trueturns an unexpected pass into a failure, so fixed bugs announce themselves. Unstrictxfailas permanent quarantine is dangerous; the pytest guide says so.- Flakiness means uncontrolled state: randomness, clocks, threads, leftover files or test order. Parallel runs and randomized ordering expose order dependence.
- Rerun plugins and quarantines mitigate; they do not fix. Isolate the source first, then control it, quarantine it with an owner, or delete the test.
Reproduce closed-book
assert random.random() < 0.5 and run it 10 times: expect about half to fail (5 failed and 5 passed in the reference run, about 0.07 s per run on a 12th Gen Intel Core i9-12900K with Python 3.14.4 and pytest 9.1.1). Seed it with random.seed(1) (first draw 0.134) for 5 passes in 5 runs, then random.seed(0) (first draw 0.844) for 5 failures: seeding makes the outcome deterministic, not correct. Then mark a known failure xfail(strict=True), fix the code, and watch the suite fail on XPASS until you remove the mark.Traps
- Rerunning the suite until green and merging. The team learns to distrust failures, so quarantine with a ticket instead.
xfailwithoutstricton a fixed bug. The suite stays green while the mark silently goes stale.- Wall-clock sleeps as synchronization. Load makes them flaky, so wait on events or poll with deadlines instead.
Practice
Missing Semester run-until-failure script with command argument
Exercises, item 12: modify the lecture flaky-test script to accept the test command as an argument
Point it at your flaky test and capture the first failing log; then fix the flake and retire the loop.
Self-test
-
A test fails one run in twenty. What is the first measurement, and what comes after?
The flake rate over repeated runs, then isolation: seed randomness, fix clocks, serialize threads, clean state, and run the test alone versus in the suite. -
When is
strictxfailthe right tool?A known bug with a test: the suite stays green while it fails and goes red withXPASSwhen fixed, which forces removal of the mark. -
A test passes alone but fails under parallel workers. What does that suggest?
Order dependence or shared state: a previous test leaves data behind, or the test needs data another test creates. -
Why does
pytest.approxbelong in a flake fix for float asserts?Exact float equality is an overly strict assertion: tiny representation differences fail otherwise identical runs.approxcompares within tolerance.
Completion criterion
Stop when
You can quote the flake rate before and after your fix, the regression test fails on the old code, and no test needs a rerun to pass.
Mark passed, then open 5.4 Only mark this when you can do it without notes.Leads to
- 5.4 Debuggers, logs, traces and competing hypotheses Turn the isolated flake or regression into a hypothesis log with tool evidence.
- 5.6 Git history, merges, rebases, bisect and recovery Bisect the history to find which commit introduced the regression.
Comes after
- 5.1 Test levels, oracles, boundary cases and test doublesPin fixed behavior as regression tests and isolate nondeterminism.
- 5.2 Property, metamorphic and differential testing with shrinkingTreat nondeterministic property failures as flakiness to isolate, not to rerun.
Resources for this unit
- pytest documentation Use Get Started, fixtures, parametrization, monkeypatch, caplog and temporary paths to build behavior and failure-path tests without order dependence.
- Hypothesis documentation Follow the Quick start and property-based testing guides, then generate operation histories and retain the minimized example for one real defect.
- The Art of Debugging Use the Unix, Python and compiled-code sections during a real seeded defect. Follow the evidence and tiny-reproducer workflow rather than reading the unfinished book front to back.
Also in this section
- The Missing Semester of Your CS Education Complete the 2026 lectures on the shell, command-line environment, development tools, debugging and profiling, Git, packaging and shipping code, and code quality. Apply every exercise to the same stage project.
- Pro Git Use Chapters 2-3 and the sections on debugging, internals and recovery while constructing a history for bisect and recovering a deliberately displaced commit.
- Stas' Python Cookbook - debugging and testing Use Part III and the packaging sections for debugger, traceback, pytest, logging, profiling and clean-environment recipes.
- AlgoMaster Python - mocking Use the Mocking lesson for
unittest.mock:MockversusMagicMock,patchtargets,autospecand the over-mocking trap. The pytest docs remain the reference for fixtures andmonkeypatch.