Iterators, generators and lazy pipelines

Data work streams. Single-pass exhaustion and silent re-iteration bugs are the usual pipeline failures.

explain
implement

You can now

Implement a reusable iterator with tests for exhaustion, repeated iteration and invalid input.
Implement a lazy generator pipeline and compare it with the eager version on peak memory, time and correctness.
Predict generator cleanup on close, throw and deallocation, and PEP 479 StopIteration handling.

Covers

Iterable and iterator protocols, exhaustion and repeated iteration
Generators, lazy pipelines, send and cleanup behavior

Study, in this order

5 steps

Comprehension forms and itertools
Stas' Python Cookbook - selected path

Comprehensions, Iterators and itertools: list/dict/set/generator forms; the essential itertools (chain, islice, groupby on sorted input); zip strict=True and zip_longest

Done
Generators and yield
Stas' Python Cookbook - selected path

Generators and yield: the read_big line pipeline, infinite integers, yield from in flatten()

Done
Generator functions and container protocols
Python language reference

Section 3.2.8.3 Generator functions (calling returns an iterator; the body runs on first next) and 3.3.7 Emulating container types (__len__, __iter__, __getitem__, __contains__)

Done
Custom Iterators
AlgoMaster Python - structured revision

The Custom Iterators lesson and its exercise; write __iter__ and __next__ from memory first

Done
Generators
AlgoMaster Python - structured revision

The Generators lesson and Exercise: Generators; then the yield Statement lesson for send and return values

Done

Focus on

An iterable makes fresh iterators; an iterator is consumed once. iter() on a list gives a new object each time, while a generator is its own single-pass iterator.
Generators suspend at yield and resume after it. send() delivers a value to the paused yield expression; a return value becomes StopIteration.value; close() raises GeneratorExit inside.
Raising StopIteration inside a generator body is a RuntimeError since Python 3.7 by PEP 479; use return to stop.
Lazy wins memory only when nothing materializes. One list() call in the middle removes the gain; measure peak with tracemalloc on the same workload.

Reproduce closed-book

Closed-book, implement a repeatable Range iterable and a single-pass variant; assert the second list() pass is full for the first and empty for the second. Then run the eager versus lazy comparison: list of 1M squares peaks near 40 MB while the genexpr sum peaks under 1 KiB with equal results, measured with tracemalloc on CPython 3.14.4 default build, 12th Gen Intel Core i9-12900K.

Traps

Re-iterating an exhausted generator and expecting fresh data. The second pass is empty with no error.
An accidental StopIteration inside a generator body. It surfaces as RuntimeError, not a quiet stop.
itertools.groupby across unsorted input. It groups consecutive items only; sort first or accept split groups.

Practice

AlgoMaster Python: Exercise: Custom Iterators

Exercises 1 to 5, especially Exercise 2: Reusable Product Catalog Iterator and Exercise 5: Independent Iterators That Skip Closed Tickets

Implement __iter__ and __next__ in your own file before checking.

AlgoMaster Python: Exercise: Generators

Exercises 1 to 5, especially Exercise 3: Consume a Generator Once and Exercise 5: Flatten Nested Lists Lazily

Add exhaustion and second-pass assertions the exercise does not ask for.

Self-test

How do you test whether an object is an iterable or an iterator?
A generator yields 1 then raises StopIteration('x'). What does the caller see?
How does a caller read a value returned by a generator, and how does it send one in?
The lazy pipeline uses the same peak memory as the eager one. What is the first suspect?

Completion criterion

Stop when

Your iterator passes exhaustion, repeat and invalid-input tests, and your pipeline matches the memory numbers with equal outputs.

Mark passed, then open 3.5 Only mark this when you can do it without notes.

Leads to

3.6 Context managers, exceptions and deterministic cleanup Pair generator cleanup with deterministic closing.
5.2 Property, metamorphic and differential testing with shrinking State iterator invariants as properties: exhaustion, repeatability, input rejection.

Comes after

3.3 Attribute lookup, object protocols and dataclassesThe iteration protocol in depth: __iter__, __next__ and exhaustion.
3.9 Reference counts, cycle collection and weak referencesLazy pipelines and weak caches both bound memory; compare the two tools.

Resources for this unit

Stas' Python Cookbook - selected path
Stas Bekmanprimarycourse
Run Part I on dictionaries, iteration, functions, closures, decorators, classes and context managers; Part II on modules, files, paths, environment and arguments; and Part IV on versions, dependencies, packaging and code quality. Finish by building one src-layout package from a blank directory.
Python language reference
Python Software Foundationrequiredreference
Check Data model, Execution model, Import system and Exceptions whenever a claim depends on Python semantics. These pages are the authority when the course and a runtime observation disagree.
AlgoMaster Python - structured revision
AlgoMastersupportpractice
Use Functions, Modules and Packages, OOP, Advanced OOP, Exception Handling, Iterators and Generators, Type Hints and Annotations only after the cookbook pass. Open it for recall practice or a diagnosed weak mechanism, not as a second complete course.

Also in this section

Packaging Python Projects
Python Packaging Authorityrequiredimplementation
Follow the packaging tutorial once from a blank directory, using a src layout, pyproject.toml, an isolated build and a clean-environment installation test.
dataclasses - data classes
Python Software Foundationreferencereference
Read the @dataclass parameters (eq, order, frozen, slots), field() with default_factory, frozen instances, post-init processing, replace, asdict and astuple. Use for the validated frozen dataclass in attributes-protocols-dataclasses.
typing - type hints and protocols
Python Software Foundationreferencereference
Read Protocols for structural typing and runtime_checkable limits, Generics for TypeVar generics, and assert_type for narrowing checks. Use for the Protocol and narrowing parts of type-hints-protocols; PEP 695 syntax itself is in the language reference.
gc - cycle garbage collector
Python Software Foundationreferencereference
Read the cycle collector interface: collect, get_count, garbage and disable/enable. Use for the cycle-collection and measurement parts of refcounting-gc-weakrefs.
weakref - weak references
Python Software Foundationreferencereference
Read weak-reference objects (ref callbacks, proxy objects) and the WeakValueDictionary example. Use for the weak-reference part of refcounting-gc-weakrefs.