Unit 5.5 · about 4 h
270 / 324Durable logs, delivery guarantees and idempotent consumers
Assume Programming's http-apis-idempotency and transactions-isolation units. Here the mechanism is a retained log and a consumer checkpoint: deduplicating a request is not enough if its effect and progress can disagree after a crash.
- explain
- implement
You can now
- Trace append, acknowledgement, effect and offset boundaries under consumer failure.
- Implement a replaying consumer with an atomic database effect-and-offset boundary.
- State an effectively-once effect contract without extending it to external systems.
Covers
- Durable logs, offsets, acknowledgements and replay
- At-most-once, at-least-once and scoped effectively-once effects
- Idempotent consumers and transactional effect boundaries
Study, in this order
2 steps
- DDIA, 2nd edition: retained logs and consumer recoveryDesigning Data-Intensive Applications, 2nd edition - distributed data path
Chapter 12, Log-Based Message Brokers through Consumer offsets, pp. 495 to 498; Fault Tolerance, including Atomic commit revisited and Idempotence, pp. 526 to 528
- Kafka 4.3 design: delivery and transaction scopeApache Kafka documentation - design
Persistence; The Consumer > Consumer Position; Message Delivery Semantics; Using Transactions
Depth · Use Persistence for the append-to-page-cache mechanism, not its hardware-specific performance numbers. Use classic consumer groups for this model, not share groups. Do not install Kafka for this unit.
Focus on
- Define the checkpoint as the next offset to process, scoped by consumer group and partition. Reading does not delete a retained log record; replay is limited by retention.
- Offset-before-effect can skip work after a crash; effect-before-offset can repeat it. A transport acknowledgement, append acknowledgement and consumer progress commit are different boundaries.
- For a single in-order consumer, store effect and next offset atomically in the same database. Concurrent owners require fencing or transactional serialization as well.
- Kafka transactions can couple Kafka output records with input offsets. External email, payments or another database do not join that transaction merely because the producer is idempotent.
Reproduce closed-book
Write a small program with Python 3.14's
sqlite3 module; Kafka is not needed. Model one retained input partition as the immutable record \((\text{offset } 0, \delta +5)\), with database tables effect(total=0) and progress(pos=0), where \(\text{pos}\) is the next offset. Use a fresh on-disk database per case, journal_mode=DELETE and synchronous=FULL. In a child process, inject os._exit(9) after committing \(\text{pos}=1\) but before the effect; reopening gives \((\text{total}, \text{pos})=(0,1)\), so restart skips the effect. Inject exit after committing the increment but before \(\text{pos}\): reopening gives \((5,0)\), and normal replay gives \((10,1)\). Finally put both updates in BEGIN IMMEDIATE/COMMIT: exit before COMMIT reopens at \((0,0)\), exit after COMMIT at \((5,1)\), and restart from the stored \(\text{pos}\) ends at \((5,1)\) in both cases. Close and reopen connections, replay only \(\text{offset} \ge \text{pos}\), and assert all four outcomes. These are process-exit tests with CPython 3.14.4 and SQLite 3.46.1, not a power-loss durability claim. Extend the model to two independent groups with separate progress; neither group advances the other's checkpoint.Traps
- Recording deduplication metadata separately from the protected effect. A crash between them can lose or duplicate work.
- Committing past a failed record when later records finish first. A single offset represents a contiguous processed prefix, not an arbitrary set of completed tasks.
- Equating append into the page cache with stable storage, or a process-exit test with proof against power failure.
Practice
The section mini-project covers this: a checksummed retained log, independent group/partition checkpoints, replay after each crash boundary, and atomic effect-plus-progress or deduplication metadata. Keep external effects outside the claimed transaction scope.
Self-test
-
Why does effect-before-offset produce a duplicate after a crash?
The durable effect survives, but the old checkpoint instructs the replacement consumer to repeat the same record. -
Is setting a value always an idempotent substitute for incrementing?
Not generally. Replaying an older set can overwrite newer state. Ordering, version checks or transactional deduplication must protect the intended effect. -
What identifies one consumed record in a partitioned log?
The log or topic identity, partition and offset. Offset alone is not globally unique, and effect deduplication must also have the right consumer/application scope. -
Does atomic database effect-and-offset commit make an email exactly once?
No. The email service is outside that transaction. It needs a cooperating idempotency boundary or another explicit delivery contract.
Completion criterion
Stop when
Your child-process tests reproduce the skip and duplicate, both transactional cases recover to total 5, and your contract names the effect store, offset scope and excluded external effects.
Mark passed, then open 5.6 Only mark this when you can do it without notes.Leads to
- 5.6 Consumer lag, watermarks and state rebuild Add retention, poison records and state reconstruction to replay.
Comes after
- 3.5 RPC schemas, versions, errors and trace contextReuse idempotent consumers and trace correlation for log delivery.
- 4.8 Write-ahead logging, recovery, checksums and durabilitySeparate durable local acknowledgment from delivery and replay effects.
- 5.4 Partitioning, hot keys, rebalancing and stale writersApply ownership epochs to consumers that resume a partition after failover.
Resources for this unit
- Designing Data-Intensive Applications, 2nd edition - distributed data path Read the selected chapters on replication, partitioning, distributed-system trouble, consistency and consensus, batch processing and stream processing. For each guarantee, write its assumptions, a confirming history and one counterexample outside the boundary.
- Apache Kafka documentation - design Read Persistence, The Consumer (Consumer Position), Message Delivery Semantics, Using Transactions, Replication and Log Compaction for a concrete log: offsets, consumer positions, the at-most-once, at-least-once and exactly-once boundaries, in-sync replicas and compaction guarantees. Map each guarantee onto your own log before trusting it.
Also in this section
- Gossip Glomers - Echo, Unique ID and Broadcast Complete Challenge 1 Echo, Challenge 2 Unique ID Generation and Challenge 3a Single-Node Broadcast. Add 3b Multi-Node Broadcast only after writing the delivery, duplicate and partition assumptions. Preserve protocol histories rather than relying on a success banner.
- MIT 6.5840 - MapReduce, RPC and consistency bridge Use the introductory MapReduce, RPC, GFS and consistency lectures after the local RPC and store gates. The MapReduce lab is optional. Do not begin Raft or sharded KV as foundation work.
- Building Distributed Systems from Scratch - one challenge path Choose one messaging, durable-log or partitioning challenge that matches a failed checkpoint. State the model, failures and evidence independently because interactive grading behavior has not been verified.
- AlgoMaster System Design - distributed fundamentals slice Use Database Replication, Consistency Models, Logical Clocks, Vector Clocks, Network Partitions, Split Brain Problem, Leader Election, Database Partitioning, Consistent Hashing, Delivery Semantics, Dead Letter Queues, Idempotency, Two-Phase Commit (2PC), Outbox Pattern and Streaming Engines as bounded review after DDIA. Attempt only the exercise that matches your implemented history, then challenge its assumptions with a node-loss, delay or duplicate case.