Deterministic network fault injection

1. Replicated Log and Stream Processor: protocol slice

   Replicated Log and Stream Processor - protocol slice

   The deterministic network fault list (delay, duplicate, disconnect, restart and companions); the protocol requirements for the single-node framed service; the delivery-contract questions

   Depth · Use only the fault list and the single-node protocol slice. Replication and consensus stay in the later project.

   Done
2. Failure modes and unknown outcomes

   AlgoMaster Networks - pcap and concurrent server labs

   A Network Call Has Several Failure Points; Timeouts Turn Unbounded Waiting into a Decision; Known Failure, Known Success, and Unknown Outcome

   Done
3. Observable overload and failure behavior

   AlgoMaster Networks - pcap and concurrent server labs

   Overload Behavior Is Observable; Failure Behavior; Common Failure Patterns

   Depth · Read for the failure-injection checklist; you already implemented the server contract in the sockets unit.

   Done

Focus on

• Deterministic means seeded and scripted: same fault schedule, same interleaving, same verdict. Random chaos finds bugs; only replayable schedules prove fixes.
• Inject above TCP when testing the protocol (duplicate frames, truncated writes, slow reads) and at the connection level when testing the path (delay, drop, restart).
• Each fault has an expected client observation: delay past deadline gives timeout with unknown outcome; duplicate gives one effect under keys; drop gives disconnect with unknown outcome.
• Restart is two faults at once: in-flight requests resolve as unknown, and post-restart state must reconcile pre-restart keys by policy.

Reproduce closed-book

Traps

• Counting a passed clean-path test as fault coverage. Timeouts, duplicates and restarts exercise different code than the happy path.
• Driving fault timing from wall-clock sleeps with tight deadlines. Slow machines flake; script explicit fault steps with generous margins.
• Logging faults only at the injector. Verdicts need client observation, server state and the schedule in one record, or failures cannot be replayed.

Practice

The section mini-project covers this: the full fault matrix (partial, coalesced, oversized, malformed, timeout, duplicate, disconnect, restart) against the versioned framed service, with replayable schedules and verdict records.

Self-test

1. The delay fault fires but the client still reports success. Is the test wrong?

   Show answer Got it Missed

2. Why must the duplicate test read twice?

   Show answer Got it Missed

3. The server restarts between request receipt and response. What may the client conclude?

   Show answer Got it Missed

Leads to

• 5.7 Distributed histories and deterministic failure simulation Lift the fault schedule to distributed histories and model checking.
• 6.5 Queue bounds, first saturation and open-loop load Separate fault behavior from overload behavior under open-loop load.

Comes after

• 3.4 Sockets, partial reads and writes, and message framingAttack the framer with scripted delay, duplicate and truncation faults.
• 3.6 Deadlines, retries and idempotency across RPCProve the deadline and duplicate handling under scripted faults.
• 3.7 Connection shutdown, pools and latency decompositionTest pool exhaustion and shutdown races under scripted faults.

Resources for this unit

• AlgoMaster Networks - pcap and concurrent server labs
  AlgoMasterrequiredpractice
  Use Reading a Packet Capture: A Real Bug and Building a Concurrent TCP Echo Server after the Beej implementation. Reproduce one packet-level observation and one admission, deadline or shutdown behavior in your independently written service.
• Replicated Log and Stream Processor - protocol slice
  Roadmap-Finalsupportproject
  Use only the protocol requirements, deterministic network fault matrix and delivery-contract questions. Build the single-node framed service now; keep replication and consensus for the later project.