Unit 3.6 · about 4 h
254 / 324Deadlines, retries and idempotency across RPC
A timeout tells the client nothing about the server. This unit turns that ambiguity into a protocol: bounded retries plus keys that make repeats safe. It assumes the Programming retry-budget and idempotency-ledger units (programming/concurrency-profiling-operations/cancellation-timeouts-retries, programming/interfaces-data-apis/http-apis-idempotency) and moves them from HTTP handlers onto the RPC wire.
- derive
- implement
You can now
- Derive a retry budget and the worst-case attempt amplification across layered retries.
- Propagate an absolute deadline across hops without extending it.
- Implement idempotency keys with a durable state machine and state exactly when an outcome is unknown.
Covers
- Deadlines, timeouts, backoff, jitter and retry budgets
- Idempotency keys, duplicate suppression and ambiguous side effects
Study, in this order
5 steps
- MIT 6.5840 Lecture 2: RPC failure ambiguityMIT 6.5840 - RPC, threads and fault vocabulary
RPC problem: what to do about failures (the client does not know if the server saw the request); best-effort RPC; at-most-once; the two-Put reordering example
- Retries, backoff and budgetsAlgoMaster Networks - pcap and concurrent server labs
Timeouts and Ambiguous Outcomes; Full Jitter; Fleet-Wide Retry Budget; Avoid Retry Amplification Across Layers; A Deadline-Aware Retry Loop
- Idempotency keys and durable executionAlgoMaster Networks - pcap and concurrent server labs
Idempotency Keys Identify One Logical Operation; Define the Key Contract (scope, fingerprint, replay, retention); The Server Needs a Durable State Machine; Atomicity Closes the Crash Window
- Deadlines and propagationgRPC Deadlines guide
Overview (deadline versus timeout); Deadlines on the Client; Deadline Propagation (remaining-time forwarding, clock-skew shielding)
- gRPC deadlines, cancellation and retriesAlgoMaster Networks - pcap and concurrent server labs
Deadlines, Cancellation, and Retries: realistic deadlines, application-owned cancellation, retry only safe-to-repeat operations
Focus on
- Every timeout hides three possible server states: never saw it, executed and failed to reply, executed and reply lost. Retry design starts by listing all three.
- Deadlines are absolute points in time; timeouts are durations. Propagate the remaining time, never a fresh full timeout, or downstream work outlives the caller.
- A retry budget bounds attempts per operation and per fleet. Unbounded per-hop retries multiply across layers into a retry storm.
- Idempotency keys need a durable state machine (received, in-progress, completed) with atomic transitions; memory-only dedup forgets across restarts.
Reproduce closed-book
startTime 0 and client timeout 100 through processing times [10,20,5] with local bounds [80,100,10]: hop deadlines [90,90,45]. Then run the deterministic proxy on loopback (CPython 3.14.4): 200 ms injected delay with a 100 ms client deadline records a client timeout although the server processed the request; the duplicate proxy leaves a second frame waiting on the connection. Reference run on kernel 7.0.0-38-generic, i9-12900K confirmed both rows.Traps
- Retrying on
DEADLINE_EXCEEDEDwithout an idempotency key. The first attempt may have completed; the retry repeats the side effect. - Resetting a full timeout at each hop. Three hops with fresh 2 s timeouts let downstream work run until about 6 s, 4 s after a 2 s caller has given up.
- Treating at-most-once as safety. Never resending avoids duplicates but turns every lost reply into a user-visible failure; choose the semantics per operation.
Practice
AlgoMaster: Propagate a gRPC Deadline
propagate with remaining-time bounds, parent cap, and -1 past the inherited deadline
The [90,90,45] hand derivation in Reproduce is the first case; the exact-deadline case returns -1.
AlgoMaster: Build an Idempotent Consumer
process returns true on first delivery, false on duplicates, without resetting recorded IDs
Extend it with an IN_PROGRESS state and a restart to reach the unit's durable contract.
Self-test
-
The client timeout fires at 100 ms; the server log shows the write committed at 150 ms. What does the client know?
Nothing about the outcome. The timeout only proves the reply did not arrive in time; the effect may have happened, and only a keyed status check can resolve it. -
Two layers each retry up to 3 times on failure. What is the worst-case attempt count for one user request?
Up to 16: the outer layer makes 1 initial attempt plus 3 retries, and each of the 4 can trigger up to 4 inner attempts. Layered retries multiply. -
Why does gRPC convert a deadline to a remaining-time timeout before forwarding?
Server clocks may disagree. Sending a duration shields propagation from clock skew; sending an absolute timestamp does not. -
A duplicate arrives while the first attempt is still
IN_PROGRESS. What does the state machine do?It never starts the effect a second time. It waits for the first attempt within its own deadline, returns a documented in-progress response, or returns an operation the client can poll; the stored result is replayed once it exists.
Completion criterion
Stop when
You derive hop deadlines and a layered retry budget by hand, and your keyed server produces one effect for any number of duplicate deliveries including a restart.
Mark passed, then open 3.7 Only mark this when you can do it without notes.Leads to
- 3.8 Deterministic network fault injection Prove the deadline and duplicate handling under scripted faults.
- 6.6 Retry budgets, circuit breaking and load shedding Generalize retry budgets to fleet-wide overload control.
Comes after
- 2.7 Cancellation, deadlines and shutdown state machinesRPC deadlines are this unit's timed wait across a network.
- 3.5 RPC schemas, versions, errors and trace contextAdd deadlines and safe retries on top of the status taxonomy.
Resources for this unit
- AlgoMaster Networks - pcap and concurrent server labs Use Reading a Packet Capture: A Real Bug and Building a Concurrent TCP Echo Server after the Beej implementation. Reproduce one packet-level observation and one admission, deadline or shutdown behavior in your independently written service.
- MIT 6.5840 - RPC, threads and fault vocabulary Read the introductory RPC and threads material plus the lab guidance. Use it to enumerate timeout ambiguity and duplicate effects. Stop before Raft and the full lab sequence.
- gRPC Deadlines guide Read Overview, Deadlines on the Client and Deadline Propagation for absolute deadlines, remaining-time forwarding and clock-skew shielding.
Also in this section
- Beej's Guide to Network Programming - sockets to multiplexing Read What is a socket?, IP addresses and data structures, Client-server background, System calls or bust, Client-server examples, Slightly advanced techniques and sendall. Build the simple TCP pair first, then replace delimiter assumptions with a bounded length-prefixed protocol.
- Beej's Guide to Network Concepts - TCP and UDP path Read the OS and sockets, protocol layering, IP, UDP and TCP chapters alongside packet captures from your own client and server. Use it to explain transport behavior rather than memorize layer names.
- Replicated Log and Stream Processor - protocol slice Use only the protocol requirements, deterministic network fault matrix and delivery-contract questions. Build the single-node framed service now; keep replication and consensus for the later project.
- Designing Data-Intensive Applications, 2nd edition - Chapter 5 Read Formats for Encoding Data (printed pp. 163 to 177) for schema merits and compatibility, and Dataflow Through Services: REST and RPC (printed pp. 180 to 186) for service dataflow, version coexistence and the idempotence pointer.
- gRPC Status Codes Read the full status list, including
DEADLINE_EXCEEDED,UNAVAILABLEand the codes never generated by the library, for the error taxonomy. - W3C Trace Context Read Section 3.2 for the
traceparentformat and Sections 3.4 to 3.5 for the legaltraceparentandtracestatemutations when forwarding.