Unit 3.5 · about 3 h
253 / 324RPC schemas, versions, errors and trace context
Schemas and status codes are the contract clients program against; trace context is how you debug it. It builds on the Programming OpenTelemetry unit (programming/concurrency-profiling-operations/logs-metrics-traces), which covered spans and bounded labels; here the wire format is the mechanism.
- explain
- implement
You can now
- Define an RPC schema with field identity, presence rules and safe evolution (add, never reuse).
- Separate transport status from RPC status when reading errors.
- Propagate W3C trace context across hops and keep correlation IDs intact.
Covers
- Serialization, schemas, protocol versions and error taxonomy
- Request IDs, correlation and end-to-end trace context
Study, in this order
5 steps
- Schemas and service dataflowDesigning Data-Intensive Applications, 2nd edition - Chapter 5
Chapter 5, Formats for Encoding Data (printed pp. 163 to 177) for schema merits and compatibility; Dataflow Through Services: REST and RPC (printed pp. 180 to 186) for version coexistence
Depth · Read the two sections only, not the whole chapter. Follow the idempotence pointer to Chapter 12 later, in the streams section.
- MIT 6.5840 Lecture 2: RPC structureMIT 6.5840 - RPC, threads and fault vocabulary
The Remote Procedure Call (RPC) section to the end: the message diagram, the
kv.goexample and the RPC failure questionsDepth · Read the RPC half only; the threading and crawler half is assumed from the concurrency stage.
- Schemas, status and evolution in gRPCAlgoMaster Networks - pcap and concurrent server labs
Protocol Buffers as a Schema; Presence and Default Values; Evolving a Protobuf Contract Safely; The Five-Byte gRPC Message Prefix; HTTP Status Is Not RPC Status; gRPC Status Codes
- The status listgRPC Status Codes
Full list of Status codes, including
DEADLINE_EXCEEDED,UNAVAILABLE, and the codes never generated by the library - Traceparent and tracestateW3C Trace Context
Section 3.2 (traceparent format and examples); Sections 3.4 to 3.5 (legal mutations when forwarding)
Focus on
- Wire identity is the field number, not the name. In the binary encoding, renaming tag 2 is compatible; changing 2 to 4 deletes a field and creates another. JSON mappings use names, so a rename breaks them.
- proto3 scalars conflate absent with default. Use
optional,oneoformessagefields when the distinction changes behavior. - HTTP 200 with a nonzero
grpc-statusis an RPC failure. Monitors that check only HTTP status report success while every call fails. - Forwarding a trace means sending
traceparentwith a newparent-idand preservingtracestateorder; a hop that regenerates thetrace-idsplits the trace.
Reproduce closed-book
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 into version 00, 16-byte trace-id, 8-byte parent-id and sampled flag, and state the legal forwarding mutations (new parent-id, optional sampled change, restart, downgrade). Then add a request-id field to your framed protocol and show it echoed unchanged in the response.Traps
- Reusing a deleted field number for a new type. Old binaries parse the new bytes as the old type; use a fresh number.
- Hiding errors inside OK responses. Clients, retry policies and metrics can only reason about errors carried in status.
- Putting secrets or high-cardinality values into trace context.
traceparentandtracestatepropagate to every downstream service, and the spec forbids personally identifying data there.
Practice
AlgoMaster: Audit Correlation ID Propagation
brokenHops and firstBrokenHop over receive-forward chains; regenerated IDs break the boundary rule
A later hop that preserves a regenerated ID is not itself broken; only the regenerating hop is.
AlgoMaster: Calculate Trace Span Exclusive Time
Exclusive time as duration minus the union of direct-child intervals; ties break by smaller ID
Subtract the union \([10,80)\), not the sum of overlapping children.
Self-test
-
Field 2 changes from
stringtoint32with the same number. Wire-compatible?No. Astringtravelslength-delimitedand anint32as avarint, so old readers cannot decode it as theirstring; most runtimes keep it as an unknown field and the value is lost to them. Add a new field number instead. -
A proxy returns HTTP 200 for an RPC that failed with
grpc-status14. What should the dashboard show?An RPC failure withUNAVAILABLE. Transport success and RPC outcome are separate channels. -
A service forwards
traceparentbyte-identical and appends itstracestateentry. Correct?No. It must setparent-idto its own current operation ID; byte-identical forwarding claims the downstream work is the same span. Newtracestateentries go first, not last. -
Why must the server application stop its own work when gRPC signals cancellation for deadline?
The runtime only signals; spawned database calls and child RPCs keep running and consuming resources until the application stops them.
Completion criterion
Stop when
You evolve a schema without breaking an old reader, classify an RPC failure by its status code, and trace one request across two hops without dropping its ID.
Mark passed, then open 3.6 Only mark this when you can do it without notes.Leads to
- 3.6 Deadlines, retries and idempotency across RPC Add deadlines and safe retries on top of the status taxonomy.
- 5.5 Durable logs, delivery guarantees and idempotent consumers Reuse idempotent consumers and trace correlation for log delivery.
Comes after
- 3.3 TLS and HTTP versionsRun RPC contracts over the TLS and HTTP substrate.
- 3.4 Sockets, partial reads and writes, and message framingPut schemas, versions and trace context inside the frames.
Resources for this unit
- AlgoMaster Networks - pcap and concurrent server labs Use Reading a Packet Capture: A Real Bug and Building a Concurrent TCP Echo Server after the Beej implementation. Reproduce one packet-level observation and one admission, deadline or shutdown behavior in your independently written service.
- MIT 6.5840 - RPC, threads and fault vocabulary Read the introductory RPC and threads material plus the lab guidance. Use it to enumerate timeout ambiguity and duplicate effects. Stop before Raft and the full lab sequence.
- Designing Data-Intensive Applications, 2nd edition - Chapter 5 Read Formats for Encoding Data (printed pp. 163 to 177) for schema merits and compatibility, and Dataflow Through Services: REST and RPC (printed pp. 180 to 186) for service dataflow, version coexistence and the idempotence pointer.
- gRPC Status Codes Read the full status list, including
DEADLINE_EXCEEDED,UNAVAILABLEand the codes never generated by the library, for the error taxonomy. - W3C Trace Context Read Section 3.2 for the
traceparentformat and Sections 3.4 to 3.5 for the legaltraceparentandtracestatemutations when forwarding.
Also in this section
- Beej's Guide to Network Programming - sockets to multiplexing Read What is a socket?, IP addresses and data structures, Client-server background, System calls or bust, Client-server examples, Slightly advanced techniques and sendall. Build the simple TCP pair first, then replace delimiter assumptions with a bounded length-prefixed protocol.
- Beej's Guide to Network Concepts - TCP and UDP path Read the OS and sockets, protocol layering, IP, UDP and TCP chapters alongside packet captures from your own client and server. Use it to explain transport behavior rather than memorize layer names.
- Replicated Log and Stream Processor - protocol slice Use only the protocol requirements, deterministic network fault matrix and delivery-contract questions. Build the single-node framed service now; keep replication and consensus for the later project.
- gRPC Deadlines guide Read Overview, Deadlines on the Client and Deadline Propagation for absolute deadlines, remaining-time forwarding and clock-skew shielding.