Unit 3.4 · about 5 h
252 / 324Sockets, partial reads and writes, and message framing
TCP hands you a byte stream; messages are your construction. This unit builds the framing layer the fault-tested service slice reuses.
- implement
You can now
- Implement a length-prefixed framed protocol that survives partial reads, partial writes and coalesced arrivals.
- Derive framing state for partial, coalesced, oversized, truncated and malformed input.
- Enforce size limits and version checks, and close cleanly on protocol errors.
Covers
- Socket lifecycle, address resolution and blocking behavior
- Partial reads, partial writes and coalesced messages
- Length-prefixed framing, limits and malformed input
Study, in this order
5 steps
- Stream sockets client and serverBeej's Guide to Network Programming - sockets to multiplexing
Section 5.7 (
sendandrecv); Sections 6.1 and 6.2 (simple stream server and client); build the pair first - Partial sends and shutdownBeej's Guide to Network Programming - sockets to multiplexing
Section 7.4 (Handling Partial sends); Section 5.9 (
closeandshutdown)Depth · Use
shutdownfor the half-close direction vocabulary only; full drain behavior belongs to the connections unit. - Stream parsing and the word serverBeej's Guide to Network Concepts - TCP and UDP path
Section 11.2 (Processing a Stream into Packets); Section 11.3.1 (multiple sentences at once); Chapter 13 (The Word Server:
get_next_word_packetandextract_word) - Framed TCP sockets in PythonAlgoMaster Networks - pcap and concurrent server labs
Why One
recv()orread()Is Not Enough; Python: Reading and Writing Framed Messages; Closing and End-of-Stream; Timeouts Bound Waiting - A production-shaped echo protocolAlgoMaster Networks - pcap and concurrent server labs
The Persistent Echo Protocol (4-byte big-endian length prefix, 64 KiB limit); Concurrency Boundaries; Admission Before Task Creation
Depth · Adopt the length prefix, size limit and 30-second frame deadline. Leave the Java port and the load client for the latency unit.
Focus on
- Every
recvcan return one byte, one frame, several frames, or anything between. The parser keeps explicit state: bytes still needed for the header, then bytes still needed for the payload. sendallis a loop, not a guarantee. A rawsendcan write fewer bytes; the remainder must be retried, not dropped.- Limits come before parsing: check the declared length against the maximum before allocating or reading the payload.
- A version or length violation is a protocol error. Reply if the protocol allows, then close. Never resynchronize by scanning payload bytes for a magic value.
Reproduce closed-book
recv(5): exactly 6 calls return all bytes. Repeat with one recv(4096): a single call returns all 28 bytes, both frames coalesced. Then feed your framer the same bytes split at every offset and check it yields the same two messages each time.Traps
- Assuming one
sendequals onerecv. Coalescing and splitting are normal; the 28-byte two-frame burst routinely arrives in onerecvon loopback. - Growing the buffer from the declared length without a cap. A corrupt 4 GB prefix becomes a memory-exhaustion attack; validate first.
- Treating a clean
FINmid-frame as a complete message. A peer that began a frame and closed sent a truncated message: a protocol error, not a short message.
Practice
Beej's Network Concepts: Project The Word Server
Chapter 13: implement get_next_word_packet and extract_word against the word-packet format
A stream-to-packet parser with the same state shape as your length-prefixed framer.
Self-test
-
recv(5)returned 3 bytes of a 4-byte header, then the nextrecvreturns 0 bytes. What happened and what does the framer do?The peer closed mid-frame. The framer reports a truncation protocol error; the 3 bytes are not a message. -
The length prefix says 100 bytes but only 60 arrived and there was no FIN. What does the framer do?
It waits with the receive deadline running. A partial payload is state (40 bytes still needed), not an error, until the deadline or close. -
Why check the length against 64 KiB before reading the payload?
To bound allocation and read work from untrusted bytes. Reading first lets a bogus prefix force a huge allocation. -
Two frames arrive in one
recv. Does the application need two reads?No. The framer consumes both from the buffered bytes and delivers two messages; read count and message count are unrelated.
Completion criterion
Stop when
Your framer passes partial, coalesced, oversized, truncated and garbage-prefix cases, and a peer that closes mid-frame yields a protocol error, never a short message.
Mark passed, then open 3.5 Only mark this when you can do it without notes.Leads to
- 3.5 RPC schemas, versions, errors and trace context Put schemas, versions and trace context inside the frames.
- 3.8 Deterministic network fault injection Attack the framer with scripted delay, duplicate and truncation faults.
Comes after
- 1.4 File descriptors, pipes, signals and process lifecycleExtend descriptor ownership and partial-I/O handling to byte-stream sockets.
- 2.9 Event loops, work stealing and structured concurrencyFraming over partial reads is the event loop's day job.
- 3.2 TCP and UDP transport behaviorBuild message framing on top of the byte stream.
Resources for this unit
- Beej's Guide to Network Programming - sockets to multiplexing Read What is a socket?, IP addresses and data structures, Client-server background, System calls or bust, Client-server examples, Slightly advanced techniques and sendall. Build the simple TCP pair first, then replace delimiter assumptions with a bounded length-prefixed protocol.
- Beej's Guide to Network Concepts - TCP and UDP path Read the OS and sockets, protocol layering, IP, UDP and TCP chapters alongside packet captures from your own client and server. Use it to explain transport behavior rather than memorize layer names.
- AlgoMaster Networks - pcap and concurrent server labs Use Reading a Packet Capture: A Real Bug and Building a Concurrent TCP Echo Server after the Beej implementation. Reproduce one packet-level observation and one admission, deadline or shutdown behavior in your independently written service.
Also in this section
- MIT 6.5840 - RPC, threads and fault vocabulary Read the introductory RPC and threads material plus the lab guidance. Use it to enumerate timeout ambiguity and duplicate effects. Stop before Raft and the full lab sequence.
- Replicated Log and Stream Processor - protocol slice Use only the protocol requirements, deterministic network fault matrix and delivery-contract questions. Build the single-node framed service now; keep replication and consensus for the later project.
- Designing Data-Intensive Applications, 2nd edition - Chapter 5 Read Formats for Encoding Data (printed pp. 163 to 177) for schema merits and compatibility, and Dataflow Through Services: REST and RPC (printed pp. 180 to 186) for service dataflow, version coexistence and the idempotence pointer.
- gRPC Deadlines guide Read Overview, Deadlines on the Client and Deadline Propagation for absolute deadlines, remaining-time forwarding and clock-skew shielding.
- gRPC Status Codes Read the full status list, including
DEADLINE_EXCEEDED,UNAVAILABLEand the codes never generated by the library, for the error taxonomy. - W3C Trace Context Read Section 3.2 for the
traceparentformat and Sections 3.4 to 3.5 for the legaltraceparentandtracestatemutations when forwarding.