QUESTION
Do one send and one recv preserve the same message boundary?
TCP is an ordered byte stream. Two sends from the sender can merge into one recv, or one send can be split across several recvs. The application must define framing such as a fixed length, delimiter, or length prefix.
When outbound data accumulates faster than the send buffer accepts it, an unbounded queue exhausts memory. At a per-connection high-water mark, stop producers or reject requests, then continue sending the remaining bytes on EPOLLOUT.
STRUCTURE
Structure diagram
TCP preserves only byte order. The receiver parser reconstructs application frames by accumulating a length header and payload bytes.
CALL PATH
Call path
Message state lives in application buffers; TCP kernel queues contain only bytes. Retain in the connection object how much of the header and payload the parser has acquired.
This diagram is not for memorizing function names. Follow which return value, file descriptor, memory mapping, or wait queue is passed to the next stage.
SOURCE COORDINATES
Linux 6.18.37 LTS source locations
Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.
| File | Function / structure | What to inspect |
|---|---|---|
| net/ipv4/tcp.c | tcp_sendmsg(), tcp_recvmsg() | Stream send/receive queues and partial progress |
| net/ipv4/tcp_output.c | tcp_write_xmit() | Send segments according to congestion/window conditions |
| net/ipv4/tcp_input.c | tcp_data_queue() | Attach received bytes to the socket receive queue |
COMPLETE PROGRAM
Complete runnable example
The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.
cc -std=c17 -Wall -Wextra -O2 frame_codec.c -o frame_codec01#define _GNU_SOURCE
02#include <arpa/inet.h>
03#include <stdint.h>
04#include <stdio.h>
05#include <string.h>
06#include <sys/socket.h>
07#include <unistd.h>
08
09static int send_frame(int fd, const void *data, uint32_t length)
10{
11 uint32_t header = htonl(length);
12 unsigned char frame[4 + 256];
13 if (length > 256)
14 return -1;
15 memcpy(frame, &header, sizeof(header));
16 memcpy(frame + sizeof(header), data, length);
17 size_t total = 0, size = sizeof(header) + length;
18 while (total < size) {
19 ssize_t n = send(fd, frame + total, size - total, MSG_NOSIGNAL);
20 if (n <= 0)
21 return -1;
22 total += (size_t)n;
23 }
24 return 0;
25}
26
27int main(void)
28{
29 int pair[2];
30 if (socketpair(AF_UNIX, SOCK_STREAM, 0, pair) < 0)
31 return 1;
32 send_frame(pair[0], "alpha", 5);
33 send_frame(pair[0], "beta", 4);
34 unsigned char bytes[64];
35 ssize_t n = recv(pair[1], bytes, sizeof(bytes), 0);
36 printf("one recv returned %zd bytes\n", n);
37 close(pair[0]); close(pair[1]);
38 return n < 0;
39}CODE NOTES
Code notes
uint32_t header = htonlMakes the wire-format length a fixed 32-bit value in network byte order. Do not send a host-dependent type such as size_t directly.
if (length > 256)Applies a protocol maximum so a length field cannot grow frame allocations and parser resources without bound.
while (total < size)Even when send accepts only part of a frame, preserves the offset and continues with the remainder. On a nonblocking fd, retain queue state and return on EAGAIN.
MSG_NOSIGNALPrevents a process-wide SIGPIPE termination on a socket closed by the peer and handles EPIPE as a return value.
recv(pair[1]Although two frames were sent, recv may return one frame, two frames, part of a header, or any other byte count. The parser must accumulate the buffer.
DETAILS
Detailed behavior
The parser is an incremental state machine
First wait for the full 4-byte header and validate the length, then emit a frame when that many payload bytes have accumulated. If bytes for the next frame remain, continue parsing within the same read event.
Define an explicit connection-close policy for a malformed length, integer overflow, or allocation failure.
Propagate backpressure to upstream producers
When the send queue exceeds its high-water mark, waiting for socket EPOLLOUT is not sufficient. Stop file reads, upstream RPC, and message generation too so memory remains bounded.
Reenable producer interest after the queue falls below the low-water mark.
A TCP ACK is not application receipt
Successful send and a TCP ACK mean only that bytes were accepted as far as the peer kernel. If the peer application must parse the frame and place it in durable storage, define an application acknowledgement.
Retries require request IDs and idempotency rules.
OBJECTS
Objects and lifetimes
| Object | Creation and release | Values to inspect |
|---|---|---|
outbound frame | Created during encoding and retained until every byte is sent or the connection aborts | buffer, total, offset |
receive parser | Retains a partial header/payload for the connection lifetime | buffered bytes, expected length |
TCP send queue | Accepted by the send syscall and retained in the kernel until ACK/abort | wmem, unacked, window |
FAILURE PATH
Failure conditions and common misconceptions
| Observed symptom | Likely causes | How to verify |
|---|---|---|
| Messages merge or split | TCP message boundaries were assumed | Inspect incremental parser state |
| Memory keeps growing | The outbound queue is unbounded | Track queued bytes/high-water per connection |
| The process terminates with SIGPIPE | send was called on a closed peer | Use MSG_NOSIGNAL or define signal policy |
LAB
Verify it yourself
- Vary the recv buffer from 1 to 7 bytes and write a parser that always reconstructs both frames exactly.
- Make the socket send buffer small and delay the receiver to observe partial send/EAGAIN and the queue high-water mark.
- Test a malicious peer that sends a frame length above the maximum and verify rejection before allocation.
./frame_codecstrace -e trace=socketpair,sendto,recvfrom,close ./frame_codecPRIMARY REFERENCES