TCP, how reliable byte streams run over unreliable IP

TCP (Transmission Control Protocol) provides three guarantees on top of the unreliable IP layer: reliable, ordered, byte-stream delivery. Vint Cerf and Bob Kahn specified it in 1974; RFC 793 standardised it in 1981, refreshed as RFC 9293 in 2022. TCP underlies HTTP/1, HTTP/2, SSH, SMTP, FTP, and most other application protocols.

IP packets can arrive duplicated, out of order, or not at all. TCP layers a reliable, ordered byte-stream abstraction on top of that, and it has been refined for decades.

A connection isn't open until both sides know two things: their own initial sequence number and the peer's. Two packets synchronize only one direction, so synchronizing both directions and proving the round trip works takes three.

Initial Sequence Numbers are chosen unpredictably rather than starting at zero. If a router delays a packet for two minutes and finally delivers it, a random ISN makes it very unlikely that the new connection accepts the stale packet. With predictable ISNs, a long-lost packet from a closed connection could inject corrupted bytes into a live database query.

The simulator below plays the entire lifecycle: three-way handshake, a single request/response data exchange, and the four-way teardown. Use the phase tabs to jump between phases, the player to scrub, and + Show wire bytes to read the segment header values at each step.

Every byte gets a number. The sequence number in a TCP header is the number of the first byte of the payload. The acknowledgement number is the next byte the receiver expects.

The ACK is cumulative: ack=1018 means "every byte up to and including 1017 has arrived." If the receiver got bytes 1–100 then bytes 201–300, it does not ack 300. It keeps acking 101, repeatedly, to tell the sender there is a hole. When the sender sees three duplicate ACKs in a row, it doesn't wait for its retransmit timer. It triggers fast retransmit and resends the missing segment immediately.

This is why packet loss over TCP shows up as a delay rather than as corrupted data.

A fast sender talking to a slow receiver can overflow the receiver's buffer, and TCP must not lose bytes the receiver's OS has already accepted. The defence is the receive window, a 16-bit field in every TCP header that advertises how much buffer space is free right now.

A mobile phone whose CPU is briefly pinned can advertise a window of zero, and the server stops sending mid-flight however big the file is. As the phone drains its buffer it advertises a larger window, possibly in a small "window update" segment, and transmission resumes. Flow control protects the receiver, while congestion control protects the network, and the two run as independent feedback loops at the same time.

The 16-bit window field maxes out at 65 535 bytes, which is small for high-bandwidth links. The window scaling option, negotiated at SYN time, lets each side multiply the field by 2ⁿ, so windows of up to about a gigabyte are possible.

Above flow control sits congestion control, the sender's own estimate of how fast it can transmit before a router drops packets. That estimate is the congestion window (cwnd). The sender may have at most min(cwnd, rwnd) bytes in flight at any moment.

TCP's strict in-order delivery is also its main architectural limitation. If you multiplex five HTTP/2 streams over a single TCP connection and the very first segment of stream one is dropped, TCP halts everything. The OS withholds the valid, already-arrived bytes for streams two to five, because it must deliver byte one to the application before byte one thousand.

This cannot be fixed inside TCP without breaking its in-order guarantee. HTTP/3 therefore moved off TCP: QUIC reimplements TCP's reliability on top of UDP, but with independent per-stream sequence numbers, so a loss on one stream stalls only that stream. See the HTTP guide for more.

Closing a connection is half-duplex. Either side can declare "I have no more data to send" with a FIN, and the other end keeps sending until it sends its own FIN. That gives four control segments: FIN, ACK, FIN, ACK. The middle two often ride together, with the server's ACK piggybacked on its FIN, but conceptually there are four phases.

After the final ACK, the closing side sits in TIME_WAIT for two maximum-segment-lifetimes (2 × MSL, typically 60–240 s). This is deliberate. A duplicate FIN delayed in some router could otherwise arrive long after the connection has closed and confuse a new connection that reuses the same four-tuple. TIME_WAIT keeps the kernel holding the four-tuple until any stragglers have expired.

Servers under enormous churn can run out of ephemeral ports waiting for TIME_WAITs to clear. Common remedies are tuning tcp_tw_reuse and reusing connections via keep-alive.

tcp_congestion_control

The algorithm. Linux defaults to CUBIC; production high-throughput setups (Google, Cloudflare) often switch to BBR. Set globally: sysctl net.ipv4.tcp_congestion_control=bbr.

tcp_rmem / tcp_wmem

Receive and send buffer sizes (min/default/max). Defaults are tuned for ~10 ms RTT; for transcontinental connections at 150 ms RTT, raising max to ~16 MB allows TCP to keep the pipe full.

tcp_fastopen

Enables TFO (RFC 7413). Saves an RTT on repeat connections. Set to 3 (both client and server). Mostly transparent to applications.

tcp_tw_reuse

Allow reuse of TIME_WAIT sockets for new connections. Important for high-RPS services hitting port exhaustion. Set to 1 cautiously (correctness depends on timestamp option being on).

tcp_keepalive_time / _intvl / _probes

Keepalive cadence. The defaults of 7200/75/9 mean the OS notices a dead peer about 2.2 hours after it goes silent. If your clients are unreliable mobile or IoT devices, tune down to 60/10/3.

net.core.somaxconn

Listen-backlog ceiling. Default 128; serving 100k+ accepts/sec needs 4096 or more.

Kernel versions. Recent Linux kernels add improvements such as TCP zero-copy (SO_ZEROCOPY) and TLS offload in the kernel (kTLS). They make little difference on a single laptop but add up at fleet scale. Cloudflare, Netflix and Meta run custom-tuned kernels.