The internet: how it was built, and what happens when you load a page

The internet is a network of networks: tens of thousands of independently run networks that agree to carry each other’s packets using the Internet Protocol. It grew out of the ARPANET, a research network funded by the US Department of Defense, and took its present shape in stages, each adding one layer of the stack that every page load still uses.

That is why the history and the page load belong on one page. The ARPANET and TCP/IP gave us packets and connections; DNS gave us names; BGP let networks find routes to each other; the web added URLs and HTTP; TLS added encryption; and QUIC and HTTP/3 rebuilt the transport to save round trips. Loading a page today runs through all of them in order, and the second half of this guide counts what each one costs.

Drag the year to see which parts of the stack existed at that point, and filter the timeline by layer. Each entry links to its source; for protocols that is the RFC that first defined them.

The ARPANET was funded by the US Defense Department’s Advanced Research Projects Agency to let research computers share resources. Its key idea was packet switching: instead of reserving a circuit for each conversation, as telephone networks did, messages were cut into packets that travelled independently through small dedicated computers called IMPs (interface message processors), built by BBN. The first IMP was installed at UCLA in September 1969, and in October a computer there sent the first host-to-host message to the Stanford Research Institute. By the end of the year the network joined four sites.

Email arrived in 1972, and quickly became the network’s most popular use. Meanwhile other packet networks appeared, including packet radio and satellite networks, and on single sites Ethernet, which Bob Metcalfe and David Boggs began developing at Xerox PARC in 1973. The ARPANET’s own protocol, NCP, assumed one network that never lost packets. Connecting different networks needed something else.

In May 1974 Vint Cerf and Bob Kahn published “A Protocol for Packet Network Intercommunication”. Its design choices still define the internet: networks stay independent and are joined by gateways (now called routers); the network only tries its best to deliver packets; and the hosts at each end take care of reliability, retransmitting what is lost and putting data back in order. The protocol was later split in two. IP carries packets between any two addresses, and TCP builds a reliable byte stream on top. UDP (1980) offers IP’s best effort with port numbers, for applications that do not want TCP’s guarantees.

IPv4 and TCP were published as RFC 791 and RFC 793 in 1981, and on 1 January 1983 the whole ARPANET switched over in one flag day. In 1986 the US National Science Foundation built NSFNET to join university supercomputer centres, and it became the backbone that most networks connected to.

The internet’s growth soon exposed a problem: when many TCP connections sent as fast as they could, routers dropped packets, senders retransmitted, and the network spent its capacity on retransmissions. After the first of these congestion collapses in 1986, Van Jacobson added slow start and congestion avoidance to TCP (1988). Every TCP connection still starts slowly and grows its sending rate as acknowledgements come back, which is why the size of a response matters on a page load (see the 14 KB first flight).

Early hosts found each other through a single file, HOSTS.TXT, that listed every machine’s name and address and was copied to every host. That could not scale. In November 1983 Paul Mockapetris published the Domain Name System: a hierarchy of name servers in which the root knows who runs each top-level domain such as .com, those servers know who runs each domain, and the domain’s own servers hold its records. Each answer can be cached for as long as its TTL allows. The DNS guide follows one lookup through it.

Routing between networks needed the same kind of decentralisation. With the Border Gateway Protocol (1989), each network, called an autonomous system, tells its neighbours which address ranges it can reach and through which path, and each network decides by its own policy which routes to accept and pass on. No one controls the whole map. CIDR (1993) replaced fixed class A, B and C address blocks with prefixes of any length, which slowed the use of addresses and let routes be aggregated. BGP’s reliance on trust between networks is also its weakness, as the outages below show; the BGP guide covers it in depth.

In March 1989 Tim Berners-Lee, at CERN, proposed a system of linked documents for sharing information between physicists. It had three parts that are still the web’s foundation: URLs to name documents, HTML to write them with links, and HTTP to fetch them. The first website, info.cern.ch, ran on his NeXT computer. The first version of HTTP was tiny: the browser sent GET and a path, and the server sent back the HTML and closed the connection.

On 30 April 1993 CERN put the web software in the public domain, and the web spread through the 1990s. In April 1995 NSFNET was shut down and traffic moved to commercial networks, which exchange traffic with each other at internet exchange points and through paid transit, the arrangement that still holds.

HTTP grew to match. HTTP/1.0 (1996) added headers, status codes and content types. HTTP/1.1 (1997) kept connections open for several requests and added the Host header, so that many websites could share one IP address. It carried the web for nearly two decades; the HTTP guide covers its messages in detail.

IPv4 addresses are 32 bits long, about 4.3 billion in total, and many ranges are reserved. IPv6 (1998) uses 128-bit addresses, but it is a separate protocol: an IPv4-only host cannot talk to an IPv6-only one, so adoption has been slow and both run side by side. On 3 February 2011 IANA handed out its last free IPv4 blocks to the regional registries.

The internet kept growing thanks to network address translation (NAT): a home router or a carrier shares one public address among many devices by rewriting addresses and ports on the way out. NAT broke the original model in which any host could reach any other directly, and it is part of the reason so much now runs over a few well-known ports and protocols. The NAT guide explains how it works.

Netscape’s SSL encrypted web traffic from the mid-1990s and was standardised as TLS in 1999. For years HTTPS was used mainly for logins and payments. Two things changed that: free, automated certificates from Let’s Encrypt (public since December 2015) and browsers marking plain HTTP pages as not secure. TLS 1.3 (2018) cut the handshake from two round trips to one and removed old, weak options. The HTTPS guide walks through the handshake.

HTTP/2 (2015) sent many requests over one connection at once, in binary frames, with compressed headers. But it still ran over TCP, where one lost packet holds up everything behind it. QUIC (2021) moved the transport into a new protocol on top of UDP, with TLS 1.3 built in: one round trip to set up both, independent streams, and connections that survive a change of network, such as moving from Wi-Fi to mobile data. HTTP/3 (2022) is HTTP over QUIC.

DNS gained encryption too: DNS over HTTPS (2018) and DNS over TLS hide queries from the networks they cross. What has not changed since the 1980s is the core: IP packets, TCP for most traffic, DNS for names and BGP for routes.

Type a URL and press Enter. Before anything appears, the browser has to do the following, and most of it is waiting for messages to cross the network and come back:

1. Parse the URL and check its caches. If the page is cached and still fresh, or a service worker answers, the network is never used.
2. Look up the name with DNS, unless the address is cached on the device.
3. Open a connection: a TCP handshake, then a TLS handshake for HTTPS. HTTP/3 does both in one QUIC handshake. If a connection to the site is already open, all of this is skipped.
4. Send the request and wait while the server builds the response.
5. Receive the response. A new TCP connection starts slowly, so a large response takes several round trips.
6. Render: parse the HTML, fetch the CSS, scripts, fonts and images it refers to (each of which may repeat the steps above for other servers), lay out the page and paint it.

The figure counts the round trips for steps 2 to 5, for one HTML document. Pick a protocol generation, a cache state and a round-trip time; everything else is worked out from the protocols’ rules.

Round trips dominate the early part of a page load because each step waits for the previous one: the browser cannot open a connection before it knows the address, or send the request before the handshake finishes. The time of a round trip is set by distance and the networks in between, and light in fibre covers about 200 km per millisecond, so no amount of bandwidth makes a far-away server close.

For a 50 KB HTML document, a 50 ms round-trip time, 100 ms of server time and the DNS answer cached at the resolver, the model gives:

A new TCP connection does not know how much the network can take, so it starts with a small congestion window and doubles it every round trip as acknowledgements arrive. Since RFC 6928 the initial window is 10 segments; at 1,460 bytes of data per segment, that is 14,600 bytes, about 14 KB. A response that fits arrives in the first round trip after the request. Beyond that, the extra round trips grow with the logarithm of the size:

The figure stops when the HTML has arrived, but the user sees nothing yet. The browser parses the HTML into the DOM as it streams in and starts fetching the resources it finds: stylesheets, scripts, fonts and images. CSS blocks rendering until it is loaded, because the browser cannot lay out the page without it, and scripts without async or defer pause parsing while they download and run. Resources on other domains need their own DNS lookups and connections, which is why sites preconnect to them or serve them from their own domain.

Once the DOM and the styles are ready, the browser computes styles, lays out every box, paints and composites the result. Google’s Core Web Vitals measure the outcome from the user’s side: Largest Contentful Paint (when the main content appears), Interaction to Next Paint (how quickly the page responds to input) and Cumulative Layout Shift (how much things jump around).

Every layer of the history above has failed in public, usually through one configuration change:

• YouTube, 24 February 2008 (BGP). Pakistan Telecom announced a /24 route inside YouTube’s /22. Its upstream provider passed the route on, and because routers prefer the most specific route, YouTube traffic from much of the internet went to Pakistan for about two hours. YouTube recovered traffic by announcing even more specific routes itself.
• Fastly, 8 June 2021 (CDN). A software bug deployed in May was triggered by a valid configuration change from a customer, and 85% of Fastly’s network returned errors. Within 49 minutes 95% of the network was working again.
• Facebook, 4 October 2021 (BGP and DNS). A command meant to assess backbone capacity disconnected all of Facebook’s data centres. Its DNS servers, unable to reach the data centres, withdrew their BGP routes as designed, so the rest of the internet could no longer find facebook.com, even though the servers were running.

The pattern is the internet’s design showing through: independent networks trust each other’s announcements, and a few large providers carry a large share of traffic.

“What happens when you type a URL into a browser and press Enter?” is one of the most common interview questions for web and backend roles, because it can go as deep as the interviewer wants. A good answer goes in order, DNS, TCP, TLS, HTTP request, server, response, render, and then dives into whichever layer the interviewer asks about: the DNS hierarchy and TTLs, the TCP handshake and slow start, the TLS handshake and certificates, HTTP/2 multiplexing, load balancers and CDNs on the server side, or the critical rendering path in the browser. Counting the round trips, as this page does, shows you know where the time goes.

History questions are rarer, but knowing why things are the way they are helps: why IPv4 ran out and what NAT broke, why BGP can be hijacked, why HTTP/3 needed a new transport rather than a better TCP.