How load balancers work

A load balancer gives a service one address that clients connect to, and passes each connection or request on to one of several servers behind it. Clients never see the individual servers, so servers can be added, removed, restarted or replaced without clients noticing.

It does three jobs. It spreads the load, choosing a server for each connection or request. It avoids failed servers, by checking their health and taking broken ones out of rotation. And it is the place where a lot of shared work can happen once: ending TLS connections, retrying failed requests, adding headers and recording logs.

This guide follows traffic through the two main kinds of load balancer and then looks at what keeps them working. The choice of server, the algorithm, is covered on the load balancer simulator page, where four algorithms run side by side.

Most web services sit behind a layer 7 load balancer, one that understands HTTP. NGINX, HAProxy, Envoy and the AWS Application Load Balancer all work this way. Here is what happens to one request for https://shop.example.com/cart.

1. DNS. The browser looks up shop.example.com and gets the load balancer's IP address, not a server's.
2. Connection and TLS. The browser opens a TCP connection to the load balancer and performs the TLS handshake with it. The load balancer holds the site's certificate and private key, so this is where TLS ends.
3. Reading the request. The load balancer decrypts and reads the HTTP request: method, path, host and headers. It can now apply routing rules, for example sending /api/ to one group of servers and everything else to another.
4. Choosing a server. Within the chosen group, it picks one healthy server using its algorithm, such as round robin or least connections.
5. Forwarding. It sends the request to that server, usually over a connection it already has open, since load balancers keep a pool of connections to each server. It adds headers such as X-Forwarded-For, which carries the client's IP address, because the server otherwise sees only the load balancer's.
6. Response. The server replies to the load balancer, which sends the response back to the browser over the original connection.

The client has two connections in this picture, one to the load balancer and one from the load balancer to the server, and the load balancer is a full participant in both. That is what lets it route by URL, retry a request on another server, compress responses, or speak HTTP/2 to browsers and HTTP/1.1 to servers. The cost is the work of decrypting and parsing every request.

Because each request is handled separately, a layer 7 load balancer can spread the requests from one long-lived connection across many servers. This matters for HTTP/2 and gRPC, which send many requests over a single connection.

A layer 4 load balancer works with TCP connections or UDP packets and never reads what is inside them. When a new connection arrives, it chooses a server and then sends every packet of that connection to the same server. It is fast, works for any protocol, and can handle very large volumes of traffic, which is why it often sits in front of layer 7 load balancers. The AWS Network Load Balancer, Linux IPVS and Kubernetes' kube-proxy work at this layer.

To send all packets of a connection to the same server, the load balancer identifies each connection by its addresses and ports and remembers its choice in a connection table. There are three common ways to get the packets to the server:

Address translation (NAT)

The load balancer rewrites each packet's destination address to the chosen server's, and rewrites the replies on the way back. Every packet in both directions passes through it. This is simple and needs nothing special on the servers, but the load balancer carries all the response traffic too.

Direct server return

The load balancer forwards incoming packets to the server without changing the destination address, and the server is set up to accept traffic for the load balancer's address. The server then sends its replies straight to the client, bypassing the load balancer. Since responses are usually much larger than requests, this removes most of the load balancer's work. Linux IPVS calls this direct routing.

Tunnelling

The load balancer wraps each packet inside another packet addressed to the chosen server, which unwraps it and replies directly to the client. This works across networks where direct server return cannot. Google's Maglev load balancer works this way.

A load balancer is only useful during failures if it notices them. Active health checks test each server at a fixed interval: open a TCP connection, or request a path such as /healthz and expect status 200. After a set number of failures in a row, the server is taken out of rotation; after a set number of successes, it is put back. Passive checks, called outlier detection in Envoy, watch real traffic and remove a server that returns too many errors.

What the health endpoint checks matters. Checking only that the process answers misses a server whose database connection is broken. Checking a shared dependency from every server can fail them all together when that dependency has a short problem, removing the whole service at once. Kubernetes separates the two questions: a liveness probe asks whether to restart a container, a readiness probe asks whether to send it traffic.

When a server is taken out on purpose, during a deployment or a scale-down, the load balancer stops sending it new requests but lets the requests in progress finish. This is connection draining. Servers should keep running for longer than the slowest request after they are marked as not ready, or those requests are cut off.

A server that has just started has cold caches and no warm connections. Sending it a full share of traffic at once can make it slow, so some load balancers ramp traffic up gradually; HAProxy has a slowstart setting and Envoy a slow start mode.

Behind a layer 7 load balancer, a server sees every connection coming from the load balancer. Applications that need the client's address for logs, rate limits or fraud checks read it from a header instead: the widespread X-Forwarded-For, or the standard Forwarded header defined in RFC 7239. These headers can be forged by clients, so servers should trust only the values added by their own load balancers.

For layer 4 traffic, where headers cannot be added, the PROXY protocol, introduced by HAProxy, sends a short line with the original addresses at the start of each connection. Some applications also need a user's requests to keep reaching the same server, which is called session affinity; the simulator page covers it.

Every request passes through the load balancer, so it must not be a single point of failure. There are three common arrangements.

• An active and a standby pair. Two load balancers share a virtual IP address. The active one holds it; if it fails, the standby takes the address over. On Linux this is often done with keepalived and the VRRP protocol.
• Many load balancers sharing one address. Several load balancers announce the same IP address to the network routers, which spread incoming traffic across them (equal-cost multipath routing). If one fails, the routers stop sending to it. Large layer 4 systems such as Google's Maglev, Facebook's Katran and GitHub's GLB work this way. Because routers can move a connection's packets to a different load balancer at any time, each load balancer must choose servers in a way that gives the same answer for the same connection, which is why they use consistent hashing schemes rather than only a local connection table.
• Managed load balancers. Cloud providers run their load balancers across many machines and zones and scale them automatically.

A large service often uses several kinds of load balancing at once:

1. Global: DNS returns the address of the nearest healthy region, or an anycast address is announced from many locations so that internet routing takes each user to the nearest one.
2. Layer 4 at the edge of each region: a fleet of layer 4 load balancers absorbs large volumes of traffic, including attacks, and spreads connections across layer 7 proxies.
3. Layer 7 proxies: these end TLS, apply routing rules and spread requests across application servers.
4. Inside the system: services calling each other balance requests themselves, through a client library or a sidecar proxy in a service mesh, instead of going through a central load balancer.

Kubernetes shows several of these: a Service is a layer 4 virtual address implemented by kube-proxy on every node, and an Ingress or Gateway controller adds a layer 7 proxy in front.

Long-lived connections

A layer 4 load balancer balances connections, not requests. Clients that keep one connection open for hours, such as gRPC services or databases with connection pools, can leave a newly added server almost idle while older ones stay busy. Balancing per request at layer 7, or having clients reconnect from time to time, fixes this.

Retries that multiply

If a client retries, the load balancer retries and the service behind it retries too, one failed request can turn into many during an outage, making it worse. Limit retries to a small share of traffic, and retry at one layer only where possible. The retry strategy simulator shows the effect of backoff and jitter.

Hashing on addresses behind a NAT

Choosing a server from a hash of the client's IP address sends everyone behind a company or mobile network gateway to the same server, because they share one address. Hash on a cookie or a user ID instead.

Health checks that are too sensitive

Removing a server after one slow check can make servers flap in and out of rotation, and if all of them flap at once there is nothing left to send traffic to. Envoy has a "panic threshold": when too few servers are healthy, it ignores health and spreads traffic across all of them.