Load balancers

Think of it like this

Analogy

A busy restaurant has a host at the door. Guests don't pick their own table. The host looks at the room, seats each party at a table that has space, and stops seating people in a section whose waiter just went home sick. Guests never need to know how many waiters are working tonight.

A load balancer is that host. Users send requests to one address. The load balancer decides which server handles each one.

The problem it solves

In the last lesson you added more servers (horizontal scaling). But that raises two questions:

• Which server should a user talk to? You can't give users a list of server addresses and hope they spread out evenly. Some servers would be swamped while others sit idle.
• What happens when a server dies? Users connected to it get errors, and they keep getting errors until something stops sending traffic there.

Without a load balancer, adding servers doesn't help much. You still have a hot spot and a single point of failure. You just have more machines.

A real example

Example

A concert ticket sale at 10:00

Tickets go on sale at 10:00. Eight app servers sit behind a load balancer using least connections, so busy servers get fewer new requests.

At 10:02 one server freezes. Its health check, GET /healthz every 5 seconds, fails twice in a row, and the load balancer stops sending it traffic. Users see a few slow requests, not an outage.

The load balancer also routes by path. /checkout goes to a separate pool, so a flood of people browsing seat maps can't slow down people who are paying. When the team deploys a fix at 10:30, each old server is drained first: it gets no new requests, but finishes the ones in flight.

How it works

Users only ever see the load balancer's address. DNS points your domain at it. For every request, the load balancer picks a healthy server, forwards the request, and passes the response back.

Architecture

Loading diagram…

> Initializing simulation kernel

> Allocating nodes

> Opening network links

• Users to DNS: lookup
• Users to Load balancer
• Load balancer to Server 1
• Load balancer to Server 2
• Load balancer to Server 3
• Server 1 to Database
• Server 2 to Database
• Server 3 to Database

A load balancer has to make three decisions over and over: how much of the request to look at, which server gets it, and whether each server is healthy.

Layer 4 or layer 7: how much it reads

Aspect	Layer 4 (transport)	Layer 7 (application)
What it sees	IP addresses and ports (TCP/UDP)	The full HTTP request: URL, headers, cookies
Can route by URL or header	No	Yes, e.g. /api/* to one group, /images/* to another
Can end HTTPS and inspect traffic	No (it passes encrypted bytes through)	Yes
Speed	Very fast, very little work per packet	A bit more work per request
Typical use	Game servers, databases, any non-HTTP traffic, huge volumes	Websites and APIs

Layer 7 load balancers are the usual choice for web apps because they understand HTTP. Layer 4 load balancers win when you need raw speed or the traffic isn't HTTP.

Choosing a server: the algorithms

• Round robin hands requests out in turn: 1, 2, 3, 1, 2, 3. Simple, and fine when servers are identical and requests take about the same time.
• Weighted round robin gives bigger servers a bigger share, for example two requests to the large server for every one to the small one.
• Least connections sends each request to the server handling the fewest right now. It adapts when some requests are slow or one server is struggling.
• IP hash hashes the client's IP address to pick a server, so the same user keeps landing on the same server. Useful if servers hold per-user data in memory, but it reshuffles almost everyone when a server is added or removed.
• Consistent hashing fixes that reshuffling (more below).

Try them. Turn on “S3 is slow”, run traffic with round robin, then switch to least connections and watch the busy pips on S3. Then crash a server mid-stream.

Two things to notice from the simulator. First, when you crash a server, a few requests still fail before the load balancer reacts. Second, with every server down, the load balancer can only return an error. It spreads load; it can't create capacity.

Health checks: knowing who is alive

Every few seconds the load balancer sends each server a small request, usually GET /health. If a server fails a few checks in a row, it is taken out of rotation. When it passes again, it is put back.

A good health endpoint checks what the server actually needs. If the server is up but can't reach its database, it should report unhealthy, or the load balancer will keep sending it requests it can only fail.

Consistent hashing: moving as little as possible

Sometimes you want the same key to keep going to the same server, for example so each cache server holds its own slice of the data. The obvious way is server = hash(key) % N. The problem: change N from 3 to 4 and almost every key gets a new server, so every cache is suddenly cold.

Consistent hashing places both servers and keys on a ring. Each key belongs to the next server clockwise. Adding a server only takes keys from its neighbor; removing one only hands its keys to the next one along. Placing each server at several points (“virtual nodes”) keeps the shares even.

Don't make the load balancer the new weak point

If every request goes through one load balancer, that box is now your single point of failure. In practice you run load balancers in pairs or clusters, or use your cloud's managed load balancer, which is already spread across several machines and data centers.

Sticky sessions

If a server keeps a user's login or shopping cart in its own memory, the user has to keep hitting that server. A sticky session (usually a cookie the load balancer sets) does that. It works, but it undoes some of the benefit: load becomes uneven, and if that server dies the user's session is lost. The better fix is to make servers stateless, which is the next lesson.

Trade-offs

What you gain

• Add or remove servers without users noticing.
• A crashed server stops getting traffic automatically.
• One public address for the whole fleet, and one place to handle HTTPS certificates.
• Layer 7 balancers can route by path, host or header and roll out new versions gradually.

What it costs you

• One more component that can fail and must itself be made redundant.
• A small amount of extra latency per request (usually well under a millisecond).
• Requests can still fail in the gap before a health check notices a dead server.
• Long-lived connections like WebSockets don't rebalance on their own when servers are added.

When not to use it

• You only have one server and no plan to add a second. DNS pointing straight at the server is simpler.
• Clients already balance themselves, as in some internal gRPC setups that pick from a list of servers.

In the cloud

Load balancingEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS3

• Application Load Balancer (opens vendor docs)

  Layer 7 (HTTP/HTTPS) load balancer that can route by path, host name or header. Part of Elastic Load Balancing.

• Network Load Balancer (opens vendor docs)

  Layer 4 (TCP/UDP) load balancer for very high throughput and static IP addresses.

• Gateway Load Balancer (opens vendor docs)

  Sends traffic through third-party network appliances such as firewalls.

Azure3

• Azure Load Balancer (opens vendor docs)

  Layer 4 load balancer for TCP and UDP traffic within a region.

• Azure Application Gateway (opens vendor docs)

  Regional Layer 7 load balancer with path-based routing and an optional web application firewall.

• Azure Front Door (opens vendor docs)

  Global Layer 7 entry point that combines load balancing, CDN caching and a firewall at the edge.

Google Cloud2

• Application Load Balancer (opens vendor docs)

  Layer 7 load balancer for HTTP/HTTPS, available as global or regional, external or internal.

• Network Load Balancer (opens vendor docs)

  Layer 4 load balancer, in proxy (can terminate TLS) or passthrough (keeps client IPs, supports UDP) modes.

Public traffic hits DNS, then a Layer 7 load balancer that spreads it across a group of identical servers.

AWSAzureGoogle Cloud

1. Route 53
2. Application Load Balancer
3. Auto Scaling group of EC2
4. Amazon RDS

Interview questions

Try answering out loud first, then open the model answer.

What's the difference between a Layer 4 and a Layer 7 load balancer, and when would you use each?+

Model answer

A Layer 4 load balancer works with TCP or UDP connections. It routes using IP addresses and ports and never reads the request content, so it's very fast and works for any protocol. A Layer 7 load balancer understands HTTP, so it can route by URL path, host name, headers or cookies, end HTTPS, and do things like gradual rollouts.

For a typical web app or API I'd use Layer 7. I'd pick Layer 4 for non-HTTP traffic (databases, game servers, MQTT) or when I need the highest throughput and static IP addresses.

How does a load balancer know a server is down, and what happens to requests in the meantime?+

Model answer

It runs health checks: a small request such as GET /health every few seconds. After a set number of failures in a row the server is marked unhealthy and gets no new traffic. Between the crash and that decision, requests routed to the server fail. To shrink the damage: use short check intervals, make clients retry idempotent requests on another server, and make the health endpoint check real dependencies like the database.

Your load balancer is a single point of failure. How do you fix that?+

Model answer

Run more than one. Common options are an active-passive pair sharing a floating IP, several active load balancers behind DNS that returns multiple addresses, or a managed cloud load balancer that is already distributed across availability zones. At global scale you add DNS-based or anycast routing in front so traffic goes to the nearest healthy region.

Why would you use consistent hashing instead of round robin?+

Model answer

When it matters which server handles a key. For example, a cache cluster where each node stores part of the data, or WebSocket servers where a user's connection lives on one machine. Round robin would send the same key to a different node each time. Plain hash % N keeps keys in place but moves almost all of them when N changes. Consistent hashing keeps keys sticky and moves only about 1/N of them when a node joins or leaves, and virtual nodes keep the load even.

Quiz

Pick an answer to see whether it’s right and why.

1. 01You run four servers behind a round-robin load balancer. One server crashes. What happens before the next health check runs?

   Nothing, the load balancer already knows it crashedAbout a quarter of requests still go to the dead server and failAll requests fail until the server restarts
2. 02One server is much slower than the others. Which algorithm adapts best?

   Round robinLeast connectionsIP hash
3. 03You need to route /api/* to one group of servers and /images/* to another. What kind of load balancer do you need?

   Layer 4Layer 7Either one works
4. 04Why do caches and sharded databases use consistent hashing instead of hash(key) % N?

   It is faster to computeWhen N changes, only a small share of keys move instead of almost all of themIt guarantees every server gets exactly the same number of keys
5. 05Your load balancer runs on a single machine. What's the risk and the usual fix?

   It encrypts too much trafficIt slows every request by secondsIt's a single point of failure, so run a redundant pair or use a managed one
6. 06You take a server out for a deploy. How do you avoid cutting users off mid-request?

   Connection draining: send it no new requests and let current ones finishTurn it off immediatelyLower the DNS TTL
7. 07What does terminating TLS at the load balancer give you?

   It removes the need for health checksCertificates live in one place and servers skip the encryption workTraffic from users becomes unencrypted

0 of 7 answered

Key terms

Load balancer

A component that receives incoming requests and spreads them across several servers.

Layer 4 (L4)

The transport layer (TCP/UDP). An L4 load balancer routes using IP addresses and ports without reading the request content.

Layer 7 (L7)

The application layer (HTTP). An L7 load balancer can read URLs, headers and cookies to decide where a request goes.

Round robin

A load-balancing algorithm that hands requests to servers in turn, one after another.

Least connections

A load-balancing algorithm that sends each request to the server currently handling the fewest requests.

Consistent hashing

A way to map keys to servers on a ring so that adding or removing a server only moves a small share of keys.

Health check

A regular probe (often an HTTP request to /health) that tells a load balancer whether a server can take traffic.

Sticky session

Sending every request from the same user to the same server, usually with a cookie. Also called session affinity.

Horizontal scaling

Adding more machines and splitting the work between them. Also called scaling out.