Prowid Photos now has three API instances, but the site has a single address — photos.prowid.com. Which of the three servers gets a request from the browser? That is decided by the load balancer — the "host at the restaurant door": it greets guests, walks them to a free table, and if a table is broken it stops seating anyone there. In this lesson we look at why it is needed, how it picks a server (algorithms), how it learns about failures (health checks), how L4 differs from L7, how it is configured in Nginx — and why the balancer itself can become a weak link.

Why a load balancer

A balancer stands between clients and servers and gives you several things at once:

  • A single entry point. Clients know one address and do not know how many servers are behind it.
  • Load distribution. Requests spread across the instances, so none of them is overloaded before the others.
  • Fault tolerance. A failed server is taken out of rotation and users do not notice.
  • Updates without downtime. A server is taken out of rotation, you wait until it finishes its current requests, update it and put it back.
  • Bonuses. A balancer often also terminates TLS (HTTPS), compresses responses and limits the request rate.

The balancer role can be played by Nginx or HAProxy on your own server, or by a managed cloud service. You already know the basics of Nginx as a reverse proxy from the lesson "Nginx as a Reverse Proxy for Containers": a balancer is a reverse proxy that has a group of servers instead of one.

Balancing algorithms

Three algorithms: round robin, least connections and IP hash
Three algorithms: round robin, least connections and IP hash

Round robin is "in turn": the first request goes to the first server, the second to the second, and so on. It is simple and fair if the servers are identical and the requests are similar:

js
const servers = ['s1', 's2', 's3'];
let next = 0;

function pickRoundRobin() {
  const server = servers[next];
  next = (next + 1) % servers.length;
  return server;
}

for (let i = 1; i <= 4; i++) {
  console.log(`request ${i} -> ${pickRoundRobin()}`);
}
// request 1 -> s1, request 2 -> s2, request 3 -> s3, request 4 -> s1

But requests are not equal: a light GET takes 5 ms, a photo upload takes 3 seconds. In turn, one server may end up buried in heavy requests while another sits idle. Weighted round robin takes the different power of servers into account: a server with weight 2 gets twice as many requests. Least connections sends a request where there are currently the fewest open connections — ideal for file uploads and WebSocket:

js
const connections = { s1: 3, s2: 1, s3: 2 };

function pickLeastConnections() {
  let best = 's1';
  for (const name of Object.keys(connections)) {
    if (connections[name] < connections[best]) best = name;
  }
  return best;
}

console.log(pickLeastConnections()); // s2: it has the fewest open connections