Scalability and Load Balancing

Your app works on launch day. Then a popular newsletter links to it, traffic grows tenfold in an hour, and the single server it runs on starts timing out. Scalability is about being ready for that moment, and load balancing is the first tool you reach for.

Vertical vs horizontal scaling

Scaling means handling more load. You can make one machine bigger (vertical scaling, or scale up) or add more machines (horizontal scaling, or scale out). Neither is universally better.

A sensible path: begin vertically because it is simple, and move to horizontal scaling when you approach the limits or when you need redundancy. Horizontal scaling also gives you availability, because one failing machine no longer takes the whole service down.

Stateless services: the key to scaling out

A service is stateless when any server can handle any request, because nothing about the user's session lives only in that server's memory. If your server stores a shopping cart in local memory, then the user must always hit that same server, and if it dies the cart is lost.

To make a service stateless, push state out to shared systems. Keep the cart or session in a shared store such as Redis or in the database, or put it in a signed token (like a JWT) sent with each request. Each choice has a cost: a shared store adds a network call per request and must itself be made reliable, while tokens cannot be revoked as easily and grow with the data you place in them.

Load balancer algorithms

A load balancer receives requests and forwards each one to a healthy backend server. The algorithm decides which server gets the next request.

A load balancer can work at Layer 4 (TCP level: fast, only sees IPs and ports) or Layer 7 (HTTP level: can route by URL path, header or cookie, terminate TLS and retry requests, at the cost of more processing). Many systems use Layer 7 for the web tier because routing /api and /images to different server pools is so useful.

upstream app_servers {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 backup;
}

server {
    listen 80;

    location / {
        proxy_pass http://app_servers;
    }
}

The same idea in a few lines of TypeScript makes the logic concrete. This round-robin picker skips servers that are not in the healthy set.

class RoundRobin {
  private index = 0;

  constructor(private servers: string[]) {}

  next(healthy: Set<string>): string {
    for (let tries = 0; tries < this.servers.length; tries++) {
      const server = this.servers[this.index % this.servers.length];
      this.index++;
      if (healthy.has(server)) return server;
    }
    throw new Error("no healthy servers available");
  }
}

servers = [A, B, C], healthy = {A, B, C}  ->  A, B, C, A, B, C ...
servers = [A, B, C], healthy = {A, C}     ->  A, C, A, C ...

Health checks and failure handling

A load balancer is only useful if it stops sending traffic to broken servers. Active health checks probe each backend on a schedule (for example GET /health every 5 seconds) and remove it after several failures. Passive checks watch real traffic and mark a server unhealthy after repeated errors or timeouts, as in max_fails above.

• Make /health reflect real readiness, such as being able to reach the database, but keep it cheap so it does not overload a struggling server.
• Require several consecutive failures before removing a server, and several successes before adding it back, to avoid flapping.
• Support graceful draining: stop sending new requests to a server being deployed, let in-flight requests finish, then restart it.
• Capacity-plan with headroom. If you need 4 servers for peak, run 5 so that losing one (N+1) does not overload the rest.

Single points of failure

A single point of failure (SPOF) is any component whose failure takes the whole system down. After adding multiple app servers, the load balancer itself becomes one. Fix it with a redundant pair (active-passive with failover through a shared virtual IP, or active-active behind DNS), or by using a managed load balancer that is itself distributed.

        Clients
           |
           v
    +-------------+
    | DNS / VIP   |
    +-------------+
       |       |
       v       v
  +------+  +------+
  | LB 1 |  | LB 2 |   active / standby
  +------+  +------+
       \       /
        v     v
  +-----+ +-----+ +-----+
  | App | | App | | App |
  +-----+ +-----+ +-----+
        \    |    /
         v   v   v
        +----------+
        | Database |  <-- next SPOF!
        +----------+

Notice the database at the bottom: scaling the stateless tier just moves the bottleneck and the SPOF downwards. The next lessons, Caching and Databases, address that.

Common mistakes

• Storing sessions or uploads on local disk. Symptom: users are randomly logged out or files go missing when the load balancer picks a different server.
• Scaling the app tier but not the database. Symptom: adding servers does not help because every request waits on the same overloaded database.
• Forgetting the load balancer is a SPOF. Symptom: the whole site is down when one box fails, even though ten app servers are healthy.
• Health checks that are too shallow or too aggressive. A check that only returns 200 hides a broken database connection; one that runs a heavy query causes outages by itself.
• No headroom. Running at 90 percent of capacity means one failure triggers a cascade as remaining servers become overloaded and fail too.

How this shows up in interviews and real projects

Almost every design question includes a line like "we put a load balancer in front of several stateless servers", and interviewers often follow up with "how do you handle sessions?" or "what if the load balancer fails?". In real projects, cloud providers offer managed load balancers, so your job is mostly choosing the algorithm, health check and failure behaviour. Keep the system design interview prep guide nearby for how these topics are asked.