Rate Limiting Explained: Why, Where and What to Limit

Imagine a pizza chain ships a mobile build with one small bug: when the order-status call fails, the app retries immediately, forever. A brief server blip now turns every open app into a machine-gun that fires as fast as the network allows, the servers get slower, more calls fail, more retries fire, and the outage feeds itself. No attacker was involved. A rate limit is the guard that stops one misbehaving client, malicious or merely buggy, from consuming capacity everyone else needs.

What a rate limit actually protects

A rate limit caps how many requests an identified caller may make in a period of time. Everything else is detail. The reasons teams add one fall into a few groups, and it helps to know which one you are solving because it decides the rest of the design.

• Capacity and stability: keep a spike from one caller from exhausting CPU, database connections or a downstream dependency (the retry storm above).
• Fairness: stop one heavy tenant from degrading the experience of all the others on shared infrastructure.
• Abuse resistance: slow down credential guessing on a login route, scraping, or one-time-code flooding. Limits slow attackers; they do not stop a determined one, and they are not a replacement for real authentication and monitoring.
• Cost control: paid downstream calls (SMS, maps, a model API) can turn a loop into a bill.
• Commercial plans: free, pro and enterprise tiers often differ mainly in their limits.

Where to enforce it

Requests pass through several layers on the way to your code, and each layer can enforce limits with different trade-offs. The cheaper and earlier you reject, the less capacity a bad request costs. The later you reject, the more you know about who is calling and what they are asking for.

Most mature setups layer them: the edge drops floods, the gateway applies a per-key ceiling, and the service enforces the rules only it can know, such as "five failed password attempts per account per hour". For how the layers fit into a bigger picture, see Scalability and Load Balancing.

What to key on: the identity problem

A limit is meaningless without a key: the identity whose requests are counted together. The choice has consequences that surprise people. The next program counts requests per key for the same traffic twice: four colleagues behind one office network address (NAT) and one script with its own key.

from collections import defaultdict

LIMIT = 5  # requests allowed per window, per key


def run(key_fn, requests):
    counts = defaultdict(int)
    verdicts = defaultdict(list)
    for req in requests:
        key = key_fn(req)
        counts[key] += 1
        verdicts[req["user"]].append("200" if counts[key] <= LIMIT else "429")
    return dict(verdicts)


# Four colleagues share one office IP (NAT), two requests each.
office = [
    {"user": u, "ip": "203.0.113.10", "api_key": "key-" + u}
    for u in ["asha", "ben", "chitra", "dev"]
    for _ in range(2)
]
# One script on its own IP sends 8 requests with a single key.
bot = [{"user": "bot", "ip": "198.51.100.7", "api_key": "key-bot"} for _ in range(8)]
traffic = office + bot

for label, fn in [("IP", lambda r: r["ip"]), ("API key", lambda r: r["api_key"])]:
    print("keyed by", label)
    for user, codes in run(fn, traffic).items():
        print(f"  {user:7}", " ".join(codes))

keyed by IP
  asha    200 200
  ben     200 200
  chitra  200 429
  dev     429 429
  bot     200 200 200 200 200 429 429 429
keyed by API key
  asha    200 200
  ben     200 200
  chitra  200 200
  dev     200 200
  bot     200 200 200 200 200 429 429 429

Keyed by IP, innocent colleagues Chitra and Dev were rejected because their office shares an address; keyed by API key, only the script was limited. The reverse failure exists too: if you key purely on an identity that is cheap to create (a free sign-up, a rotating key), an abuser simply makes more of them. Each key type fails differently.

Real policies usually combine two keys, such as per-IP and per-account on a login route. And a warning about IP keys behind proxies: you must take the client address from a header like X-Forwarded-For only from proxies you operate, and use the address those proxies added, not whatever the client sent, otherwise anyone can pick their own identity by setting the header.

A first limit in a reverse proxy

For a quick, first layer you do not need to write code. This nginx excerpt limits each client IP on a login route to about 5 requests per second with a short burst allowance. It is not run here, and the directive names are worth checking against your nginx version. Note limit_req_status: without it nginx answers rejected requests with 503, which looks like a server fault rather than a rate limit.

http {
    limit_req_zone $binary_remote_addr zone=login_zone:10m rate=5r/s;

    server {
        location /login {
            limit_req zone=login_zone burst=10 nodelay;
            limit_req_status 429;
            proxy_pass http://app_backend;
        }
    }
}

Common mistakes

• Limiting only by IP for a B2B product. A customer whose staff share an office gateway starts getting 429s together. Symptom: support tickets from one company all at the same hour.
• Trusting a client-supplied forwarding header. Anyone can send X-Forwarded-For: 1.2.3.4 and slip past a per-IP limit. Symptom: a limit that works in tests but not against a script that rotates the header.
• Returning 503 or 500 for a rejection. Clients and monitors then treat it as a server outage and retry harder. Use 429 and say when to come back.
• Applying one global number to every route. A static asset and a report export have wildly different costs. Symptom: either the cheap route is over-restricted or the expensive one is unprotected.
• Forgetting that the limiter is on the hot path. A limiter that does a slow database query per request becomes the bottleneck it was meant to prevent.

How this shows up in real projects and interviews

"Design a rate limiter" is a staple system design question, and a strong answer starts with exactly this lesson's questions before drawing boxes: who is the caller, what are we protecting, where do we enforce, what do we do when the limit is hit. In real projects, the first limit is usually added after an incident, so a good habit is to ask during design which routes are expensive and which callers are unauthenticated, and protect those first. Our system design interview guide covers how to frame such a question.