Design a Rate Limiter: Distributed Throttling for a Busy API

The core job is to cap how many calls a given caller can make within some period. The caller might be identified by a logged-in user ID, an API key issued to a partner, or simply an IP address for anonymous traffic. Supporting all three matters because each catches a different kind of abuse: a scraping script without an account shows up only by IP, while a partner integration misbehaving is best tracked by its key.

Excess requests should be rejected immediately with HTTP 429, not queued, because queueing would hold server resources for traffic we already decided not to serve. Limits also need to be data, not code. A free tier at 100 requests per minute and a paid tier at 10,000 should be adjustable by an operator without shipping a new build. Finally, good clients deserve feedback: headers that show remaining quota and a Retry-After value let them slow down gracefully. DDoS mitigation at the network layer and monthly billing quotas are separate concerns and stay out of scope.

Because the limiter sits in the path of every call, its own latency is a tax on the entire API. A budget of roughly a millisecond, and certainly under two, keeps that tax invisible. This one constraint rules out anything that needs a slow database lookup per request.

Accuracy across servers is the second pressure. If fifty gateway instances each kept a private counter, a determined client could spread traffic across them and receive fifty times its quota. The limit has to be global. Third, the limiter should fail open: when its backing store is unhealthy, letting traffic through briefly is far better than rejecting every customer. Losing rate limiting for a minute is an inconvenience; losing the API is an outage. Throughput completes the list, since the limiter handles the full request volume of the platform.

Assume a peak of about 1 million requests per second, all of which need a decision. Assume roughly 10 million active clients across users, keys and IPs. Each client needs only a small record, around 100 bytes, so the whole working set is about 1 GB. That fits comfortably in memory on a single machine.

Throughput is a different story. If one Redis shard handles roughly 100,000 operations per second, then 1 million checks per second calls for about 10 shards. The estimate therefore tells us where to spend design effort: storage is a non-issue, while per-request speed and making each check-and-update a single indivisible operation are the real problems to solve.

Three entities carry the design. The client ID names who is being limited. A rule describes the limit that applies, such as a match on plan and endpoint plus a limit, a window and an algorithm. The token bucket is the live state for one client, recording how much allowance is left.

Internally the gateway calls something like POST /ratelimit/check with the client ID, the endpoint and an optional cost, and receives whether the call is allowed, the remaining quota and a reset time. The optional cost is useful when one expensive endpoint should consume several units at once. Externally, a rejected call such as GET /api/orders returns 429 with Retry-After: 2, which tells a polite client exactly how long to back off. Operators manage policies through PUT /rules/{id}, so a new plan or a tighter limit on a hot endpoint is a configuration change rather than a release.

Traffic flows from clients through a load balancer into a pool of stateless API gateways. A throttling filter runs as the first step of request handling. The filter loads rules from a rules store once and keeps them cached in memory, refreshing periodically, so reading a policy costs nothing on the hot path.

For each request the filter makes one call to a Redis cluster that both checks and updates the client's bucket atomically. If allowance remains, the request continues to backend services such as orders or payments and on to their database. If not, the gateway answers 429 with a Retry-After header and the backend never sees the call. That is the payoff of placing the check up front: rejected traffic costs a single Redis round trip, not a database query.

The key decision is that counters live in shared Redis rather than inside each gateway. Every instance reads and writes the same bucket for a given client, so a caller gets one limit no matter how the load balancer spreads its requests.

A fixed window counter is the simplest choice: count calls per minute and reset at the boundary. Its flaw is the edge. A client can spend its full allowance at 12:00:59 and again at 12:01:00, landing twice the intended rate in two seconds. A sliding window log fixes this by storing every request timestamp, but memory grows with traffic. A sliding window counter blends the current and previous windows cheaply, at the cost of being an estimate.

The token bucket wins here. Each client has a bucket with a capacity and a refill rate; each request spends a token. With a capacity of 10 and a refill of 1 token per second, a client can burst 10 calls after a quiet period but sustains only one per second over time. That matches how real apps behave, since page loads arrive in small bursts.

Refill is computed lazily. On each request we take the time since the last refill, multiply by the rate, add those tokens up to capacity, then spend one. No background timers run, and the state is just two numbers per client: tokens remaining and the last refill timestamp.

The danger with a shared counter is a read-then-write race. Two gateways read the same bucket showing one token, both decide to allow, and both write back zero. The client just got two requests for the price of one, and under heavy concurrency this leak grows large.

The fix starts with one Redis key per client and endpoint, for example rl:{client}:{endpoint}, holding the token count and last refill time. Rather than separate GET and SET calls, the gateway sends a small Lua script that reads the bucket, applies the refill, makes the decision and writes the result in a single call. Redis executes scripts one at a time, so no other command can slip between the read and the write. Atomicity comes from the execution model, with no distributed lock required.

Two housekeeping steps keep the system lean. Every key carries an expiry, so buckets for clients that go quiet disappear on their own and memory stays near the 1 GB estimate. Redis Cluster spreads keys by hash across the roughly 10 shards. If Redis becomes unreachable, the gateway fails open and applies a coarse in-process limit until the cluster recovers.

Client-side throttling is a courtesy, not a control. A well-written SDK can pace itself, but anyone can strip that logic out, so the server must never rely on it. The API gateway is the main enforcement point because it is one place that guards every service, and it rejects bad traffic before that traffic consumes any backend capacity.

Limits inside individual services still have a role. A service understands business cost, such as a report endpoint being far heavier than a profile lookup, but duplicating full limiter logic in every service is hard to maintain. The practical answer is layers: an edge proxy or web application firewall drops floods by IP, the gateway enforces per-user and per-plan quotas, and critical services keep a simple self-protection limit of their own as a last line of defence.

Accuracy competes with latency. A shared Redis check gives a precise global count but adds a network hop; purely local counters are faster but drift as instances disagree. We accept the hop because a sub-millisecond call inside the same data center fits the budget. Fail open versus fail closed is a choice between availability and strictness, and for most APIs availability wins, with the understanding that limits briefly lapse during an outage.

The token bucket is cheap and tolerant of bursts, while a sliding window is smoother and stricter; pick the window if smooth traffic matters more than burst friendliness. Centralising at the gateway keeps rules in one place, but the gateway tier must scale with total traffic. The resulting design: token buckets, updated by atomic Lua in Redis, enforced at the gateway.