Designing Rate Limit Policies: Bursts, Fairness and Abuse Handling

A bulk-import tool and a small shop's checkout page call the same invoicing API. During a nightly import, the tool sends fifty calls in a burst every second. The shop sends three. If the service only has one global limit of twenty requests per second, who gets served? The next program sets the arrival order as badly as it can be, with the importer always first, and compares a global limit alone to a global limit plus a per-tenant limit.

class Bucket:
    def __init__(self, capacity, rate):
        self.capacity, self.rate = capacity, rate
        self.tokens, self.last = float(capacity), 0.0
    def take(self, now):
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

def simulate(per_tenant):
    shared = Bucket(capacity=20, rate=20)                    # whole-service cap: 20 req/s
    tenants = {"bulk-importer": Bucket(10, 10), "small-shop": Bucket(10, 10)}
    served = {"bulk-importer": 0, "small-shop": 0}
    for second in range(1, 6):
        # worst-case arrival order: the heavy tenant's 50 calls land first
        arrivals = ["bulk-importer"] * 50 + ["small-shop"] * 3
        for who in arrivals:
            if per_tenant and not tenants[who].take(float(second)):
                continue
            if shared.take(float(second)):
                served[who] += 1
    return served

print("global limit only    :", simulate(per_tenant=False))
print("global + per-tenant  :", simulate(per_tenant=True))

global limit only    : {'bulk-importer': 100, 'small-shop': 0}
global + per-tenant  : {'bulk-importer': 50, 'small-shop': 15}

With only a global ceiling, the heavy tenant drains the whole budget every second and the small shop is served zero requests out of fifteen. Adding a per-tenant limit capped the importer at ten per second and left the shop every one of its calls. This is the core fairness idea: a global limit protects the server, a per-tenant limit protects the other tenants. You need both, and in that order: reject over-limit tenants first so their rejected requests do not even touch the shared budget.

Where the numbers come from

A number such as "100 per minute" feels objective but is a decision. A defensible way to set it starts from capacity, not from guesses. Measure how much traffic the system sustains before latency degrades, reserve headroom for deploys and failures, and divide the remainder across the callers you expect to be active at once. Then check the result against the observed traffic of your well-behaved clients: your limit should sit comfortably above their normal peak, so only abnormal behaviour hits it.

Service: invoice-export endpoint, shared by paying tenants
Assumed measured capacity before latency degrades : 200 requests/s
Reserve for deploys, failover, internal jobs (30%) : 60 requests/s
Budget for external tenants                       : 140 requests/s

Expected simultaneously active tenants (peak)     : 20
Naive fair share per tenant                       : 140 / 20 = 7 requests/s

Observed p95 peak of normal tenants               : 4 requests/s
Chosen sustained rate (token refill)              : 6 requests/s   (above normal, under share)
Chosen burst (bucket capacity)                    : 20 requests    (about 3 seconds of rate)

Check: if 20 tenants all ran at 6 r/s = 120 r/s  -> inside the 140 budget.
Risk : 30 active tenants at the limit = 180 r/s  -> over budget; the global
       limit sheds load, and capacity planning must be revisited.

Everything in that block is made up to show the reasoning, so replace each figure with measurements from your own system. The shape matters: capacity, then headroom, then fair share, then sanity-check against real usage, then name the scenario in which the policy breaks.

Bursts, weights and tiers

Real clients are bursty: a page load fires six calls at once, a mobile app syncs after reconnecting. Set the burst (bucket capacity) to cover a legitimate burst and the rate to the sustained average you want to allow. A limit of 10 per second with a capacity of 10 gives no burst tolerance at all, while a capacity of 100 allows ten seconds of work in an instant, which might hurt a fragile backend.

Not every request costs the same. If a cheap lookup and a heavy report share one bucket, either the lookup is over-limited or the report is under-limited. The fix is weighted cost: the report takes five tokens instead of one. The token bucket from the algorithms lesson supports that by changing the check from tokens >= 1 to tokens >= cost. Tiers (free, pro, enterprise) are then just different rate, capacity and weight settings per plan.

Abuse handling that does not punish honest users

Rate limits are the cheapest layer of abuse defence, and good ones are graduated. For a login route: count failures per account and per IP, so a distributed attack on one account and a single machine trying many accounts are both slowed. Start with delays or CAPTCHA-style challenges before hard blocks; escalate block durations for repeat offenders; and never reveal in the response whether the account exists. Do not lock accounts permanently on failure counts alone, because an attacker can then lock out the victim on purpose. Allow-list your own health checks and internal services so monitoring is not limited into a false alarm.

Being a good client: backoff with jitter

The policy is only half the system; the other half is how clients react to a 429. A client that retries on a fixed schedule synchronises with every other client and hits the limit in waves. The recommended behaviour is to wait at least the Retry-After period, otherwise use exponential backoff with random jitter so retries spread out, and give up after a bounded number of attempts. Publish that guidance and ship it in your own SDKs. Documenting limits clearly, with the numbers, the headers and an example of the retry loop, prevents more support tickets than any code change.

Common mistakes

• A single global limit for everyone. One heavy tenant consumes it, as the simulation showed. Symptom: unrelated small customers report timeouts during someone else's batch job.
• Limits set below real peak usage. Honest customers hit 429 during normal busy hours. Symptom: a spike in rejections that correlates with business hours, not with attacks.
• Burst size equal to rate, or enormous. No burst tolerance rejects normal page loads; a huge burst lets a client dump a long backlog onto your database at once.
• Permanent lockouts as the abuse response. Attackers use them to deny service to real users. Prefer escalating delays and challenges.
• No visibility. Without metrics per key and per route, you cannot tell whether the limit is protecting you or hurting customers. Track allowed, rejected and the top rejected keys.

How this shows up in real projects and interviews

Interviewers often ask how you would choose the limits and what you would do about a noisy neighbour. A good answer walks through capacity, headroom, fair share, observed usage and the failure scenario, as in the worked example, and mentions per-tenant limits, burst, weighting and shadow rollout. In real work, the policy ends up as a table in a design document and as configuration in a gateway, and the harder part is the conversation with the customers whose integrations will break. For authenticating the callers you limit, see JWT Authentication, and for how limits fit into an API contract see REST API Development.