Rate Limiting
Capping how many requests a client can make in a window — the fundamental control that protects APIs from abuse, overload, and runaway costs.
What is rate limiting?
Rate limiting caps how many requests a given client can make within a time window — say, 100 requests per minute per API key. Exceed it and further requests are rejected, classically with HTTP status 429 Too Many Requests. It’s one of the most fundamental protective controls in any networked system: it defends against abuse, prevents accidental or malicious overload, ensures fair resource sharing among clients, and — increasingly important — caps runaway costs when each request triggers expensive downstream work.
The algorithms, and why they differ
A few standard algorithms trade precision against simplicity. Fixed window counts requests per clock interval — simple, but allows bursts at window boundaries (200 requests split across a window edge slip through two counters). Sliding window smooths this by considering a rolling time range, at more bookkeeping cost. Token bucket — the most popular — refills tokens at a steady rate up to a bucket size; each request spends a token, so it permits controlled bursts (up to the bucket) while enforcing a sustained average rate. Leaky bucket enforces a strictly smooth output rate. The choice matters: token bucket’s burst tolerance suits real user traffic, while leaky bucket suits protecting a downstream that needs steady flow. In distributed systems, the limiter’s counter usually lives in a shared fast store like Redis so all API nodes enforce one consistent limit.
Where it’s enforced — and the AI cost angle
Rate limiting typically lives at the edge: an API gateway, load balancer, or reverse proxy, so bad traffic is rejected before reaching application servers. It’s a core defense against credential stuffing and brute-force attacks (limiting login attempts), scraping, and denial-of-service, and it complements WAF and DDoS protection. The 2026-relevant angle: with LLM APIs, each request can cost real money in tokens, so rate limiting (and its cousin, token/quota limiting) is now a direct cost-control mechanism, not just an availability one — an unthrottled AI endpoint is a budget vulnerability. Good implementations return Retry-After and rate-limit headers so well-behaved clients can back off gracefully rather than hammer.
What people get wrong
- Fixed-window boundary bursts — naive per-minute counters let double the limit through at window edges; use sliding window or token bucket.
- Per-node limits in a cluster: without a shared counter, each server enforces its own limit and the real cap is N× higher than intended.
- No client feedback — returning 429 without
Retry-Afterheaders pushes clients to retry aggressively, worsening the overload.
Primary source: MDN: 429 Too Many Requests
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.