Md Nayeem Hossain | Software Engineer

Rate Limiting at Scale: Token Buckets, Sliding Windows and Redis#

Every public API eventually meets a client that sends far more traffic than it should — a misconfigured retry loop, a scraper, a brute-force attempt on the login endpoint. Rate limiting is how you keep one bad actor from degrading the service for everyone else.

The concept is simple. Doing it correctly across many server instances is where the engineering lives.

Choosing the algorithm#

Fixed window#

Count requests per key per minute, reset at the minute boundary. It is trivial to build with INCR and EXPIRE, but it allows bursts at the edges: a client can send the full limit at 12:00:59 and again at 12:01:00, doubling the effective rate.

Sliding window#

Smooths that edge effect by weighting the previous window. A good approximation:

text

1estimated = current_window_count + previous_window_count × (1 − elapsed_fraction)

It is accurate enough for most APIs and uses only two counters per key.

Token bucket#

Each client has a bucket of tokens that refills at a steady rate. Each request spends a token; an empty bucket means the request is rejected. It allows controlled bursts (up to the bucket size) while enforcing a long-term average rate. This is my default for user-facing APIs, because real users are bursty.

A distributed token bucket in Redis#

With several API instances, the counter must live in shared storage. The check and the update must also be atomic, or two instances can both approve the last token. A Lua script gives you atomicity inside Redis:

lua

1-- KEYS[1] = bucket key2-- ARGV: capacity, refill_per_sec, now_ms, cost3local capacity = tonumber(ARGV[1])4local rate     = tonumber(ARGV[2])5local now      = tonumber(ARGV[3])6local cost     = tonumber(ARGV[4])7 8local state  = redis.call("HMGET", KEYS[1], "tokens", "ts")9local tokens = tonumber(state[1]) or capacity10local ts     = tonumber(state[2]) or now11 12tokens = math.min(capacity, tokens + (now - ts) / 1000 * rate)13 14local allowed = tokens >= cost15if allowed then tokens = tokens - cost end16 17redis.call("HSET", KEYS[1], "tokens", tokens, "ts", now)18redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / rate * 1000))19 20return { allowed and 1 or 0, tokens }

Call it from middleware and translate the result into HTTP:

1const [allowed, remaining] = await redis.evalsha(sha, 1, `rl:${userId}`, 60, 1, Date.now(), 1);2 3res.setHeader("RateLimit-Limit", "60");4res.setHeader("RateLimit-Remaining", Math.floor(remaining));5 6if (!allowed) {7  res.setHeader("Retry-After", "1");8  return res.status(429).json({ message: "Too many requests" });9}

What to key on#

The key decides who shares a limit:

Authenticated user or API key for normal endpoints. This is the fairest.
IP address for anonymous traffic, knowing that offices and mobile carriers put many users behind one IP.
Account plus endpoint for sensitive routes — login, password reset, OTP verification — with much stricter limits.

Layer them: a generous per-user limit across the API, plus a tight limit on authentication endpoints. Brute-force protection on login is not optional.

Where to enforce#

Rate limiting belongs as close to the edge as possible. A CDN or API gateway can reject abusive traffic before it costs you any compute. Application-level limits are still valuable for business rules — "five password reset emails per hour" — that the edge does not know about.

Failure modes#

Your rate limiter is now a dependency on every request. Decide what happens when Redis is unavailable:

Fail open (allow traffic) for general endpoints, so a cache outage does not become a full outage.
Fail closed (reject) for high-risk endpoints such as login, where letting unlimited attempts through is worse.

Also watch for hot keys: one very active API key hitting a single Redis shard. A local in-memory pre-check per instance can absorb most of that load.

Communicate limits clearly#

Good rate limiting is part of the API's contract:

Return 429 Too Many Requests with a Retry-After header.
Expose remaining quota in response headers so clients can pace themselves.
Document the limits, and give paying customers higher ones as a product feature.

Conclusion#

A solid rate limiter is a token bucket with atomic updates in shared storage, keyed thoughtfully, enforced at the edge where possible, with a deliberate failure mode. Build it once, put it in front of every endpoint, and the next runaway client becomes a log line instead of an incident.