Design a Rate Limiter: A System Design Interview Walkthrough

Start with requirements, not algorithms

Before naming an algorithm, ask what is being protected and from whom. Is the limit per API key, per user, per IP address, or per tenant plan? Is it one rule, such as 100 requests per minute, or a sustained rate plus an allowed burst? Must the limit never be exceeded, or is approximately right acceptable? Does it apply in one region or globally? The answers decide the data structure, where the state lives, and how much accuracy you pay for.

Carry one concrete rule through the rest of the answer. This walkthrough uses a public API where each API key may burst 5 requests and then sustain 2 requests per second; the numbers are small so the trace fits on screen, and real limits come from configuration. State the non-functional requirements too: the check runs on every request, so it needs a small, bounded latency cost, and the limiter must not become the one component whose failure takes the whole API down.

• Identity: an API key or user ID for authenticated traffic; an IP address only as a coarse fallback.
• Rule shape: requests per window, or a rate plus a burst; per endpoint or for the whole API.
• Accuracy: a hard cap or an approximation, and whether rejected calls count.
• Scope: one gateway instance, one region, or every region.

Decide where the check runs

A client-side limiter is advisory only, because you cannot trust the client. A gateway or reverse proxy rejects excess traffic before it reaches expensive code: NGINX's limit_req module describes its method as a leaky bucket, while Envoy's local rate limit filter applies a token bucket. Note a default that trips people up: NGINX rejects with 503 unless limit_req_status changes it, whereas Envoy returns 429. A limiter inside the service knows business context, such as a tenant's plan or the cost of a query.

Next comes local versus global state. A per-instance limiter is fast, but behind a load balancer with N gateway instances, a client whose traffic spreads evenly can get up to N times its limit. A global limiter keeps the count in shared storage, usually Redis, and every instance consults it, as the figure shows. A practical hybrid keeps a generous local limit to shed floods cheaply and a precise global limit for the contract you publish to customers.

• NGINX documentation: ngx_http_limit_req_module
• Envoy documentation: local rate limit filter

Compare the five algorithms on one rule

A fixed window counter keeps one counter per key per window, such as per minute. It is cheap, but a client can send 100 requests at 12:00:59 and another 100 at 12:01:00, so 200 land within about a second while each window looks compliant. A sliding window log stores a timestamp per request and is exact, at the cost of memory that grows with the limit. A sliding window counter weights the previous window: with a limit of 100 per minute, 80 requests last minute, 30 so far, and a quarter of this minute elapsed, the estimate is 80 × 0.75 + 30 = 90, so the request passes.

A token bucket holds up to a fixed capacity of tokens, refills at a steady rate, and spends one token per request, so it admits a burst up to the capacity and then the sustained rate. A leaky bucket models a queue that drains at a constant rate; requests beyond the queue are rejected, which smooths output but can delay requests. In NGINX, the burst parameter sizes that queue, and nodelay serves requests within the burst at once instead of delaying them. Interviewers mostly want each algorithm's burst behavior and its weak spot.

Run a token bucket with an injected clock

Injecting the clock makes the limiter testable without sleeping: the test sets the time, then calls try_acquire. Refill is lazy, computed from elapsed time on each call, so no background timer is needed. The method returns whether the call is allowed and, when it is not, how long until enough tokens exist. That second value is what the gateway turns into Retry-After, rounded up to whole seconds, because the header carries an integer number of seconds or a date.

Running it with Python 3.12 prints three lines: t= 0.0: YYYYYnn retry_after=0.5, then t= 1.0: YYn retry_after=0.5, then t=10.0: YYYYYn retry_after=0.5. The first five calls drain the bucket, and one token takes 0.5 seconds at 2 per second. At t=1.0 exactly two tokens have refilled. At t=10.0, 18 tokens would have accrued, but the cap holds the bucket at 5, so idle time cannot be banked beyond capacity. The class is not thread-safe; across processes, this state belongs in shared storage.

class TokenBucket:
    def __init__(self, capacity, rate_per_sec, clock):
        self.capacity, self.rate, self.clock = capacity, rate_per_sec, clock
        self.tokens, self.last = float(capacity), clock()

    def try_acquire(self, cost=1):
        """Return (allowed, seconds until enough tokens)."""
        now = self.clock()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True, 0.0
        return False, (cost - self.tokens) / self.rate


class FakeClock:
    now = 0.0

    def __call__(self):
        return self.now


clock = FakeClock()
bucket = TokenBucket(capacity=5, rate_per_sec=2, clock=clock)
for t, n in [(0.0, 7), (1.0, 3), (10.0, 6)]:
    clock.now = t
    results = [bucket.try_acquire() for _ in range(n)]
    allowed = "".join("Y" if ok else "n" for ok, _ in results)
    print(f"t={t:>4}: {allowed} retry_after={results[-1][1]}")

Make the shared check atomic

The classic bug is read-then-write. Two gateway instances read a count of 99, both decide the request is allowed, and both increment, leaving 101. Redis INCR avoids this because it returns the new value atomically, so decide on the returned value rather than on a separate read. The Redis documentation's own rate limiter pattern shows a subtler race: incrementing a counter and setting its expiry as two separate commands can leave a key with no TTL if the client fails between them.

The fix the documentation gives is to run both steps in one Lua script sent with EVAL; Redis guarantees a script executes atomically, blocking other commands while it runs. A token bucket follows the same shape: store tokens and the last refill time in one hash, and let one script refill, decide, and write. Pass every key the script touches as an explicit key argument, which the Redis docs require for correct execution in both standalone and clustered deployments. Expect one network round trip per request, and measure that latency in your own environment.

• Redis documentation: INCR and the rate limiter pattern
• Redis documentation: scripting with Lua

Define the 429 contract and the failure mode

RFC 6585 defines 429 Too Many Requests for a client that sent too many requests in a given time. The response may include Retry-After and must not be stored by a cache, and RFC 9110 allows Retry-After as an HTTP date or a number of seconds. For remaining quota there is no RFC yet: as of November 2026, the IETF draft defining the RateLimit-Policy and RateLimit fields is at revision -11, dated May 2026, and is still a working-group Internet-Draft. Many APIs publish their own headers; GitHub, for example, documents x-ratelimit-remaining and returns 403 or 429 when its primary limit is exceeded.

Then decide what happens when Redis is slow or down. Failing open allows traffic and keeps the API available, at the risk of overload or abuse during the outage. Failing closed rejects traffic, which protects costly operations but turns a limiter outage into an API outage. Choose per endpoint: an endpoint that sends SMS codes should fail closed, while a read endpoint can fail open with a local fallback limit. Put a tight timeout on the limiter call and record every allow, deny, and fallback decision as a metric.

• RFC 6585, Section 4: 429 Too Many Requests
• IETF HTTPAPI draft: RateLimit header fields for HTTP
• GitHub Docs: rate limits for the REST API

Practice the five-minute answer

Say it in the same order every time: requirements, placement, algorithm with its weak spot, atomic shared state, client contract and failure mode. Then rehearse the follow-ups interviewers reach for. How do limits differ by plan? Store the rule per key and load it with the bucket. What about several regions? Either give each region a share of the global quota and accept some unfairness, or pay cross-region latency for a single store. Name the trade-off rather than claiming one design always wins.

Rewrite the token bucket from memory with an injected clock and check that it prints the same three lines. If you review system design with RecallDeck flashcards, make one card per decision, such as fail open versus fail closed, rather than one card per algorithm name. The articles below cover neighboring topics: retries that interact with your limiter, and the caching layer that often shares the same Redis deployment.

• 52 system design interview questions and answers
• API idempotency keys: safe retries under concurrency
• Redis cache-aside and cache stampede
• Practice the backend interview track

Quick answers

Frequently asked questions