Bursts Are a Feature: Rate Limiting with a Token Bucket

"100 requests per second" sounds like a single number, but it hides a real design choice. Does a client that sat idle for ten seconds get to fire 100 requests right now, or are they still pinned to one-every-10ms? Real traffic is bursty: a page loads twelve assets at once, then nothing for a while. A limiter that forbids all bursts feels broken; a limiter that allows infinite bursts is not a limiter. The token bucket threads that needle.

The mental model

Imagine a bucket that holds up to burst tokens. Every request must take one token to proceed. Tokens drip back into the bucket at a steady rps rate, and the bucket never overflows past burst. So:

• A client that has been quiet finds a full bucket and can burst up to burst requests immediately.
• Sustained traffic can only go as fast as tokens refill, which is rps.
• The bucket size is the burst allowance, the rate is the steady ceiling. Two knobs, clearly separated.

In Go, the bucket is a channel

This is one of those problems where Go's concurrency primitives line up so well it feels like cheating. A buffered channel of capacity burst is the bucket. Its buffered slots are tokens.

type Limiter struct {
    tokens chan struct{}
}

func New(rps, burst int) *Limiter {
    l := &Limiter{tokens: make(chan struct{}, burst)}
    for range burst {
        l.tokens <- struct{}{} // start full
    }
    go func() {
        t := time.NewTicker(time.Second / time.Duration(rps))
        defer t.Stop()
        for range t.C {
            select {
            case l.tokens <- struct{}{}: // drip one token in
            default:                     // bucket full, drop it on the floor
            }
        }
    }()
    return l
}

A background goroutine ticks every 1/rps seconds and tries to add a token. The select with a default is the "never overflow" rule: if the channel buffer is full, the send fails instantly and the tick is discarded. No counter, no clock arithmetic, no manual "how many tokens should there be by now" math. The channel buffer counts for you.

Taking a token has two natural flavors:

func (l *Limiter) Allow() bool {
    select {
    case <-l.tokens:
        return true   // got a token
    default:
        return false  // empty bucket, reject now
    }
}

func (l *Limiter) Wait(ctx context.Context) error {
    select {
    case <-l.tokens:
        return nil
    case <-ctx.Done():
        return ctx.Err() // caller gave up or timed out
    }
}

Allow is the non-blocking "reject if over limit" used at an API edge. Wait is the blocking "queue politely until a token frees up," and because it selects on ctx.Done(), a caller can bound how long it is willing to wait. Same bucket, two policies.

Why not just a fixed window?

The naive alternative is "count requests per one-second window, reset the counter each second." It is simpler, and it has a famous failure: a client can send limit requests in the last millisecond of one window and limit more in the first millisecond of the next, doubling your intended rate at the boundary. The token bucket has no boundary to exploit because refill is continuous. It smooths instead of resetting.

The takeaways

• Two knobs, two meanings. burst is how much saved credit a quiet client may spend at once; rps is the long-run ceiling. Tune them independently.
• The channel is the state. Using a buffered channel as the bucket means the limiter is correct under concurrency for free: a token is taken by exactly one receiver, no lock required.
• Pick your policy at the call site. Allow to shed load, Wait to shape it.

It is a tiny amount of code for a primitive that protects databases, upstream APIs, and your own service from a stampede of clients all being reasonable at the same time.

Build it yourself. Solve the Token-Bucket Rate Limiter challenge on barehands and get graded on correctness and speed. No libraries, just the standard toolchain.