Thundering Herd, One Survivor: Deduplicating Work with Singleflight

Picture a popular product page. Its details are cached, the cache entry expires, and in that one millisecond a thousand concurrent requests all look, all miss, and all decide to rebuild it from the database. Your database, which was happily serving from cache a moment ago, now takes a thousand identical queries to the face. This is the cache stampede, or the thundering herd, and it takes services down in production all the time.

The fix is almost embarrassingly small. When many callers want the same thing at the same time, let one of them actually do the work, and have everyone else wait for that one result. That is singleflight.

The whole idea in one method

func (g *Group[T]) Do(key string, fn func() (T, error)) (T, error, bool) {
    g.mu.Lock()
    if c, ok := g.inflight[key]; ok {
        g.mu.Unlock()
        c.wg.Wait()                  // someone is already doing this; ride along
        return c.val, c.err, true    // shared == true
    }
    c := &call[T]{}
    c.wg.Add(1)
    g.inflight[key] = c              // I am the one doing the work
    g.mu.Unlock()

    c.val, c.err = fn()              // the single real call
    c.wg.Done()                      // wake everyone waiting

    g.mu.Lock()
    delete(g.inflight, key)          // next stampede starts fresh
    g.mu.Unlock()
    return c.val, c.err, false
}

The data structure is a map from key to an in-flight call, and each call carries a sync.WaitGroup with a count of one. The first caller for a key creates the call and runs fn. Every later caller finds the existing call, blocks on c.wg.Wait(), and when the worker calls Done(), they all wake up and return the same val and err. The boolean tells each caller whether the result was shared (true) or freshly computed (false).

That is it. One query hits the database. Nine hundred and ninety-nine goroutines park cheaply on a WaitGroup and then return the cached-by-proxy answer.

Why this beats just adding a lock

A plain mutex around the rebuild would also prevent concurrent queries, but it would serialize them: the second caller waits for the first to finish, then runs its own query, then the third runs its own, and so on. You still do a thousand queries, just one at a time. Singleflight does one query total and fans the single result out to everyone. The difference between "slow" and "fixed."

The deliberate design choices

• It dedups in-flight work, it does not cache. The moment fn returns, the key is deleted from the map. Singleflight is not a cache, it is a stampede guard that sits in front of one. The next request after the work completes will run fn again, which is exactly what you want when paired with a real cache that the worker just populated.
• Errors are shared too. If the one real call fails, all the waiters get that same error. Usually correct (the dependency is down for everyone), but worth knowing: one transient blip becomes everyone's blip for that instant.
• The lock is held briefly. Notice the mutex only guards the map, never the slow fn call. Waiters block on the WaitGroup, not the mutex, so the lock is never a bottleneck.

This pattern is the heart of Go's golang.org/x/sync/singleflight, which came out of groupcache. It shows up wherever identical expensive work can pile up: cache fills, DNS lookups, config fetches, token refreshes, "regenerate this thumbnail" jobs.

One gotcha worth a comment

Because the result is shared, a slow or hung fn makes every waiter slow or hung too. If your rebuild can wedge, give it a timeout inside fn, or use a variant that lets late callers opt out. The flip side of "everyone shares one result" is "everyone shares one stall."

The lesson is the nice kind: a map, a WaitGroup, and a delete turn a stampede into a single orderly query. Concurrency problems often look like they need heavy machinery, when really they need you to notice that a thousand callers are asking the exact same question.

Build it yourself. Solve the Singleflight challenge on barehands and get graded on correctness and speed. No libraries, just the standard toolchain.