Fail Fast on Purpose: The Circuit Breaker

A dependency goes down. Maybe the payments service is overloaded. Your code calls it, waits for a 30-second timeout, fails, and (if you are unlucky) retries. Now multiply that by your traffic: thousands of goroutines all parked on doomed calls, each holding a connection and a timeout, your own latency climbing, your own health checks starting to fail. A single sick service has made you sick too. This is a cascading failure, and the circuit breaker is the standard guard against it.

The idea is borrowed straight from electrical circuits. When current spikes, the breaker trips and the circuit goes dead, on purpose, to protect everything downstream. You flip it back later once things look safe.

Three states

A software breaker is a little state machine with three states:

• Closed: normal. Calls pass through. Failures are counted.
• Open: tripped. Calls are rejected instantly with an error, no attempt made. This is the "fail fast" part. After a cooldown, it gets a chance to recover.
• Half-open: testing the water. Exactly one probe call is allowed through. If it succeeds, close the breaker and resume normal service. If it fails, open again and wait another cooldown.

func (b *Breaker) Do(fn func() error) error {
    ok, isProbe := b.allow()   // decide under the lock
    if !ok {
        return ErrOpen         // fail fast, do not even call fn
    }

    err := fn()

    b.mu.Lock()
    defer b.mu.Unlock()
    if isProbe {
        b.probing = false
        if err == nil {
            b.state = stateClosed   // recovered
            b.failures = 0
        } else {
            b.state = stateOpen     // still broken, back off
            b.openUntil = time.Now().Add(b.cooldown)
        }
        return err
    }
    // closed-state bookkeeping
    if err == nil {
        b.failures = 0
        return nil
    }
    b.failures++
    if b.failures >= b.maxFailures {
        b.state = stateOpen          // trip
        b.openUntil = time.Now().Add(b.cooldown)
        b.failures = 0
    }
    return err
}

The subtle part: exactly one probe

The transition that people get wrong is half-open. When the cooldown elapses, you do not want to suddenly let all the queued-up traffic through, because if the dependency is still down you just stampeded it again and re-tripped. You want a single, gentle probe.

case stateOpen:
    if time.Now().Before(b.openUntil) {
        return false, false       // still cooling down: reject
    }
    b.state = stateHalfOpen       // cooldown over
    b.probing = true
    return true, true             // admit exactly ONE probe

case stateHalfOpen:
    if b.probing {
        return false, false       // a probe is already out; everyone else waits
    }

The probing flag is the bouncer. The first call after the cooldown becomes the probe and flips probing to true; every other call sees probing is set and gets rejected until the probe resolves. One request decides the fate of the breaker, and the herd is held back until it does.

Why this is the right behavior

• It protects the dependency. A struggling service that gets a break, instead of a constant hammering, can actually recover. Without a breaker, your retries are a denial-of-service attack on something already on its knees.
• It protects you. Failing in microseconds with ErrOpen beats failing in 30 seconds with a timeout. Your goroutines stay free, your latency stays flat, the failure stops at the boundary instead of climbing your call stack.
• It is honest fast. A breaker turns "slowly, painfully unavailable" into "immediately and clearly unavailable," which is a much better thing to hand to a caller (or a fallback).

This is the pattern behind Hystrix, resilience4j, and the resilience layers of most service meshes. The three states and the single-probe rule are the whole essence; everything else (rolling failure windows, percentages, half-open concurrency limits) is refinement on top.

The mindset shift is the valuable bit: sometimes the most reliable thing your code can do is refuse to try.

Build it yourself. Solve the Circuit Breaker challenge on barehands and get graded on correctness and speed. No libraries, just the standard toolchain.