Don't Hang Up Mid-Sentence: Graceful HTTP Shutdown

You push a deploy. The orchestrator sends your old pod a SIGTERM and starts the new one. If your server reacts to that signal by calling os.Exit or just letting the process fall over, every request that was in flight, the ones halfway through a database write or streaming a response, dies on the spot. The client sees a connection reset or a 502. Multiply by your request rate and every deploy becomes a little spike of errors.

Graceful shutdown is the fix: stop accepting new connections, but let the in-flight ones finish before you exit. Go's http.Server has this built in, and the interesting part is wiring it up correctly.

Shutdown, with a deadline and a fallback

func Run(ctx context.Context, srv *http.Server, ln net.Listener, drain time.Duration) error {
    serveErr := make(chan error, 1)
    go func() {
        err := srv.Serve(ln)
        if errors.Is(err, http.ErrServerClosed) {
            err = nil // expected: we asked it to stop
        }
        serveErr <- err
    }()

    select {
    case err := <-serveErr:
        return err          // server stopped on its own (e.g. listener died)
    case <-ctx.Done():      // someone asked us to shut down
    }

    shutdownCtx, cancel := context.WithTimeout(context.Background(), drain)
    defer cancel()
    if err := srv.Shutdown(shutdownCtx); err != nil {
        if errors.Is(err, context.DeadlineExceeded) {
            _ = srv.Close()  // drain timed out: force the stragglers closed
            return ErrForced
        }
        return err
    }
    return nil
}

There are three moving parts here, and each one is a lesson.

srv.Shutdown is the graceful part. It immediately stops the listener so no new connections are accepted, then waits for every active request to return before unblocking. That single call is the whole "let them finish" behavior.

The drain deadline is your safety net. Some request might be stuck (a slow upstream, a client that stopped reading). You cannot wait forever, because the orchestrator will hard-kill you after its own grace period anyway. So Shutdown gets a timeout. If draining finishes in time, great. If it does not, Shutdown returns DeadlineExceeded, and you fall back to srv.Close(), which yanks the remaining connections closed so you exit on your own terms instead of being killed mid-cleanup.

http.ErrServerClosed is success, not failure. This is the trap everyone hits once. srv.Serve always returns a non-nil error, and when you shut down cleanly that error is http.ErrServerClosed. Treating it as a real error means your clean shutdown logs a scary message and maybe exits non-zero. Check for it and translate it to nil.

How it fits a real deploy

In Kubernetes the sequence is: the pod is marked not-ready and removed from the Service endpoints (so new traffic stops arriving), then SIGTERM is delivered, then after terminationGracePeriodSeconds a SIGKILL if you are still alive. Your job is to catch SIGTERM, call into Run so it triggers Shutdown, and pick a drain timeout comfortably shorter than the grace period. Do that and rolling deploys produce zero client-visible errors. The old pod quietly finishes its last few requests and then steps off the stage.

Worth knowing

• Long-lived connections need a plan. Shutdown waits for active requests, but a streaming response or a WebSocket can stay "active" indefinitely. For those, propagate the shutdown context into the handler so it can wind itself down, or accept that the force-close fallback will catch them.
• Order matters in front of a load balancer. Stop receiving new traffic first (readiness off), then drain. If you drain while the load balancer is still sending you requests, you will keep getting new work you are trying to finish.

Graceful shutdown is not glamorous, but it is the line between "deploys are a non-event" and "we only deploy during low traffic because it spikes errors." A few lines of orchestration around one built-in method.

Build it yourself. Solve the Graceful HTTP Shutdown challenge on barehands and get graded on correctness and speed. No libraries, just the standard toolchain.