Every distributed system eventually develops the same slow-motion disaster: an upstream client gives up after 5 seconds, but the downstream services keep grinding away for another 30. The database is running queries nobody will ever read. Connection pools are exhausted. CPU is pegged. The request is dead, but nobody told the machinery.
This is the timeout propagation problem, and it bites everyone eventually. The fix isn’t complicated — but it requires being deliberate about every layer, not just slapping a timeout on your HTTP client and calling it done.
Why Your Timeouts Are Probably Broken
Let’s be honest about what "setting timeouts" looks like in most codebases:
# Service A (edge/API gateway)
response = requests.get("http://service-b/data", timeout=5)
# Service B (middleware)
response = requests.get("http://service-c/data", timeout=5)
# Service C (calls database)
cursor.execute(query, timeout=5)
Looks fine, right? Three services, each with a 5-second timeout. What could go wrong?
The problem: these are independent timeouts, not a budget. Service A gives up at T+5s. But Service B doesn’t know that — it started its own 5-second clock from its receive time. Which is maybe T+0.2s. So Service B will wait until T+5.2s. Service C until T+5.4s. And the database query runs to T+10.4s, cheerfully returning results to a connection that has been dead for 5 seconds.
In a real microservice chain with 6-8 hops, you can have zombie work running 30-40 seconds after the original caller hung up. This is how you get cascading failures: every timeout causes a pile-up of orphaned inflight requests that eat connections, goroutines, and thread pool slots.
The Right Model: Deadline Budgets
The correct mental model is a deadline, not a duration. The edge sets an absolute point in time by which the entire request must complete. Every downstream hop gets a remaining budget: deadline - now. If that budget hits zero or goes negative before you even start the downstream call, you fail fast and return immediately. No work done, no resources consumed.
T=0ms Edge receives request, sets deadline = now + 5000ms
T=10ms Service B receives call, remaining = 4990ms
T=40ms Service C receives call, remaining = 4960ms
T=4990ms Edge deadline fires — *everything* stops
This is what gRPC does natively with its deadline metadata. HTTP has no built-in mechanism, but you can build it yourself in 20 lines of middleware.
Implementing It in Go
Go’s context package was designed for exactly this. context.WithDeadline creates a context that auto-cancels at a specific moment. Pass it through every layer. Never discard it.
Edge: Setting the Budget
func (s *APIServer) HandleRequest(w http.ResponseWriter, r *http.Request) {
// Total budget for this entire request tree: 5 seconds
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()
// Propagate via header so downstream HTTP services can read it
deadline, _ := ctx.Deadline()
r = r.WithContext(ctx)
result, err := s.serviceB.GetData(ctx, r.URL.Query().Get("id"))
if err != nil {
if errors.Is(err, context.DeadlineExceeded) {
http.Error(w, "upstream timeout", http.StatusGatewayTimeout)
return
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
json.NewEncoder(w).Encode(result)
}
Middleware: Inheriting the Deadline
When Service B receives the call, it needs to pick up the deadline from the incoming context rather than creating a fresh one.
func (c *ServiceBClient) GetData(ctx context.Context, id string) (*Data, error) {
// Check remaining budget BEFORE doing any work
deadline, ok := ctx.Deadline()
if ok && time.Until(deadline) < 50*time.Millisecond {
// Less than 50ms left — not worth starting, fail fast
return nil, fmt.Errorf("insufficient budget: %w", context.DeadlineExceeded)
}
// Pass the same ctx — it carries the original deadline
req, err := http.NewRequestWithContext(ctx, "GET",
fmt.Sprintf("http://service-c/item/%s", id), nil)
if err != nil {
return nil, err
}
resp, err := c.httpClient.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
// ...
}
The key line is http.NewRequestWithContext(ctx, ...). The Go HTTP client respects context cancellation. If the deadline fires mid-flight, the client aborts the TCP connection immediately.
Database Layer: The Most Common Failure Point
This is where people most often break the chain. ORMs and database drivers frequently have their own timeout configuration that’s divorced from request context. You must explicitly pass the context.
With database/sql:
func (r *Repo) FetchItem(ctx context.Context, id string) (*Item, error) {
// QueryContext uses the context for cancellation — not QueryRow
row := r.db.QueryRowContext(ctx,
"SELECT id, name, created_at FROM items WHERE id = $1", id)
var item Item
if err := row.Scan(&item.ID, &item.Name, &item.CreatedAt); err != nil {
if errors.Is(ctx.Err(), context.DeadlineExceeded) {
return nil, fmt.Errorf("db query exceeded budget: %w", err)
}
return nil, err
}
return &item, nil
}
Never use QueryRow when you have a context. Always QueryRowContext. Same for Exec/ExecContext, Query/QueryContext.
With Redis (go-redis):
val, err := r.client.Get(ctx, key).Result()
// The ctx deadline is respected — connection returns early if fired
With MongoDB:
result := r.collection.FindOne(ctx, bson.M{"_id": id})
// Same pattern — context flows through
Propagating Deadlines Over HTTP
When you call a non-gRPC service, the context deadline lives in-process. You need to serialize it into the HTTP request so the other service can reconstruct it.
There’s no standard HTTP header for this — gRPC has grpc-timeout, but plain HTTP has nothing. Two approaches are common in the wild:
Option 1: Absolute deadline as a Unix timestamp header
// Outbound middleware — attach deadline to all egress calls
func DeadlineTransport(next http.RoundTripper) http.RoundTripper {
return roundTripperFunc(func(req *http.Request) (*http.Response, error) {
if deadline, ok := req.Context().Deadline(); ok {
// millisecond precision is enough
req.Header.Set("X-Request-Deadline",
strconv.FormatInt(deadline.UnixMilli(), 10))
}
return next.RoundTrip(req)
})
}
// Inbound middleware — reconstruct context from header
func DeadlineMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if raw := r.Header.Get("X-Request-Deadline"); raw != "" {
ms, err := strconv.ParseInt(raw, 10, 64)
if err == nil {
deadline := time.UnixMilli(ms)
if time.Until(deadline) <= 0 {
// Already expired before we even started
http.Error(w, "request deadline already passed",
http.StatusGatewayTimeout)
return
}
ctx, cancel := context.WithDeadline(r.Context(), deadline)
defer cancel()
r = r.WithContext(ctx)
}
}
next.ServeHTTP(w, r)
})
}
Option 2: Relative remaining time in milliseconds
// Simpler, avoids clock skew issues between hosts
if deadline, ok := req.Context().Deadline(); ok {
remaining := time.Until(deadline).Milliseconds()
if remaining > 0 {
req.Header.Set("X-Timeout-Ms", strconv.FormatInt(remaining, 10))
}
}
Relative time is safer across machines with imperfect NTP sync. A 10ms clock skew on a 100ms budget is a 10% error — annoying. On a 5000ms budget, it’s irrelevant.
gRPC: This Already Works, If You Don’t Break It
gRPC propagates deadlines natively via the grpc-timeout metadata header. The server stub auto-cancels the handler’s context when the deadline fires. You get this for free — the only way to break it is to not pass the context.
// WRONG — creates a new independent context, breaks the chain
ctx := context.Background()
resp, err := client.GetUser(ctx, &pb.GetUserRequest{Id: id})
// RIGHT — pass the incoming context downstream
resp, err := client.GetUser(ctx, &pb.GetUserRequest{Id: id})
// where ctx came from the gRPC server handler parameter
gRPC client interceptors are the right place to enforce minimum-budget checks:
func MinBudgetInterceptor(minBudget time.Duration) grpc.UnaryClientInterceptor {
return func(ctx context.Context, method string, req, reply interface{},
cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
if deadline, ok := ctx.Deadline(); ok {
if remaining := time.Until(deadline); remaining < minBudget {
return status.Errorf(codes.DeadlineExceeded,
"insufficient budget %v for call to %s", remaining, method)
}
}
return invoker(ctx, method, req, reply, cc, opts...)
}
}
Gotchas
Gotcha #1: Setting the same timeout at every layer
The classic mistake. If every layer has a 5s timeout and you have 4 layers, the maximum total time is 20 seconds from the user’s perspective when everything times out in sequence. Pick one budget at the edge and propagate it.
Gotcha #2: Creating a new context at the wrong layer
// This silently breaks the chain
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
context.Background() throws away any deadline from the incoming request. Use the parent context as the base. In HTTP handlers, that’s r.Context(). In gRPC handlers, it’s the ctx parameter passed to you.
Gotcha #3: Not checking ctx.Err() before expensive operations
func processItems(ctx context.Context, items []Item) error {
for _, item := range items {
// If ctx is already cancelled, bail immediately
if err := ctx.Err(); err != nil {
return fmt.Errorf("context cancelled mid-loop: %w", err)
}
if err := processItem(ctx, item); err != nil {
return err
}
}
return nil
}
If you iterate over 10,000 items and the context fires at item 5,000, you want to stop immediately — not after the loop finishes.
Gotcha #4: Background goroutines that outlive the request
// WRONG — goroutine has no deadline, runs forever
go func() {
doHeavyWork(context.Background())
}()
// RIGHT — pass the request context
go func(ctx context.Context) {
doHeavyWork(ctx)
}(ctx)
Any goroutine spawned to serve a request should inherit the request context. Goroutines with context.Background() that were started inside a request handler are timeout leaks.
Gotcha #5: Insufficient buffer for network overhead
If the remaining budget is 2ms, you can’t make a TCP connection to a remote host. Your minimum-budget check should account for realistic network RTT. For local-network calls, 20-50ms is a reasonable floor. For cross-region calls, 100-200ms.
Gotcha #6: Propagating dead deadlines from retry logic
// If the first attempt took 4.9s out of a 5s budget,
// retrying with the same context will immediately DeadlineExceeded.
// Either use a fresh context per attempt (with your own shorter timeout)
// or don't retry on deadline exceeded at all.
for attempt := 0; attempt < 3; attempt++ {
if ctx.Err() != nil {
break // Budget exhausted, stop retrying
}
result, err = call(ctx)
if err == nil {
break
}
}
Production-Ready Patterns
Pattern 1: Always log the remaining budget at service boundaries
When a timeout fires and you’re debugging why, you want to know how much budget arrived at each hop. Structured logging at ingress:
func LogBudget(logger *slog.Logger) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if deadline, ok := r.Context().Deadline(); ok {
remaining := time.Until(deadline)
logger.InfoContext(r.Context(), "request budget",
"remaining_ms", remaining.Milliseconds(),
"path", r.URL.Path,
)
}
next.ServeHTTP(w, r)
})
}
}
Pattern 2: Never return a 500 for a deadline exceeded — use 504
func httpStatusFromError(err error) int {
if errors.Is(err, context.DeadlineExceeded) ||
errors.Is(err, context.Canceled) {
return http.StatusGatewayTimeout // 504
}
return http.StatusInternalServerError // 500
}
504 vs 500 matters for clients and SLO dashboards. A 504 is a timeout. A 500 is a crash. They have different remediation paths.
Pattern 3: Reserve a slice of budget for cleanup and response writing
Some operations need a bit of headroom after the main work: flushing buffers, closing transactions, writing the response body. A common pattern:
const cleanupBudget = 200 * time.Millisecond
func (s *Server) handle(ctx context.Context) error {
// Work with a slightly shorter deadline to leave room for cleanup
workCtx, cancel := context.WithTimeout(ctx,
timeUntilDeadline(ctx) - cleanupBudget)
defer cancel()
result, err := s.doWork(workCtx)
// Even if workCtx expired, the outer ctx still has 200ms
return s.writeResponse(ctx, result, err)
}
func timeUntilDeadline(ctx context.Context) time.Duration {
deadline, ok := ctx.Deadline()
if !ok {
return 30 * time.Second // No deadline set, use a safe fallback
}
return time.Until(deadline)
}
Pattern 4: Set conservative database statement timeouts as a backstop
Even with correct context propagation, a runaway query that ignores cancellation signals (some drivers have this problem) can hold a connection indefinitely. Set server-side statement timeouts in your database config as a hard backstop:
-- PostgreSQL: kill any query running longer than 10 seconds
SET statement_timeout = '10s';
-- Or per connection in your pool config
// With pgx, you can set this per connection at open time
config.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error {
_, err := conn.Exec(ctx, "SET statement_timeout = '10000'")
return err
}
This catches the cases where context cancellation doesn’t reach the database side: network partitions, driver bugs, queries that check cancellation less frequently.
Checklist Before Shipping
Run through this for any service that makes downstream calls:
- Context is threaded from the inbound handler all the way to the DB/cache call — no
context.Background()in the hot path - Deadline is serialized into outbound HTTP headers and deserialized at ingress
- Minimum-budget guard fires before starting expensive operations
- 504 is returned (not 500) when
context.DeadlineExceededbubbles up - Background goroutines that serve a request inherit the request context
- Database calls use
*Contextvariants (QueryContext,ExecContext) - Integration tests include a case where the upstream deadline fires mid-request
The test case is important. The most confident way to validate your propagation is to write a test that sets a 50ms deadline and injects a 200ms artificial delay at each layer in turn. If the timeout fires correctly and returns 504 instead of completing, you’re done. If it hangs for 200ms before returning — you found the break in the chain.
Timeout correctness is one of those things that looks fine in nominal load testing and completely falls apart the moment you hit a slow third-party API or a database under write pressure. Get it right before you need it.