Concurrency in Go is one of its strongest selling points — until you actually have to handle errors across goroutines. The naive approach (launch N goroutines, collect results through a channel, hope nothing panics) works fine for toy examples. In production it falls apart fast: you leak goroutines when something fails early, you swallow errors, and cancellation is bolted on as an afterthought.
The errgroup package from golang.org/x/sync solves the structural problem. But even with errgroup, a lot of people miss the bounded-concurrency half of the story and accidentally DoS their own database by spawning 10,000 goroutines in a loop.
This article covers both halves: using errgroup correctly, and pairing it with a semaphore to cap parallelism — with cancellation that actually works when something goes wrong.
GitHub repo for the x/sync package: https://github.com/golang/sync
Why the WaitGroup + channel pattern is not enough
Before reaching for errgroup, you’ve almost certainly written something like this:
var wg sync.WaitGroup
errs := make(chan error, len(items))
for _, item := range items {
wg.Add(1)
go func(item Item) {
defer wg.Done()
if err := process(item); err != nil {
errs <- err
}
}(item)
}
wg.Wait()
close(errs)
for err := range errs {
// now what? you have N errors, the loop already ran
}
Several things are wrong here:
You get all errors, but you wanted the first one. If 800 out of 1000 items fail, logging 800 errors is noise. You care about whether the batch succeeded or not.
There’s no cancellation. If item 3 fails, items 4–1000 keep running anyway. Their work is wasted, their connections stay open, their database queries keep firing.
The buffered channel size is a guess. If you undersize it, goroutines block on send and the wg.Wait() never returns. Classic deadlock.
Semaphore? What semaphore? Nothing stops you from spawning more goroutines than your downstream can handle.
errgroup fixes the first two issues cleanly. The third you can fix yourself with a semaphore channel. Let’s go through it step by step.
errgroup basics
Install the package:
go get golang.org/x/sync/errgroup
The simplest usage:
package main
import (
"fmt"
"golang.org/x/sync/errgroup"
)
func main() {
var g errgroup.Group
g.Go(func() error {
return fetchUser(1)
})
g.Go(func() error {
return fetchUser(2)
})
g.Go(func() error {
return fetchUser(3)
})
if err := g.Wait(); err != nil {
fmt.Println("failed:", err)
}
}
g.Wait() blocks until all goroutines finish and returns the first non-nil error. The rest are discarded. That’s intentional — you get a clean Go-style single error return, not a slice of errors that you have to iterate over.
If you need all errors, errgroup is not the right tool. Look at golang.org/x/sync/errgroup‘s source or write a small multi-error collector yourself.
Adding context cancellation
The real power kicks in when you create the group with a context:
package main
import (
"context"
"fmt"
"time"
"golang.org/x/sync/errgroup"
)
func main() {
ctx := context.Background()
// WithContext returns a derived context that gets cancelled
// when the first goroutine returns a non-nil error.
g, ctx := errgroup.WithContext(ctx)
g.Go(func() error {
return doWork(ctx, "task-1", 100*time.Millisecond)
})
g.Go(func() error {
// This one fails fast — ctx will be cancelled for the others.
return doWork(ctx, "task-2", 10*time.Millisecond)
})
g.Go(func() error {
return doWork(ctx, "task-3", 200*time.Millisecond)
})
if err := g.Wait(); err != nil {
fmt.Println("error:", err)
}
}
func doWork(ctx context.Context, name string, d time.Duration) error {
select {
case <-time.After(d):
fmt.Println(name, "done")
return nil
case <-ctx.Done():
fmt.Println(name, "cancelled")
return ctx.Err()
}
}
When task-2 finishes first and returns an error (or when any goroutine errors), the context is cancelled. task-1 and task-3 are listening on ctx.Done(), so they exit immediately instead of running to completion.
This is the pattern you want for any fan-out operation: HTTP calls, database queries, filesystem operations. One failure should stop the rest.
Gotcha: errgroup.WithContext cancels the context when the first error is returned, and also when all goroutines finish successfully. The context is not a long-lived one — don’t store it outside the g.Wait() call.
Gotcha: If your goroutine ignores the context (doWork that just does time.Sleep instead of select), cancellation does nothing. Your goroutines must actively check ctx.Done(). No magic here.
The bounded concurrency problem
Consider this code:
g, ctx := errgroup.WithContext(context.Background())
for _, url := range tenThousandURLs {
url := url // capture loop variable (pre-Go 1.22)
g.Go(func() error {
return fetch(ctx, url)
})
}
return g.Wait()
This spawns 10,000 goroutines simultaneously. Each one holds an open TCP connection. Your target server (or your own database connection pool) will see a thundering herd. If you’re calling an external API with rate limits, you’ll get 429s. If it’s Postgres, you’ll hit max_connections and start getting errors.
The fix is a semaphore — a buffered channel used as a token bucket:
// sem is a channel of empty structs acting as a semaphore.
// Buffer size = max concurrent goroutines.
sem := make(chan struct{}, maxConcurrency)
Acquiring a token: sem <- struct{}{} (blocks if full)
Releasing a token: <-sem
Combine it with errgroup:
package main
import (
"context"
"fmt"
"net/http"
"time"
"golang.org/x/sync/errgroup"
)
const maxConcurrency = 10
func fetchURLs(ctx context.Context, urls []string) error {
g, ctx := errgroup.WithContext(ctx)
sem := make(chan struct{}, maxConcurrency)
for _, url := range urls {
url := url // loop variable capture — required before Go 1.22
// Acquire semaphore token — respect context cancellation.
// If ctx is cancelled before we get a slot, we bail out
// without launching the goroutine at all.
select {
case sem <- struct{}{}:
case <-ctx.Done():
return g.Wait()
}
g.Go(func() error {
defer func() { <-sem }() // release slot when done
return fetch(ctx, url)
})
}
return g.Wait()
}
func fetch(ctx context.Context, url string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return fmt.Errorf("build request %s: %w", url, err)
}
client := &http.Client{Timeout: 5 * time.Second}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("fetch %s: %w", url, err)
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
return fmt.Errorf("fetch %s: HTTP %d", url, resp.StatusCode)
}
return nil
}
Walk through what happens here:
- We allocate a semaphore channel with capacity
maxConcurrency. - Before launching each goroutine, we try to put a token in the channel. If all
maxConcurrencyslots are taken,sem <- struct{}{}blocks until one goroutine finishes and releases its token. - The
selectwithctx.Done()means: if the context is already cancelled (because a previous goroutine errored), we stop the loop entirely rather than continue to enqueue more work. - Inside the goroutine,
defer func() { <-sem }()releases the token exactly once, regardless of whether the function returns an error or nil.
Gotcha: The defer func() { <-sem }() pattern with a closure is intentional. If you write defer <-sem directly, Go evaluates the channel expression immediately at the defer statement, not at the time of release. It compiles but behaves differently in more complex setups. Use the closure form.
Gotcha: The semaphore acquire (sem <- struct{}{}) happens in the launching goroutine (the loop), not inside the spawned goroutine. This means the loop itself blocks when all slots are busy — you never hold more than maxConcurrency goroutines in-flight at once. If you move the acquire inside g.Go(...), you just spawn all goroutines immediately and the semaphore does nothing useful.
Production pattern: worker pool with errgroup
For long-running services or processing queues, a static worker pool is cleaner than spawning per-item goroutines:
package main
import (
"context"
"fmt"
"golang.org/x/sync/errgroup"
)
// ProcessBatch processes items using a fixed pool of workers.
// Returns the first error encountered; workers respect context cancellation.
func ProcessBatch(ctx context.Context, items []Item, workers int) error {
g, ctx := errgroup.WithContext(ctx)
itemCh := make(chan Item)
// Feed goroutine: sends items into the channel,
// stops if context is cancelled.
g.Go(func() error {
defer close(itemCh)
for _, item := range items {
select {
case itemCh <- item:
case <-ctx.Done():
return ctx.Err()
}
}
return nil
})
// Worker goroutines: fixed count, pull from shared channel.
for i := 0; i < workers; i++ {
g.Go(func() error {
for item := range itemCh {
if err := processItem(ctx, item); err != nil {
return fmt.Errorf("item %v: %w", item.ID, err)
}
}
return nil
})
}
return g.Wait()
}
This is the producer-consumer pattern with errgroup as the lifecycle manager. A few things to note:
- Closing
itemChin the producer’sdeferis critical. When the producer exits (normally or via cancellation), workers will drain the channel and then exit theirrangeloop cleanly. - Workers don’t need a semaphore because there’s a fixed number of them — the channel itself is the backpressure mechanism.
- If any worker returns an error, the context is cancelled. The producer will see
ctx.Done()and stop sending. Workers waiting onitemChwill drain remaining items and exit, but sincectxis passed intoprocessItem, they can also short-circuit mid-item.
When to use the semaphore pattern vs. the worker pool:
- Semaphore: you have a slice of items up front, items vary in duration, and you want to saturate N slots without pre-allocating goroutines.
- Worker pool: you have a stream of incoming work, you want strict goroutine count control, or your processing state is expensive to initialize per-goroutine (e.g., a database connection or a gRPC client).
Collecting results alongside errors
errgroup only propagates errors, not return values. For results, you need a slice pre-allocated to the same length as your input, with each goroutine writing to its own index. No mutex needed if each goroutine owns exactly one slot:
func fetchAll(ctx context.Context, urls []string) ([]Result, error) {
results := make([]Result, len(urls))
g, ctx := errgroup.WithContext(ctx)
sem := make(chan struct{}, 20)
for i, url := range urls {
i, url := i, url // capture
select {
case sem <- struct{}{}:
case <-ctx.Done():
return nil, g.Wait()
}
g.Go(func() error {
defer func() { <-sem }()
r, err := fetch(ctx, url)
if err != nil {
return err
}
results[i] = r // safe: each goroutine owns index i
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
Gotcha: This is only safe because each goroutine writes to a unique index. The moment two goroutines can write to the same index (e.g., because you’re appending to a shared slice), you need a mutex. Pre-allocated fixed-size slices with index ownership are your friend.
SetLimit: the built-in semaphore (Go 1.20+)
Since x/sync v0.1.0 (released alongside Go 1.20), errgroup has SetLimit built in:
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(10) // max 10 goroutines in flight
for _, url := range urls {
url := url
g.Go(func() error {
return fetch(ctx, url)
})
}
return g.Wait()
With SetLimit, g.Go blocks the caller until a slot is available — the same behavior as the manual semaphore, but with less ceremony. Internally it’s implemented almost exactly the same way.
Use SetLimit when:
- You’re on a recent enough Go version (check your
go.mod). - The blocking-on-Go behavior is what you want.
Stick with the manual semaphore when:
- You need to respect context cancellation before launching goroutines (the
selecttrick from earlier).SetLimit‘sg.Godoesn’t accept a context — it blocks unconditionally. - You’re on an older codebase locked to an older version of
x/sync.
Gotchas summary
Loop variable capture. Before Go 1.22, for _, item := range items { g.Go(func() error { use(item) }) } is a bug — all goroutines capture the same variable. The fix: item := item before the g.Go call. Go 1.22+ fixes this in the language spec, but if your go.mod says go 1.20, the old behavior applies.
The derived context lives longer than you think. errgroup.WithContext creates a child context. If you pass this context to a connection pool or cache that stores it, that context gets cancelled when the group finishes — even on success. Create a fresh context for anything that should outlive the group.
g.Wait() returns only one error. If you need to know all failure reasons (e.g., for partial retry logic), errgroup is the wrong tool. Write a custom collector with a mutex-protected slice or use a buffered error channel.
Panics are not errors. errgroup does not recover panics. A panicking goroutine will crash the whole process. If you’re calling untrusted code or handling external plugins, wrap with a recover-and-return-error pattern inside your goroutine function.
Don’t reuse a group after Wait. Once g.Wait() returns, the group is done. Creating a new group is cheap — just do it.
Putting it all together
Here’s a complete, production-ready function that crawls a list of URLs with bounded concurrency, context cancellation, and result collection — using SetLimit for Go 1.22+ projects:
package crawler
import (
"context"
"fmt"
"io"
"net/http"
"time"
"golang.org/x/sync/errgroup"
)
type PageResult struct {
URL string
Body []byte
}
// Crawl fetches all URLs concurrently, capping parallelism at concurrency.
// Returns results in the same order as the input slice.
// Cancels remaining fetches on the first error.
func Crawl(ctx context.Context, urls []string, concurrency int) ([]PageResult, error) {
results := make([]PageResult, len(urls))
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(concurrency)
client := &http.Client{Timeout: 10 * time.Second}
for i, url := range urls {
i, url := i, url
g.Go(func() error {
body, err := fetchBody(ctx, client, url)
if err != nil {
return fmt.Errorf("crawl %s: %w", url, err)
}
results[i] = PageResult{URL: url, Body: body}
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
func fetchBody(ctx context.Context, client *http.Client, url string) ([]byte, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
}
return io.ReadAll(resp.Body)
}
Caller:
results, err := crawler.Crawl(ctx, urls, 15)
if err != nil {
log.Fatal(err)
}
for _, r := range results {
fmt.Printf("%s: %d bytes\n", r.URL, len(r.Body))
}
Clean, no goroutine leaks, no manual WaitGroup, no deadlocks from undersized channels.
Choosing your concurrency limit
There’s no universal number. Some heuristics that work in practice:
- External HTTP APIs: 5–20. Most rate-limited APIs will complain above this. Check the API docs first.
- Internal microservices: 50–200. Your service mesh or load balancer is the bottleneck; measure under load.
- Database queries: match your connection pool size. If
pgxpoolhas 10 connections, 10 concurrent goroutines is the ceiling. More is pointless — they’ll queue in the pool. - CPU-bound work:
runtime.NumCPU(). More goroutines just adds context-switching overhead. - Disk I/O: depends on storage (SSD vs. HDD, NVMe vs. NAS). Start at
runtime.NumCPU() * 2, measure, adjust.
Always make the concurrency limit configurable — hardcoding it means a flag to redeploy every time ops wants to tune throughput.
errgroup is one of those packages that solves a real, recurring problem with almost no surface area. Once you understand the WithContext + semaphore combination, you’ll stop reaching for raw goroutines and channels for fan-out patterns. The code gets shorter, the error handling is correct by construction, and cancellation propagates the way it should.