errgroup

Run related tasks at the same time, wait for all of them, and get back the first error any of them returned.

Syntaxvar g errgroup.Group; g.Go(func() error { return nil }); err := g.Wait()

sync.WaitGroup waits for a set of goroutines to finish, but a goroutine cannot return anything, so there is nowhere for its error to go. errgroup.Group fills that gap. It waits like a WaitGroup and hands back the first error.

It lives in golang.org/x/sync/errgroup, a separate module from the standard library. Add it with go get golang.org/x/sync.

How it works

g.Go(f func() error) runs f in a new goroutine. g.Wait() error blocks until every function passed to Go has returned, then returns the first non-nil error from them, or nil if there was none. The zero errgroup.Group is ready to use.

errgroup.WithContext(ctx) returns (*Group, context.Context). That returned context is cancelled the first time one of the functions returns a non-nil error, or when Wait returns, whichever is first. Pass it to the work inside each function and the siblings stop as soon as one fails.

g.SetLimit(n) caps how many of the group's goroutines run at once. Go then blocks until a slot is free. g.TryGo(f) returns false instead of blocking when the group is full.

What trips people up

Wait gives you one error. The rest are discarded, so log inside each function if you need them all. The group gives you no help with shared data: two functions writing the same variable is still a data race.

Example

Example · go
package main

import (
	"errors"
	"fmt"

	"golang.org/x/sync/errgroup"
)

// fetch stands in for a network call. Record 2 is missing.
func fetch(id int) (string, error) {
	if id == 2 {
		return "", errors.New("record 2 not found")
	}
	return fmt.Sprintf("record %d", id), nil
}

func main() {
	var g errgroup.Group
	// Each task writes only its own slot, so no lock is needed.
	results := make([]string, 4)
	for i := range results {
		g.Go(func() error {
			v, err := fetch(i)
			results[i] = v
			return err
		})
	}
	// Wait returns the first non-nil error any task returned.
	if err := g.Wait(); err != nil {
		fmt.Println("error:", err)
	}
	for _, r := range results {
		if r != "" {
			fmt.Println(r)
		}
	}
}

// Output:
// error: record 2 not found
// record 0
// record 1
// record 3

When to use it

  • Fanning out three independent database queries for one HTTP response and failing the whole response if any query fails.
  • Checking a list of URLs or S3 objects concurrently with a cap of eight in flight.
  • Starting an HTTP server, a metrics server and a background worker in main, and shutting all of them down when the first one returns an error.

More examples

WithContext, so one failure stops the others

This is the everyday use: one HTTP request needs three independent queries, so run them together and give up on all of them if any fails. Shadowing <code>ctx</code> with the group's context is deliberate, because a child that keeps using the outer context will not be cancelled. Wrapping each error with <code>fmt.Errorf</code> and <code>%w</code> matters here, since Wait returns the error with no clue which task produced it.

Example · go
func loadPage(ctx context.Context, db *sql.DB, id int) (*Page, error) {
	g, ctx := errgroup.WithContext(ctx)
	var page Page

	g.Go(func() error {
		// ctx here is the group's context, not the caller's.
		p, err := loadProfile(ctx, db, id)
		if err != nil {
			return fmt.Errorf("profile: %w", err)
		}
		page.Profile = p
		return nil
	})
	g.Go(func() error {
		posts, err := loadPosts(ctx, db, id)
		if err != nil {
			return fmt.Errorf("posts: %w", err)
		}
		page.Posts = posts
		return nil
	})

	if err := g.Wait(); err != nil {
		return nil, err
	}
	return &page, nil
}

SetLimit to avoid opening a thousand connections

Starting a goroutine per item is cheap, but the resource behind it is not: ten thousand goroutines means ten thousand sockets or database connections. <code>SetLimit</code> makes the group a worker pool with no extra machinery. Note that the loop variables <code>i</code> and <code>u</code> are captured directly; since Go 1.22 each iteration gets fresh copies, so the old <code>i := i</code> line is no longer needed.

Example · go
func fetchAll(ctx context.Context, urls []string) ([]int, error) {
	g, ctx := errgroup.WithContext(ctx)
	g.SetLimit(8) // at most 8 requests in flight

	codes := make([]int, len(urls))
	for i, u := range urls {
		g.Go(func() error {
			// Go blocks here until one of the 8 slots is free.
			code, err := head(ctx, u)
			codes[i] = code
			return err
		})
	}
	if err := g.Wait(); err != nil {
		return nil, err
	}
	return codes, nil
}

TryGo for work you can afford to drop

<code>Go</code> blocks when the group is full, which is usually what you want. <code>TryGo</code> returns false instead, so the caller stays responsive and decides what to do with the work it could not start. Reach for it in a loop that must keep draining a queue or answering a health check rather than sitting in a blocked call.

Example · go
var g errgroup.Group
g.SetLimit(4)

skipped := 0
for _, job := range jobs {
	// TryGo reports whether the goroutine was started.
	started := g.TryGo(func() error { return process(job) })
	if !started {
		skipped++ // group is full; this job waits for the next pass
	}
}
if err := g.Wait(); err != nil {
	log.Printf("batch failed: %v", err)
}
log.Printf("skipped %d jobs", skipped)

Related

go waitgroupgo context cancelgo context timeoutgo error wrappinggo goroutine leaks

Discussion

  • Be the first to comment on this lesson.