Goroutine Leaks
A goroutine blocked forever on a send or receive never exits, holding its stack and everything it references.
runtime.NumGoroutine() intA goroutine ends only when its function returns. If it is blocked waiting for something that will never happen, it waits for the life of the process. That is a goroutine leak.
The usual cause involves a channel, a pipe between goroutines where one side sends a value and the other receives it. On a plain channel the sender waits until a receiver takes the value. If the receiver gave up early, or returned on an error path, the sender waits forever.
Why it matters
Each leaked goroutine keeps its stack, and the stack keeps every value it still refers to: buffers, request data, database rows. One leak per request means memory climbing all day and an eventual kill by the host. Leaks are not slow code, they are a slow leak of memory.
Spotting them
runtime.NumGoroutine() returns how many goroutines exist right now. If that number only ever rises while load is steady, you have a leak. For the stacks themselves, import net/http/pprof and read /debug/pprof/goroutine?debug=2, which prints every goroutine with the line it is parked on. Hundreds of goroutines stopped on the same line is the answer.
What trips people up
Make sure every value sent is received, or give the channel room so the send cannot block. Later lessons add select and cancellation, which is the general fix.
Example
package main
import (
"fmt"
"sync"
)
// double starts one goroutine and then receives the single value it
// sends. A channel is a pipe between goroutines: <- sends on the left
// form and receives on the right.
//
// If the receive below were skipped, by an early return or a forgotten
// line, the goroutine would stay parked on its send forever: a leak.
func double(n int) int {
result := make(chan int)
go func() {
result <- n * 2
}()
return <-result
}
func main() {
totals := make([]int, 3)
var wg sync.WaitGroup
for i := range totals {
wg.Go(func() {
totals[i] = double(i + 1)
})
}
wg.Wait()
fmt.Println("totals:", totals)
}
// Output:
// totals: [2 4 6]When to use it
- Explain a service whose memory climbs all day while CPU and request rate stay flat.
- Audit a fan-out helper that returns the first successful response and abandons the rest.
- Add a test that fails when a new code path forgets to drain a worker channel.
More examples
The leak, and the one-line fix
This shape, start several workers and take the first answer, leaks every slower worker. Giving the channel one slot per sender costs a little memory and lets the losers finish and exit. Reach for it whenever a caller may stop reading before the senders are done.
package main
import "errors"
func leaky(ids []int) (int, error) {
out := make(chan int)
for _, id := range ids {
go func() {
out <- id * 2 // blocks until someone receives
}()
}
if len(ids) == 0 {
return 0, errors.New("no ids")
}
// Only one value is ever received. Every other goroutine is stuck
// on its send for the rest of the process's life.
return <-out, nil
}
func fixed(ids []int) (int, error) {
if len(ids) == 0 {
return 0, errors.New("no ids")
}
// Buffered: the channel holds len(ids) values, so every send
// completes even though we read only the first one.
out := make(chan int, len(ids))
for _, id := range ids {
go func() {
out <- id * 2
}()
}
return <-out, nil
}Exposing the goroutine profile
The blank import is for its side effect: the package's init function registers handlers on the default mux. Then curl http://localhost:6060/debug/pprof/goroutine?debug=2 dumps every goroutine's stack, or go tool pprof http://localhost:6060/debug/pprof/goroutine groups them. Read it twice a few minutes apart: the group that grew is the leak.
package main
import (
"log"
"net/http"
_ "net/http/pprof" // registers /debug/pprof/ on the default mux
)
func main() {
// A separate port, not reachable from the internet, so the
// profiles are not public.
go func() {
log.Println(http.ListenAndServe("localhost:6060", nil))
}()
// ... the real service runs here ...
select {}
}Counting goroutines in a test
A rough count like this catches a leak introduced by a refactor, which is otherwise found months later in production. The small tolerance keeps it from failing on unrelated background goroutines; tighten it only if you control everything the package starts.
package worker
import (
"runtime"
"testing"
"time"
)
func TestNoLeak(t *testing.T) {
before := runtime.NumGoroutine()
for i := 0; i < 50; i++ {
if err := Process(i); err != nil {
t.Fatalf("Process(%d): %v", i, err)
}
}
// Goroutines exit shortly after their last statement, so allow a
// moment before comparing.
time.Sleep(100 * time.Millisecond)
if after := runtime.NumGoroutine(); after > before+2 {
t.Errorf("goroutines grew from %d to %d", before, after)
}
}
Discussion