Go

Every place a goroutine waits, and what lets it go

A Node server has to ask which thread pays for each piece of work. A Go server does not: when a goroutine waits — for a channel, a lock, the network — the runtime parks it and runs another goroutine on the same thread, so waiting costs a few kilobytes of memory and nothing else. That moves the risk somewhere else. The useful question about a line of Go is not "is it fast?" but "if this waits, what will wake it — and what happens if nothing ever does?". A goroutine that nothing will wake is not an error: no log line, no crash. It sits in memory with its stack and everything it points to until the process restarts. Here are fourteen places a goroutine waits. Later sections cite the rows by number.

The scheduler: why waiting is cheap, and what a container does to it

A goroutine is not a thread. It starts with a stack of a few kilobytes that grows as needed, so a server starts one per request, or one per connection, without pooling them. The runtime runs goroutines on real operating-system threads, and its scheduler has three parts, named by letters in the source code. G is a goroutine. M is a machine — an OS thread. P is a processor: permission to run Go code, with its own queue of runnable goroutines. There are exactly GOMAXPROCS of them, so that number decides how many goroutines run at the same instant; the rest wait in the queues.

When a goroutine waits on a channel, a lock or a timer, the scheduler parks it and the P takes the next one from its queue; the thread never stops. Network waits cost no thread either. The runtime hands every socket to the operating system’s readiness interface — epoll on Linux, kqueue on macOS, the same mechanism Node uses — and wakes the goroutine when its data arrives. A P with an empty queue steals half the queue of a busy one, so work spreads across cores without anyone assigning it. And since Go 1.14 the scheduler can interrupt a goroutine in the middle of a tight loop; before that, a loop with no function calls could hold its thread and stall the garbage collector for the whole process.

Row 14 is the one kind of wait the scheduler cannot hide. A blocking system call, such as reading a file, or a call into C through cgo holds a real thread until it returns. The runtime detaches the P from that thread and gives it to another one — starting a new thread if none is idle — so the other goroutines keep running. The cost is one OS thread per call in progress. A thousand concurrent slow file reads mean a thousand threads, and past 10,000, the default limit in debug.SetMaxThreads, the program crashes. Bound such calls with a semaphore — a buffered channel of size N is enough — rather than letting the number of requests decide.

GOMAXPROCS inside a container

Until Go 1.25, GOMAXPROCS defaulted to the number of CPU cores on the machine — and inside a container that is the wrong number. A Kubernetes CPU limit of 2 does not give the container two cores. It gives it a quota: 200 ms of CPU time in every 100 ms window, enforced by the Linux scheduler. A Go process on a 64-core node still started 64 Ps. When work arrived in a burst — or when the garbage collector, which takes a quarter of the Ps for its own workers, started a cycle — 64 threads spent the whole quota in about 3 ms. The kernel then paused every thread in the container for the remaining 97. The Go team’s own description is that throttling "completely pauses application execution for the remainder of the throttling period".

Go 1.25 reads the container’s CPU limit and sets GOMAXPROCS to it, rounding up, and re-checks it periodically, because an orchestrator can change the limit while the process runs. It considers the limit, not the CPU request. The automatic value switches off as soon as GOMAXPROCS is set by hand, so a deployment that set it years ago keeps whatever it set. Before 1.25 the same effect came from Uber’s automaxprocs package, imported for its side effect; on 1.25 or later it can go. The symptom of the old default is worth recognising, because it still runs in older images: p99p99latency for the slowest sliceThe response time that 99% of requests come in under — p95 and p50 are the same measure at 95% and at the median. The average hides exactly the requests that make people leave, which is why these are the numbers anyone serious watches.All terms latency in steps of about 100 ms, a CPU graph far below the limit, and the cgroup’s throttled-periods counter climbing.

Every goroutine needs an owner: leaks, context and errgroup

A goroutine has no parent. The function that started it can return, and the goroutine carries on; nothing in the language links the two lifetimes. That is why rows 1–3 and 8 are the most common Go leak. The Go 1.26 release notes show the textbook case. A function starts one goroutine per work item, each sending its result on an unbuffered channel, and collects the results in a loop. When one result is an error, the function returns early — and every goroutine still holding a result blocks on its send forever, because nobody is receiving any more.

The leak grows with traffic, not with time, so it survives every test and appears a week after launch as a memory graph that only climbs. There are three fixes, each matching a shape of code. If the caller may stop reading, give the channel a buffer as large as the number of senders, so every send completes whether anyone listens or not. If the work should stop when the caller gives up, pass a context and make every blocking step a select that also waits on ctx.Done(). And for "start N tasks, stop at the first error", use errgroup.WithContext from golang.org/x/sync: it cancels the shared context on the first failure, and Wait returns that error once every goroutine has finished.

Finding a leak in production

The first signal is a number: runtime.NumGoroutine(), exported as a metric. A healthy service’s count rises and falls with traffic; a leaking one ratchets up and never comes back down. The second is a goroutine profile — /debug/pprof/goroutine from net/http/pprof — which groups thousands of goroutines by the line they are stuck on, so a leak shows up as one line holding forty thousand of them. Go 1.27 adds a third, built for exactly this. The goroutineleak profile, contributed by Vlad Saioc at Uber, uses the garbage collector to find goroutines blocked on a channel or lock that no running goroutine can reach — and that therefore nothing can ever release. It was an experiment in 1.26 and is stable in 1.27, at /debug/pprof/goroutineleak. It cannot see a leak through a global variable, and its documentation says so. In tests, Uber’s goleak package fails any test that leaves goroutines behind.

Shared memory: data races, maps and the loop variable

Go’s slogan is "do not communicate by sharing memory; share memory by communicating", and new Go developers often read it as a rule to use channels for everything. It is advice about ownership, not a ban on locks. When a value is handed from one goroutine to another, a channel is the clearest way to do it: whoever receives the value owns it. When several goroutines simply read and update one shared structure — a cache, a counter, a map of sessions — a sync.Mutex is shorter, faster and far easier to follow than a protocol built from channels. Use channels to hand work over and to signal; use a mutex to protect state.

Either way, the bug to fear is a data race: two goroutines touching the same memory at the same time, at least one of them writing. The compiler does not stop it, and the program does not crash at the moment it happens. It produces a wrong value, a torn read of a multi-word value such as a slice or an interface, or a crash somewhere else much later. Go’s answer is a tool. Building with -race instruments every memory access and reports both conflicting stacks when a race actually happens during the run. By the documentation’s own estimate it costs 5–10 times the memory and 2–20 times the run time, so it belongs in CICI/CDContinuous Integration / Continuous Delivery — automatic build, test and deliveryEvery commit is built and put through the tests automatically (CI), and a build that passes travels to the servers the same way (CD). The point is not automation for its own sake: it is that a breakage shows up in minutes instead of on release day.Full article →All terms and on a small slice of production traffic rather than in every process — and it can only report the races your tests make happen.

How common races are in a large Go code base was measured in 2022. Uber ran the race detector across its Go monorepo — about 50 million lines and some 2,100 services — for six months and found around 2,000 data races, of which about 1,100 were fixed by 210 engineers. The authors also measured that Go services expose about eight times more concurrency than Uber’s Java services. The patterns they listed are the ones to search for: a variable captured by reference in a goroutine’s closure — a loop variable, an err, a named result; a slice appended to from several goroutines; a map written concurrently; a sync.Mutex copied by value, so each copy guards nothing; and a WaitGroup whose Add runs inside the goroutine instead of before it.

The loop variable, before and after Go 1.22

Until Go 1.22 a for loop declared its variable once, and every iteration reused it. A closure or a pointer that captured it — a goroutine started in the loop, &v stored in a slice — saw whatever the variable held last, not the value of its own iteration. It was the most common Go mistake of all, and the Go team’s 2023 post on fixing it quotes a production example from Let’s Encrypt (in the cases below). Since Go 1.22 each iteration gets a fresh variable. But the change applies per module, by the go line in go.mod: a module that still says go 1.21 keeps the old behaviour on a new compiler. Check that line before trusting the fix, and leave the v := v lines in old code alone until it changes.

The language: errors as values, a small design, and its releases

Go has no exceptions. A function that can fail returns an error beside its result, and the caller checks it on the next line. People argue about this more than about anything else in Go, and the argument tends to miss what it buys: every place a failure can happen is visible, at the line where it happens, and no hidden path carries control out of a function. The cost is repetition — the three-line if err != nil check that appears everywhere.

Modern Go makes the repetition carry information. Wrapping with fmt.Errorf("load user %d: %w", id, err) adds context and keeps the original error inside, so a failure ten layers down arrives as one readable sentence rather than "connection refused". errors.Is asks whether a particular error is anywhere in that chain; errors.As — and its generic form errors.AsType, added in Go 1.26 — extracts one of a given type; errors.Join (Go 1.20) combines several. The pattern to aim for: low-level code adds context and returns, and exactly one layer near the edge of the program decides — log it, retry it, or turn it into a status code.

The rest of the language is small on purpose, and the omissions are the design rather than an oversight. There is no inheritance: types embed other types, and satisfy an interface simply by having its methods, without declaring it. There are no constructors, no operator overloading, no default arguments, and for the first twelve years there were no generics. The point is that Go written by a stranger looks like Go written by you, which is what keeps a large code base readable by people who did not write it.

The standard library is unusually complete for a backend: an HTTPHTTPHyperText Transfer Protocol — the protocol of the webThe protocol browsers and servers talk in: the client sends a method and an address, the server answers with a status code and a body. It keeps no memory between requests — every request stands alone, which is why anything that must be remembered is carried explicitly.All terms server and client, JSONJSONJavaScript Object Notation — text format for dataA text format for passing data around: key–value pairs, lists, numbers and strings. It came out of JavaScript, but every language reads it now, and that is why most APIs answer in it rather than in anything more capable.All terms, TLS, the SQL interfaces, templates, testing, fuzzing and profiling all ship with the compiler, so a production service can genuinely have a short go.mod. Go 1.27 replaced the engine under encoding/json with the new encoding/json/v2 — stricter by default in the new package, and much faster at decoding in both — while keeping the old package’s behaviour.

Releases, the compatibility promise, and which Go you are looking at

Go ships a major release every six months, in February and August, and supports each one until two newer ones exist — about a year. Patch releases (1.27.1, 1.27.2…) carry security fixes for the two supported lines only. Since Go 1, in 2012, the team has promised that code written for an earlier release keeps compiling and running. When a fix has to change behaviour, it ships behind a GODEBUG setting that keeps the old behaviour for modules declaring an older Go version. The table lists the releases you are most likely to meet in code.

Memory: GOGC, GOMEMLIMIT and a container’s limit

Go’s garbage collector is tuned for short pauses rather than maximum throughput. It marks live objects while the program keeps running and stops every goroutine only for brief phases, typically well under a millisecond. It pays for that with CPU. While a cycle runs, it takes about a quarter of the Ps for its own workers, and a goroutine that allocates quickly is made to help with the marking — the "assist" that shows up as latency in whatever request it was serving. Go 1.26 made the new Green Tea collector the default, which the release notes expect to cut garbage-collection overhead by 10–40% in programs that lean on it.

One number decides how often a cycle runs. GOGC, 100 by default, means the heap may grow by 100% of the live data before the next cycle starts: a service holding 1 GB of live objects runs at about 2 GB of heap. Doubling GOGC roughly halves the collector’s CPU and adds the same amount of memory again. That trade is fine on a machine with spare memory and dangerous in a container, because GOGC knows nothing about the container’s limit. A service with 1.5 GB of live data in a 2 GB container lets its heap head for 3 GB before the collector thinks about running — and the kernel kills it on the way.

Go 1.19 added the other half: GOMEMLIMIT, a soft limit on all the memory the runtime manages. As the total approaches it, the collector runs more often, whatever GOGC says. Set it to the container’s limit minus some headroom — the Go garbage-collector guide suggests 5–10% for memory the runtime cannot see, such as C libraries — so GOMEMLIMIT=1800MiB in a 2 GiB container. The limit is soft on purpose. If the live data itself outgrows it, the runtime caps the collector at roughly half the CPU rather than collecting non-stop, so a real leak still ends in an out-of-memory kill — just later, and after a stretch of high CPU that is worth an alert of its own.

Shipping: one binary, and a clean shutdown

The operational story is why Go took over infrastructure. go build produces one executable with no runtime to install, and with CGO_ENABLED=0 it is statically linked, so a container image can be the binary alone — a few megabytes on a scratch or distroless base. It starts in milliseconds, and GOOS and GOARCH cross-compile it for another platform from any machine. Docker, Kubernetes, Terraform and Prometheus are written in Go for exactly these reasons. A scratch image has no shell and no CA certificates, so copy the certificate bundle in, or the first outgoing HTTPS call fails.

A clean stop takes six lines. signal.NotifyContext turns SIGTERM into a cancelled context; when it fires, srv.Shutdown(ctx) stops accepting connections, closes the idle ones and waits for requests in flight, up to the deadline you give it. Make that deadline shorter than the orchestrator’s grace period — 30 seconds by default in Kubernetes — and pass the same context down, so long-running work notices too. And set the server’s timeouts while you are there: an http.Server built as a bare struct has none, which is row 9 waiting for a slow client.

Five cases, and what each one was missing

Each case is documented by the people it happened to — engineering blogs, a certificate authority’s incident report, a research paper. Between them they cover time, the garbage collector twice, the loop variable and data races. Twitch and Uber are told above, where their mechanism lives; the other three follow.

The first case is a clock. At the end of 2016 a leap second was added to the last minute of the year, and the wall clock on Cloudflare’s servers stepped back. RRDNS, Cloudflare’s DNS server, written in Go, measured how fast its upstream resolvers answered by subtracting two time.Now() readings — and across the leap second one of those durations came out negative. Later, the code that picked a resolver weighted by those timings was fed the negative number and panicked. A recover caught the panic, so the process survived, but CNAME lookups failed: at the peak about 0.2% of DNS queries to Cloudflare and under 1% of HTTP requests returned errors, and the fix was out worldwide within seven hours. Go’s time.Now() then read only the wall clock, which can jump. Go 1.9, in August 2017, made every time.Time carry a monotonic clock reading as well, so time.Since and Sub no longer go backwards. The lesson outlives the fix: a duration is measured and a timestamp is read, and they come from different clocks.

The fourth case is the loop variable in production. On 29 February 2020 Let’s Encrypt found that Boulder, its certificate authority software, written in Go, had been checking CAA records — the DNS record in which a domain owner says which authorities may issue certificates for it — on only one name of a multi-name certificate. When a request held N names that needed rechecking, Boulder checked one of them N times. The Go blog later quoted the code: the loop copied its key, kCopy := k, but passed &v to a function that kept pointers into it, so every result pointed at the same variable. About 3 million certificates were affected. Let’s Encrypt replaced and revoked about 1.7 million by the 5 March deadline and let the rest expire, since they lived 90 days. Go 1.22 made the loop variable per iteration, which removes this whole class of bug — for modules whose go.mod says so.

Where it degrades

Go is generous in one direction and silent in the other. Concurrency is cheap enough to use without thinking, so a service written plainly scales a long way. But nothing in the language says when a goroutine ends, the standard library sets no timeouts, and the race detector reports only what the tests made happen. Almost every failure is one of these:

• Goroutines started with no owner, which leak whenever their caller stops listening (rows 1–3, 8).
• A context accepted as a parameter and ignored, so cancellation never reaches the work (row 12).
• An http.Server or http.Client built with no timeouts (rows 9–10).
• Result sets left open, so the connection pool drains and every query waits (row 11).
• Shared maps and slices written from several goroutines, with -race never run in CI.
• A container image built before Go 1.25, running with GOMAXPROCS set to the host’s core count.
• No GOMEMLIMIT in a container whose limit is less than twice the live heap.
• panic and recover used as exceptions for ordinary, expected failures.
• Errors returned without context, so a failure arrives as "connection refused" with no clue where.
• A go line in go.mod older than the fixes the team believes it has — the loop variable, the timers.

The honest limit is the garbage collector and the deliberately small type system. A service whose heap is tens of millions of long-lived pointers — a huge in-memory cache, an index — pays for every cycle with CPU and latency that tuning can reduce but not remove, which is where Discord left. And a domain that wants to encode its rules in types, with exhaustive matching and no nil, will find Go’s types a cage; that is Rust’s territory, at the price of a much steeper language. For the network services, gateways and tools most teams actually write, the trade runs the other way, which is why so much of the infrastructure underneath them is written in Go.

The practical check takes ten minutes. Run go version against the production binary and find its row in the releases table. Read the go line in go.mod. Look at the goroutine count on a dashboard over a week. Compare GOMAXPROCS and GOMEMLIMIT with the container’s limits. Search the code for http.Client{ and http.Server{ and read their timeouts. And confirm that CI runs the tests with -race.

When to use it

Related pages

• DevelopmentRustRust is a compiled systems language that frees memory without a garbage collector by checking, at compile time, who owns each value and who may borrow it. That check is why safe Rust has no use-after-free and no data races — and why newcomers spend their first weeks arguing with the compiler. The page opens with the twelve compiler errors everyone meets first and the rule behind each, then covers ownership and borrowing, what the compiler cannot check, threads and async, errors and panics, releases and editions up to Rust 1.99, five documented cases, and where Rust is the wrong tool.
• DevelopmentJavaJava is a statically typed language compiled to bytecode for the JVM, a runtime that compiles hot code as it runs and frees memory with a garbage collector. For thirty years it has been the default for large back-end systems — banks, airlines, telecoms — and what changed most in the last five is how it is written and how it waits: records, sealed types and pattern matching in the language, virtual threads in the runtime. The page opens with every building block on one screen, the current way beside the one it replaced, then covers releases and licences, the JVM, memory in a container, Spring and its proxies, threads and virtual threads, and five documented cases.
• DevelopmentNode.jsNode.js is a server runtime for JavaScript that runs all of your code on one thread and hands every wait to the operating system or a small thread pool. That one design is why a single process holds ten thousand idle connections, and why one slow function delays every request at once. The page opens with seventeen things a Node server does and which thread pays for each, then covers the event loop and the pool, releases and support — Node 27 moves to one major a year — what is built in now, scaling and a clean shutdown, five documented cases, and where it degrades.