Most Go developers have a vague mental model: goroutines are "like threads but cheap," and the runtime "somehow" schedules them. That’s enough to write working code — until you hit a performance cliff, a goroutine leak, or a program that inexplicably pegs one CPU while the others idle. At that point, "somehow" stops being acceptable.
This article cracks open the Go runtime scheduler. We’ll cover the G/M/P model, how work-stealing keeps your CPU cores fed, how preemption evolved, and what all this means for how you write real software.
The runtime source lives at https://github.com/golang/go/tree/master/src/runtime. proc.go is where most of the scheduler lives — it’s dense but readable once you know the model.
The Problem With OS Threads
An OS thread carries about 8 MB of stack by default on Linux, involves kernel context switches, and you can realistically create maybe a few thousand of them before the system starts sweating. Go programs routinely spawn hundreds of thousands of goroutines.
Goroutines solve this by living in userspace. A freshly created goroutine starts with a 2–8 KB stack that grows on demand. Context switching between goroutines is done by the Go runtime, not the kernel. The kernel only sees OS threads — Go multiplexes goroutines onto those threads.
That multiplexing is the scheduler. And the scheduler is built on three primitives: G, M, and P.
The Three Players
G — Goroutine
A G is the runtime’s representation of a goroutine. It carries the goroutine’s stack, its current program counter, scheduler bookkeeping, and a status field. Think of it as a task descriptor.
Key fields inside runtime.g (simplified from runtime/runtime2.go):
type g struct {
stack stack // current stack bounds [lo, hi)
sched gobuf // saved SP, PC, etc. for context switching
goid int64 // unique goroutine ID
atomicstatus uint32 // _Gidle, _Grunnable, _Grunning, _Gwaiting, _Gdead...
m *m // current M, if running
...
}
A goroutine can be in one of these statuses at any moment:
_Gidle— freshly allocated, not initialized yet_Grunnable— ready to run, sitting in a run queue_Grunning— currently executing on an M_Gwaiting— blocked (channel op, syscall, GC, etc.)_Gdead— finished; runtime may recycle the struct
The status transitions are what the scheduler manipulates constantly.
M — Machine (OS Thread)
An M is an OS thread. The runtime creates and destroys Ms as needed, but it keeps a pool to avoid thrashing. Each M needs a P to run Go code. Without a P, an M can only run special runtime code (like syscall wrappers) using a dedicated scheduling stack called g0.
type m struct {
g0 *g // scheduling goroutine (stack for runtime code)
curg *g // current user goroutine
p puintptr // attached P (nil if not running Go code)
nextp puintptr // P to acquire on next wakeup
spinning bool // looking for work, no goroutines to run
...
}
The g0 goroutine is important: every call into the scheduler (goroutine switching, memory allocation, garbage collection) runs on g0‘s stack, not the user goroutine’s stack. That’s how the runtime avoids stack growth issues inside the scheduler itself.
P — Processor
P is the least intuitive part of the trio. It’s not a CPU core — it’s a scheduling context. A P owns a local run queue, caches for memory allocation, and a handful of other per-P resources. An M must hold a P to schedule and run goroutines.
type p struct {
status uint32
m muintptr // back-pointer to the M holding this P
runqhead uint32
runqtail uint32
runq [256]guintptr // circular local run queue
runnext guintptr // shortcut: run this next before the queue
mcache *mcache // per-P memory allocator cache
...
}
The local run queue (runq) is a fixed-size ring buffer of 256 goroutines. This is a lockless structure — no mutex needed because only the owning P touches it (with one exception: work stealing, which we’ll get to).
The number of Ps is controlled by GOMAXPROCS. Set it to 4, you get 4 Ps, and at most 4 goroutines can run in parallel. Default is runtime.NumCPU().
How They Fit Together
The normal steady-state looks like this:
┌───────────────────────────────────────────┐
│ Global Run Queue │
└───────────────────────────────────────────┘
↑ overflow / steal ↓
┌─────────────────┐ ┌─────────────────┐
│ P1 │ │ P2 │
│ runq: [G3,G4] │ │ runq: [G7] │
│ runnext: G5 │ │ runnext: nil │
└────────┬────────┘ └────────┬────────┘
│ │
M1 M2
│ │
G2 (running) G6 (running)
Each P is attached to one M. Each M runs one goroutine at a time (the curg). When that goroutine yields, blocks, or gets preempted, the M dips into its P’s local run queue to grab the next one.
The global run queue exists for overflow (when a P’s LRQ hits 256) and for goroutines that aren’t tied to any P.
Work Stealing
Work stealing is why GOMAXPROCS=8 on an 8-core machine actually uses all 8 cores even when goroutines are unevenly distributed.
When a P runs out of local work:
- It tries to take a batch from the global run queue (takes min(len/GOMAXPROCS + 1, len/2))
- If the global queue is empty, it picks a random victim P and steals half of that P’s local run queue
- If everything is empty, the M parks itself (goes to sleep)
The steal operation reads from the head of the victim’s ring buffer — the victim reads from the tail. This design minimizes contention so that stealing rarely requires a full lock.
This is the mechanism that makes Go’s concurrency performant for bursty workloads. A P that creates a bunch of goroutines can’t hoard them — idle Ps will steal them.
Goroutine Lifecycle in Detail
go myFunc()
│
▼
newproc() → allocate G struct (or reuse dead one)
│
▼
runqput() → P.runnext = G (or LRQ, or GRQ if LRQ full)
│
▼
Schedule() → pick next G from runnext → LRQ → GRQ → steal
│
▼
execute(G) → gogo() assembly → run G on M
│
▼
goroutine finishes → goexit() → G marked _Gdead → recycled
runnext is a small optimization: when a goroutine spawns a child, the child is placed in runnext and runs before anything else in the queue. This improves cache locality — the parent and child often work on the same data.
Blocking and Syscalls
This is where the model really earns its design. When a goroutine makes a blocking OS syscall (say, reading from a file with no io_uring), the M executing it gets stuck in the kernel. If P stayed attached to that M, no other goroutines on this P would run.
The runtime’s solution: handoff.
When the runtime detects a goroutine entering a blocking syscall, it detaches the P from the M, and another idle M (or a newly spawned one) picks up that P and continues draining its run queue.
When the syscall completes:
- Original M tries to re-acquire a P
- If no idle P is available, the goroutine is put in the global run queue and M parks itself
For non-blocking I/O (network sockets, pipes), the runtime uses a poller (netpoller) backed by epoll on Linux. Goroutines that block on network operations aren’t stuck in a syscall — they’re parked in a _Gwaiting state. The netpoller thread (also an M without a P) calls epoll_wait and wakes up the relevant goroutines when I/O is ready, putting them back in a run queue.
This is why network-heavy Go programs are so efficient: most goroutines waiting for network I/O don’t tie up Ms at all.
Preemption: Then and Now
Early Go had cooperative preemption. A goroutine could only be scheduled away at specific safe points — primarily function call preambles. The compiler inserted stack growth checks at every function call, and the scheduler piggybacked on those checkpoints.
The problem: a tight loop with no function calls was unpreemptable.
func spin() {
for { } // pre-1.14: this goroutine holds its M forever
}
Go 1.14 introduced asynchronous preemption via signals. A background goroutine called sysmon (more on it below) detects goroutines that have been running for more than 10ms. It sends SIGURG to the M running that goroutine. The signal handler saves the goroutine’s state and yields it. This works even inside tight loops.
The signal SIGURG was chosen deliberately — it’s rarely used by real programs and not subject to SA_RESTART, which would make it hard to interrupt syscalls safely.
sysmon: The Background Watchdog
sysmon is a goroutine that runs without a P. It’s the scheduler’s janitor and runs on its own M. Every 10–60ms (adaptive sleep) it:
- Retakes Ps from goroutines running too long (triggers async preemption)
- Schedules goroutines that the netpoller has unblocked
- Forces GC if too much heap has been allocated
- Checks if Ms stuck in blocking syscalls have been there too long and triggers handoff
You can’t directly observe sysmon, but its work shows up everywhere in profiling.
Observing the Scheduler
Stop guessing, start measuring. These are the tools you should know.
GODEBUG=schedtrace=1000 — prints scheduler state every 1000ms:
SCHED 1001ms: gomaxprocs=8 idleprocs=5 threads=10 spinningthreads=1 \
needspinning=0 idlethreads=2 runqueue=0 [0 0 1 0 0 0 0 2]
The numbers in brackets are per-P local queue lengths. Lopsided numbers mean work isn’t being stolen effectively.
GODEBUG=scheddetail=1 pairs with schedtrace for per-goroutine info — very verbose, only useful for narrow debugging.
go tool trace — the nuclear option. Instrument with runtime/trace:
import "runtime/trace"
f, _ := os.Create("trace.out")
trace.Start(f)
defer trace.Stop()
Then: go tool trace trace.out opens a browser UI showing goroutine scheduling, GC, and syscalls on a timeline. This is the right tool for diagnosing scheduler fairness issues and latency spikes.
Goroutine profile via pprof:
curl https://cd-linux.club:6060/debug/pprof/goroutine?debug=2 > goroutines.txt
This dumps stack traces of all live goroutines. A file with 50,000 goroutines all sitting on the same select{} is a leak. Diagnosis is straightforward.
Gotchas
Goroutine leaks are real and silent. A goroutine waiting on a channel that nobody will ever close just sits there forever, holding its stack (and anything the stack references). runtime.NumGoroutine() trending upward over time is a red flag. Use goleak in tests.
GOMAXPROCS=1 is not single-threaded. Ms can still be created for blocking syscalls. A GOMAXPROCS=1 program under heavy file I/O might have dozens of Ms — it just can’t run more than one Go goroutine at a time.
Stack growth is not free. Goroutines start small and grow by allocating new stack segments (historically "split stacks," now contiguous stacks with copying). A function that bounces near the stack boundary triggers frequent growth and shrink cycles — "stack thrashing." Hot paths with deep recursion in tight loops are the usual suspect. Use //go:nosplit sparingly and only when you have profiling data to justify it.
runtime.Gosched() only yields within the current scheduling cycle. It puts the current goroutine at the back of the run queue and lets the scheduler pick something else. It does not block or sleep. If you’re trying to throttle CPU-heavy goroutines, Gosched is often the wrong tool — structured rate limiting or batching is better.
Work stealing doesn’t work well for tiny goroutines with high spawn rate. If you’re spawning a goroutine per request for very short-lived work, the overhead of G allocation and scheduling can dominate. Worker pools (bounded goroutine counts) outperform unbounded spawning when tasks are sub-microsecond.
Blocking in channel ops doesn’t block an M. Goroutines waiting on channels are in _Gwaiting state and are not using an M. This is expected behavior, not a bug — but it surprises people who assume GOMAXPROCS limits all concurrency. You can have a million goroutines waiting on channels and still only use GOMAXPROCS Ms for runnable work.
The runnext optimization can cause priority inversion. If goroutine A repeatedly spawns goroutine B which spawns goroutine C, goroutines at the front of the run queue may starve if the spawn chain is fast enough. In practice this is rare, but if you observe starvation in benchmarks, runtime/trace will show it clearly.
Production-Ready Practices
Set GOMAXPROCS explicitly in containers. The default reads /proc/cpuinfo, which in a container often returns the host CPU count, not the container’s CPU quota. Use automaxprocs — it reads cgroup limits and sets GOMAXPROCS correctly:
import _ "go.uber.org/automaxprocs"
One import line. No excuses for not having it.
Limit goroutine count for CPU-bound work. For tasks that burn CPU (image processing, encryption, compression), pool goroutines at GOMAXPROCS. Spawning more goroutines than Ps creates scheduler overhead with no throughput benefit.
sem := make(chan struct{}, runtime.GOMAXPROCS(0))
for _, item := range items {
sem <- struct{}{}
go func(item Item) {
defer func() { <-sem }()
process(item)
}(item)
}
Profile before you tune. Nine times out of ten, the bottleneck isn’t the scheduler — it’s lock contention, GC pressure, or an algorithm. go tool pprof with CPU profiling will show you. Scheduler tuning without profiling data is noise.
Use runtime/trace for latency, not throughput issues. CPU profiles tell you where time is spent. Execution traces tell you when goroutines run and why they wait. Latency tail problems — P99, P999 — almost always need trace data.
Avoid sync.Mutex in hot paths across Ps. A mutex causes goroutines to park in _Gwaiting, which is cheap individually but adds up. For hot counters, prefer sync/atomic. For per-CPU data, consider sharding by P index (accessible via runtime_procPin() if you’re comfortable with unsafe).
Putting It Together
The GMP model is elegant once you see the full picture: Gs are cheap tasks, Ms are expensive OS threads, and Ps are the connective tissue that lets you run many Gs on few Ms without the kernel’s involvement.
Work stealing keeps cores fed without coordination overhead. Async preemption (post-1.14) makes the scheduler fair even for CPU-bound code. The netpoller keeps I/O-heavy programs from burning Ms on sleeping goroutines. And sysmon watches all of it from the background.
Understanding this model changes how you write Go. You stop treating goroutines as threads and start treating them as tasks. You stop guessing at GOMAXPROCS and start measuring. You reach for go tool trace instead of fmt.Println when concurrency goes sideways.
The scheduler isn’t magic. It’s a well-documented piece of engineering you can read in src/runtime/proc.go. Start there when the mental model isn’t enough.