Go: Go Goroutines and WaitGroups
Goroutines are the cornerstone of concurrent programming in Go—they are neither threads nor coroutines, but rather lightweight concurrency units managed by the Go runtime. With 2 KB of stack space, they make concurrency on the order of millions possible.
Go's concurrency model is second to none: launching a goroutine requires nothing more than the go keyword, while the Go runtime efficiently maps thousands upon thousands of goroutines to a small number of OS threads behind the scenes.
1. You will learn
- The Concept of Goroutines and the
goKeyword sync.WaitGroupwaits for all goroutines to finish- Goroutine vs. OS خيط (stack size/creation overhead)
GOMAXPROCSand the GMP Scheduling Model- Causes and Troubleshooting of Goroutine Leaks
runtime.NumGoroutinemonitoring- Use goroutines to process 100,000 log entries concurrently
2. A Data Engineer's True Story
(1) Pain Point: 100,000 log entries take 5 hours to process in a single خيط
Charlie is a data engineer who processes 100,000 خادم logs every day:
"Every day at dawn, the log cleanup task runs, processing 100,000 lines one after another—it takes over five hours. During the morning meeting, the PM always asks, 'When will yesterday's data be ready?' I say, 'At 3:00 p.m.,' and they reply, 'Why can't it be ready by 9:00 a.m.?'"
He opened the current code:
GO
// Serial processing: 100k entries × 200ms/entry = 5.5 hours
func processLogs(logs []LogEntry) []Result {
results := make([]Result, 0, len(logs))
for _, log := range logs {
result := processSingleLog(log) // 200ms each, including network IO
results = append(results, result)
}
return results
}
Processing each log entry involves one external API call (with a wait time of approximately 200 ms), but the actual CPU computation takes less than 1 ms—99.5% of the time is spent waiting for the network.
(2) The Go Solution: Goroutine Concurrency
GO
// log_processor.go
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
type LogEntry struct {
ID int
Message سلسلة
Level سلسلة
}
type Result struct {
ID int
Status سلسلة
}
func processSingleLog(log LogEntry) Result {
time.Sleep(200 * time.Millisecond)
return Result{ID: log.ID, Status: "processed"}
}
func processLogsConcurrent(logs []LogEntry, workerCount int) []Result {
results := make([]Result, 0, len(logs))
var mu sync.Mutex
var wg sync.WaitGroup
// Use channels for task distribution
jobs := make(chan LogEntry, len(logs))
for _, log := range logs {
jobs <- log
}
close(jobs)
// Start workerCount workers
for w := 0; w < workerCount; w++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
for log := range jobs {
result := processSingleLog(log)
mu.Lock()
results = append(results, result)
mu.Unlock()
}
}(w)
}
wg.Wait()
return results
}
func main() {
// Simulate 100 log entries (demo; can scale to 100k)
logs := make([]LogEntry, 100)
for i := 0; i < 100; i++ {
logs[i] = LogEntry{ID: i, Message: fmt.Sprintf("log-%d", i), Level: "INFO"}
}
start := time.Now()
results := processLogsConcurrent(logs, 10)
elapsed := time.Since(start)
fmt.Printf("Processed %d logs in %v\n", len(results), elapsed)
fmt.Printf("Current goroutine count: %d\n", runtime.NumGoroutine())
}
Output:
TEXT
📖 Display only
Processed 100 logs in 2.05s
Current goroutine count: 1
(3) Performance: Sequential vs. Concurrent
| Processing Method | 100 logs | 100,000 logs | Number of goroutines |
|---|---|---|---|
| Serial | 20s | 5.5h | 1 |
| 10 concurrent workers | 2s | 33min | ~12 |
| 100 concurrent workers | 0.2s | 3.3min | ~102 |
💡 Tip: More goroutines aren't necessarily better. For I/O-intensive tasks, we recommend
worker count = GOMAXPROCS * 2–10; for CPU-intensive tasks, we recommend worker count = GOMAXPROCS.
3. Goroutine Basics
(1) The go keyword
GO
package main
import (
"fmt"
"time"
)
func printNumbers() {
for i := 1; i <= 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Printf("%d ", i)
}
}
func printLetters() {
for _, c := range "ABCDE" {
time.Sleep(150 * time.Millisecond)
fmt.Printf("%c ", c)
}
}
func main() {
go printNumbers() // Execute concurrently in a goroutine
go printLetters()
time.Sleep(1 * time.Second)
fmt.Println()
}
Output (may vary each time):
TEXT
📖 Display only
1 A 2 B 3 C 4 D 5 E
(2) Anonymous goroutine
GO
package main
import (
"fmt"
"time"
)
func main() {
go func() {
fmt.Println("Anonymous goroutine running")
}()
go func(msg سلسلة) {
fmt.Println("Anonymous goroutine with parameter:", msg)
}("hello")
time.Sleep(100 * time.Millisecond)
fmt.Println("main done")
}
🔥 Common Mistake: When the main function returns, all goroutines are forcibly terminated. The
time.Sleep above is intended to wait for the goroutine to finish—in production code, you should use sync.WaitGroup.
4. sync.WaitGroup: Wait for synchronization
(1) Three Methods for WaitGroup
GO
package main
import (
"fmt"
"sync"
)
func worker(id int, wg *sync.WaitGroup) {
defer wg.Done() // 3. Tell WaitGroup this worker is done
fmt.Printf("Worker %d started\n", id)
// Simulate work...
fmt.Printf("Worker %d done\n", id)
}
func main() {
var wg sync.WaitGroup
for i := 1; i <= 3; i++ {
wg.Add(1) // 1. Increment counter
go worker(i, &wg)
}
wg.Wait() // 2. Block until all workers finish
fmt.Println("All workers done!")
}
| طريقة | دالة |
|---|---|
wg.Add(delta int) |
Increments the counter (typically called before starting a goroutine) |
wg.Done() |
Decrements the counter (typically a defer call inside a goroutine) |
wg.Wait() |
Blocks until the counter reaches zero |
▶ Example: WaitGroup + Closure Pitfalls
GO
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
// ❌ Wrong: Closure captures loop variable
for i := 1; i <= 3; i++ {
wg.Add(1)
go func() {
defer wg.Done()
fmt.Printf("Wrong i=%d\n", i) // All print 3 or 4
}()
}
wg.Wait()
// ✅ Correct: Pass parameter (creates a copy)
for i := 1; i <= 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
fmt.Printf("Correct i=%d\n", id)
}(i)
}
wg.Wait()
}
🔥 Common Mistake (Classic Closure Trap): A goroutine closure directly captures the loop variable
i—by the time the goroutine starts, the loop may have already finished, and the value of i may have been overwritten. You must pass a copy of the parameter.
5. Goroutines vs. OS Threads
(1) Key Differences
GO
package main
import (
"fmt"
"runtime"
)
func main() {
// View current GOMAXPROCS
fmt.Printf("GOMAXPROCS: %d\n", runtime.GOMAXPROCS(0))
fmt.Printf("CPU cores: %d\n", runtime.NumCPU())
fmt.Printf("Current goroutine count: %d\n", runtime.NumGoroutine())
}
| Dimension | OS Thread | Goroutine |
|---|---|---|
| Initial stack size | 1–8 MB | 2 KB |
| Maximum Stack | Fixed at 1–8 MB | Dynamically expands to 1 GB |
| Creation overhead | ~1 µs | ~0.1 µs |
| Context Switch | Kernel mode (~1 µs) | User mode (~0.1 µs) |
| Maximum number | ~10,000 | Millions |
| Scheduler | OS Kernel Scheduling | Go Runtime GMP |
(2) GMP Scheduling Model
graph TB
G[G: Goroutine Queue] --> P[P: Logical Processor<br/>GOMAXPROCS count]
P --> M[M: OS Thread<br/>Scheduled by kernel]
M --> CPU[CPU Core]
subgraph Global Queue
GQ[(Global Goroutine Queue)]
end
GQ --> P
style P fill:#e1f5fe
style M fill:#fff3e0
GMP: G (Goroutine) — P (Processor, logical processor; number = GOMAXPROCS) — M (Machine, OS خيط). The Go runtime schedules Gs to the local queue of P, and P then binds to M for execution.
6. GOMAXPROCS and Concurrency Control
(1) GOMAXPROCS Settings
GO
package main
import (
"fmt"
"runtime"
)
func main() {
fmt.Printf("Default GOMAXPROCS: %d\n", runtime.GOMAXPROCS(0))
// CPU-intensive: GOMAXPROCS = NumCPU
// IO-intensive: GOMAXPROCS = NumCPU * 2~10
runtime.GOMAXPROCS(4) // Set to 4
fmt.Printf("After setting GOMAXPROCS: %d\n", runtime.GOMAXPROCS(0))
}
▶ Example: The Impact of GOMAXPROCS on Performance
GO
package main
import (
"fmt"
"runtime"
"sync"
"time"
)
func cpuIntensiveTask() {
sum := 0
for i := 0; i < 10000000; i++ {
sum += i
}
}
func benchmarkGOMAXPROCS(n int) time.Duration {
runtime.GOMAXPROCS(n)
var wg sync.WaitGroup
start := time.Now()
for i := 0; i < 8; i++ {
wg.Add(1)
go func() {
defer wg.Done()
cpuIntensiveTask()
}()
}
wg.Wait()
return time.Since(start)
}
func main() {
for _, p := range []int{1, 2, 4} {
elapsed := benchmarkGOMAXPROCS(p)
fmt.Printf("GOMAXPROCS=%d: %v\n", p, elapsed)
}
}
7. Goroutine Leaks and Troubleshooting
(1) Leak Scenarios
GO
package main
import (
"fmt"
"runtime"
"time"
)
// Leaky goroutine: reads from a channel that is never closed
func leakyGoroutine() {
ch := make(chan int)
go func() {
<-ch // Blocks forever—no one writes to ch
}()
}
func main() {
for i := 0; i < 10; i++ {
leakyGoroutine()
}
time.Sleep(100 * time.Millisecond)
fmt.Printf("Goroutine count after leak: %d\n", runtime.NumGoroutine())
// Output: Goroutine count after leak: 11 (10 leaked + 1 main)
}
▶ Example: 4 Ways to Start a Goroutine
GO
package main
import (
"fmt"
"sync"
"time"
)
func say(msg string) {
fmt.Println(msg)
}
func main() {
var wg sync.WaitGroup
// Style 1: Named function
wg.Add(1)
go func() {
defer wg.Done()
say("Style 1: Named function")
}()
// Style 2: Anonymous function
wg.Add(1)
go func() {
defer wg.Done()
fmt.Println("Style 2: Anonymous function")
}()
// Style 3: Anonymous function with parameter (recommended—avoids closure trap)
wg.Add(1)
go func(msg string) {
defer wg.Done()
fmt.Println(msg)
}("Style 3: With parameter")
// Style 4: Function as value
wg.Add(1)
fn := func() {
defer wg.Done()
fmt.Println("Style 4: Function variable")
}
go fn()
wg.Wait()
}
▶ Example: Comparison of Goroutines and OS Thread Stacks
GO
package main
import (
"fmt"
"runtime"
"sync"
)
func main() {
var memStats runtime.MemStats
var wg sync.WaitGroup
num := 100000 // 100k goroutines
runtime.GC()
runtime.ReadMemStats(&memStats)
before := memStats.HeapAlloc
for i := 0; i < num; i++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
_ = n
}(i)
}
wg.Wait()
runtime.ReadMemStats(&memStats)
after := memStats.HeapAlloc
perGoroutine := float64(after-before) / float64(num)
fmt.Printf("Started %d goroutines\n", num)
fmt.Printf("Memory increase: %.2f MB\n", float64(after-before)/1024/1024)
fmt.Printf("Per goroutine: ~%.2f KB\n", perGoroutine/1024)
fmt.Printf("Total goroutine count: %d\n", runtime.NumGoroutine())
}
Output:
TEXT
📖 Display only
Started 100000 goroutines
Memory increase: 21.45 MB
Per goroutine: ~0.22 KB
Total goroutine count: 1
💡 Tip: Each goroutine is initially allocated only 2 KB of stack space, while OS threads are allocated 1–8 MB by default. 100,000 goroutines ≈ 21 MB, and 100,000 threads ≈ 80 GB.
▶ Example: Monitoring the Goroutine Lifecycle
GO
package main
import (
"fmt"
"runtime"
"time"
)
func safeWorker(done chan struct{}) {
<-done // Wait for exit signal
}
func main() {
done := make(chan struct{})
// Start 5 workers
for i := 0; i < 5; i++ {
go safeWorker(done)
}
fmt.Printf("Goroutine count after start: %d\n", runtime.NumGoroutine())
// Send exit signal
close(done)
time.Sleep(10 * time.Millisecond)
fmt.Printf("Goroutine count after close: %d\n", runtime.NumGoroutine())
}
(5) Leak Prevention Checklist
| Scenario | Prevention Methods |
|---|---|
| Reading from a channel but no one is writing to it | Use a buffered channel or select with the default option |
| Writing to a channel but no one is reading it | Make sure there is a consumer or use select with a default value |
| Goroutine Infinite Loops | Controlling Exit Via a done channel or context |
time.After causes a memory leak in a loop |
Create a new timer in each iteration using time.NewTimer + Stop |
select {} empty |
Ensure there is an exit path |
8. Complete Example: Concurrent Processing of 100,000 Log Entries
GO
// log_processor.go
package main
import (
"fmt"
"math/rand"
"runtime"
"sync"
"time"
)
type LogLevel int
const (
Info LogLevel = iota
Warn
Error
Debug
)
type LogLine struct {
Timestamp time.Time
Level LogLevel
Message سلسلة
Source سلسلة
}
type ProcessedLog struct {
Original LogLine
Severity سلسلة
Alert bool
ProcessedAt time.Time
}
func (l LogLine) Process() ProcessedLog {
// Simulate processing time (IO wait)
time.Sleep(time.Duration(50+rand.Intn(50)) * time.Millisecond)
severity := "low"
alert := false
switch l.Level {
case Error:
severity = "critical"
alert = true
case Warn:
severity = "medium"
}
return ProcessedLog{
Original: l,
Severity: severity,
Alert: alert,
ProcessedAt: time.Now(),
}
}
// Serial processing
func processSerial(logs []LogLine) []ProcessedLog {
results := make([]ProcessedLog, 0, len(logs))
for _, log := range logs {
results = append(results, log.Process())
}
return results
}
// Concurrent processing: worker pool pattern
func processConcurrent(logs []LogLine, workers int) []ProcessedLog {
jobs := make(chan LogLine, len(logs))
results := make(chan ProcessedLog, len(logs))
// Fill jobs
for _, log := range logs {
jobs <- log
}
close(jobs)
var wg sync.WaitGroup
for w := 0; w < workers; w++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
for log := range jobs {
results <- log.Process()
}
}(w)
}
// Wait for all workers to finish, then close results
go func() {
wg.Wait()
close(results)
}()
// Collect results
processed := make([]ProcessedLog, 0, len(logs))
for r := range results {
processed = append(processed, r)
}
return processed
}
func main() {
const logCount = 10000 // 10k for demo; can scale to 100k
levels := []LogLevel{Info, Warn, Error, Debug}
sources := []سلسلة{"api-gateway", "user-service", "payment", "قاعدة بيانات"}
logs := make([]LogLine, logCount)
for i := 0; i < logCount; i++ {
logs[i] = LogLine{
Timestamp: time.Now(),
Level: levels[rand.Intn(len(levels))],
Message: fmt.Sprintf("log-entry-%d", i),
Source: sources[rand.Intn(len(sources))],
}
}
fmt.Printf("Log count: %d\n", logCount)
fmt.Printf("CPU cores: %d, GOMAXPROCS: %d\n", runtime.NumCPU(), runtime.GOMAXPROCS(0))
// Serial processing
start := time.Now()
serialResults := processSerial(logs)
serialTime := time.Since(start)
fmt.Printf("\nSerial: %v (%d entries/sec)\n", serialTime,
int(float64(logCount)/serialTime.Seconds()))
// Concurrent processing (10 workers)
runtime.GC()
start = time.Now()
concurrentResults := processConcurrent(logs, 10)
concurrentTime := time.Since(start)
fmt.Printf("Concurrent(10): %v (%d entries/sec)\n", concurrentTime,
int(float64(logCount)/concurrentTime.Seconds()))
// Concurrent processing (50 workers)
runtime.GC()
start = time.Now()
concurrentResults = processConcurrent(logs, 50)
concurrentTime = time.Since(start)
fmt.Printf("Concurrent(50): %v (%d entries/sec)\n", concurrentTime,
int(float64(logCount)/concurrentTime.Seconds()))
fmt.Printf("\nResult validation: serial=%d, concurrent=%d\n",
len(serialResults), len(concurrentResults))
fmt.Printf("Current goroutine count: %d\n", runtime.NumGoroutine())
// Count alerts
alertCount := 0
for _, r := range concurrentResults {
if r.Alert {
alertCount++
}
}
fmt.Printf("Alert count: %d\n", alertCount)
}
Expected Output:
TEXT
📖 Display only
Log count: 10000
CPU cores: 8, GOMAXPROCS: 8
Serial: 5.2s (1923 entries/sec)
Concurrent(10): 520ms (19230 entries/sec)
Concurrent(50): 110ms (90909 entries/sec)
Result validation: serial=10000, concurrent=10000
Current goroutine count: 1
Alert count: 2453
🔥 Common Mistake: A panic inside a goroutine will cause the entire program to crash and will not be caught by an outer
recover. The entry point of each goroutine should be protected with defer recover(). Lesson 14 on channels will explore this pattern in more detail.
❓ FAQ
Q What is the difference between a goroutine and an OS خيط?
A A goroutine is a user-mode "coroutine" managed by the Go runtime. Its stack is initially only 2 KB (compared to 1–8 MB for a خيط), and it is 10 times faster to create and 10 times faster to switch between. Having millions of goroutines is commonplace in Go, whereas a system with tens of thousands of threads would struggle to handle the load.
Q How do I wait for all goroutines to finish?
A Use
sync.WaitGroup—wg.Add(n) increments the count, wg.Done() decrements it, and wg.Wait() blocks until the count reaches zero. Note that Add must be called in the outer نطاق or before launching a goroutine, and Done should be wrapped in defer to ensure it executes.Q How do I troubleshoot goroutine leaks?
A Use
runtime.NumGoroutine() to check if the count is continuously increasing; use net/http/pprof to examine the goroutine stack traces. Common causes of leaks: blocked channel reads or writes, select statements without a default clause, and for loops that launch goroutines without controlling their exit.Q What is the purpose of
runtime.Gosched?A It proactively yields the P, giving other goroutines a chance to execute. There is rarely a need to call it manually—Go automatically schedules it during scenarios such as I/O waits, channel operations, and
time.Sleep.Q What is the upper limit on the number of goroutines?
A Theoretically, it is limited by memory—each goroutine's stack starts at 2 KB, so 4 GB of memory ≈ 2 million goroutines. Practical recommendations: For CPU-intensive applications, ≤ GOMAXPROCS; for I/O-intensive applications, ≤ 1,000–10,000.
Q What should the GOMAXPROCS value be set to?
A In Go 1.5 and later, the default is NumCPU (number of CPU cores). Generally, no change is needed: Keep the default for CPU-intensive workloads; for I/O-intensive workloads, you can increase it appropriately (2–10 times), but the fundamental bottleneck lies in I/O speed rather than the number of parallel processes.
Q Does a panic in a goroutine affect other goroutines?
A Yes! An unrecovered panic in any goroutine will cause the entire process to crash. Therefore, every goroutine entry point should include
defer func() { if r := recover(); r != nil { ... } }().Q Do functions called with the
go keyword have to be parameterless?A No, they can accept parameters:
go myFunc(arg1, arg2). Be careful of cyclic متغير traps when capturing variables in closures—use parameter passing (which creates copies) instead of direct capture.📖 Summary
- The
gokeyword launches a goroutine, enabling lightweight user-mode concurrency - Goroutine stacks start at 2 KB and grow dynamically; OS threads are fixed at 1–8 MB
sync.WaitGroup: The three methods—Add,Done, andWait—manage the goroutine lifecycle- When a goroutine إغلاق captures a حلقة متغير, the parameter must be passed by copy
GOMAXPROCScontrols the degree of parallelism; default = NumCPU- GMP Scheduling Model: Goroutine → Processor → Machine
- Goroutine leaks: blocked channels, infinite loops with no exit conditions
runtime.NumGoroutine()monitors the current number of goroutines
📝 Exercises
-
Basic Problem (Difficulty ⭐): Write a program that starts 5 goroutines, each of which prints its own number and "Hello." Use
sync.WaitGroupto wait for all of them to finish, and verify that the execution order of the goroutines is random. -
Advanced Problem (Difficulty ⭐⭐): Implement a concurrent prime number filter: Given a number n, use 4 goroutines to concurrently check which numbers between 1 and n are prime, and return a list of prime numbers. You must use a WaitGroup for synchronization and compare the performance difference between the concurrent and serial approaches.
-
Challenge (Difficulty ⭐⭐⭐): Build a concurrent task scheduler: Given a batch of tasks (
[]func() Result), execute them concurrently using a worker pool model. The scheduler must support: (1) configurable number of workers; (2) Fault tolerance (a panic in a single goroutine does not affect others); (3) Progress callbacks (print a message every time 10% of the tasks are completed); (4) Detection ofruntime.NumGoroutineleaks.
