Thread pools, how servers run tasks on a fixed set of threads.

A thread pool is a fixed set of worker threads that pull tasks off a shared queue and run them. The pattern dates to the 1990s; modern implementations include Java's ExecutorService, Go's goroutine scheduler, Rust's Tokio runtime, and the kernel's workqueue subsystem. Pool size is the main tuning knob. CPU-bound work sizes near NumCPU, and I/O-bound work sizes much higher.

A thread pool separates how work arrives from how it runs. Producers push tasks onto a queue; a fixed number of long-lived worker threads loop, dequeue and run. No thread is created or destroyed per task, so the cost (typically ~100 μs to spawn, plus stack allocation) is paid once, at startup.

Both the worker count and the queue need a bound. With unbounded workers the server keeps creating threads under load until it runs out of memory or thrashes. With an unbounded queue it replies ever more slowly and never pushes back. Bounding both makes the server fail predictably, much like a ring buffer with backpressure.

Submit jobs of varying size. The dispatcher hands them to free workers, and busy workers count down. The queue grows when arrivals exceed pool capacity, which is how back-pressure forms. (See rate limiting for the upstream version.)

The right size depends on what the workers do. For CPU-bound work (compression, parsing, math) you want N ≈ number of cores. Beyond that, threads time-share, adding context-switch overhead without finishing more work.

For I/O-bound work (database, HTTP, disk) the worker spends most of its time waiting. N should be much higher, and Brian Goetz's formula is N = cores × (1 + wait/compute). A worker that waits 90 ms per 10 ms of compute can have 10× cores in the pool before saturating.

An unbounded queue is a common cause of production incidents. It absorbs load until the process runs out of memory, or until tail latency stretches into hours because the queue holds minutes of work. Bound the queue and pick a rejection policy.

A single shared queue creates contention as N grows: every dispatch touches the same lock or atomic. Work-stealing pools (Java's ForkJoin, Go's runtime, Rust's Tokio multi-thread) give each worker its own deque. Workers push and pop their own end (no contention); when one runs dry, it steals from a busy neighbour's tail.

This works because most tasks come from local recursion (a worker submitting a child of its own work), so touching another worker's queue is the rare path. ForkJoinPool, Go's scheduler and Tokio all use it to scale to dozens of cores without lock contention dominating.

The classic Tomcat-style server gives each request a thread, blocking on I/O. This is fine up to a few thousand concurrent requests, and beyond that the per-thread stack (1 MB default) and context-switching cost dominate. To go higher you either move to async I/O (one thread serves many connections via an event loop on epoll/kqueue) or to user-space threads.

Java 21's virtual threads, Go's goroutines, and Kotlin's coroutines sit between these approaches. The code looks blocking, but the runtime parks the lightweight task while a small pool of OS threads carries the work. The dispatch model is the same, with a different unit of work.

Active workers tells you how saturated the pool is. Sitting at N in steady state leaves no headroom for spikes. Queue depth tells you whether you are absorbing or shedding load, and a depth above zero for a sustained interval means back-pressure is forming. Rejection rate is the early warning, and if your rejection policy is "abort", every spike will trigger an alert.

Also track p99 latency for the dispatch step itself (queue time, not just task time). The Little's Law identity, concurrency = arrival rate × latency, follows directly from these metrics.

Java ExecutorService

Doug Lea's ThreadPoolExecutor (j.u.concurrent, 2004) is the standard reference. It has configurable core and max threads, a work queue and a rejection policy. Executors.newFixedThreadPool(N) uses an unbounded queue, which hides back-pressure until the JVM runs out of memory. In production, configure it explicitly with new ThreadPoolExecutor and a bounded queue.

Java ForkJoinPool / virtual threads

ForkJoinPool (Java 7) does work-stealing. Java 21's virtual threads (Project Loom) make a thread-per-request model viable again. Millions of cheap virtual threads run on a small carrier-thread pool, which removes much of the reason to size a thread pool tightly.

.NET ThreadPool

Auto-scales from Environment.ProcessorCount up to a high ceiling (~32k by default). The CLR injects threads at ~500ms intervals when the queue is non-empty, which is the source of the "ThreadPool starvation" stall. Modern .NET code usually relies on Task.Run and async instead of manual pool sizing.

Rust Tokio runtime

Multi-threaded by default, work-stealing across N worker threads (defaults to NumCPU). Each worker has a local queue (256 entries) plus a shared injection queue. Tokio's blog posts on its scheduler are useful background.

Go runtime (GMP)

Not strictly a thread pool: goroutines are user-mode tasks scheduled onto OS threads via the GMP model (see the goroutine scheduler simulator). Effectively a thread pool of GOMAXPROCS workers, each draining a per-P run queue with work stealing.

Python concurrent.futures

ThreadPoolExecutor(max_workers=N). Defaults to min(32, os.cpu_count() + 4). Because of the GIL, CPU-bound code doesn't benefit from threads, so use ProcessPoolExecutor for that.

Unbounded queue leading to OOM (the most common). Many Java services have died because Executors.newFixedThreadPool(N) uses an unbounded LinkedBlockingQueue. The producer rate exceeds the consumer rate, the queue grows without limit, the JVM runs out of memory and restarts, and the cycle repeats.

Pool starvation by recursive submit. A pool of N threads where a task submits another task to the same pool and waits for its result. With N concurrent root tasks, all N threads are blocked waiting for sub-tasks that can never run. Use a separate pool, or ForkJoinPool, which handles this case.

Slow downstream dragging the whole pool. One downstream service goes slow; threads in the pool block on downstream calls; queue depth grows; new requests get rejected even though the local CPU is idle. Per-downstream bulkhead pools (the pattern Hystrix popularised) or async I/O with a small thread pool avoid this.

Resource leaks on exceptions. Tasks that throw don't always release resources cleanly, particularly when the rejection handler itself blocks. Wrap submitted tasks in a try/finally that releases anything they hold.

Thread pools often fail by growing or blocking without raising any error. The three metrics from Part 07 (active workers, queue depth, rejection rate) are the early-warning signals for these four failure modes.