A thread for every task
A server receives requests. The simplest way to handle many at once is to start a new thread for each one. The thread does the work, sends the reply and ends.
It is easy to write and it works well for a handful of requests.
Threads
Start. Every task gets a brand new thread of its own, created when the task arrives and destroyed when it finishes.
- Threads alive
- 0
- Most at once
- 0
- Waiting in queue
- 0
- Done
- 0 / 12
Twelve tasks, twelve threads, all alive at once. Now imagine a spike of ten thousand requests.
Why that falls over
Threads are not free.
- Each one needs memory. A thread has its own stack, typically a megabyte or so reserved. Ten thousand threads reserve gigabytes just to exist.
- Creating and destroying them takes time. For a short task, setting up the thread can cost more than the task.
- Switching between them is work. With far more threads than processor cores, the operating system spends its time swapping threads in and out. See Do More Threads Make It Faster?.
- There is no limit. The busier the server gets, the more threads it creates, the slower each one runs. Under the heaviest load, it uses the most resources to do the least work, and then runs out of memory.
A fixed team and a queue
A thread pool separates two things that the first design tied together: the tasks to be done and the threads that do them.
Create a fixed number of worker threads once, at the start
They are never destroyed.
Put incoming tasks in a queue
A task is just a description of work to do.
Each worker takes a task from the queue, runs it, and comes back for the next
If the queue is empty, the worker sleeps until something arrives.
Threads
Queue
Start. Three worker threads are created once and kept. Tasks that arrive while all three are busy wait in a queue.
- Threads alive
- 3
- Most at once
- 3
- Waiting in queue
- 0
- Done
- 0 / 12
Three threads, the whole time. The burst did not create more threads. It made the queue longer for a moment.
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=3) as pool:
results = pool.map(handle, requests)What you gain, and what you pay
Good at
- Resource use is bounded: the number of threads never grows
- No cost of creating a thread per task
- A burst of work is absorbed by the queue
Where it bites
- Tasks wait when all workers are busy
- The pool size must be chosen, and the right size depends on the work
- One stuck task occupies a worker for as long as it is stuck
In the animation the pool took longer than a thread per task. With twelve tasks that is true. The pool wins when there are thousands, because the alternative does not finish at all.
Threads
Queue
Start. Three worker threads are created once and kept. Tasks that arrive while all three are busy wait in a queue.
- Threads alive
- 3
- Most at once
- 3
- Waiting in queue
- 0
- Done
- 0 / 12
How big should the pool be?
It depends on what the tasks spend their time doing.
| The tasks mostly | A sensible pool size | Why |
|---|---|---|
| Compute | About the number of processor cores | More threads than cores cannot run at the same time anyway |
| Wait (network, database, disk) | Many more than the number of cores | A waiting thread does not use a core |
Example
A rough way to size it
Each task spends 90 milliseconds waiting on a database and 10 milliseconds computing. A thread is only using a core a tenth of the time.
So one core can keep about 10 such threads busy. On 8 cores, around 80 threads is a reasonable starting point. Then measure and adjust.
Two things that go wrong
The queue needs a limit too. A pool caps the threads. If the queue can grow without bound, a sustained overload just piles up tasks until memory runs out, with every one of them getting later. Give the queue a maximum size and decide what happens when it is full. See Queues and Backpressure.
A task must not wait for another task in the same pool. If every worker is busy with a task that is waiting for a second task, and those second tasks are sitting in the queue with no worker free to run them, nothing can ever proceed. The pool has deadlocked itself. Keep such tasks in separate pools.
Pools are everywhere
The same idea, a fixed set of reusable things with a queue in front, appears all over software.
- Web servers handle requests with a pool of workers.
- Database connection pools reuse a fixed set of connections. See Connection Pools.
- Languages' async runtimes run thousands of lightweight tasks on a small pool of real threads.
Key takeaways
- A thread per task is simple and collapses under load, because threads cost memory and switching time.
- A thread pool keeps a fixed number of workers and queues the tasks.
- A burst makes the queue longer, not the number of threads larger.
- Size the pool near the core count for computing work, and larger for work that mostly waits.
- Limit the queue as well, and never make a task wait on another task in the same pool.