Pointer Walk

Thread Pools

A thread per task works until there are ten thousand tasks. A fixed team of workers and a queue does the same job with a fraction of the cost.

ConcurrencyIntermediate8 min read

A thread for every task

A server receives requests. The simplest way to handle many at once is to start a new thread for each one. The thread does the work, sends the reply and ends.

It is easy to write and it works well for a handful of requests.

A new thread for every taskInteractive

Threads

idle

Start. Every task gets a brand new thread of its own, created when the task arrives and destroyed when it finishes.

Threads alive
0
Most at once
0
Waiting in queue
0
Done
0 / 12

Twelve tasks, twelve threads, all alive at once. Now imagine a spike of ten thousand requests.

Why that falls over

Threads are not free.

  • Each one needs memory. A thread has its own stack, typically a megabyte or so reserved. Ten thousand threads reserve gigabytes just to exist.
  • Creating and destroying them takes time. For a short task, setting up the thread can cost more than the task.
  • Switching between them is work. With far more threads than processor cores, the operating system spends its time swapping threads in and out. See Do More Threads Make It Faster?.
  • There is no limit. The busier the server gets, the more threads it creates, the slower each one runs. Under the heaviest load, it uses the most resources to do the least work, and then runs out of memory.

A fixed team and a queue

A thread pool separates two things that the first design tied together: the tasks to be done and the threads that do them.

  1. Create a fixed number of worker threads once, at the start

    They are never destroyed.

  2. Put incoming tasks in a queue

    A task is just a description of work to do.

  3. Each worker takes a task from the queue, runs it, and comes back for the next

    If the queue is empty, the worker sleeps until something arrives.

A pool of 3 workersInteractive

Threads

idle
idle
idle

Queue

empty

Start. Three worker threads are created once and kept. Tasks that arrive while all three are busy wait in a queue.

Threads alive
3
Most at once
3
Waiting in queue
0
Done
0 / 12

Three threads, the whole time. The burst did not create more threads. It made the queue longer for a moment.

python
from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=3) as pool:
    results = pool.map(handle, requests)

What you gain, and what you pay

Good at

  • Resource use is bounded: the number of threads never grows
  • No cost of creating a thread per task
  • A burst of work is absorbed by the queue

Where it bites

  • Tasks wait when all workers are busy
  • The pool size must be chosen, and the right size depends on the work
  • One stuck task occupies a worker for as long as it is stuck

In the animation the pool took longer than a thread per task. With twelve tasks that is true. The pool wins when there are thousands, because the alternative does not finish at all.

A pool of 3 workersInteractive
Strategy

Threads

idle
idle
idle

Queue

empty

Start. Three worker threads are created once and kept. Tasks that arrive while all three are busy wait in a queue.

Threads alive
3
Most at once
3
Waiting in queue
0
Done
0 / 12

How big should the pool be?

It depends on what the tasks spend their time doing.

The tasks mostlyA sensible pool sizeWhy
ComputeAbout the number of processor coresMore threads than cores cannot run at the same time anyway
Wait (network, database, disk)Many more than the number of coresA waiting thread does not use a core

Example

A rough way to size it

Each task spends 90 milliseconds waiting on a database and 10 milliseconds computing. A thread is only using a core a tenth of the time.

So one core can keep about 10 such threads busy. On 8 cores, around 80 threads is a reasonable starting point. Then measure and adjust.

Two things that go wrong

The queue needs a limit too. A pool caps the threads. If the queue can grow without bound, a sustained overload just piles up tasks until memory runs out, with every one of them getting later. Give the queue a maximum size and decide what happens when it is full. See Queues and Backpressure.

A task must not wait for another task in the same pool. If every worker is busy with a task that is waiting for a second task, and those second tasks are sitting in the queue with no worker free to run them, nothing can ever proceed. The pool has deadlocked itself. Keep such tasks in separate pools.

Pools are everywhere

The same idea, a fixed set of reusable things with a queue in front, appears all over software.

  • Web servers handle requests with a pool of workers.
  • Database connection pools reuse a fixed set of connections. See Connection Pools.
  • Languages' async runtimes run thousands of lightweight tasks on a small pool of real threads.

Key takeaways

  • A thread per task is simple and collapses under load, because threads cost memory and switching time.
  • A thread pool keeps a fixed number of workers and queues the tasks.
  • A burst makes the queue longer, not the number of threads larger.
  • Size the pool near the core count for computing work, and larger for work that mostly waits.
  • Limit the queue as well, and never make a task wait on another task in the same pool.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress