Pointer Walk

Queues and Backpressure

Producers that are 1% faster than consumers build a queue that never ends. Backpressure is how the slow side says stop.

System DesignIntermediate9 min read

Why put a queue in the middle

Two parts of a system rarely work at the same speed. A web server accepts an upload in milliseconds. Turning that upload into five video sizes takes minutes. If the web server had to wait, every user would wait.

So the fast part writes a note saying "process this" and moves on. The notes go into a queue. The slow part takes them out one at a time, at its own pace.

  • The part that adds to the queue is the producer.
  • The part that takes from it is the consumer.

A queue is very good at one thing: absorbing a burst. Ten thousand uploads in one minute and none in the next is no problem. The queue fills, then drains.

What a queue cannot fix

Now make one small change. The producer is not bursty. It is just steadily, slightly faster than the consumer: 101 items per second against 100. A gap of 1%.

Each tick below is one minute, so the whole run shows an hour.

No limit · producer at 101 per secondInteractive

Producer

101

per second

Queue (no limit)

0

Consumer

100

per second

Queue length, minute by minute

The rule: The queue accepts everything and grows as long as it has to. The producer sends 101 items per second and the consumer handles 100.

Minute
0 / 60
Waiting in queue
0
Wait for a new item
0 s
Thrown away
0

After one hour, 3,600 items are waiting and a new item waits 36 seconds before anyone looks at it. Nothing is broken. No error was logged. And it does not stop there.

Example

The same gap, left running

One extra item per second is 86,400 extra items per day. After a day, a new item waits more than 14 minutes. After a week, over an hour and a half. The queue has no level it settles at. It grows for as long as the gap exists.

Compare that with a producer that is 1% slower than the consumer:

No limit · producer at 99 per secondInteractive

Producer

99

per second

Queue (no limit)

0

Consumer

100

per second

Queue length, minute by minute

The rule: The queue accepts everything and grows as long as it has to. The producer sends 99 items per second and the consumer handles 100.

Minute
0 / 60
Waiting in queue
0
Wait for a new item
0 s
Thrown away
0

Empty, the whole time. The difference between a healthy system and a doomed one is which side of the line the rates fall on.

Fix 1: limit the queue and drop the overflow

Every queue should have a maximum size. The first thing you can do when it is full is throw new items away.

Limit, then drop · producer at 101 per secondInteractive

Producer

101

per second

Queue (holds 1,000)

0

Consumer

100

per second

Queue length, minute by minute

queue is full

The rule: The queue holds 1,000 items. When it is full, new items are thrown away. The producer sends 101 items per second and the consumer handles 100.

Minute
0 / 60
Waiting in queue
0
Wait for a new item
0 s
Thrown away
0

The wait is now capped: the queue holds 1,000 items, so nothing waits more than 10 seconds. The price is lost work. This is the right trade when a fresh item is worth more than a late one.

Good at

  • The wait can never grow past a known limit
  • Simple, and the producer is not affected

Where it bites

  • Work is lost, so it only suits data you can afford to lose

Fix 2: backpressure

The other choice is to tell the producer the truth. When the queue is full, the producer is made to wait until there is room. It cannot send faster than the consumer can receive. This is backpressure: the slow side pushing back on the fast side.

Limit, then push back · producer at 101 per secondInteractive

Producer

101

per second

Queue (holds 1,000)

0

Consumer

100

per second

Queue length, minute by minute

queue is full

The rule: The queue holds 1,000 items. When it is full, the producer is made to wait. The producer sends 101 items per second and the consumer handles 100.

Minute
0 / 60
Waiting in queue
0
Wait for a new item
0 s
Held at producer
0

Nothing is lost, the wait is capped, and the producer has been slowed to 100 per second. The problem has not vanished. It has been moved to the producer, which now has to decide what to do with work it cannot send. That is the point: the pressure travels back up the chain until it reaches something that can act on it, such as a user seeing "busy, try again".

python
import queue

jobs = queue.Queue(maxsize=1000)   # a queue with a limit

def produce(item):
    jobs.put(item)                 # waits here while the queue is full

def consume():
    while True:
        handle(jobs.get())

Good at

  • Nothing is lost and the wait stays bounded
  • The overload becomes visible where it starts, instead of hiding in a queue

Where it bites

  • The producer slows down, and must be able to cope with that
  • Everything upstream needs to respect the signal, or the pile just forms one step earlier

Which one, and what else

Your items areDo this when the queue is full
Replaceable, and stale ones are worthless (metrics, live video frames)Drop
Valuable, and the producer can wait (orders, file processing)Backpressure
Arriving faster than you can ever handleNeither is enough: add consumers, or make each item cheaper

Dropping and backpressure keep the system alive. Only more consuming power closes the gap.

Try it yourself

No limit · producer at 101 per secondInteractive
Producer
Queue

Producer

101

per second

Queue (no limit)

0

Consumer

100

per second

Queue length, minute by minute

The rule: The queue accepts everything and grows as long as it has to. The producer sends 101 items per second and the consumer handles 100.

Minute
0 / 60
Waiting in queue
0
Wait for a new item
0 s
Thrown away
0

Key takeaways

  • A queue lets a fast producer and a slow consumer work at their own pace.
  • It absorbs bursts. It cannot absorb a producer that is steadily faster.
  • With a steady gap, even 1%, a queue with no limit grows for ever.
  • Always give a queue a maximum size.
  • When it is full, either drop new items or push back on the producer.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress