Why put a queue in the middle
Two parts of a system rarely work at the same speed. A web server accepts an upload in milliseconds. Turning that upload into five video sizes takes minutes. If the web server had to wait, every user would wait.
So the fast part writes a note saying "process this" and moves on. The notes go into a queue. The slow part takes them out one at a time, at its own pace.
- The part that adds to the queue is the producer.
- The part that takes from it is the consumer.
A queue is very good at one thing: absorbing a burst. Ten thousand uploads in one minute and none in the next is no problem. The queue fills, then drains.
What a queue cannot fix
Now make one small change. The producer is not bursty. It is just steadily, slightly faster than the consumer: 101 items per second against 100. A gap of 1%.
Each tick below is one minute, so the whole run shows an hour.
Producer
101
per second
Queue (no limit)
0
Consumer
100
per second
Queue length, minute by minute
The rule: The queue accepts everything and grows as long as it has to. The producer sends 101 items per second and the consumer handles 100.
- Minute
- 0 / 60
- Waiting in queue
- 0
- Wait for a new item
- 0 s
- Thrown away
- 0
After one hour, 3,600 items are waiting and a new item waits 36 seconds before anyone looks at it. Nothing is broken. No error was logged. And it does not stop there.
Example
The same gap, left running
One extra item per second is 86,400 extra items per day. After a day, a new item waits more than 14 minutes. After a week, over an hour and a half. The queue has no level it settles at. It grows for as long as the gap exists.
Compare that with a producer that is 1% slower than the consumer:
Producer
99
per second
Queue (no limit)
0
Consumer
100
per second
Queue length, minute by minute
The rule: The queue accepts everything and grows as long as it has to. The producer sends 99 items per second and the consumer handles 100.
- Minute
- 0 / 60
- Waiting in queue
- 0
- Wait for a new item
- 0 s
- Thrown away
- 0
Empty, the whole time. The difference between a healthy system and a doomed one is which side of the line the rates fall on.
Fix 1: limit the queue and drop the overflow
Every queue should have a maximum size. The first thing you can do when it is full is throw new items away.
Producer
101
per second
Queue (holds 1,000)
0
Consumer
100
per second
Queue length, minute by minute
The rule: The queue holds 1,000 items. When it is full, new items are thrown away. The producer sends 101 items per second and the consumer handles 100.
- Minute
- 0 / 60
- Waiting in queue
- 0
- Wait for a new item
- 0 s
- Thrown away
- 0
The wait is now capped: the queue holds 1,000 items, so nothing waits more than 10 seconds. The price is lost work. This is the right trade when a fresh item is worth more than a late one.
Good at
- The wait can never grow past a known limit
- Simple, and the producer is not affected
Where it bites
- Work is lost, so it only suits data you can afford to lose
Fix 2: backpressure
The other choice is to tell the producer the truth. When the queue is full, the producer is made to wait until there is room. It cannot send faster than the consumer can receive. This is backpressure: the slow side pushing back on the fast side.
Producer
101
per second
Queue (holds 1,000)
0
Consumer
100
per second
Queue length, minute by minute
The rule: The queue holds 1,000 items. When it is full, the producer is made to wait. The producer sends 101 items per second and the consumer handles 100.
- Minute
- 0 / 60
- Waiting in queue
- 0
- Wait for a new item
- 0 s
- Held at producer
- 0
Nothing is lost, the wait is capped, and the producer has been slowed to 100 per second. The problem has not vanished. It has been moved to the producer, which now has to decide what to do with work it cannot send. That is the point: the pressure travels back up the chain until it reaches something that can act on it, such as a user seeing "busy, try again".
import queue
jobs = queue.Queue(maxsize=1000) # a queue with a limit
def produce(item):
jobs.put(item) # waits here while the queue is full
def consume():
while True:
handle(jobs.get())Good at
- Nothing is lost and the wait stays bounded
- The overload becomes visible where it starts, instead of hiding in a queue
Where it bites
- The producer slows down, and must be able to cope with that
- Everything upstream needs to respect the signal, or the pile just forms one step earlier
Which one, and what else
| Your items are | Do this when the queue is full |
|---|---|
| Replaceable, and stale ones are worthless (metrics, live video frames) | Drop |
| Valuable, and the producer can wait (orders, file processing) | Backpressure |
| Arriving faster than you can ever handle | Neither is enough: add consumers, or make each item cheaper |
Dropping and backpressure keep the system alive. Only more consuming power closes the gap.
Try it yourself
Producer
101
per second
Queue (no limit)
0
Consumer
100
per second
Queue length, minute by minute
The rule: The queue accepts everything and grows as long as it has to. The producer sends 101 items per second and the consumer handles 100.
- Minute
- 0 / 60
- Waiting in queue
- 0
- Wait for a new item
- 0 s
- Thrown away
- 0
Key takeaways
- A queue lets a fast producer and a slow consumer work at their own pace.
- It absorbs bursts. It cannot absorb a producer that is steadily faster.
- With a steady gap, even 1%, a queue with no limit grows for ever.
- Always give a queue a maximum size.
- When it is full, either drop new items or push back on the producer.