Retrying sounds harmless
Networks drop things. A request fails, you send it again, it works. Every client library can do this, and for one client talking to one server it is the right thing to do.
The trouble starts when a thousand clients do it at the same time, to a server that is already having a bad moment.
The server stalls for a few seconds
A slow disk, a deploy, a pause in the runtime. It happens, and it passes by itself.
Every request sent during the stall fails
They all time out at about the same moment, when the clients stop waiting.
Every client retries
The server wakes up to its normal traffic plus everything it missed, all at once.
That is more than it can handle, so most of those fail too
And failed requests are retried. The server is healthy again, and it is buried under requests that exist only because earlier requests failed.
This is a retry storm. The original fault lasted seconds. The retries can keep the outage going for as long as clients keep trying.
Watch it happen
Five new requests arrive each tick. The server can handle eight, so it has room to spare. At tick 8 it stalls for four ticks, then comes back as good as new. Clients retry a failed request on the next tick, up to five times.
Requests sent, tick by tick
Server: healthy- new requests
- retries
- sent during the stall
The rule: A failed request is sent again on the very next tick. 5 new requests arrive each tick and the server can handle 8. At tick 8 it stalls for 4 ticks.
- Answered
- 0
- Waiting to retry
- 0
- Gave up
- 0
The server is fine from tick 12 onwards. Look at what it faces: around 25 requests a tick, and most of the bar is pink. Those are not new users. They are the same requests coming back.
Fix 1: wait longer each time
If the server is struggling, the kindest thing a client can do is leave it alone for a bit. Exponential backoff means doubling the wait after each failure.
Example
Doubling the wait
The first retry waits 2 ticks. If that fails, the next waits 4, then 8, then 16, then 32. Five retries are spread over a minute instead of being fired in five consecutive ticks.
def call_with_backoff(request, attempts=6):
for attempt in range(attempts):
try:
return send(request)
except TemporaryError:
time.sleep(2 ** (attempt + 1)) # 2, 4, 8, 16, 32 seconds
raise GaveUp()Requests sent, tick by tick
Server: healthy- new requests
- retries
- sent during the stall
The rule: Wait 2 ticks before the first retry, then 4, then 8, doubling each time. 5 new requests arrive each tick and the server can handle 8. At tick 8 it stalls for 4 ticks.
- Answered
- 0
- Waiting to retry
- 0
- Gave up
- 0
Much better: the server is not buried. But look at the shape. There are tall spikes with calm in between. Every client that failed at the same moment waits exactly the same time, so they all come back together. Then the ones that fail again wait exactly twice as long, and come back together again. Backoff moved the herd further apart in time. It did not break the herd up.
Fix 2: add randomness
The spikes exist because the clients are in step with each other. Jitter means making each wait random, so they fall out of step.
Example
The same waits, spread out
Instead of "wait 4 ticks", each client waits a random time between 1 and 8 ticks. Twenty clients that failed together now return a few at a time across that whole span.
import random
def call_with_backoff_and_jitter(request, attempts=6):
for attempt in range(attempts):
try:
return send(request)
except TemporaryError:
limit = 2 ** (attempt + 2)
time.sleep(random.uniform(0, limit)) # anywhere up to the limit
raise GaveUp()Requests sent, tick by tick
Server: healthy- new requests
- retries
- sent during the stall
The rule: Double the wait each time, and pick a random moment inside it. 5 new requests arrive each tick and the server can handle 8. At tick 8 it stalls for 4 ticks.
- Answered
- 0
- Waiting to retry
- 0
- Gave up
- 0
No spikes. The backlog arrives as a gentle bump that the server's spare capacity soaks up, and a few ticks later everything is back to normal with nobody giving up.
Compare them yourself
Requests sent, tick by tick
Server: healthy- new requests
- retries
- sent during the stall
The rule: A failed request is sent again on the very next tick. 5 new requests arrive each tick and the server can handle 8. At tick 8 it stalls for 4 ticks.
- Answered
- 0
- Waiting to retry
- 0
- Gave up
- 0
More ways to keep retries safe
Backoff with jitter is the baseline. Busy systems add a few more rules.
- Limit the attempts. Three tries is common. A request that has failed three times is unlikely to work on the fourth, and each try adds load.
- Retry at one layer only. If the app retries 3 times, and the service it calls retries 3 times, and the database driver retries 3 times, one click becomes 27 queries.
- Use a retry budget. Allow retries only up to a small share of normal traffic, for example 10%. When everything is failing, the budget runs out and the retries stop.
- Only retry what is safe to repeat. Reading a page twice is harmless. Charging a card twice is not. Make such requests safe to repeat first, for example by sending a unique id that the server uses to ignore duplicates.
- Do not retry "no". A
400or404will fail the same way every time. Retry timeouts and503, and honourRetry-Afterwhen the server sends it.
Key takeaways
- A retry storm is an outage kept alive by retries after the original fault has gone.
- An overloaded server does less useful work, so piling on makes recovery harder.
- Exponential backoff doubles the wait after each failure and takes the pressure off.
- Without jitter, clients that failed together retry together, in spikes.
- Random jitter spreads them out, and the server absorbs the backlog.
- Limit attempts, retry at one layer, and only retry requests that are safe to repeat.