Pointer Walk

Cascading Failures and Load Shedding

One server dies, the rest go over 100% and fall in turn. Refusing some requests keeps the others alive.

System DesignIntermediate9 min read

One failure should not matter

You run four servers so that losing one is not a disaster. The load balancer notices the dead server and sends its traffic to the other three. Users should never know.

Whether that works depends on a number that is easy to overlook: how busy the servers were before.

Example

The arithmetic of losing one

Four servers each run at 80% of what they can handle. That feels safe: everyone has 20% to spare.

One fails. Its traffic is shared among the remaining three. The same total load divided by three is 107% each.

Every surviving server is now being asked for more than it can do.

Watch it spread

No load sheddingInteractive

Load on each server

80%
A
80%
B
80%
C
80%
D

Share of requests answered, tick by tick

now: 100%

All four servers are up, at 80% each. The dashed line is 100%, the most a server can handle. Server A is about to fail.

Servers up
4 / 4
Answered
0%
Refused quickly
0%
Lost
0%
  1. Server A fails

    The reason does not matter: a bad disk, a bug, a routine restart.

  2. The other three go to 107%

    They slow down. Requests queue up inside them and memory fills.

  3. The weakest of them falls over

    Now two servers share everything: 160% each.

  4. They fail almost at once

    The last server standing receives 320% and lasts one tick.

  5. Restarting does not help

    A server that comes back is alone. It gets all the traffic immediately and goes down again.

This is a cascading failure: one failure causes the next. A small fault took out a system that had four times the redundancy it seemed to need. And the last step is what makes these outages so long. The system cannot be restarted piece by piece, because each piece is crushed the moment it appears.

Why overload kills a server

A server pushed past its limit does not simply answer what it can and ignore the rest. It accepts everything and starts work on all of it. Every request gets slower. Slow requests pile up, holding memory and connections. Clients time out and retry, adding more. Eventually it runs out of something and crashes.

In short, an overloaded server that tries to help everyone ends up helping no one.

The fix: refuse what you cannot handle

Load shedding means a server knows its own limit and says "no" to anything beyond it, immediately. A refusal costs almost nothing, so the server keeps its strength for the requests it accepted.

With load sheddingInteractive

Load on each server

80%
A
80%
B
80%
C
80%
D

Share of requests answered, tick by tick

now: 100%

All four servers are up, at 80% each. The dashed line is 100%, the most a server can handle. Server A is about to fail.

Servers up
4 / 4
Answered
0%
Refused quickly
0%
Lost
0%

Server A fails just as before. The other three go to their limit and stop there. They refuse the extra 7% with a quick error. About 94% of requests are still answered, no server is damaged, and when A is restarted it rejoins a working system.

Compare the two outcomes. Without shedding, almost every request failed for the rest of the run. With shedding, 6% of requests got a quick error for a few ticks.

python
MAX_IN_PROGRESS = 100
in_progress = 0

def handle(request):
    global in_progress
    if in_progress >= MAX_IN_PROGRESS:
        return Response(503, headers={"Retry-After": "2"})   # full: refuse at once
    in_progress += 1
    try:
        return do_the_work(request)
    finally:
        in_progress -= 1

Other protections

  • Leave real headroom. With 4 servers, each must stay at or below 75% to survive losing one, because three servers at 100% carry exactly what four carried at 75%. In general: with N servers, keep each below (N − 1) ÷ N.
  • Circuit breakers. A caller that sees a service failing stops calling it for a while, instead of adding to the pile.
  • Careful retries. Retries multiply load exactly when a system is weakest. See Retries, Backoff and Jitter.
  • Bring traffic back gradually. After an outage, let a restarted server take a small share first and ramp up.

Try both

No load sheddingInteractive
Overloaded

Load on each server

80%
A
80%
B
80%
C
80%
D

Share of requests answered, tick by tick

now: 100%

All four servers are up, at 80% each. The dashed line is 100%, the most a server can handle. Server A is about to fail.

Servers up
4 / 4
Answered
0%
Refused quickly
0%
Lost
0%

Key takeaways

  • When a server fails, its load moves to the others. If they were already busy, that pushes them over their limit.
  • An overloaded server slows down and fails, moving even more load to those that remain.
  • After a full collapse, restarted servers are crushed one at a time by the full traffic.
  • Load shedding refuses the excess quickly, so most requests are served and no server is harmed.
  • Keep enough headroom to lose a server: with N servers, stay below (N − 1) ÷ N each.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress