One failure should not matter
You run four servers so that losing one is not a disaster. The load balancer notices the dead server and sends its traffic to the other three. Users should never know.
Whether that works depends on a number that is easy to overlook: how busy the servers were before.
Example
The arithmetic of losing one
Four servers each run at 80% of what they can handle. That feels safe: everyone has 20% to spare.
One fails. Its traffic is shared among the remaining three. The same total load divided by three is 107% each.
Every surviving server is now being asked for more than it can do.
Watch it spread
Load on each server
Share of requests answered, tick by tick
now: 100%
All four servers are up, at 80% each. The dashed line is 100%, the most a server can handle. Server A is about to fail.
- Servers up
- 4 / 4
- Answered
- 0%
- Refused quickly
- 0%
- Lost
- 0%
Server A fails
The reason does not matter: a bad disk, a bug, a routine restart.
The other three go to 107%
They slow down. Requests queue up inside them and memory fills.
The weakest of them falls over
Now two servers share everything: 160% each.
They fail almost at once
The last server standing receives 320% and lasts one tick.
Restarting does not help
A server that comes back is alone. It gets all the traffic immediately and goes down again.
This is a cascading failure: one failure causes the next. A small fault took out a system that had four times the redundancy it seemed to need. And the last step is what makes these outages so long. The system cannot be restarted piece by piece, because each piece is crushed the moment it appears.
Why overload kills a server
A server pushed past its limit does not simply answer what it can and ignore the rest. It accepts everything and starts work on all of it. Every request gets slower. Slow requests pile up, holding memory and connections. Clients time out and retry, adding more. Eventually it runs out of something and crashes.
In short, an overloaded server that tries to help everyone ends up helping no one.
The fix: refuse what you cannot handle
Load shedding means a server knows its own limit and says "no" to anything beyond it, immediately. A refusal costs almost nothing, so the server keeps its strength for the requests it accepted.
Load on each server
Share of requests answered, tick by tick
now: 100%
All four servers are up, at 80% each. The dashed line is 100%, the most a server can handle. Server A is about to fail.
- Servers up
- 4 / 4
- Answered
- 0%
- Refused quickly
- 0%
- Lost
- 0%
Server A fails just as before. The other three go to their limit and stop there. They refuse the extra 7% with a quick error. About 94% of requests are still answered, no server is damaged, and when A is restarted it rejoins a working system.
Compare the two outcomes. Without shedding, almost every request failed for the rest of the run. With shedding, 6% of requests got a quick error for a few ticks.
MAX_IN_PROGRESS = 100
in_progress = 0
def handle(request):
global in_progress
if in_progress >= MAX_IN_PROGRESS:
return Response(503, headers={"Retry-After": "2"}) # full: refuse at once
in_progress += 1
try:
return do_the_work(request)
finally:
in_progress -= 1Other protections
- Leave real headroom. With 4 servers, each must stay at or below 75% to survive losing one, because three servers at 100% carry exactly what four carried at 75%. In general: with N servers, keep each below (N − 1) ÷ N.
- Circuit breakers. A caller that sees a service failing stops calling it for a while, instead of adding to the pile.
- Careful retries. Retries multiply load exactly when a system is weakest. See Retries, Backoff and Jitter.
- Bring traffic back gradually. After an outage, let a restarted server take a small share first and ramp up.
Try both
Load on each server
Share of requests answered, tick by tick
now: 100%
All four servers are up, at 80% each. The dashed line is 100%, the most a server can handle. Server A is about to fail.
- Servers up
- 4 / 4
- Answered
- 0%
- Refused quickly
- 0%
- Lost
- 0%
Key takeaways
- When a server fails, its load moves to the others. If they were already busy, that pushes them over their limit.
- An overloaded server slows down and fails, moving even more load to those that remain.
- After a full collapse, restarted servers are crushed one at a time by the full traffic.
- Load shedding refuses the excess quickly, so most requests are served and no server is harmed.
- Keep enough headroom to lose a server: with N servers, stay below (N − 1) ÷ N each.