One copy is not enough
Keep your data on one server and you have two problems. If that server dies, the data is unavailable, perhaps gone. And every read in the world has to go to that one machine.
The answer is replication: keep the same data on several servers. Lose one and the others carry on. Reads can be shared between them.
It sounds simple. The hard part is hidden in the word "same". Data changes, and a change cannot reach every copy at the same instant.
Leader and followers
The most common arrangement gives the copies different roles.
- One server is the leader. All writes go to it.
- The others are followers, also called replicas. They receive every change from the leader and apply it in the same order.
- Reads can go to any of them.
With a single place where writes happen, there is never an argument about which change came first.
How does a change get from leader to follower? The leader sends its log of changes, the same log it uses for crash safety. See The Write Path and the Write-Ahead Log.
Copy afterwards: fast, and briefly wrong
The leader can confirm a write as soon as it has saved it locally, and send it to followers a moment later. This is asynchronous replication.
A user changes their name from Ana to Anna. The data is kept on two servers.
The write was quick. But a read that reached the follower before the change did got the old value. The time between a write on the leader and its arrival on a follower is replication lag. It is usually milliseconds. Under heavy load or a slow network it can be seconds or more.
Example
The edit that disappears
You change your profile name and press Save. The page reloads and shows your old name. You save again, confused.
Your write went to the leader. Your reload was served by a follower that had not caught up. The change was never lost. You just read from a copy that was behind.
A system that behaves this way is called eventually consistent: stop writing, wait a moment, and all copies agree.
Wait for the copy: safe, and slower
The alternative is for the leader to wait until a follower has confirmed before it tells the app the write succeeded. This is synchronous replication.
A user changes their name from Ana to Anna. The data is kept on two servers.
No reader can see stale data from that follower, and every confirmed write exists on two machines. The cost is speed: each write waits for a round trip. And if the follower is slow or down, writes stall.
Many systems compromise: wait for one follower, and let the rest catch up afterwards.
What happens when the leader dies
If the leader fails, a follower is promoted to take its place. This is a failover. With asynchronous replication it has a sharp edge.
A user changes their name from Ana to Anna. The data is kept on two servers.
A write that was confirmed to the user existed only on the leader. The leader died before passing it on. The new leader never saw it.
This is the real trade-off of replication, and every system has to pick a side.
| Copy afterwards | Wait for a copy | |
|---|---|---|
| Write speed | Fast | Slower |
| Reads from followers | May be stale | Fresh from the one that confirmed |
| Leader crashes | Recent writes can be lost | Confirmed writes survive |
| A follower is down | No effect on writes | Writes may stall |
A user changes their name from Ana to Anna. The data is kept on two servers.
Living with lag
Most applications use asynchronous replication for speed and handle the awkward cases deliberately.
Read your own writes from the leader
After a user changes something, send that user's reads to the leader for a short while. Everyone else can read from followers.
Do not jump between followers mid-session
A user who reads from an up-to-date follower and then a lagging one sees time go backwards. Keep each user on one.
Watch the lag
Measure how far behind each follower is, and stop sending reads to one that falls too far back.
Send critical reads to the leader
Checking a balance before a payment is not the place for a stale answer.
Other arrangements
One leader is the simplest design, and writes are limited by that one machine. Two alternatives exist, both harder to get right:
- Several leaders, each accepting writes, often one per region. Two leaders can accept conflicting changes to the same data, and something must decide who wins.
- No leader at all. A write is sent to several servers at once and counts as done when enough of them confirm. See Split Brain and Quorum.
Key takeaways
- Replication keeps copies of data on several servers, for safety and to share reads.
- With a leader and followers, all writes go to the leader and are copied to the rest.
- Copying afterwards is fast but followers lag, so reads can be stale.
- Waiting for a copy before confirming is safer and slower.
- If a leader dies before copying a write, that write can be lost in the failover.