Pointer Walk

Replication and Replication Lag

Copies of the data on several servers. Writes go to one and are copied to the rest, and for a moment the copies disagree.

Distributed SystemsIntermediate10 min read

One copy is not enough

Keep your data on one server and you have two problems. If that server dies, the data is unavailable, perhaps gone. And every read in the world has to go to that one machine.

The answer is replication: keep the same data on several servers. Lose one and the others carry on. Reads can be shared between them.

It sounds simple. The hard part is hidden in the word "same". Data changes, and a change cannot reach every copy at the same instant.

Leader and followers

The most common arrangement gives the copies different roles.

  • One server is the leader. All writes go to it.
  • The others are followers, also called replicas. They receive every change from the leader and apply it in the same order.
  • Reads can go to any of them.

With a single place where writes happen, there is never an argument about which change came first.

How does a change get from leader to follower? The leader sends its log of changes, the same log it uses for crash safety. See The Write Path and the Write-Ahead Log.

Copy afterwards: fast, and briefly wrong

The leader can confirm a write as soon as it has saved it locally, and send it to followers a moment later. This is asynchronous replication.

Asynchronous replicationInteractive
App
Leader
name = Ana
Follower
name = Ana

A user changes their name from Ana to Anna. The data is kept on two servers.

The write was quick. But a read that reached the follower before the change did got the old value. The time between a write on the leader and its arrival on a follower is replication lag. It is usually milliseconds. Under heavy load or a slow network it can be seconds or more.

Example

The edit that disappears

You change your profile name and press Save. The page reloads and shows your old name. You save again, confused.

Your write went to the leader. Your reload was served by a follower that had not caught up. The change was never lost. You just read from a copy that was behind.

A system that behaves this way is called eventually consistent: stop writing, wait a moment, and all copies agree.

Wait for the copy: safe, and slower

The alternative is for the leader to wait until a follower has confirmed before it tells the app the write succeeded. This is synchronous replication.

Synchronous replicationInteractive
App
Leader
name = Ana
Follower
name = Ana

A user changes their name from Ana to Anna. The data is kept on two servers.

No reader can see stale data from that follower, and every confirmed write exists on two machines. The cost is speed: each write waits for a round trip. And if the follower is slow or down, writes stall.

Many systems compromise: wait for one follower, and let the rest catch up afterwards.

What happens when the leader dies

If the leader fails, a follower is promoted to take its place. This is a failover. With asynchronous replication it has a sharp edge.

Losing a write in a failoverInteractive
App
Leader
name = Ana
Follower
name = Ana

A user changes their name from Ana to Anna. The data is kept on two servers.

A write that was confirmed to the user existed only on the leader. The leader died before passing it on. The new leader never saw it.

This is the real trade-off of replication, and every system has to pick a side.

Copy afterwardsWait for a copy
Write speedFastSlower
Reads from followersMay be staleFresh from the one that confirmed
Leader crashesRecent writes can be lostConfirmed writes survive
A follower is downNo effect on writesWrites may stall
Asynchronous replicationInteractive
Scenario
App
Leader
name = Ana
Follower
name = Ana

A user changes their name from Ana to Anna. The data is kept on two servers.

Living with lag

Most applications use asynchronous replication for speed and handle the awkward cases deliberately.

  1. Read your own writes from the leader

    After a user changes something, send that user's reads to the leader for a short while. Everyone else can read from followers.

  2. Do not jump between followers mid-session

    A user who reads from an up-to-date follower and then a lagging one sees time go backwards. Keep each user on one.

  3. Watch the lag

    Measure how far behind each follower is, and stop sending reads to one that falls too far back.

  4. Send critical reads to the leader

    Checking a balance before a payment is not the place for a stale answer.

Other arrangements

One leader is the simplest design, and writes are limited by that one machine. Two alternatives exist, both harder to get right:

  • Several leaders, each accepting writes, often one per region. Two leaders can accept conflicting changes to the same data, and something must decide who wins.
  • No leader at all. A write is sent to several servers at once and counts as done when enough of them confirm. See Split Brain and Quorum.

Key takeaways

  • Replication keeps copies of data on several servers, for safety and to share reads.
  • With a leader and followers, all writes go to the leader and are copied to the rest.
  • Copying afterwards is fast but followers lag, so reads can be stale.
  • Waiting for a copy before confirming is safer and slower.
  • If a leader dies before copying a write, that write can be lost in the failover.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    8 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress