Replication

Think of it like this

Analogy

A teacher writes the day's notes on the main whiteboard. Students copy them into their notebooks. Anyone who missed class can borrow a notebook instead of crowding around the board. If the whiteboard gets wiped, the notes still exist in many notebooks.

But someone who copies slowly might still have yesterday's version when you borrow their notebook. That's replication lag.

The problem it solves

A single database server is a single point of failure and a single bottleneck. If its disk dies, you lose data. If it's busy, every read waits. You need more than one copy, but copies must be kept up to date, and that introduces delays and choices.

A real example

Example

A news site on election night

Journalists publish updates to the leader database. Five read replicas serve the millions of readers refreshing the results page.

At 23:40 the leader's disk fails. Monitoring notices, a replica with the most recent data is promoted, and the app switches to it in about 30 seconds. Readers barely notice, because the replicas kept serving reads the whole time.

There's a catch. Replication is asynchronous, so a story published a moment before the crash may not have reached the new leader. And on a normal night, a journalist who publishes and immediately refreshes can see the old version for a second, because their read went to a replica that was slightly behind.

How it works

Replication continuously copies changes from one database to others.

Leader–follower (single leader)

The most common setup, also called primary–replica.

• All writes go to the leader, which records them in a log.
• Followers (read replicas) replay that log to stay in sync, and serve reads.
• If the leader fails, a follower is promoted. That's failover.

Architecture

Loading diagram…

> Initializing simulation kernel

> Allocating nodes

> Opening network links

• App servers to Leader: writes
• Leader to Follower 1: change log
• Leader to Follower 2: change log

Synchronous or asynchronous

• Synchronous: the leader waits for a follower to confirm before telling the client “saved”. No data loss if the leader dies, but writes are slower and stall if that follower is unavailable.
• Asynchronous: the leader confirms immediately and followers catch up. Fast, but followers lag, and if the leader dies, its last few writes may be lost.
• Semi-synchronous: wait for one follower, the rest async. A common compromise, and similar to what managed “high availability” options do with a standby in another zone.

Replication lag and stale reads

With async replication, followers are usually milliseconds behind, but under heavy load it can be seconds or more. That replication lag causes visible bugs. Try it:

Ways to hide lag:

• Read-your-writes: after a user writes, send their reads to the leader for a while (or until the follower has caught up to that write).
• Monotonic reads: pin each user to one follower so they never see time go backwards.
• Read critical data from the leader: balances, inventory, anything just written.

Multi-leader

Several nodes, often one per region, accept writes and sync with each other. Users write to a nearby leader, so writes are fast everywhere and a region can go offline. But two leaders can change the same record at the same time, and the system must resolve the conflict: last-write-wins (simple, loses data), merge rules, or special data types designed to merge (CRDTs). Multi-leader setups are powerful and hard to get right.

Leaderless

Some databases (Cassandra, DynamoDB's internals) write to several replicas at once and read from several, using quorums so that enough of them agree (lesson 21).

Failover is the risky moment

Promoting a follower sounds simple, but it raises questions. Is the old leader really dead, or just slow? If it comes back thinking it's still the leader, you have two leaders (“split brain”). Async followers may be missing the last writes. Managed databases automate this with health checks and fencing, but you should still test failover regularly.

Trade-offs

What you gain

• Survives a machine or zone failing without losing the database.
• Spreads read traffic across followers.
• Followers in other regions give nearby users fast reads, and backups can be taken from a follower without loading the leader.

What it costs you

• Async followers can serve stale data.
• Doesn't scale writes: every write still goes through one leader.
• Failover can lose recent writes or cause split brain if done badly.
• Multi-leader adds write conflicts.

When not to use it

• Replication doesn't fix write bottlenecks or too much data for one machine. That's sharding (next lesson).
• Don't send reads that must be up to date (like “did my payment go through?”) to async followers.

In the cloud

Replicated and multi-region databasesEvery major cloud sells this as a managed service, so you configure it rather than build it.

AWS2

• Amazon Aurora Global Database (opens vendor docs)

  One primary region with read-only copies in other regions, usually under a second behind.

• Amazon DynamoDB global tables (opens vendor docs)

  Multi-region, multi-writer tables; replicas converge with last-writer-wins.

Azure2

• Azure Cosmos DB (multi-region) (opens vendor docs)

  Replicate to any region, optionally write everywhere, and pick one of five consistency levels.

• Azure SQL Database active geo-replication (opens vendor docs)

  Readable secondary databases in other regions with manual or automatic failover.

Google Cloud2

• Spanner (opens vendor docs)

  Globally distributed relational database with strong consistency across regions.

• Cloud SQL cross-region read replicas (opens vendor docs)

  Read-only copies of a Cloud SQL instance in another region, promotable on disaster.

Writes go to a leader; replicas in other zones or regions serve reads and take over on failure.

AWSAzureGoogle Cloud

1. App
2. Aurora writer
3. Aurora readers
4. Secondary region (Global Database)

Interview questions

Try answering out loud first, then open the model answer.

How does leader–follower replication work, and what are its failure modes?+

Model answer

The leader takes all writes and appends them to a replication log; followers stream and apply the log in order and serve reads. Failure modes: replication lag giving stale reads; data loss on failover if async followers missed the last writes; split brain if the old leader comes back without being fenced off; and a leader bottleneck since writes don't scale. Mitigations are semi-sync replication to at least one standby, automatic failover with fencing, read-your-writes routing, and monitoring lag.

How would you serve users in Europe and the US with low latency?+

Model answer

For read-heavy data, keep one leader and add read replicas in each region, so reads are local while writes go to the leader's region; use read-your-writes for a user's own changes. If writes must also be local, use a database built for multi-region writes, such as DynamoDB global tables, Cosmos DB with multi-region writes, or Spanner, and decide how conflicts resolve. Or partition users by home region so each user's data has one leader near them.

What is split brain and how do you prevent it?+

Model answer

Split brain is when two nodes both believe they're the leader and accept writes, so the data diverges. It happens when a network problem makes a healthy leader look dead and a follower is promoted. Prevent it with a consensus-based election that needs a majority (quorum), fencing tokens or epochs so storage rejects writes from an old leader, and shutting down or isolating the old leader before promoting a new one.

Quiz

Pick an answer to see whether it’s right and why.

1. 01A user updates their profile and the page reloads showing the old value. Reads go to followers. What's happening?

   A cache bugReplication lagThe write failed
2. 02In leader–follower replication, which node accepts writes?

   Any nodeOnly the leaderOnly followers
3. 03What's the main new problem multi-leader replication introduces?

   Reads become slowerTwo leaders can accept conflicting writes to the same dataFollowers can't serve reads
4. 04With synchronous replication to one follower, what do you gain and lose?

   Gain speed, lose durabilityGain a guaranteed up-to-date copy, lose some write latency and availability if that follower is downNothing changes
5. 05The leader fails and an async follower is promoted. What can be lost?

   Nothing, everWrites the leader confirmed but hadn't yet sent to that followerAll of the data
6. 06Someone runs a DELETE on the wrong table. Do replicas save you?

   No. The delete is copied to every replica, so you need backupsYes, replicas keep old dataOnly synchronous replicas do
7. 07After a network split, two nodes both believe they're the leader and accept writes. What's this called?

   A hot partitionA cache stampedeSplit brain

0 of 7 answered

Key terms

Replication

Keeping copies of the same data on more than one machine.

Leader–follower replication

One copy (the leader) takes all writes and streams them to read-only copies (followers).

Multi-leader replication

Several copies accept writes and sync with each other, which needs a way to resolve conflicts.

Replication lag

The delay between a write on the leader and that write appearing on a follower.

Failover

Switching to a backup when the primary fails, such as promoting a follower to leader.

Read replica

A read-only copy of a database used to spread read traffic.

Read-your-writes

A guarantee that after you change something, you see your change, even if others don't yet.