Replication: High-Availability Architecture

Replica Sets are the cornerstone of MongoDB high availability—mastering them allows you to build production clusters with 99.99% availability.

1. What You'll Learn

• Replica Set Concept (Primary / Secondary / Arbiter)
• Election mechanism (based on Raft)
• Data Synchronization (initial sync / oplog tailing)
• Read-Write Separation (readPreference)
• Deploying a 3-node replica set with Docker
• Failover and Recovery

2. Replica Set Architecture

Concept Overview: A Replica Set is the foundation of MongoDB’s high-availability architecture. It consists of multiple mongod processes—1 Primary node + N Secondary nodes + an optional Arbiter node. The Primary node accepts all write operations, which are asynchronously replicated to the Secondary nodes via the oplog, thereby ensuring data redundancy and automatic failover.

How It Works: The Primary records every write operation to the oplog (a fixed-size capped collection), while the Secondary continuously pulls and replays write operations by tailing the oplog to maintain data consistency with the Primary. When the Primary fails, the remaining nodes elect a new Primary using an election protocol (based on Raft), and the application automatically reconnects, achieving 99.99% availability.

The Raft Protocol and Election Mechanism: MongoDB’s election process is based on a variant of the Raft protocol, with the core rule being “majority rule”—only a candidate who receives votes from more than half of the nodes can become the Primary. This ensures that there is at most one Primary at any given time (to prevent split-brain) and that the new Primary possesses the most complete data.

graph TB
    App[Applications] --> P[Primary<br/>Master Node<br/>Accept all posts]
    P -.Copy oplog.-> S1[Secondary 1<br/>From node<br/>Readable+Backup]
    P -.Copy oplog.-> S2[Secondary 2<br/>From node<br/>Readable+Backup]
    A[Arbiter<br/>Arbitration Node<br/>Vote Only]

    subgraph "Data Flow"
        W[Write] --> P
        P -->|oplog| S1
        P -->|oplog| S2
    end

    subgraph "Election"
        P -->|Heartbeat| S1
        P -->|Heartbeat| S2
        P -->|Heartbeat| A
    end

    style P fill:#d4edda
    style S1 fill:#cce5ff
    style S2 fill:#cce5ff
    style A fill:#fff3cd

graph TB
    App[Applications] --> P[Primary<br/>Master Node<br/>Accept all posts]
    P -.Copy oplog.-> S1[Secondary 1<br/>From node<br/>Readable+Backup]
    P -.Copy oplog.-> S2[Secondary 2<br/>From node<br/>Readable+Backup]
    A[Arbiter<br/>Arbitration Node<br/>Vote Only]

    subgraph "Data Flow"
        W[Write] --> P
        P -->|oplog| S1
        P -->|oplog| S2
    end

    subgraph "Election"
        P -->|Heartbeat| S1
        P -->|Heartbeat| S2
        P -->|Heartbeat| A
    end

    style P fill:#d4edda
    style S1 fill:#cce5ff
    style S2 fill:#cce5ff
    style A fill:#fff3cd

Comparison of Node Roles:

Replica Sets vs. Single Nodes:

3. Start the replica set

Concept Explanation: Starting a replica set involves two steps: 1) Start the mongod process with the --replSet parameter; 2) Execute rs.initiate() in mongosh to initialize the replica set configuration. During initialization, specify the addresses of all members, and MongoDB will automatically elect a Primary.

How It Works: rs.initiate() writes the replica set configuration to each node’s local database, triggering the election protocol. The first node to initialize typically becomes the Primary; the other nodes automatically synchronize the configuration and begin tailing the oplog.

# === Start the first node(Raid Mode)===
mongod --replSet rs0 --port 27017 --dbpath /data/db1 --bind_ip localhost

# === mongosh Initialization ===
mongosh
> rs.initiate({
    _id: 'rs0',
    members: [
      { _id: 0, host: 'localhost:27017' },
      { _id: 1, host: 'localhost:27018' },
      { _id: 2, host: 'localhost:27019' }
    ]
  });

Initialization Configuration Parameters:

4. Adding/Removing Nodes

Concept Explanation: Replica sets support adding and removing nodes online without requiring a service outage. After a node is added, the new node automatically performs an initial sync (full sync) from the Primary, and then switches to oplog tailing (incremental sync).

Data Synchronization Process:

sequenceDiagram
    participant New as New Node
    participant P as Primary
    participant Oplog as oplog

    New->>P: Request to Join a Raid Group
    P-->>New: Confirm + Current Configuration

    Note over New,P: Phase 1: Initial Sync(Full Synchronization)
    New->>P: Request Full Data
    P-->>New: All Aggregated Data + Index

    Note over New,P: Phase 2: Oplog Tailing(Incremental Synchronization)
    loop Continuous
        New->>Oplog: Get the latest oplog Entry
        Oplog-->>New: Incremental Operations
        New->>New: Replay oplog Operation
    end

    Note over New: Once synchronization is complete, you can participate in the election.

sequenceDiagram
    participant New as New Node
    participant P as Primary
    participant Oplog as oplog

    New->>P: Request to Join a Raid Group
    P-->>New: Confirm + Current Configuration

    Note over New,P: Phase 1: Initial Sync(Full Synchronization)
    New->>P: Request Full Data
    P-->>New: All Aggregated Data + Index

    Note over New,P: Phase 2: Oplog Tailing(Incremental Synchronization)
    loop Continuous
        New->>Oplog: Get the latest oplog Entry
        Oplog-->>New: Incremental Operations
        New->>New: Replay oplog Operation
    end

    Note over New: Once synchronization is complete, you can participate in the election.

// === Add a New Node ===
rs.add('localhost:27020');

// === Add Arbiter(Do not save data)===
rs.addArb('localhost:27021');

// === Remove Node ===
rs.remove('localhost:27020');

// === View Replica Set Status ===
rs.status();

Key Points Analysis:

1. During the initial sync, new nodes do not participate in elections or read services; once synchronization is complete, they automatically become Secondary nodes.
2. The initial sync of large datasets takes a long time (100 GB may take several hours), so it is recommended to add them during off-peak hours.
3. Arbiter does not store data; it is used solely to ensure an odd number of nodes participate in voting, and its priority is 0.

5. Election Mechanism (Based on Raft)

Concept Explanation: The election mechanism is central to the high availability of a replica set—when the Primary fails, the Secondary automatically initiates an election to select a new Primary. MongoDB’s election is based on a variant of the Raft protocol, with the core guarantees being “at most one Primary at any given time” (to prevent split-brain) and “the new Primary has the most complete data.”

Raft Election Process:

1. Fault Detection: The Secondary did not receive a heartbeat from the Primary within the electionTimeout (default 10 seconds)
2. Initiate an election: The available Secondary with the highest priority becomes a candidate, and its term is incremented.
3. Request a Vote: The candidate sends a vote request to all nodes
4. Voting Rules: Each node casts only one vote per term, for the candidate with the most up-to-date data.
5. Majority Election: The candidate who receives more than half of the votes (> N/2) becomes the new Primary.
6. Application Reconnection: The driver automatically detects the new primary and switches transparently

sequenceDiagram
    participant S1 as Secondary 1<br/>(Candidates)
    participant S2 as Secondary 2
    participant A as Arbiter

    Note over S1: Primary Heartbeat Timeout 10s<br/>Call for an Election

    S1->>S1: Auto-increment term = 2<br/>Vote for yourself
    S1->>S2: RequestVote(term=2, lastOplogTime)
    S1->>A: RequestVote(term=2, lastOplogTime)

    S2->>S1: GrantVote ✅<br/>(The data is recent enough)
    A->>S1: GrantVote ✅

    Note over S1: receive a majority of the votes (2/3)<br/>Become a New Primary

    S1->>S2: Heartbeat(term=2, role=PRIMARY)
    S1->>A: Heartbeat(term=2, role=PRIMARY)

    Note over S1,S2: The election is over，App Auto-Reconnect

sequenceDiagram
    participant S1 as Secondary 1<br/>(Candidates)
    participant S2 as Secondary 2
    participant A as Arbiter

    Note over S1: Primary Heartbeat Timeout 10s<br/>Call for an Election

    S1->>S1: Auto-increment term = 2<br/>Vote for yourself
    S1->>S2: RequestVote(term=2, lastOplogTime)
    S1->>A: RequestVote(term=2, lastOplogTime)

    S2->>S1: GrantVote ✅<br/>(The data is recent enough)
    A->>S1: GrantVote ✅

    Note over S1: receive a majority of the votes (2/3)<br/>Become a New Primary

    S1->>S2: Heartbeat(term=2, role=PRIMARY)
    S1->>A: Heartbeat(term=2, role=PRIMARY)

    Note over S1,S2: The election is over，App Auto-Reconnect

Key Election Parameters:

graph LR
    A[Primary Failure] --> B[Secondary Testing]
    B --> C{receive a majority of the votes?}
    C -->|Yes| D[Promoted to Primary]
    C -->|No| E[Waiting]
    D --> F[App Reconnection]

    style D fill:#d4edda

graph LR
    A[Primary Failure] --> B[Secondary Testing]
    B --> C{receive a majority of the votes?}
    C -->|Yes| D[Promoted to Primary]
    C -->|No| E[Waiting]
    D --> F[App Reconnection]

    style D fill:#d4edda

Why an Odd Number of Nodes Is Needed: A system with 3 nodes can tolerate 1 failure (2/3 majority vote); a system with 4 nodes can also tolerate 1 failure (3/4 majority vote); and a system with 5 nodes can tolerate 2 failures (3/5 majority vote). An even number of nodes does not increase fault tolerance but does increase costs, so an odd number of nodes is recommended.

▶ Example 1: Election Priority Configuration

// TechCorp:Give priority to the more powerful server Primary
rs.reconfig({
  _id: 'rs0',
  members: [
    { _id: 0, host: 'mongo1:27017', priority: 3 },   // The Strongest Server，Elected by a landslide
    { _id: 1, host: 'mongo2:27017', priority: 2 },   // Second choice
    { _id: 2, host: 'mongo3:27017', priority: 1 },   // Lowest Priority
    { _id: 3, host: 'mongo4:27017', priority: 0 }    // Never Elected(Cold Standby)
  ]
});

// priority: 0 That node will never be elected Primary，Suitable for use as a cold standby or reporting server
// Adjustment priority This will trigger a re-election(If the current Primary No longer the best option)

6. Separation of Read and Write Operations

Concept Explanation: Replica sets inherently support read-write separation—write operations are routed to the Primary, while read operations can be distributed to the Secondary. By configuring the read routing policy via readPreference, the load on the Primary can be significantly reduced in read-intensive scenarios.

How It Works: The Mongoose/Mongo driver uses the readPreference parameter to determine which node a read operation is sent to. primary ensures strong consistency, while secondary reduces the load on the Primary node but introduces replication latency.

readPreference Details:

Consistency vs. Performance Trade-off:

graph LR
    A[primary<br/>Strong consistency<br/>High latency] --> B[primaryPreferred<br/>Strong consensus]
    B --> C[secondaryPreferred<br/>Weak consensus]
    C --> D[secondary<br/>Weak consistency<br/>Low latency]
    D --> E[nearest<br/>Not sure<br/>Lowest Latency]

    style A fill:#d4edda
    style D fill:#fff3cd

graph LR
    A[primary<br/>Strong consistency<br/>High latency] --> B[primaryPreferred<br/>Strong consensus]
    B --> C[secondaryPreferred<br/>Weak consensus]
    C --> D[secondary<br/>Weak consistency<br/>Low latency]
    D --> E[nearest<br/>Not sure<br/>Lowest Latency]

    style A fill:#d4edda
    style D fill:#fff3cd

// === Default:All read and write operations take place in Primary ===
const user = await User.findById(userId);

// === Read from a child node(Reduce Primary Pressure)===
const products = await Product.find().read('secondary');

// === Read Preference Options ===
await Product.find().read('primary');              // Master node only
await Product.find().read('primaryPreferred');     // Preferred Primary Node
await Product.find().read('secondary');            // From the node alone
await Product.find().read('secondaryPreferred');    // Priority Node
await Product.find().read('nearest');              // Recent Online Trends

Key Points Analysis:

1. Reading from a node may involve a replication delay (typically < 1 second, though it may be longer in the event of network issues).
2. In secondary mode, if all Secondary nodes are unavailable, the query will return an error.
3. For scenarios where consistency is not a major concern, such as reports and search index building, we recommend secondary or secondaryPreferred.

7. Deploying Docker on 3 Nodes

Concept Explanation: Docker Compose is the most convenient way to develop and test replica sets locally. A 3-node configuration (1 Primary + 2 Secondary) is the minimum recommended production configuration, providing high availability and automatic failover.

Deployment Architecture:

graph TB
    subgraph "Docker Compose"
        M1[mongo1:27017<br/>Primary/Secondary]
        M2[mongo2:27017<br/>Primary/Secondary]
        M3[mongo3:27017<br/>Primary/Secondary]
    end

    App[Applications] --> M1
    App --> M2
    App --> M3

    M1 <--> M2
    M2 <--> M3
    M1 <--> M3

    style M1 fill:#d4edda
    style M2 fill:#cce5ff
    style M3 fill:#cce5ff

graph TB
    subgraph "Docker Compose"
        M1[mongo1:27017<br/>Primary/Secondary]
        M2[mongo2:27017<br/>Primary/Secondary]
        M3[mongo3:27017<br/>Primary/Secondary]
    end

    App[Applications] --> M1
    App --> M2
    App --> M3

    M1 <--> M2
    M2 <--> M3
    M1 <--> M3

    style M1 fill:#d4edda
    style M2 fill:#cce5ff
    style M3 fill:#cce5ff

# docker-compose.yml
version: '3.8'
services:
  mongo1:
    image: mongo:7.0
    command: mongod --replSet rs0 --port 27017
    ports:
      - "27017:27017"

  mongo2:
    image: mongo:7.0
    command: mongod --replSet rs0 --port 27017
    ports:
      - "27018:27017"

  mongo3:
    image: mongo:7.0
    command: mongod --replSet rs0 --port 27017
    ports:
      - "27019:27017"

// === Initialize the replica set ===
rs.initiate({
  _id: 'rs0',
  members: [
    { _id: 0, host: 'mongo1:27017' },
    { _id: 1, host: 'mongo2:27017' },
    { _id: 2, host: 'mongo3:27017' }
  ]
});

Deployment Steps:

8. Failover

Concept Explanation: Failover is the most critical capability of a replica set—when the Primary node fails, the remaining nodes automatically elect a new Primary, and the application reconnects transparently. The entire process typically completes within 10–30 seconds; during this time, writes are unavailable, but reads can be degraded to the Secondary node.

Failover Process:

sequenceDiagram
    participant App as Applications
    participant P as Primary (mongo1)
    participant S1 as Secondary (mongo2)
    participant S2 as Secondary (mongo3)

    Note over P: mongo1 Failure

    P-xApp: Heartbeat Disconnected
    P-xS1: Heartbeat Disconnected
    P-xS2: Heartbeat Disconnected

    Note over S1,S2: 10Not received yetPrimaryHeartbeat

    S1->>S2: Call for an Election
    S2->>S1: Vote in favor
    S1->>S1: receive a majority of the votes(2/3)

    Note over S1: mongo2 Become a New Primary

    App->>S1: Automatic Reconnection → Writing has returned to normal
    App->>S2: Continue reading(secondaryPreferred)

    Note over App,S1: Total downtime: 10-30 seconds

sequenceDiagram
    participant App as Applications
    participant P as Primary (mongo1)
    participant S1 as Secondary (mongo2)
    participant S2 as Secondary (mongo3)

    Note over P: mongo1 Failure

    P-xApp: Heartbeat Disconnected
    P-xS1: Heartbeat Disconnected
    P-xS2: Heartbeat Disconnected

    Note over S1,S2: 10Not received yetPrimaryHeartbeat

    S1->>S2: Call for an Election
    S2->>S1: Vote in favor
    S1->>S1: receive a majority of the votes(2/3)

    Note over S1: mongo2 Become a New Primary

    App->>S1: Automatic Reconnection → Writing has returned to normal
    App->>S2: Continue reading(secondaryPreferred)

    Note over App,S1: Total downtime: 10-30 seconds

# === Primary Downtime Simulation ===
# kill mongo1 Container
docker stop mongo1

# === Automatic Election(30 Completed in seconds)===
# mongo2 or mongo3 Automatically promoted to Primary

# === App Auto-Reconnect(mongoose Layout)===
mongoose.connect('mongodb://mongo1,mongo2,mongo3/shopdb?replicaSet=rs0');

Application Layer Troubleshooting:

9. Best Practices for Replica Sets

Concept Overview: Best practices for replica sets cover aspects such as the number of nodes, security configurations, and monitoring and alerts. Following these practices can help prevent common replica set failures in production environments.

Key Practices:

writeConcern Comparison:

Monitoring Checklist:

▶ Example: Docker Deployment of a 3-Node Replica Set + Failover Testing

# === 1. docker-compose.yml ===
# version: '3.8'
# services:
#   mongo1:
#     image: mongo:7.0
#     command: mongod --replSet rs0 --bind_ip_all --port 27017
#     ports: ["27017:27017"]
#     networks: [mongo-net]
#
#   mongo2:
#     image: mongo:7.0
#     command: mongod --replSet rs0 --bind_ip_all --port 27017
#     ports: ["27018:27017"]
#     networks: [mongo-net]
#
#   mongo3:
#     image: mongo:7.0
#     command: mongod --replSet rs0 --bind_ip_all --port 27017
#     ports: ["27019:27017"]
#     networks: [mongo-net]
#
# networks:
#   mongo-net:
#     driver: bridge

# Start 3 Node
docker-compose up -d

# === 2. Initialize the replica set ===
mongosh --port 27017 --eval '
  rs.initiate({
    _id: "rs0",
    members: [
      { _id: 0, host: "localhost:27017" },
      { _id: 1, host: "localhost:27018" },
      { _id: 2, host: "localhost:27019" }
    ]
  });
'

# === 3. Verify the replica set status ===
mongosh --port 27017 --eval 'rs.status();'
# Output:
# {
#   set: 'rs0',
#   members: [
#     { name: 'localhost:27017', stateStr: 'PRIMARY' },
#     { name: 'localhost:27018', stateStr: 'SECONDARY' },
#     { name: 'localhost:27019', stateStr: 'SECONDARY' }
#   ]
# }

# === 4. Writing Test Data(Automatically copy to Secondary)===
mongosh --port 27017 --eval '
  db.products.insertOne({ sku: "PHONE-001", title: "Smartphone X", price: 599 });
'

# === 5. Verify Data Replication ===
mongosh --port 27018 --eval '
  db.setSecondaryOk();  // Read Permitted Secondary
  db.products.findOne({ sku: "PHONE-001" });
'
# Return to the same document，Proof that the copy was successful

# === 6. Failover Testing ===
# Stop Primary
docker stop mongo1

# View on mongo2
mongosh --port 27018 --eval 'rs.status();'
# Output:mongo2 Automatically promoted to PRIMARY(30 Completed in seconds)

# === 7. App Auto-Reconnect(mongoose)===

// mongoose The chain automatically discovers all nodes
const mongoose = require('mongoose');
await mongoose.connect(
  'mongodb://localhost:27017,localhost:27018,localhost:27019/shopdb?replicaSet=rs0'
);

// Even if Primary fails, App automatically connects to new Primary
const products = await Product.find();  // Automatic Retry，No errors

// === 8. Read-Write Separation Configuration ===
// Write operation complete Primary
await Product.create({ sku: 'PHONE-002', title: 'Phone 2' });

// Read operations can proceed Secondary(Reduce Primary Pressure)
const products = await Product.find().read('secondaryPreferred');

// === 9. Restore a Failed Node ===
docker start mongo1
# mongo1 Automatically set to Secondary Join a Raid Group，Start Over Primary Synchronize Data

Output: A 3-node replica set provides high availability; in the event of a primary failure, a failover occurs automatically within 30 seconds, with no impact on the application.

❓ FAQ

📖 Summary

• Replica Set: Primary + Secondary + Arbiter
• Election mechanism: Based on Raft; requires a majority vote
• Data Synchronization: initial sync + oplog tailing
• Read-write separation: readPreference
• Docker Deployment on 3 Nodes
• Failover: Automatic election + application reconnection

📝 Exercises

1. Basic Question (⭐): Deploy a 3-node replica set using Docker Compose.
2. Basic Question (⭐): Initialize the replica set and verify its status (rs.status()).
3. Advanced Exercise (⭐⭐): Test read-write separation (readPreference: secondary).
4. Advanced Exercise (⭐⭐): Simulate a Primary failure and observe the automatic election process.
5. Challenge (⭐⭐⭐): Deploy a complete replica set (3 nodes + replica set monitoring + failure recovery testing).

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

About Us