mongodb database replication

Database Replication

valikhani@sirangweb.com@mehdivk

Mehdi Valikhani

Replication

Process of keeping IDENTICAL COPY of your data on different servers.

Replication offersMinimum Downtimein case of hardware failure, software failure or maintenance

Replication is highly recommended for all databases in production from the very first day.

Replica SetMongoDB’s Terminology for Replication

A replica set is a group of servers with one primary, the server taking client requests, and multiple secondaries, servers that keep copies of the primary’s data.

Give A Go To MongoDB’s Replica SetIn60 Seconds!

#ToBeTriedOnlyAtHome

Step 1: Setup Replica Set

$: mongo --nodb

$: myReplica = new ReplSetTest({nodes: 3});

$: myReplica.startSet();

$: myReplica.initiate();

Congratulations!3 member of replica set running on ports 31000, 31001 and 31002

Step 2: Connect To Primary

$: mongo --port 31000

testReplSet:PRIMARY> use surryhills_db

testReplSet:PRIMARY> db.offices.insert({name: ‘MongoDB’});

testReplSet:PRIMARY> show dbs

testReplSet:PRIMARY> show collections

testReplSet:PRIMARY> db.offices.find().pretty()

Pro Tip :] Auto Format Query Response

Add DBQuery.prototype._prettyShell = true;to your ~/.mongorc.js

Step 3: Connect To A Secondary

$: mongo --port 31001

testReplSet:SECONDARY> db.slaveOk()

testReplSet:SECONDARY> show dbs

testReplSet:SECONDARY> use surryhills_db

testReplSet:SECONDARY> show collections

testReplSet:SECONDARY> db.offices.find()

Step 4: Shutdown Replica Set

$: myReplica.stopSet()

Setup Your Replica SetPart 1: Run replica members

$: mongod --port 31000 --fork --logpath ./path/to/log --dbpath /path/to/srv1 --replSet myReplica

$: mongod --port 31001 --fork --logpath ./path/to/log --dbpath /path/to/srv2 --replSet myReplica

$: mongod --port 31002 --fork --logpath ./path/to/log --dbpath /path/to/srv3 --replSet myReplica

Setup Your Replica SetPart 2: Configure the set

$: mongo --port 31000

$: rs.initiate();

$: rs.add(‘my-host:31001’);

$: rs.add(‘my-host:31002’);

Setup Your Replica SetPart 2: Example Configuration$: rs.config();{ "_id" : "myReplica",

"version": 3, "members" : [ { "_id" : 0, "host" : "my-host:31000" }, { "_id" : 1, "host" : "my-host:31001" }, { "_id" : 2, "host" : "my-host:31002" } ]}

Auto Connect To Current Primary

$: mongo --host myReplica/my-host:31000,my-host:31001,my-host:31002

Replica Set Good To Knows

1. Always start your instance with `replSet`

2. You can’t initiate a replica set with more than one member with existing data on it.

3. If one of set members has data, you must run rs.initiate() against that member

4. `system.replset` collection on `local` database of all members contains set configuration.

5. Replica sets can have up to 50 members, 7 voting

6. By default, MongoDB reads from primary

Replica Set ReconfigurationAdd new member

$: mongo --port 31000 # current primary

$: rs.add(‘my-host:31003’) # accepts object as well.

Replica Set ReconfigurationRemove existing member

$: mongo --port 31000 # current primary

$: rs.remove(‘my-host:31003’)

Replica Set ReconfigurationUpdate existing member

$: mongo --port 31000 # current primary

$: config = rs.config()

$: config.members[2].priority = 0;

$: rs.reconfig(config);

How Replication WorksMongoDB keeps a log of every write operations that primary does in database

Logs are saved in “oplog.rs” collection of “local” db

Secondaries apply changes by reading from oplog.rs and applieing to their data

Secondaries maintain their own “oplog.rs” as well.

“oplog.rs” Facts“oplog.rs” is a capped collection which means it can save limited amount of operation logs.

Bulk operations fill “oplog” quickly. A bulk removal of 1K documents adds 1K documents to “oplog”

It’s important to have a “oplog.rs” to keep logs for atleast last 24 hrs.

“Stale” Secondary :(Secondary misses a copy of last operation it applied in syncing source.

Secondary queries other members to find another source to continue syncing.

Secondary missed some operations permanently. Admin must resyncs from scratch.

HeartbeatMembers PING each other every 2 seconds.

Members should reply within 10 seconds otherwise they’re counted as unreachable.

Change in replica set configuration

Current primary steps down

Replica Set ElectionWhen It Happens?

Current primary is unreachable

MajorityThe Single Most Important Fact About Replica Set

More than half of All Voting Members in the set must be reachable.

You need a majority of voting members to elect a primary.

A primary can only stay primary so long as it can reach a majority of voting members.

Majority

Replica Set with two members.

Three Common Replica Set Design Mistakes :(

Even number of members in two data center.

A set with more than one arbiter

Keep a majority of members in preferred data center and rest in the other one.

Two Common Replica Set Design Patterns :)

Keep even number of members in two data center and a tie breaker in third one.

Meet Arbiter Budget Friendly Member Of Replica Set

Does not hold data

It’s a lightweight process

Can vote on elections

Start An Arbiter Process

$: mongod --port 31005 --replSet myReplica --nojournal --smallfiles

Arbiter does not save data, no need for journaling

Don’t waste disk space and use --smallfiles

Add An Arbiter To Replica Set

$: rs.addArb(‘arbiter-host-name:port’);

ArbiterGood To Know

You can’t change an arbiter to non-arbiter member

If possible, use normal members instead of arbiters

At most you need one arbiter in your set

Do not add an arbiter to a set with odd members

Replica SetUseful Commands

$: rs.config()

$: rs.reconfig()

$: rs.isMaster()

$: rs.status()

$: rs.stepDown()

Rollback :(Primary does a write and goes down before at least one secondary has a chance to replicate the write.

RollbackMain Reasons

Network Partitions

Different write rates between primary and secondary

Rollback Process

1. Former primary reverts write operations

2. Saves affected documents into /rollback directory

3. Admin browses documents using mongorestore

4. Admin applies changes to the current primary

DurabilityWrite Concerns And Replica Set

1. Unacknowledged

2. Acknowledged

3. Journaled

4. Replica Acknowledged

5. Custom Writer Concern

Improve Durabilitydb.runCommand({getLastError:1}) options

Returns the error status of the preceding write operation on the current connection.

j: boolean

w: number | majority | custom role

wtimeout: number (milliseconds)

fsync: boolean

“wtimeout” GOOD TO KNOW

MongoDB does not “rollback” or ”undo modifications” made before the “wtimeout” interval expired.

Custom Write ConcernsStep 1: Tags

$: config = rs.config()

$: config.members[0].tags = {dc: ‘dc-1’};

$: config.members[3].tags = {dc: ‘dc-2’};

$: rs.reconfig(config);

Custom Write ConcernsStep 2: Custom Last Error Mode

$: config = rs.config()

$: config.settings.lastErrorModes = [{

“each-dc”: {dc: 2}

}];

$: rs.reconfig(config);

Custom Write ConcernsStep 3: Use it!

$: db.runCommand({getLastError:1, w: ‘each-dc’})

Replica Set & Mixed Storage Engines

You can have both WiredTiger & MMAPv1

● Due to migration● Due to different use cases

Keen To Go Further?

Register for M102: MONGODB FOR DBAS course!

Starts tomorrow!It’s awesome!

https://university.mongodb.com/courses/M102/about

Keen To Go Further?docs.mongodb.org/manual/replication

MongoDBThe Definite Guide

Stay In Touch● Follow me on Twitter

@mehdivk

● Add me to your Linkedin network https://au.linkedin.com/in/valikhani

● Subscribe to my blogblog.mehdivk.net

● Send me an emailvalikhani@sirangweb.com

Thank You!