mongodb in production - amazon web...
TRANSCRIPT
![Page 2: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/2.jpg)
HIGH AVAILABILITY SCENARIOS
![Page 3: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/3.jpg)
Single Node
•Will have downtime•If node crashes human intervention might be needed
![Page 4: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/4.jpg)
How MongoDB Replication works
Member 1
Member 2
Member 3
•Set is made up of 2 or more nodes
![Page 5: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/5.jpg)
How MongoDB Replication works
Member 1
Member 2
PRIMARY
Member 3
•Election establishes the PRIMARY•Data replication from PRIMARY to SECONDARY
![Page 6: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/6.jpg)
How MongoDB Replication works
Member 1
Member 2
DOWN
Member 3
negotiate new master
•PRIMARY may fail•Automatic election of new PRIMARY if majority exists
![Page 7: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/7.jpg)
How MongoDB Replication works
Member 1
Member 2
DOWN
Member 3
PRIMARY
•New PRIMARY elected•Replication Set re-established
![Page 8: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/8.jpg)
How MongoDB Replication works
Member 1
Member 2RECOVERING
Member 3
PRIMARY
•Automatic recovery
![Page 9: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/9.jpg)
How MongoDB Replication works
Member 1
Member 2
Member 3
PRIMARY
•Replication Set re-established
![Page 10: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/10.jpg)
DEMO
![Page 11: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/11.jpg)
Replica Set 0•Two Node•Network failure can cause the nodes to slip which will result in the the whole system going READ_ONLY
NOT RECOMMENDED!
![Page 12: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/12.jpg)
Replica Set 1
Arbiter
•Single datacenter•Single switch & power•Points of failure:•Power•Network•Datacenter•Two node failure
•Automatic recovery of single node crash
![Page 13: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/13.jpg)
Replica Set 2
Arbiter
•Single datacenter•Multiple power/network zones•Points of failure:•Datacenter•Two node failure
•Automatic recovery of single node crash•w=2 not viable as losing 1 node means no writes
![Page 14: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/14.jpg)
Replica Set 3•Single datacenter•Multiple power/network zones•Points of failure:•Datacenter•Two node failure
•Automatic recovery of single node crash•w=2 viable as 2/3 online
![Page 15: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/15.jpg)
![Page 16: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/16.jpg)
Replica Set 4
•Multi datacenter
•DR node for safety
•Can’t do multi data center durable write safely since only 1 node in distant DC
![Page 17: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/17.jpg)
Replica Set 5
delayed •Three data centers•Can survive full data center loss •Can do w= { dc : 2 } to guarantee write in 2 data centers
![Page 18: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/18.jpg)
HARDWARE SIZING
![Page 19: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/19.jpg)
Collection 1
Index 1
![Page 20: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/20.jpg)
Virtual Address Space 1
Collection 1
Index 1
![Page 21: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/21.jpg)
Virtual Address Space 1
Collection 1
Index 1 This is your virtual memory size (mapped)
![Page 22: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/22.jpg)
Virtual Address Space 1
Physical RAM
Collection 1
Index 1
![Page 23: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/23.jpg)
Virtual Address Space 1
Physical RAM
Collection 1
Index 1
This is your resident memory size
![Page 24: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/24.jpg)
Virtual Address Space 1
Physical RAM
DiskCollection 1
Index 1
![Page 25: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/25.jpg)
Virtual Address Space 1
Physical RAM
DiskCollection 1
Index 1
100 ns
10,000 ns
=
=
![Page 26: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/26.jpg)
Sizing RAM and Disk• Working set• Document Size• Memory versus disk• Data lifecycle patterns
• Long tail• pure random• bulk removes
![Page 27: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/27.jpg)
Figuring out working Set> db.wombats.stats() { "ns" : "test.wombats", "count" : 1338330, "size" : 46915928, "avgObjSize" : 35.05557523181876, "storageSize" : 86092032, "numExtents" : 12, "nindexes" : 2, "lastExtentSize" : 20872960, "paddingFactor" : 1, "flags" : 0, "totalIndexSize" : 99860480, "indexSizes" : { "_id_" : 55877632, "name_1" : 43982848 },
Size of data
Size on disk (and in memory!)
Size of all indexes
Average document size
Size of each index
![Page 28: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/28.jpg)
Disk configurations
~200 seeks / second
Single Disk
![Page 29: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/29.jpg)
Disk configurations
~200 seeks / second
~200 seeks / second ~200 seeks / second ~200 seeks / second
Single Disk
RAID 0
![Page 30: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/30.jpg)
Disk configurations
~200 seeks / second
~200 seeks / second ~200 seeks / second ~200 seeks / second
~400 seeks / second ~400 seeks / second ~400 seeks / second
Single Disk
RAID 0
RAID 10
![Page 31: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/31.jpg)
SSD?• Seek time of 0.1ms vs 5ms
(200 seeks / sec => 10000 seeks / sec)
• But expensive
![Page 32: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/32.jpg)
Takeaway• Know how important page faults are
–If you want low latency, avoid page faults• Size memory appropriately
–To avoid page faults, fit everything in RAM–Collection Data + Index Data
• Provision disk appropriately–RAID10 is recommended–SSD’s are fast, if you can afford them
![Page 33: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/33.jpg)
MONITORING
![Page 34: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/34.jpg)
Monitoring Toolsmongostat -
MMS! - http://mms.10gen.com
munin, cacti, nagios -http://www.mongodb.org/display/DOCS/Monitoring+and+Diagnostics
![Page 35: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/35.jpg)
Monitoring is your friend!
![Page 36: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/36.jpg)
SUMMARY
![Page 37: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/37.jpg)
Typical DeploymentsUse? Set
sizeData Protection
High Availability Notes
X One No No Must use --journal to protect against crashes
Two Yes No On loss of one member, surviving member is read only
Three Yes Yes - 1 failureOn loss of one member, surviving two members can elect a new primary
X Four Yes Yes - 1 failure** On loss of two members, surviving two members are read only
Five Yes Yes - 2 failuresOn loss of two members, surviving three members can elect a new primary
![Page 38: MongoDB in Production - Amazon Web Servicesxebia-video.s3-website-eu-west-1.amazonaws.com/2012-02/... · 2012-02-14 · MongoDB in Production. HIGH AVAILABILITY SCENARIOS. Single](https://reader035.vdocument.in/reader035/viewer/2022070710/5ec59409101d3f6b2c55f267/html5/thumbnails/38.jpg)
Summary
• Reads from Primary are always consistent
• Reads from Secondaries are eventually consistent
• Automatic failover if a Primary fails
• Understand your Working Set
• Monitoring is your Friend!