back to basics: build something big with mongodb
DESCRIPTION
TRANSCRIPT
![Page 1: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/1.jpg)
Solution Architect, MongoDB
Sam Weaver
#MongoDBBasics
‘MongoDB Back to Basics’
Build something Big with MongoDB
![Page 2: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/2.jpg)
Agenda
• Replica Sets Lifecycle
• Developing with Replica Sets
• Scaling your database
![Page 3: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/3.jpg)
Q&A
• Virtual Genius Bar– Use chat to post
questions– Solution Architecture /
Support Team are on hand
– Make use of them during the sessions!!!
![Page 4: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/4.jpg)
Recap
• Introduction to MongoDB
• Thinking in documents
![Page 5: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/5.jpg)
Deployment Considerations
![Page 6: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/6.jpg)
Working Set Exceeds Physical Memory
![Page 7: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/7.jpg)
Why Replication?
• How many have faced node failures?
• How many have been woken up from sleep to do a fail-over(s)?
• How many have experienced issues due to network latency?
• Different uses for data– Normal processing– Simple analytics
![Page 8: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/8.jpg)
Replica Set Lifestyle
![Page 9: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/9.jpg)
Replica Set – Creation
![Page 10: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/10.jpg)
Replica Set – Initialize
![Page 11: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/11.jpg)
Replica Set – Failure
![Page 12: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/12.jpg)
Replica Set – Failover
![Page 13: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/13.jpg)
Replica Set – Recovery
![Page 14: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/14.jpg)
Replica Set – Recovered
![Page 15: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/15.jpg)
Developing with Replica Sets
![Page 16: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/16.jpg)
Strong Consistency
![Page 17: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/17.jpg)
Delayed Consistency
![Page 18: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/18.jpg)
Write Concern
• Network acknowledgement
• Wait for error
• Wait for journal sync
• Wait for replication
![Page 19: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/19.jpg)
Unacknowledged
![Page 20: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/20.jpg)
MongoDB Acknowledged (wait for error)
![Page 21: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/21.jpg)
Wait for Journal Sync
![Page 22: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/22.jpg)
Wait for Replication
![Page 23: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/23.jpg)
Tagging
• Control where data is written to, and read from
• Each member can have one or more tags– tags: {dc: "ny"}– tags: {dc: "ny", subnet: "192.168", rack:
"row3rk7"}
• Replica set defines rules for write concerns
• Rules can change without changing app code
![Page 24: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/24.jpg)
{
_id : "mySet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}],
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.blogs.insert({...})
> db.runCommand({getLastError : 1, w : "someDCs"})
Tagging Example
![Page 25: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/25.jpg)
Wait for Replication (Tagging)
![Page 26: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/26.jpg)
Read Preference Modes
• 5 modes– primary (only) - Default– primaryPreferred– secondary– secondaryPreferred– Nearest
When more than one node is possible, closest node is used for reads (all modes but primary)
![Page 27: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/27.jpg)
Tagged Read Preference
• Custom read preferences
• Control where you read from by (node) tags– E.g. { "disk": "ssd", "use": "reporting" }
• Use in conjunction with standard read preferences– Except primary
![Page 28: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/28.jpg)
Our application
//connect to a replica set, with auto-discovery of the primary, supply a seed list of members
MongoClient mongoClient = new MongoClient(Arrays.asList(new ServerAddress("localhost", 27017), new ServerAddress("localhost", 27018), new ServerAddress("localhost", 27019)));
DB db = mongoClient.getDB( "mydb" );
![Page 29: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/29.jpg)
Scaling
![Page 30: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/30.jpg)
Working Set Exceeds Physical Memory
![Page 31: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/31.jpg)
• When a specific resource becomes a bottle neck on a machine or replica set• RAM• Disk IO• Storage• Concurrency
When to consider Sharding?
![Page 32: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/32.jpg)
Vertical Scalability (Scale Up)
![Page 33: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/33.jpg)
Horizontal Scalability (Scale Out)
![Page 34: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/34.jpg)
Partitioning
• User defines shard key
• Shard key defines range of data
• Key space is like points on a line
• Range is a segment of that line
![Page 35: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/35.jpg)
Initially 1 chunk
Default max chunk size: 64mb
MongoDB automatically splits & migrates chunks when max reached
Data Distribution
![Page 36: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/36.jpg)
Architecture
![Page 37: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/37.jpg)
What is a Shard?
• Shard is a node of the cluster
• Shard can be a single mongod or a replica set
![Page 38: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/38.jpg)
Meta Data Storage
• Config Server– Stores cluster chunk ranges and locations– Can have only 1 or 3 (production must have
3)– Not a replica set
![Page 39: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/39.jpg)
Routing and Managing Data
• Mongos– Acts as a router / balancer– No local data (persists to config database)– Can have 1 or many
![Page 40: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/40.jpg)
Sharding infrastructure
![Page 41: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/41.jpg)
Cluster Request Routing
• Targeted Queries
• Scatter Gather Queries
• Scatter Gather Queries with Sort
![Page 42: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/42.jpg)
Cluster Request Routing: Targeted Query
![Page 43: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/43.jpg)
Routable request received
![Page 44: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/44.jpg)
Request routed to appropriate shard
![Page 45: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/45.jpg)
Shard returns results
![Page 46: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/46.jpg)
Mongos returns results to client
![Page 47: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/47.jpg)
Cluster Request Routing: Non-Targeted Query
![Page 48: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/48.jpg)
Non-Targeted Request Received
![Page 49: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/49.jpg)
Request sent to all shards
![Page 50: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/50.jpg)
Shards return results to mongos
![Page 51: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/51.jpg)
Mongos returns results to client
![Page 52: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/52.jpg)
Cluster Request Routing: Non-Targeted Query with Sort
![Page 53: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/53.jpg)
Non-Targeted request with sort received
![Page 54: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/54.jpg)
Request sent to all shards
![Page 55: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/55.jpg)
Query and sort performed locally
![Page 56: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/56.jpg)
Shards return results to mongos
![Page 57: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/57.jpg)
Mongos merges sorted results
![Page 58: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/58.jpg)
Mongos returns results to client
![Page 59: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/59.jpg)
Shard Key
![Page 60: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/60.jpg)
Shard Key
• Shard key is immutable
• Shard key values are immutable
• Shard key must be indexed
• Shard key limited to 512 bytes in size
• Shard key used to route queries– Choose a field commonly used in queries
• Only shard key can be unique across shards– `_id` field is only unique within individual shard
![Page 61: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/61.jpg)
Summary
![Page 62: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/62.jpg)
Things to remember
• Size appropriately for your working set
• Shard when you need to, not before
• Pick a shard key wisely
![Page 63: Back to Basics: Build Something Big With MongoDB](https://reader035.vdocument.in/reader035/viewer/2022062511/54b6d9a94a79597f788b4592/html5/thumbnails/63.jpg)
Thank you