mongodb training for java software engineers
TRANSCRIPT
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MongoDBfor Java Software Engineers
Copyrights © Moshe [email protected]
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MongoDBfor Java Software Engineers
Moshe KaplanScale Hacker
http://top-performance.blogspot.comhttp://blogs.microsoft.co.il/vprnd
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
About Me: It’s all About
3
Scale
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
NOSQL. ANSWER A NEEDIntroduction
4
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
5
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
The Consumer Revolution
6
http://topyaps.com/wp-content/uploads/2013/03/You-are-the-product.-You-feeling-something.jpg
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
At the fraction of the cost…
7
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
8
http://lifehacker.com/5697167/if-youre-not-paying-for-it-youre-the-product
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Transportation
9
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Moovit
10
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
The Medical Market Opportunities
11
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MediSafe
12
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
13
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Askem
14
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Major Enablers: Mobile, Cloud and IT Commoditization
15
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
The Prime Suspect
16
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
17
Assumptions…
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Where did it Fail?Get an Answer, Fast and Cheap
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Where did it Fail?I Just Want “Class Persistency
Storage” and Changing Schema on Demand
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Where did it Fail?Be Always Available, Even w/ an Old
Answer
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Where did it Fail?Get Me Fast and Good Enough
Answer
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Where did it Fail?Data is Too Big, and Storage is $$$
But CPU and Network are Even More
http://www.powerbyte.com/Isilon.html
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Software Providers
23
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
It is all great, but…I Need to Meet Compliance
http://www.vision7.com/app_system/lib/image/content/PCI_compliance.jpg
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
It is all great, but…I Need a Vendor
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
It is all great, but…I Need Reporting
http://www.novell.com/communities/node/5851/get-ready-sentinel-61
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
It is all great, but…I Need Transactions
http://www.novell.com/communities/node/5851/get-ready-sentinel-61
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
It is all great, but…We Need Training for the Data Analysts
db.article.aggregate(
{ $group : {
_id : "$author",
docsPerAuthor : { $sum : 1 },
viewsPerAuthor : { $sum : "$pageViews" }
}}
);
< SUM(pageViews)
< SUM(1) = N< GROUP BY author
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
NOSQL MARKETIntroduction
29
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
When Should I Choose NoSQL?
• Eventually Consistent
• Document Store
• Key Value
30
http://guyharrison.squarespace.com/blog/tag/nosql
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Key Value Store
• insert
• get
• multiget
• remove
• truncate
31
<Key, Value>http://wiki.apache.org/cassandra/API
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Redis
• Very simple protocol (SMTP like)
• Amazing Performance (60Kqps ops on 1 CPU machine)
• Persistency to disk
• Very little security
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Column Family Stores: Key Value Store (with benefits)
• insert
• get
• multiget
• remove
• truncate
33
http://wiki.apache.org/cassandra/API
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Cassandra• Simple protocol
• Very Good Performance
• You have indexes (but limited)
• Data Model is a pain
• You need to design you data for queries:“Table per Query”
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Document Databasesvar mydoc = {
_id: ObjectId("5099803df3f4948bd2f98391"), name: { first: "Alan", last: "Turing" }, birth: new Date('Jun 23, 1912'), death: new Date('Jun 07, 1954'), contribs: [
"Turing machine", "Turing test", "Turingery"
], views : NumberLong(1250000)
}
35
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Database for Software Engineers
Class
Subclass
Document
Subdocument
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
37
MapReduce
http://blogs.microsoft.co.il/blogs/vprnd
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
HELLO. MY NAME IS MONGODBIntroduction
38
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
39
#5 Most Popular DB Engine
http://db-engines.com/en/ranking
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Who is Using mongoDB?
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Who is Behind mongoDB
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Why MongoDB?What? Why?
JSON End to End
No Schema “No DBA”, Just Serialize
Write 10K Inserts/sec on virtual machine
Read Similar to MySQL
HA 10 min to setup a cluster
Sharding Out of the Box
GeoData Great for that
No Schema None: no downtime to create new columns
Buzz Trend is with NoSQL
42
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
What mongoDB is Made of?
43http://www.10gen.com/products/mongodb
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Installation: Give Yourself 5min• Add to /etc/yum.repos.d/10gen.repo
• [10gen]
• name=10gen Repository
• baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
• gpgcheck=0
• enabled=1
• yum –y install mongo-10gen mongo-10gen-server
• The Packages:
• mongo-10gen: tools
• mongo-10gen-server: mongod and mongos
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
The Ubuntu Way
• sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get -y update
sudo apt-get install -y mongodb-org
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Installation w/ Authentication• /etc/mongod.conf
• > mongo
• use admin db.createUser( {
user: "siteUserAdmin", pwd: “Pss0rdxxx", roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
} )
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Mastering a New Query Language
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Connect to the Database• Connect:
• > mongo
• Show current database:• >> db
• Show Databases• >> show databases;
• Show Collections• >> show collections; or show tables;
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Databases Manipulation: Create & Drop• Change Database:
• >> use <database>
• Create Database
• Just switch and create an object…
• Delete Database
• > use mydb;
• > db.dropDatabase();
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Collections Manipulation
• Create Collcation
>db.createCollection(collectionName)
• Delete Collection
> db.collectionName.drop()
Or just insert to it
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
SELECT: No SQL, just ORM…
• Select All
• db.things.find()
• WHERE
• db.posts.find({“comments.email” : ”[email protected]”})
• Pattern Matching
• db.posts.find( {“title” : /mongo/i} )
• Sort
• db.posts.find().sort({email : 1, date : -1});
• Limit
• db.posts.find().limit(3)
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
NoSQL and Data ModelingWhat is the Difference
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Database for Software Engineers
Class
Subclass
Document
Subdocument
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Same Terminology
• Database Database
• Table Collection
• Row Document
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
A Blog Case Study in MySQL
http://www.slideshare.net/nateabele/building-apps-with-mongodb
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
as a SW Engineer would like it to be…
http://www.slideshare.net/nateabele/building-apps-with-mongodb
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Migration from RDBMS to NoSQLHow to do that?
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Data Migration• Map the table structure
• Export the data and Import It
• Add Indexes
58
http://igcse-geography-lancaster.wikispaces.com/1.2+MIGRATION
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Selected Migration Tool
59
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Usage Details> Install ruby
> gem install mongify
… Modify the code to your needs
… Create configuration files
> mongify translation db.config > translation.rb
> mongify process db.config translation.rb
60
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Date Functions
• Year(), Month()… function included
• … buy only in the JavaScript engine
• Solution: New fields!• [original field]
• [original field]_[year part]
• [original field]_[month part]
• [original field]_[day part]
• [original field]_[hour part]
61
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
NO SCHEMA IS A GOOD THING BUT…Schemaless
62
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Default Values
• No Schema
• No Default Values
• App Challenge
• Timestamps… No single source of truth
63
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Casting and Type Safety
• No Schema
• No …
• App Challenge
64
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Auto Numbers• Start using _id{
"_id" : 0,"health" : 1,"stateStr" : "PRIMARY","uptime" : 59917
}
• Counter tables • Dedicated database• 1:1 Mapping• Counter++ using findAndModify
65
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
ODM Solution
66
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Data Analysts
67http://www.designersplayground.com/pr/internet-meme-list/data-analyst-2/
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Data Analysts
68
Pentaho
RockMongoMongoVUE RoboMongo
MongoChef
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
No Joins
• Do in the application
• Leverage the power of NoSQL
69http://www.slideshare.net/nateabele/building-apps-with-mongodb
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Limited Resultset
70
• 16MB document size
• GridFS
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Bottom Line
• Powerful tool
• Embrace the Challenge
• Schema-less limitations: counters, data types
• Tools for Data Scientists
• Data design
71
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Mastering a New Query Language
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Connect to the Database• Connect:
• > mongo
• Show current database:• >> db
• Show Databases• >> show databases;
• Show Collections• >> show collections; or show tables;
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Databases Manipulation: Create & Drop• Change Database:
• >> use <database>
• Create Database
• Just switch and create an object…
• Delete Database
• > use mydb;
• > db.dropDatabase();
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Collections Manipulation
• Create Collcation
>db.createCollection(collectionName)
• Delete Collection
> db.collectionName.drop()
Or just insert to it
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
SELECT: No SQL, just ORM…
• Select All
• db.things.find()
• WHERE
• db.posts.find({“comments.email” : ”[email protected]”})
• Pattern Matching
• db.posts.find( {“title” : /mongo/i} )
• Sort
• db.posts.find().sort({email : 1, date : -1});
• Limit
• db.posts.find().limit(3)
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Specific fields
Select Alldb.users.find(
{ },
{ user_id: 1, status: 1, _id: 0 }
)
1: Show; 0: don’t show
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
WHERE
• != “A” { $ne: "A" }
• > 25 { $gt: 25 }
• > 25 AND <= 50 { $gt: 25, $lte: 50 }
• Like ‘bc%’ /^bc/
• < 25 OR >= 50 { $or : [ { $lt: 25 }, { $gte : 50 } ] }
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Join
• Wrong Place…
• Or Map Reduce
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
80
db.article.aggregate(
{ $group : {
_id : { author : "$author“, name : “$name” },
docsPerAuthor : { $sum : 1 },
viewsPerAuthor : { $sum : "$pageViews" }
}}
);
GROUP BY
< GROUP BY author, name
< SUM(pageViews)
< SUM(1) = N
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
81
db.Movie.aggregate([
{$match: {SeriesType : "F", MovieID : {$in : arrMovies}}
},{$project:
{MovieID: "$MovieID", SeriesType: "$SeriesType",Genres: "$Genres"}
},
{$unwind : "$Genres" },
{$group : { _id : "$Genres" , count : { $sum : 1 } } },
{$sort : { count: -1 }}
GROUP BY
WHERE
Keep some fields
Genres is an array
Counting and sorting
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Aggregation Framework OperatorsOperator Description
$project Adding/Removing fields
$match WHERE
$redact Changes document based on Doc content/structure
$limit First N documents
$skip Skips N docs
$unwind Turns array into a multiple documents
$group Group
$sort Sort
$geoNear Geo spatial
$out Write Output to collection
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
83
db.posts.update({“comments.email”: ”[email protected]”},{$set : {“comments.email”: ”[email protected]”}}}
SET age = age + 3• db.users.update(• { status: "A" } ,• { $inc: { age: 3 } },• { multi: true }• )
UPDATE
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
84
j = { name : "mongo" }
k = { x : 3 }
db.things.insert( j )
db.things.insert( k )
INSERT
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
85
db.users.remove(
{ status: "D" }
)
DELETE
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
86
Every operation on a document is atomic
Two Phase Commit implementation is up to you
Atomic Transactions: Single Row
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
87
Multiple documents at oncedb.foo.update({ status : "A" , $isolated : 1 },{ $inc : { count : 1 } },{ multi: true }
)
Disclaimers:
• Sharding is not supported
• Not all or nothing (no roll back on failure)
Atomic Transactions: $isolated
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
88
t = db.transactions.findAndModify({ query: { state: "initial“
}, update: {
$set: { state: "pending"
}, $currentDate: { lastModified: true }
}, new: true})
Atomic Transactions: findAndModify
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
89
If it is about complex transactions.
Simplify the case.
or Consider keeping w/ RDBMS
Atomic Transactions: Bottom Line
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
90
Failure and order:
• db.collection.initializeOrderedBulkOp()
• db.collection.initializeUnorderedBulkOp()
1000 ops/bulk:var bulk = db.items.initializeUnorderedBulkOp();bulk.insert( { item: "abc123", defaultQty: 100, status: "A", points: 100 } );bulk.insert( { item: "ijk123", defaultQty: 200, status: "A", points: 200 } );bulk.insert( { item: "mop123", defaultQty: 0, status: "P", points: 0 } );bulk.execute();
Bulk Operations
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
91
Create a new project
Get the Maven configuration for MongoDB Java Driver
• http://mongodb.github.io/mongo-java-driver/
Project Setup
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
92
List l = new ArrayList();
/**** Insert ****/
// create a document to store key and value
for (int i = 1; i < 1000000; ++i) {
Document document = new Document()
.append("name", "Moshe Kaplan")
.append("age", 36 + i)
.append("createdDate", new Date());
l.add(document);
}table.insertMany(l);
Bulk Ops in Java
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
93
List<String> continentList = Arrays.asList(new String[]{"Africa", "Europe", "Asia"});
DBObject match = new BasicDBObject("$match", new BasicDBObject("continent.name", new BasicDBObject("$in", continentList)));
DBObject projectFields = new BasicDBObject("continent.name", 1);
projectFields.put("area", 1);
projectFields.put("_id", 0);
DBObject project = new BasicDBObject("$project", projectFields );
DBObject groupFields = new BasicDBObject( "_id", "$continent.name");
groupFields.put("average", new BasicDBObject( "$avg", "$area"));
DBObject group = new BasicDBObject("$group", groupFields);
List agList = new ArrayList();
agList.add(match);
agList.add(project);
agList.add(group);
MongoCursor<Document> cursor = countries.aggregate(agList).iterator();
while (cursor.hasNext()) {
System.out.println(cursor.next());
}
Aggregation Framework in Java
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Performance TuningMake a Change
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MONGODB TUNING
95
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
journalCommitInterval = 300:
Write to disk: 2ms <= t <= 300ms
Default 100ms, increase to 300ms to save resources
Disk
The Journal
96
Memory
Journal Data
1 2
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
RAM Optimization: dataSize + indexSize < RAM
97
OS
Data Index
Journal
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
PROFILING AND SLOW LOG
98
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Profiling Configuration• Enable:
• mongod --profile=1 --slowms=15
• db.setProfilingLevel([level] , [time])
• How much:
• 0 (none) 1 (slow queries only) 2 (all)
• 100ms: default
• Where:
• system.profile collection @ local db
99
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Profiling Results Analysis• Last 5 >1ms: show profile• w/o commands:db.system.profile.find( { op: { $ne : 'command' } } ).pretty()• Specific database:db.system.profile.find( { ns : 'mydb.test' } ).pretty()• Slower than:db.system.profile.find( { millis : { $gt : 5 } } ).pretty()• Between dates:db.system.profile.find({ts : {
$gt : new ISODate("2012-12-09T03:00:00Z") ,$lt : new ISODate("2012-12-09T03:40:00Z")
}}).pretty()
100
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Explain> db.courses.find().explain();{ "cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 11, “nscannedObjects" : 11, "nscanned" : 11,"nscannedObjectsAllPlans" : 11, "nscannedAllPlans" : 11,
"scanAndOrder" : false, "indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 0,
"indexBounds" : {},
"server" : "primary.domain.com:27017"
}
101
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
INDEXES
102
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Index Management• Regular Index
• db.users.createIndex( { user_id: 1 } )
• db.users.ensureIndex( { user_id: 1 } )
• Multiple + DESC Index• db.users.ensureIndex( { user_id: 1, age: -1 } )
• Sub Document Index• db.users.ensureIndex( { address.zipcode: 1 } )
• Unique Index• db.users.ensureIndex( { address.zipcode: 1 } , { unique : true } )
• List Indexes• db.users.getIndexes()
• Drop Indexes• db.users.dropIndex(“indexName”)
103
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Known Index Issues
• Bound filter should be the last (in the index as well).
• BitMap Indexes not really working
• You should design your indexes carefully
104
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Dex: The Index Analyzer• Installation:
• sudo apt-get -y install python-pipsudo pip install dex
• Running:
• dex [mongodb_uri] (-f <logfile_path> | -p) [<options>]
• dex -w -p -n "testdb.*" mongodb://127.0.0.1/testdb -f /var/log/mongodb/mongod.log
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
mtools: Visualize and Analyze Logs
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Capped Collections• Fixed size collections
• Circular buffers like
• High throughput operations
• Order guarantee
db.createCollection("mycoll", {capped: true, size:100000})db.cappedCollection.find().sort( { $natural: -1 } )
• Case studies:
• Logs
• Cache
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
TTL• Remove Old Data Automatically
• db.log_events.createIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } )
• db.log_events.insert( {"expireAt": new Date('July 22, 2013 14:00:00'),"logEvent": 2,"logMessage": "Success!“
} )
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
ENVIRONMENT TUNING
109
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
TTL
• # For SSD only• blockdev --setra 16 /dev/sdb
• blockdev --setra 16 /dev/dm-2
• # For all cluser mongod & mongos• for i in /sys/kernel/mm/*transparent_hugepage/enabled; do echo never > $i; done
• for i in /sys/kernel/mm/*transparent_hugepage/defrag; do echo never > $i; done
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
STATS &SCHEMA DESIGN
111
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Sparse Matrix? I don’t Think so
• mongostat
• > db.stats();
• > db.collectionname.stats();
• Fragmentation if storageSize/size > 2
• db.collectionanme.runCommand(“compact”)
• Padding (wrong design) if paddingFactor > 2
112
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
High AvailabilityGoing Real Time
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
(Do Not) Master/Slave
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
• In mongo.conf
• # Replication Options
• replSet=myReplSet
• > rs.initiate()
• > rs.conf()
• > rs.add(“host:port")
• rs.reconfig()
Replication Set
115
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
• rs.addArb(“host:port")
• Also:
• Low Priority
• Hidden
• (Weighted) Voting
Arbiter
116
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Show Status: rs.status();• {"set" : “myReplSet", "date" : ISODate("2013-02-05T10:23:28Z"),
• "myState" : 1,
• "members" : [
• {
• "_id" : 0, "name" : "primary.example.com:27017",
• "health" : 1, "state" : 1,
• "stateStr" : "PRIMARY","uptime" : 164545,
• "optime" : Timestamp(1359901753000, 1),
• "optimeDate" : ISODate("2013-02-03T14:29:13Z"), "self" : true
• },
• {
• "_id" : 1, "name"
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Replica Set Recovery
• Create a new mongod• Either install a plain vanilla
• Or duplicate existing mongod (better)
• Connect to the system• Use the previous machine IP
• Or change configuration to remove old and add new
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Read Preferences• primary
• primaryPreferred
• secondary
• secondaryPreferred
• Nearest
mongo.setReadPreference(ReadPreference.nearest());
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Write Concern: Unacknowledged
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Write Concern: Acknowledged
mongo.setWriteConcern(WriteConcern.NORMAL);
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Write Concern: Journaled
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Write Concern: Replica Aacknowledged
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Sharding and Scale out:Make a big Change
Map Reduce and Aggregation
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Secondary Read Enabling
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
The Strategy : Sharding
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MongoDB Implementation
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Step 1: Create a Config ReplicaSet
• mkdir /data/configdb
• mongod --configsvr --dbpath /data/configdb --port 27019
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Step 2: Install Mongos
• mongos --configdb config01:27019, config02:27019, config03:27019
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Step 3: Add Shards
• Connect a mongos
• Add Shard
• sh.addShard( "rs1/mongodb0.example.net:27017" )
• sh.addShard( "mongodb0.example.net:27017" )
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Step 4: Enable Sharding
• sh.enableSharding("<database>")
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Step 5: Sharding Colleciton• sh.shardCollection("<database>.<collection>", shard-key-pattern)
• sh.shardCollection("records.people", { "zipcode": 1, "name": 1 } )
• Keys:
• High Cardinality to enable split
• Use common query field
• Use Compound indexes for sharding
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
BACKUP AND MONITORING
133
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
First Option – Single ServerLogical Backup Physical Backup
Method mongodump Point in time snapshot (using LVM tools) or disk image/copy(using AWS or Azure “external” tools)
Pros Low costs Low costs
Cons • Downtime: Long;• Duration: Long (slow backup since logical data needs to be
extracted);• Performance impact: High (slows the disks and may stuck the
machine on heavy used machines);• Data consistency: Intact;• Differential: Supported;• Sharding: Supported;
• Downtime: OS and/or infrastructure depended;• Duration: Short (faster backup since only data blocks are
copied);
• Performance impact: Unknown (depends on OS and/orinfrastructure);
• Data consistency: Unknown state;• Differential: Infrastructure depended;• Sharding: Unsupported;
134
Sharding: is a type of database partitioning that separates very large databasesthe into smaller, faster, more easily managed parts called data shards. The wordshard means a small part of a whole..
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
SECOND Option – REPLICA SET
Logical Backup Physical Backup
Method mongodump Stop slave and copy its disk
Pros • Downtime: None (backup is performed using Slave server –Master server is always up);
• Duration: Not significant (backup is performed using Slaveserver);
• Performance impact: None (backup is performed using Slaveserver – Master server is not impacted);
• Data consistency: Intact;• Differential: Supported;• Sharding: Supported;
• Downtime: None• Duration: Not significant
Cons Very high costs – requires two additional servers. A slaveserver of the same type and size as the master server; and asmall arbiter server (used as a secondary verification forMaster server availability tests and “voting”).
• Costs: Requires a dedicated server per replica set
135
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
THIRD OPTION - MongoDB MMS• Part of the MongoDB Enterprise Edition or as a Cloud Service• The Cloud Service offer
• $50/month/node• $2.5/GB/Month backup.• A valid go to market way of MongoDB
for upsale• MMS Features• Point in time recovery• Daily snapshots• Detailed monitoring• Alerts
136
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
How to Enable Incremental Backup
• In Backup• Use the --oplog flag when doing mongodump• Dump each hour the local.oplog collection
• In recovery• mongorestore --oplogReplay• applyOps to implement hourly dump
137
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
mongostat
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
mongotop
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
db.serverStatus()
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
db.stats() and db.collection.stats()
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
rs.status()
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
STORAGE ENGINES
143
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MMAPv1
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MongoDB 3.0 and WiredTiger• MongoDB version 3.0 supports new storage engine
(WiredTiger):• Disk Compression• Heavy write• Document level locking• File per collection
• Server wide selection: • config.yaml• launch w/ --storageEngine = wiredTiger
145
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MongoDB Pluggable Architecture
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Engines Comparison
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
YAML Based Configurationstorage:
dbPath: "/var/lib/mongodbwt"
directoryPerDB: true
engine: "wiredTiger"
wiredTiger:
engineConfig:
cacheSizeGB: 16
journalCompressor: zlib
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
systemLog:
destination: file
path: "/var/log/mongodb/mongod.log"
logAppend: true
timeStampFormat: iso8601-local
processManagement:
fork: true
pidFilePath: "/var/run/mongodb.pid"
#security:
# keyFile: "/etc/mongo.key"
# authorization: "enabled"
replication:
replSetName: "arp0"
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
SECURITY
149
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Providing Permissions• use admin
db.createUser( { user: "siteUserAdmin", pwd: "password", roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
} )
• use recordsdb.createUser( {
user: "recordsUserAdmin", pwd: "password", roles: [ { role: "userAdmin", db: "records" } ]
} )
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Roles
ReadreadWritedbAdmindbOwneruserAdminclusterAdmin, clusterManager, …backup, restorereadAnyDatabase, readWriteAnyDatabaseroot
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Granular Actionsuse admindb.createRole(
role: "manageOpRole",privileges: [
{ resource: { cluster: true }, actions: [ "killop", "inprog" ] },{ resource: { db: "", collection: "" }, actions: [ "killCursors" ] }
],roles: []
})
© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
Thank You !Moshe Kaplan