big fast data with mongodb - squarespace · mongodb redis bigtable couchdb neo4j datomic db2 files...

Big Fast Data with MongoDB

Clint Combs

Thursday, April 12, 12

Polyglot Persistence

• Same idea as Polyglot Programming

• DB Variety ~= Data Structure Variety

• Don’t use a list when we need a map.

• The same goes for data persistence.


Many Options

OracleMongoDB

Redis

BigTable CouchDB

Neo4j

Datomic

DB2

Files

CSV

Excel

Logs

MySQL


Many Options

Oracle

MongoDB

Redis

BigTable CouchDB

Neo4j

Datomic

DB2

Files

CSV

Excel

Logs

MySQL


MongoDB


MongoDB

A Scalable High-PerformanceDocument-Oriented Database


When is MongoDB a Good Fit?

• when you have a large data set

• when you have a dynamic schema

• when you need speed

• when you can tolerate eventual consistency


Wordnik

• Migrated from MySQL

• Primary reason: performance

• 500k requests/hour - 4x that during peak

• 12 billion documents in MongoDB

• 3 TB per node


Wordnik (cont.)

• insert 8k documents per second with bursts to 50k per second

• each java client can sustain 10MB/second reading from a mongod


Wordnik (cont.)

• Every type of retrieval is faster

• sample fetch: 400 ms to 60 ms

• dictionary entries: 20 ms to 1 ms

• document metadata: 30 ms to 0.1 ms

• spelling suggestions: 10 ms to 1.2 ms


Wordnik (cont.)

• MongoDB built-in caching benefits

• removed memcached, freed GBs of RAM

• average speed increase of 1-2 ms per request under load

• not all data in RAM, so 60ms avg. includes disk access


Wordnik (cont.)

• full story: http://blog.wordnik.com/12-months-with-mongodb

• others: http://www.mongodb.org/display/DOCS/Production+Deployments


http://blog.wordnik.com/12-months-with-mongodb




http://www.mongodb.org/display/DOCS/Production+Deployments




MongoDB Applications

• archiving

• event logging

• document and CMS systems

• geospatial indexing

• CQRSCommand Query Responsibility Separation


CQRS

System of Record(Oracle)

Query Data(MongoDB)

Command Model

QueryModel

Client

Change Events


What does it store?

• JSON documents up to 16MB in size

• stored in BSON format

• larger documents can be stored with GridFS

• all documents stored in collections


Data Types

• JSON types: string, integer, boolean, double, null, array, and object

• other: date, object id, binary data, regular expressions, code

• timestamps are for internal use only

• JavaScript code into system.js


Java Type MappingsMongoDB Type Java Type

string, int, boolean, double String, int, boolean, doubleObject ID com.mongodb.ObjectId

Regular Expression java.util.regex.PatternDates/Times java.util.Date

Database References com.mongodb.DBRefBinary Data byte[]

Timestamp Data BSONTimestampCode Data Code, CodeWScope

Embedded Documents BasicDBObjectArrays anything that extends List


mongo shell

• ‘mongo’ starts the command-line interface

• use <database-name>

• show collections - displays a list of collections in the current database


Collections

• collections of BSON documents

• schema-free

• can be organized in namespaces for user convenience (really a flat model in DB)


Object IDs

• every document in a collection has a unique _id field

• _id can be any type

• _id created as a BSON ObjectID by the database if not supplied


Collections

• create: db.createCollection(“users”)

• rename:db.users.renameCollection(“allUsers”)

• drop: db.users.drop()

• count: db.users.count()


newhart

• example docs in this talk are from newhart

• newhart is an open source project that provides audit tracking with storage in MongoDB

• uses capped collections for audit trails

• will be available atgithub.org/ClintCombs/newhart


newhart document{ "_id" : ObjectId("4f84b0473004b2a7bf9dcdad"), "origin" : "org.newhart.example.AuditUsers", "originKey" : "mary2012-create-1334095943422", "originKeyType" : "User", "criticality" : "major", "auditTS" : ISODate("2012-04-10T22:12:23.426Z"), "createTS" : ISODate("2012-04-10T22:12:23.424Z"), "updateTS" : ISODate("2012-04-10T22:12:23.424Z"), "msg" : "new user created: mary2012", "data" : { "loginName" : { "text" : "mary2012" } }, "errors" : [ ], "warnings" : [ ], "labels" : [ "security", "create" ]}


Insert

> use newhart> show collections> db.users.insert({origin:"shell"})


Queries

> db.users.find();> db.users.findOne();


Queries (cont.)

> db.users.find({origin:"shell"})

Field Selection:> db.users.find({origin:"org.newhart.example.AuditUsers"},{labels:1})

Arrays> db.users.find({labels:"authenticate"});


Skip and Limit

> db.users.find({labels:"failed"},{"data.loginName.text":1}).sort({"data.loginName.text":1}).skip(5).limit(3)


Updates

• db.users.save({"_id" : ObjectId("4f84fecae2f8776e5cdc3ba7"), "origin" : "Clint"})

• update if exists, otherwise insert

• http://www.mongodb.org/display/DOCS/Updating


http://www.mongodb.org/display/DOCS/Updating




Delete

remove all items> db.users.remove()

same as above> db.users.remove({})

remove all items with minor criticality> db.users.remove({criticality:"minor"})


Capped Collections

• Fixed-size, FIFO collections

• create:db.createCollection(“users”,{capped:true, size:1000000});

• size is number of bytes, including database headers


Sorting

> db.users.find({labels:"failed"},{"data.loginName.text":1}).sort({"data.loginName.text":1})

> db.users.find().sort({$natural:-1}).limit(1)


Indexes> db.users.ensureIndex({origin:1})

Unique Index> db.contacts.ensureIndex({email:1},{unique:true})

Indexing Embedded Fields> db.users.ensureIndex({"data.loginName.text":1})

Compound Keys> db.contacts.ensureIndex({last:1, first:1})


Optimization with Explain

> db.users.find({"data.loginName.text":"mary2012"}).explain(){ "cursor" : "BtreeCursor data.loginName.text_1", "nscanned" : 6, "nscannedObjects" : 6, "n" : 6, "millis" : 0, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : false, "indexOnly" : false, "indexBounds" : { "data.loginName.text" : [ [ "mary2012", "mary2012" ] ] }}


Fault Tolerance & ScalingReplica Sets and Sharding


Replica Sets

R1

R2

R3

R4

Arbiter

Client


Replica Sets

R1 R3

R4

Arbiter

Client


Replica Sets

R1 R3

R4

Arbiter

Client

R2mongod

logs

data

journal

file lock


Replica Sets

R1 R3

R4

Arbiter

Client


Replica Sets

R1

R2

R3

R4

Arbiter

Client


Replica Sets

R1

R2

R3

R4

Arbiter

Client

R2


Replica Sets

R1

R2

R3

R4

Arbiter

Client

R3

R2


Replica Sets

R1

R2

R3

R4

Arbiter

Client

R3R3

R2


Replica Sets

R1

R2

R3

R4

Arbiter

Client

R3

R2

R3

R2


Replica Sets

R1

R2

R3

R4

Arbiter

Client

R2

R3

R2

R3

R2


Replica Sets

R1

R2

R3

R4

Arbiter

Client


slaveOk

• used when querying a replica set

• drivers route requests to master by default

• set slaveOk to query a secondary member of the replica set

• rs.slaveOk()


Read from Secondary

R1

R2

R3

R4

Arbiter

Client


Sharding

Client

R1

R2

R3

R1

R2

R3

R1

R2

R3

R1

R2

R3

Replica Set

mongod c2mongos mongos mongos

mongod c1

Shard 1A-F

Shard 2G-L

Shard 3M-R

Shard 4T-Z


UI Clients

• MongoVUE

• JMongoBrowser

• others...


Drivers

• Standard Java Driver: http://github.com/mongodb/mongo-java-driver

• Hammersmith: high-performance async Java driver (Akka 2.0 durable mailboxes)https://github.com/bwmcadams/hammersmith

• Casbah Scala Driver:http://github.com/mongodb/casbah


http://github.com/mongodb/mongo-java-driver




https://github.com/bwmcadams/hammersmith




http://github.com/mongodb/casbah

http://github.com/mongodb/casbah

mongodb.org Drivers

• C

• C++

• Erlang

• Haskell

• JavaScript

• .NET

• Perl

• PHP

• Python

• Ruby


Community Drivers

• ActionScript

• Clojure

• ColdFusion

• D

• Dart

• Fantom

• F#

• Go

• Groovy

• Lisp

• Smalltalk

• and more...


Quick Reference Cards and Cookbook

• http://www.10gen.com/reference

• Commands

• Queries

• Indexing

• Replica Sets

• http://cookbook.mongodb.org/


http://www.10gen.com/reference

http://www.10gen.com/reference

http://cookbook.mongodb.org/

http://cookbook.mongodb.org/

Summary

• MongoDB is one of many persistence alternatives

• Document-Oriented

• Big and Fast

• Flexible - no schema


Contact Me

• Twitter: @ClintCombs

• Presentation posted at:http://ccombs.net/presentations


http://ccombs.net/presentation

http://ccombs.net/presentation