partner webinar: the scaling checklist for mongodb - 100gb and beyond

www.mongohq.com Scaling Checklist for MongoDB

Scaling Checklist for MongoDB

100GB & Beyond

MongoHQwww.mongohq.com | @mongohq

MongoHQ is a fully-managed platform used by developers to deploy, host and scale open-source databases.

Chris Winslettchris@mongohq.com

I’ve spoken at a number of MongoDB conferences on optimizing queries. I’ve been with MongoHQ for two

years – prior to that I built applications for the education and technical sectors.

• 100GB of data is relatively big data• MongoDB has comparative advantages• MongoDB has absolute constraints• Know the MongoDB gauges• Surpassing 100GB requires:– Understanding absolute constraints.– Knowledge of application’s data consumption– Optimization of data consumption to comparative

advantages

Audience Survey

What is your data size? Choose the biggest bucket.

A. < 10GBB. < 50GBC. < 75GBD. < 100GBE. > 100 GB

100 GB Checklist

1. Identify your data behavior2. Use MongoDB for comparative advantages3. Know the MongoDB indexing constraints4. Refactor schema to simplify queries5. Remove data that does not fit MongoDB6. Separate hot and cold data7. Stop using `mongodump`8. Check your gauges9. Avoid queries causing page faults10. Track and monitor slow queries11. Buying time with hardware

100 GB Checklist

Identify your data behavior

1. Small v. Large – type of data2. Fast v. Slow – behavior of data3. Complex v. Simple – type of queries4. Known v. Unknown – behavior of queries5. Queuing v. Application data

This can happen at planning, staging, or production phase.

Patterns of your Data

Small Large

Modern applications have all patterns

Main application collections

ApplicationMetadata

SecondaryApplication Collections

Internal metrics

Event logs and event

Queues, OLTP,

Messages

Rendered in background

Small Large

Where doesn’t MongoDB excel?

ApplicationMetadata

Internal metrics

Queues, OLTP,

Messages

4th dimension is time

Today’s Data

Last week’s data

Small Large

Data-types to avoid with MongoDB

ApplicationMetadata

Internal metrics

Queues, OLTP,

Messages

Small Large

What type of queries do you have?

Unknown Known

Simple

Complex

Unknown Known

Simple

Complex

Modern applications have all types of queries

Data discovery

Application search

Keyvalue

SingleRange Query

User generated

search

Internal metrics

Multi-Range Query

Unknown Known

Simple

Complex

Queries to Avoid with MongoDB

Data discovery

Application search

Keyvalue

SingleRange Query

User generated

search

Internal metrics

Multi-Range Query

Unknown Known

Simple

Complex

4th Dimension is Time

Real-time core of

application

Today’s Data

Last week’s data

MongoDB

Queries and MongoDB

Elastic SearchSQL

Elastic Search

Unknown Known

Simple

Complex

100 GB Checklist

MongoDB’s Technical Comparative Advantage

• Expressive data structure allows simplification of complex data relationships

• Create simple, known queries and return expressive relationships

• On-the-fly addition of attributes / columns

• Total Cost of Ownership*

100 GB Checklist

MongoDB Indexing Constraints

• Only one index can be used per query

• Only one range operator can be used per index

• Range operator must be the last field on index

• Know how to use the right side of indexes

100 GB Checklist

What does it mean to optimize?

Unknown Known

Simple

Complex

Scaling to 100GBinvolves moving queries fromcomplex to simple and unknown to known

Finish

Example of simplifying a query.

Naïve Query:

db.messages.find({$or: [{recipient_id: <id>}, {sender_id: <id>}]}).sort({_id: -1})

Find the most recent messages for a person’s message stream.

Second attempt:

db.messages.find({participant_ids: <id>}).sort({_id: -1})

Best approach

db.users.find({_id: <id>})

Naïve Query

{ _id: <id>, message: “Wow, this pizza is good!”, sender_id: <user_id>, recipient_id: <user_id>}

db.messages.find({$or: [{recipient_id: <id>}, {sender_id: <id>}]}).sort({_id: -1})

Document

Second Attempt

Document

{ _id: <id>, message: “Wow, this pizza is good!”, sender_id: <sender_id>, recipient_id: <recipient_id>, participant_ids: [<sender_id>,<recipient_id>]}

db.messages.find({participant_ids: <id>}).sort({_id: -1})

Best approach

Document

Hint: use the $push, $sort, $slice for the last 50

{ _id: <id>, name: “Clarke Kent”, recent_messages: [ <…50 denormalized messages…> ]}

db.users.find({_id: <id>})

How did we optimize?

Unknown Known

Simple

Complex

We took a known, complex query and made it simple.

Finish

Methods for Simplifying Queries• Bucket values

• Create summary attributes

• Pre-compute values

• Use expressive documents structures

• Sort and filter at the application level

• Create summary documents

• Divide and measure (more on this later)

100 GB Checklist

Small Large

Remove “unrefactorable” data

ApplicationMetadata

Internal metrics

Queues, OLTP,

Messages

MongoDB

Move up and right, or find another tool

Unknown Known

Simple

Complex

Data discovery

Application search

User generated

search

Multi-Range Query

100 GB Checklist

Unknown Known

Simple

Complex

4th Dimension is Time

Real-time core of

application

Today’s Data (fast)

Last week’s data (slower)

Separate Data with Cross Purposes

• If this today’s data must be fast, and last week’s data can be slow:– Rollout today’s data using TTL collections– Use another database for last weeks data– Use high-RAM ratio and SSD backed machines for

this today’s data– Use cheaper hardware for last week’s data

MongoDB Doesn’t have Joins

Data doesn’t have to be adjacent.

Divide, measure, conquer.

100 GB Checklist

Stop Use `mongodump`

`mongodump` is long running tablescan that exports all documents. This disrupts RAM and causes performance issues.

Self-hosting: use the MongoDB MMS and Backup

As-a-service: ask your vendor about backup alternatives

100 GB Checklist

Configure MMS Now!

100 GB Checklist

Avoid Page Faults like the Plague

50% Table Scans 1% Table Scans 0% Table Scans0

MongoDB Operations / Second

MongoDB

What type of queries cause page faults?

Unknown Known

Simple

Complex

Data discovery

Application search

User generated

search

Multi-Range Query

100 GB Checklist

Track & Remove Slow Queries

• system.profile collection – link

• MongoDB professor – link

• Dex – link

• MongoHQ Slow Query Tracker and Profiler - link

100 GB Checklist

Buying time with hardware has a limited life

• Don’t get addicted to buying more hardware.

• Before any purchasing decision, always– consider optimization– investigate separating, paring data

100 GB Checklist

Thank you!

For any questions:chris@mongohq.com

www.mongohq.com@mongohq

partner webinar: the scaling checklist for mongodb - 100gb and beyond

mongodb mongodb queries

mongodb scaling checklist

mongodb datatypes

big data mongodb

mongodb unknown

application data

mongodb mongohq

mongodb gauges

Technology

mongodb monitoring become a mongodb dba - monitoring ......

2011 mongo fr - scaling with mongodb

scaling with mongodb - netways€¦ · • o/s tuning •...

humongous mongodb · agenda • scaling mongodb - concepts...

scaling with mongodb

scaling the server side of occasionally-connected, mobile...

torodb: scaling postgresql like mongodb by Álvaro...

back to basics 3: scaling 30,000 requests a second with...

shortcuts around the mistakes i've made scaling mongodb

foxsathdr manual 100gb

scaling mongodb for real time analytics

mongodb linux porting, performance measurements and and...

mongodb world 2016: scaling mongodb with docker and cgroups

scaling mongodb - presentation at mtp

torodb: scaling postgresql like mongodb

scaling mongodb with docker and cgroups

salvatore incandela "loyalty cashback - scaling with...

scaling mongodb on amazon web services (dat209) | aws...

mongodb on aws - amazon s3 · availability and scaling 10...

multi 100gb campusngfw - internet2