how we fixed our mongodb problems

Eric Lubow

@elubow

[email protected]

#MongoDBDays

How We Fixed OurMongoDB Problems

How We Fixed Our MongoDB

Problems

Eric Lubow @elubow #MongoDBDays

OverviewThe Secret

SimpleReach

Usage Patterns

Tools

Architecture Implementation

Questions

•

•

•

•

•

•


Problems



Problems


The 2 Truths


Problems


Even with the right tools, 80% of the workof building a big data system is acquiringand refining the raw data into usable data.

The Real Truth


Problems



Problems


Millions of URLs per dayOver 1.25 billion page views per month500m events per day (~6k events/second)Auto-scale 125-160 machines depending on trafficBuilt a predictive measurement algorithm for the social web

SimpleReach


Problems


And It Goes Like This...

C*Vertica


Problems



Problems


Why Mongo?Fast and easy prototyping

Low barrier to entry

B-Tree indexes and range queries

Aggergation

Everything is JSON

TTLs

MongoID

•

•

•

•

•

•

•


Problems


GoalsHighly available

Speed

Repeatability

Data accuracy (across storage engines)

Clients should have minimal architecture knowledge

Controlled Data Flow Patterns

Control data set size

Restore capabilities for non-ephemeral data

•

•

•

•

•

•

•

•


Problems


Availability and SpeedInternal service architecture

Mongos on every server that talks to Mongo

Server distribution across data centers

Latest version isn’t always the greatest version

Understand how usage patterns affect Mongo

•

•

•

•

•


Problems


Repeatability - Sharded Replica SetSHARD0000A

MONGOS

PRIMARY SECONDARY

BASE AMI

ORGANIZATIONAL BASE

BASE IMAGE

LAYOUT

APPLICATION GROUP

AMAZONLINUX

MONITORING

USERS

MONGOD

MONGOD-ARBITER

SHARD0000B

MONGOS

AMAZONLINUX

MONITORING

USERS

MONGOD

APPLICATION


Problems


Availability - Architecture DistributionUS-EAST-

1a

MONGO-SHARD-0001-B

MONGO-SHARD-0000-A

CASSANDRA-0001

CASSANDRA-0010

REDIS-0001A

VERTICA-0001

iAPI-0001

US-EAST-1b

MONGO-SHARD-0002-B

MONGO-SHARD-0001-A

CASSANDRA-0002

CASSANDRA-0011

REDIS-0001B

iAPI-0002

US-EAST-1e

MONGO-SHARD-0002-A

MONGO-SHARD-0000-B

CASSANDRA-0003

CASSANDRA-0012

VERTICA-0003

iAPI-0003

VERTICA-0002


Problems


The Schrute of the Problem


Problems


ReleasesReasons why I update software:

Because I want the latest version

To get rid of the reminder


Problems


Usage PatternsMongos uses TCP-based flow control

Separate DBs to deal with DB level locking

Consistent access patterns

Schema design

Proper indexing

Avoid scatter/gather and aim for targeted

•

•

•

•

•

•


Problems


Consistent Access Patterns

realtime_score(‘score’, ‘realtime’)

score.realtime

srt


Problems


Schema DesignRandomly pre-populate consistent document structures

Use SetOnInsert to pre-populate

Shard keys

Separate DBs to deal with DB level locking (volume based)

TTL

Hashed shard keys

$inc when possible, $set is expensive

•

•

•

•

•

•

•


Problems


Hourly Stats Documents{

"_id": BinData(5, "OWQ5NzQ0ZjgxZGUwYTdmMzM3Y2U0NDkzZGFlMGY0NTc="),

"account_id": ObjectId("5165905f4240cf9182000069"),

"hour": ISODate("2013-06-02T23:00:00Z"),

"content_id": "56250f88530ecc21233be5d2384679b2",

"totals": {

"facebook_likes": 0,

"facebook_shares": 1,

"facebook_referrals": 0,

"pageviews": 10134,

"twitter_tweets": 16,

"twitter_referrals": 3045,

"social_actions": 17,

"social_referrals": 3045

}

}


Problems


Daily Stats Documents{

"_id": BinData(5, "OWQ5NzQ0ZjgxZGUwYTdmMzM3Y2U0NDkzZGFlMGY0NTc="),

"account_id": ObjectId("5165905f4240cf9182000069"),

"day": ISODate("2013-06-02T00:00:00Z"),

"content_id": "56250f88530ecc21233be5d2384679b2",

"totals": {

"pageviews": 10134,


"social_actions": 17

},

"00": {

"pageviews": 283,


"social_actions": 10

},

"01": {

"pageviews": 9851,


"social_actions": 6

}

}


Problems


Path of a Packet

INTERNET

Inte

rnal

API

Solr

C*

Mongo

Redis

VerticaCo

nsum

ers

Que

ue

FIREHOSE

EC

API

SC


Problems


NSQ by Bit.lyDistributed and de-centralized topology

At least once delivery guaranteed

Multicast style message routing

Runtime discovery for consumers to find producers

Allow for maintenance windows with no downtime

Ephemeral channels for testing

•

•

•

•

•

•


Problems


Controlled Data Flow

Social EventCollector

Social Data

Batch & WriteProcessed Data

Batch & WriteRaw Data

Calculate Score Write

NSQ Multicast NSQ NSQ


Problems


Problems?

Big Architectures for Big Data Eric Lubow @elubow #Cassandra13

Service Architecture

Internal API

Solr

Real-time

C*

C*

Vertica


Problems


Anatomy of an Endpoint

MONGO

MONGO

VERTICA

C*

C*

HO

URL

YCO

NTE

NT MONGO

MONGO

VERTICA

C*

C*TEN

MIN

UTE

CON

TEN

T

QU

ERY

ING

MA

CHIN

ES

HELENUS

HELENUS

PYVERTICA

PYMONGO

PYMONGO

PYVERTICA


Problems


Endpoint Breakout AdvantagesAvailability

Consistent Access Patterns

Minimal downtime changes

Smaller code deploys

Non-monolithic code base

No async necessary

•

•

•

•

•

•


Problems


DevOpsMonitor: Nagios, Statsd, and Cloudwatch

Manage: Chef, OpsWorks, cSSHx, Vagrant

Know failure cases

Turn off balancer on backups

Restart EVERYTHING on upgrade

Extensive use of AWS

•

•

•

•

•

•


Problems


Cloud Specificsblockdev --setra 256

Use ephemeral storage, not EBS volumes

Use MMS

Cloudwatch Metrics are important and easily scriptable

Don’t use spots but always expect instance loss

Kernel tuning

•

•

•

•

•

•


Problems


SummaryUnderstand your usage patterns

Know the common failure cases

Architecture distribution

Homogeneous Distribution

Monitoring & Automation

•

•

•

•

•


Problems


We’reHiring

(Ask about Food ComaFridays)


Problems


Questions are guaranteed in life.Answers aren’t.

Eric Lubow

@elubow

[email protected]

#Cassandra13

Thank you.

how we fixed our mongodb problems

Technology

mongodb problems eric

mongodbdays goals

mongodbdays path

mongodbdays nsq

mongodbdays schema design

mongodbdays millions

mongodbdays releases

b mongoshard