confessions of a relational addict

Confessions of a relational addict

Chandra Patni

Twitter @cpatni

Architecture

http://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf

Volume 13 / Number 6 / June 1970

Promise of ACID

Atomicity

Consistency

Isolation

Durability

READ COMMITTED may block

ANSI/ISO SQL...

READ UNCOMMITTED to allow for non-blocking reads

But allowed dirty reads...

Isolation. Really?

Locking

Locking

Lock Lock

Escalation

Escalation

Read Lock

Read Lock

Write Lock

Write Lock

Table Lock

Table Lock

MVCCFor transactional memory

Readers don’t block readers

Readers don’t block writers

Writers don’t block writers

Writers block writers (serialized)

Living in a bliss

Until...

need redundancy

scale reads

scale writes

horizontally

Solutions...

replication to scale reads

cache through writes

expensive solution for redundancy

mongodb

Distributed database from ground up

replication in a minute (HA)

built-in cache through writes

replicate if you care about your data

sharding (scale out)

Effective Design

Domain objects rarely fit in a table row

join to assemble domain objects

mongo documents as pre-assembled domain objects

in place updates ($set, $push)

New hotness

Multikeys

Map reduce

map = function () { for (var i = 0; i < this.tags.length; i++) { emit(this.tags[i], this.state == "published" ? 1 : 0); } }reduce = function (key, values) { var total = 0; for (var i = 0; i < values.length; i++) { total += values[i]; } return total; }

Capped collections

Geospatial

GridFS

Rubber meets the road

Realtime Dashboard

Realtime Dashboard

Realtime stats for 1 sec, 5 secs, 15 secs

4K updates per second

pywebsocket

IGN Media: Editorial Generated Content

Content in MongodbMaster/slave over filer for instant backups

100 ops

Ruby on Rails backend

MongoMapper

Integrate Solr Search with MongoMapper: http://github.com/tsxn26/customized-mongomapper-search

XML/JSON serialization: http://github.com/rubyorchard/mongomapper

RESTful API

Game Schema DesignRapid Schema Revision

Optimize canonical use-case

IGN Social: User Generated Content

Manish Pandit

Activity Streams

3M documents @10-15 qps, @5 updates/sec

Moving to activity propagation: 4 documents for every user, i.e. a total of 2M documents with arrays of 500 activities each

Move to replica sets on 1.7

Meta+Level Games

Shadelight

Data in MongoDB and Redis

Replicated in real-time to slave servers

Master and slave are in different colo

Ben Myles wrote awesome library called MongoMatic http://mongomatic.com/

Mongomatic

Ruby datamapper for MongoDB

Simple and embraces MongoDB

Shadelight and Mongo Machine (http://mongomachine.com/) use it

We’re hiringhttp://corp.ign.com/http://labs.ign.com/