the where and when of nosql platforms

PRESENTS

Moderator

Christina Warren,

Panelists

Harry Heymann, Head of Server Engineering at

Michael Bryzek, CTO & Founder at

Frank Weigel, Director of Products at

Matt Pfeil, VP of Customer Solutions and Co-Founder at

Dave Connors, VP of Operations at

Dwight Merriman, CEO at 10Gen at

The Old World

Pros• Transactional integrity• Sequential nature

Cons• Expensive• Scale up typically not out

The New World

Why ??

• Huge volume of content• Distributed Infrastructure• Relaxed and agile• Throw the RD rules out the window• Speed of development• No DBA needed• Elastic scaling (out not up)• Major cost savings

Platforms

•

•

•

Companies

Michael Bryzek CTO & Founder

GILT GROUPE is an innovative e-commerce company offering highly coveted products and experiences at insider prices.

One of the Most Innovating Companies“Gilt Groupe, the online shopping phenomenon”--Alexandra Kotur, Vogue

“The t-shirt arrived the very next day in a clean box with nicely folded black tissue paper and a friendly note from the founders thanking me for my purchase. They're thanking me? No, Gilt Groupe, thank you.”--Damien Nunes, GQ (on his first purchase)

“The French invented the online sample sale for designer merchandise, spawning numerous American competitors, but none boasts the pedigree of Gilt Groupe.”--Fortune, (from their story “Dotcom Divas”)

“People with a taste for high-end fashion items have turned Gilt into a guiltless addiction.”--Roy Furchgott, The New York Times

10,000 foot view

The Shopping Cart

Harry HeymannHead of Server Engineering

mongodb: some numbers

• 8 clusters

o some sharded, some not

o some master/slave, some replica sets

• ~40 machines (68GB, m2.4xl on EC2)

• 2.3 billion records

• ~15k QPS

Why MongoDB?

Biggest reason (by far): auto sharding:

• Started on a single SQL database. Eventually split to two nodes: 1 for check-ins (our biggest dataset), 1 for everything else.

• It was clear that check-ins would grow beyond what a single machine could handle.

• Major efficiency gains by outsourcing the development of the sharding layer to consultants at 10gen.

Dave Connors, VP of Operations

• Customers are Small Businesses• Email, Event, Survey & now Social Media• Over 450k paying customers

• Business model• Many customers pay as little as $15 a month• ~2 million database transactions per minute

• Business problem• Social Media up to 100 times more data• Challenge with our business model

Constant Contact

Implementation

• Monitoring• Dev took lead with Munin

• Logging• Ops took lead with Scribed

• Roles & Responsibilities• DBA?• Still in progress…

Cultural Challenges: NoSQL

Traditional Roles

• Switchable modes• Mirroring• Dial-able traffic

Managing Risk: Phased Rollout

ConclusionBusiness Value Comparison

Traditional RDBMS NoSQL

Agility

System Standup 4 weeks 4 hours

Add Capacity 1 week 1 hour

Upgrade DB 1 month 3 days

Time to Market for App 9 months 3 months

Estimated 1st Year Cost $2.5M $250k

Technologists

Dwight Merriman CEO

Past:RDBMS

one size fits all

BI / Data warehousing

RDBMS

other segmentse.g. search

RDBMS


BI / Data warehousingaster, greenplum, neteeza, teradata, vertica, hadoop

RDBMS

NoSQL

RDBMS


BI / Data warehousingaster, greenplum, neteeza, teradata, vertica, hadoop

RDBMS

NoSQL

• a few gigantic queries• visual client tools important -> a

need for SQL• scales horizontally (better ones)

• very complex transactional semantics• legacy projects• projects requiring SQL

• agile• programmer friendly data modle• horizontall scalable• works for operational data (lots of small

reads and writes)

Frank Weigel, Director of Products

moxi

11211 11210

Memcached Managed Cacheprotocol listener/sender

engine interface

memcapable 1.0 memcapable 2.0

21100 – 2119943698091

httpRE

ST m

anag

emen

t API

/Web

UI

Hea

rtbe

at

Proc

ess

mon

itor

Glo

bal s

ingl

eton

sup

ervi

sor

Confi

gura

tion

man

ager

on each node

Erlang/OTP

Reba

lanc

e or

ches

trat

or

Nod

e he

alth

mon

itor

one per cluster

vBuc

ket s

tate

and

repl

icati

on m

anag

er

HTTP distributed erlangerlang port mapper

Couchbase Server Architecture

Couchbase Storage EngineDat

a M

anag

er

Clus

ter M

anag

er

• Example of online migration of existing application.

• First migrated large or slow performing tables and frequently updated fields from MySQL to Couchbase

: Migrating to Couchbase

memcachedprotocol listener/sender

engine interface

Couchbase Storage Engine

TAP

Web Server

Apache + PHP

Client-side Moxi

MySQL


engine interface


TAP


engine interface


TAP

Web Server

Apache + PHP

Client-side Moxi

Matt Pfeil, VP of Customer Solutions

Architecture

Better technology

• Multi-master, multi-DC• Linearly scalable• Larger-than-memory datasets• Best-in-class performance (not just writes!)• Fully durable• Integrated caching• Tuneable consistency

A performance retrospective

Credit: Nathan Hurst, 2010

QUESTIONS FROM THE AUDIENCE

THANK YOU FOR JOINING US!We look forward to seeing you at our upcoming events

the where and when of nosql platforms

Technology