the where and when of nosql platforms
Post on 17-Dec-2014
1.281 Views
Preview:
DESCRIPTION
TRANSCRIPT
PRESENTS
Moderator
Christina Warren,
Panelists
Harry Heymann, Head of Server Engineering at
Michael Bryzek, CTO & Founder at
Frank Weigel, Director of Products at
Matt Pfeil, VP of Customer Solutions and Co-Founder at
Dave Connors, VP of Operations at
Dwight Merriman, CEO at 10Gen at
The Old World
Pros• Transactional integrity• Sequential nature
Cons• Expensive• Scale up typically not out
The New World
Why ??
• Huge volume of content• Distributed Infrastructure• Relaxed and agile• Throw the RD rules out the window• Speed of development• No DBA needed• Elastic scaling (out not up)• Major cost savings
Platforms
•
•
•
Companies
Michael Bryzek CTO & Founder
GILT GROUPE is an innovative e-commerce company offering highly coveted products and experiences at insider prices.
One of the Most Innovating Companies“Gilt Groupe, the online shopping phenomenon”--Alexandra Kotur, Vogue
“The t-shirt arrived the very next day in a clean box with nicely folded black tissue paper and a friendly note from the founders thanking me for my purchase. They're thanking me? No, Gilt Groupe, thank you.”--Damien Nunes, GQ (on his first purchase)
“The French invented the online sample sale for designer merchandise, spawning numerous American competitors, but none boasts the pedigree of Gilt Groupe.”--Fortune, (from their story “Dotcom Divas”)
“People with a taste for high-end fashion items have turned Gilt into a guiltless addiction.”--Roy Furchgott, The New York Times
10,000 foot view
The Shopping Cart
Harry HeymannHead of Server Engineering
mongodb: some numbers
• 8 clusters
o some sharded, some not
o some master/slave, some replica sets
• ~40 machines (68GB, m2.4xl on EC2)
• 2.3 billion records
• ~15k QPS
Why MongoDB?
Biggest reason (by far): auto sharding:
• Started on a single SQL database. Eventually split to two nodes: 1 for check-ins (our biggest dataset), 1 for everything else.
• It was clear that check-ins would grow beyond what a single machine could handle.
• Major efficiency gains by outsourcing the development of the sharding layer to consultants at 10gen.
Dave Connors, VP of Operations
• Customers are Small Businesses• Email, Event, Survey & now Social Media• Over 450k paying customers
• Business model• Many customers pay as little as $15 a month• ~2 million database transactions per minute
• Business problem• Social Media up to 100 times more data• Challenge with our business model
Constant Contact
Implementation
• Monitoring• Dev took lead with Munin
• Logging• Ops took lead with Scribed
• Roles & Responsibilities• DBA?• Still in progress…
Cultural Challenges: NoSQL
Traditional Roles
• Switchable modes• Mirroring• Dial-able traffic
Managing Risk: Phased Rollout
ConclusionBusiness Value Comparison
Traditional RDBMS NoSQL
Agility
System Standup 4 weeks 4 hours
Add Capacity 1 week 1 hour
Upgrade DB 1 month 3 days
Time to Market for App 9 months 3 months
Estimated 1st Year Cost $2.5M $250k
Technologists
Dwight Merriman CEO
Past:RDBMS
one size fits all
BI / Data warehousing
RDBMS
other segmentse.g. search
RDBMS
other segmentse.g. search
BI / Data warehousingaster, greenplum, neteeza, teradata, vertica, hadoop
RDBMS
NoSQL
RDBMS
other segmentse.g. search
BI / Data warehousingaster, greenplum, neteeza, teradata, vertica, hadoop
RDBMS
NoSQL
• a few gigantic queries• visual client tools important -> a
need for SQL• scales horizontally (better ones)
• very complex transactional semantics• legacy projects• projects requiring SQL
• agile• programmer friendly data modle• horizontall scalable• works for operational data (lots of small
reads and writes)
Frank Weigel, Director of Products
moxi
11211 11210
Memcached Managed Cacheprotocol listener/sender
engine interface
memcapable 1.0 memcapable 2.0
21100 – 2119943698091
httpRE
ST m
anag
emen
t API
/Web
UI
Hea
rtbe
at
Proc
ess
mon
itor
Glo
bal s
ingl
eton
sup
ervi
sor
Confi
gura
tion
man
ager
on each node
Erlang/OTP
Reba
lanc
e or
ches
trat
or
Nod
e he
alth
mon
itor
one per cluster
vBuc
ket s
tate
and
repl
icati
on m
anag
er
HTTP distributed erlangerlang port mapper
Couchbase Server Architecture
Couchbase Storage EngineDat
a M
anag
er
Clus
ter M
anag
er
• Example of online migration of existing application.
• First migrated large or slow performing tables and frequently updated fields from MySQL to Couchbase
: Migrating to Couchbase
memcachedprotocol listener/sender
engine interface
Couchbase Storage Engine
TAP
Web Server
Apache + PHP
Client-side Moxi
MySQL
memcachedprotocol listener/sender
engine interface
Couchbase Storage Engine
TAP
memcachedprotocol listener/sender
engine interface
Couchbase Storage Engine
TAP
Web Server
Apache + PHP
Client-side Moxi
Matt Pfeil, VP of Customer Solutions
Architecture
Better technology
• Multi-master, multi-DC• Linearly scalable• Larger-than-memory datasets• Best-in-class performance (not just writes!)• Fully durable• Integrated caching• Tuneable consistency
A performance retrospective
Credit: Nathan Hurst, 2010
QUESTIONS FROM THE AUDIENCE
THANK YOU FOR JOINING US!We look forward to seeing you at our upcoming events
top related