why nosql? why riak? - goto conferencegotocon.com/dl/jaoo-brisbane-2010/slides/justinsheehy... ·...

Why NoSQL?

Justin [email protected]

Why Riak?

1

What's all of this NoSQL nonsense?

2

MongoDB

CouchDB

Cassandra

Voldemort

Neo4j

MembaseRedis

HBaseRiak

(and the list goes on...)

What went wrong with SQL databases?

3

Nothing! They are great tools.

What went wrong with SQL databases?

4

Nothing! They are great tools.

NoSQL came before SQL.

5

IBM IMS MUMPS

Honeywell IDS

Cincom TOTAL

dbm

VAX DBMS


PICK

But that's not really NoSQL, is it?

6

IBM IMS MUMPS

Honeywell IDS

Cincom TOTAL

dbm

VAX DBMS


PICK

The start of today's NoSQL

7

Amazon: Dynamo (2007)

Google: Bigtable (2006)

Technology is not the important part!

8

Amazon sells books.(esp. 3+ years ago)

Why not just use Oracle/MySQL/etc?

Technology is not the important part!most


9

Amazon sells books.(esp. 3+ years ago)

Why not just use Oracle/MySQL/etc?

Business needs (re)created demand foralternative database technologies.


What's all of this NoSQL nonsense?

10

MongoDB

CouchDB

Cassandra

Voldemort

Neo4j

MembaseRedis

HBaseRiak

Do we really need all of these?

It's still about business needs.

11

Redis

Riak

use cas

es drive

decisio

ns!

It's still about business needs.

12

Redis

Riak

what solves

your re

al problem

choose

13

what solves your real problemhow do you know ?

two places to start:

➡ innate/exposed data model➡ distribution/operational model

14

data model differences

requirements usually from app developers

Native Data Structures

Semi-Structured Documents

Bigtable Column FamilyTabular/Relational

HierarchicalGraph Traversal Key/Value

Indexed Query

15

data model differencesnot much useful ordering

Bigtable Column Family

Graph Traversal

which one is

more powerfu

l?

16

ColFamGraphK/V etc etc

data model differences

Cassandra

Voldemort

Neo4j

17

distribution model differencesrequirements usually from biz or operations

Locally Embedded

Single Server

Distributed System

Server Replication

18

distribution model differencesrequirements usually from biz or operations

Locally Embedded

Single Server

Distributed System

Server Replication

19

starting to decide

Locally Embedded

Single Server

Distributed System

Server Replication


20

starting to decide

Locally Embedded

Single Server

Distributed System

Server Replication


We need a K/Vor ColFam store.

21

starting to decide

Locally Embedded

Single Server

Distributed System

Server Replication


We need a K/Vor ColFam store.

We need to recoverquickly from server failure.

22

Distributed System

Server Replication

ColFamK/V

now we're getting somewhere!

23

Distributed System

Server Replication

ColFamK/V

we can make a shorter list...Riak, Voldemort, CouchDB, Cassandra, HBase...

24

Distributed System

Server Replication

ColFamK/V


then we can narrow it more...protocols, licensing, benchmarking, simplicity...

25

Distributed System

Server Replication

ColFamK/V



and make a choice!

26

Distributed System

Server Replication

ColFamK/V



and make a choice!

How about MySQL?

I believe we could, Bob.

27

How about MySQL?


wait, WHAT?

28

How about MySQL?


wait, WHAT?

Great tools are still great tools.

29

Great tools are still great tools.

Understand your needsbefore you choose.

30

So what's so special about

?

31

Locally Embedded

Single Server

Distributed System

Server Replication


Riak KV

32

Locally Embedded

Single Server

Distributed System

Server Replication


Riak KVRiak Search

33

Locally Embedded

Single Server

Distributed System

Server Replication


Riak KVRiak Search

Riak Core

34

Locally Embedded

Single Server

Distributed System

Server Replication

ColFamGraphK/V etc ?

improvements flow upward

Riak Core can flow sideways

35

Locally Embedded

Single Server

Distributed System

Server Replication


Will the market need this?

36

Locally Embedded

Single Server

Distributed System

Server Replication


Will the market need this?

I know how to get there.

37

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

storage engine

The Riak key/value stack:

38

client application

protobuf http

riak_client

dynamo model FSMs

vnode master

k/v vnode

storage engine

client application

protobuf http

riak_client

dynamo model FSMs

the cluster nodes are united by riak core via gossip, consistent hashing, etc

vnode master

k/v vnode

storage engine

client application

protobuf http

riak_client

dynamo model FSMs

vnode master

k/v vnode

storage engine

39

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

storage engine

just a local k/v store:

40

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

storage engine

just an abstract k/v store:

41

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

storage engine

a distributed system at heart:

42

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

storage engine

a distributed system at heart:

virtual nodes

gossip

failure detection

vector clocks

sloppy quorums

remote dispatchdynamic membership

43

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

storage engine

carefully managed complexity...

44

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

storage engine

allows simplicity at the edges

45

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

k/v storage engine

let's make a dist. search system!

46

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

k/v storage engine


adapted FSMs

47

client application

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

k/v storage engine


adapted FSMs

search storage

search vnode

48

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

k/v storage engine

adapted FSMs

search storage

search vnode

search client

solr lucene

done!

49

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

k/v storage engine

adapted FSMs

search storage

search vnode

search client

solr lucene

the sum

is greater than

the parts

50

I don't know what's next

51

protobuf http

riak_client

dynamo model FSMs

riak core

vnode master

k/v vnode

k/v storage engine

adapted FSMs

search storage

search vnode

search client

solr lucene

I don't know what's next,

but I know how to build it.

more FSMs

X client

another interface

X storage

X vnode


52

riak core

I don't know what's next,

but I know how to build it.


scalabilityfault-tolerance

ease of operations

interoperability

pluggable protocolsflexible storage

predictable availability

I know how to build it

53

scalabilityfault-tolerance

ease of operations

interoperability

pluggable protocolsflexible storage

predictable availability

I know how to build itI know how to build it,

but you don't have to.

54

our earlier evaluation criteria:


55


fully distributed system

56


fully distributed system

flexible and growing

57

Justin [email protected]

http://www.basho.com/