couchconf london: couchbase server in production

1

Couchbase Server in Production

Perry KrugSr. Solutions Architect

2

Typical Couchbase production environment

Application users

Load Balancer

Application Servers

Servers

3

We’ll focus on App-Couchbase interaction …

Application users

Load Balancer

Application Servers

Servers

4

… at each step of the application lifecycle

Dev/Test Size Deploy Monitor Manage

5

KEY CONCEPTS

6

Reading, Writing and Arithmetic

Reading Data Writing Data

Server

Give medocument A

Here is document A

Application Server

A

Server

Please storedocument A

OK, I storeddocument A

Application Server

A

(We’ll save the arithmetic for the sizing section : )

7

Server

Reading data

RAM

DISK

Application Server

Give me document A

A

Here is document A

If document A is in memory return document A to the applicationElse add document to read queue reader eventually loads document from disk into memory return document A to the application

A

Reading Data

8

Keeping working data set in RAM is key to read performance

Your application’s working set should fit in RAM…

Server

RAM

DISK

Application Server

Give me document A

A

Here is document A

If document A is in memoryreturn document A to the application

Elseadd document to read queuereader eventually reads document

from disk to memoryreturn document A to the application

A

… or else! (because you don’t want the “else” part happening very often – it is MUCH slower than a memory read and you could have to

wait in line an indeterminate amount of time for the read to happen.)

Reading Data

9

Working set ratio depends on your application

Server Server Server

Late stage social gameMany users no longer

active; few logged in at any given time.

Ad NetworkAny cookie can show up

at any time.

Business applicationUsers logged in during

the day. Day moves around the globe.

working/total set = 1working/total set = .01 working/total set = .33

Reading Data

10

Server

Couchbase in operation: Writing data

RAM

DISK

Application Server

Store document A

A

OK, it is stored

If there is room for the document in RAM Store the document in RAMElse Eject other document(s) from RAM Store the document in RAMAdd the document to the replication queue Replicator eventually transmits documentAdd the document to write queue Writer eventually writes document to disk

A

Writing Data

11

Server

Flow of data when writing

Writing Data

Application ServerApplication Server Application Server

Applications writing to Couchbase

Couchbase writing to disk

network

Couchbase transmitting replicas

12

Server

Queues build if aggregate arrival rate exceeds drain rates

Writing Data


network

Replication queue Disk write queue

13

ServerServer Server

Scaling out permits matching of aggregate flow rates so queues do not grow


network networknetwork

14

DEVELOPMENT

Dev-Test Size Deploy Monitor Manage

15

Couchbase Data

• Couchbase uses (and is completely compatible with) the memcached protocol.

• While you can use any standard memcached library, Couchbase also provides it’s own libraries for a variety of languages.

• Couchbase is document-oriented

• See http://www.couchbase.com/develop

16

Couchbase SDKs

Java SDK

.Net SDK

PHP SDK

Ruby SDK

…and many more

Java client API

User Code

Couchbase Server

CouchbaseClient cb = new CouchbaseClient(listURIs,"aBucket", "letmein");// this is all the same as beforecb.set("hello", 0, "world");cb.get("hello");

http://www.couchbase.com/develop

Couchbase Java Library (spymemcached)

17

Farm Town Wars App Code

Applic

ati

on s

erv

er

Couch

base

Serv

er

Couchbase JavaClient library

Couchbase Server

11210

(“smart”) library

Farm Town Wars App Code

Applic

ati

on s

erv

er

Couch

base

Serv

er

Memcached Client

Moxi (Couchbase proxy)

11210

Client-side Moxi

OR80918091

Couchbase Client Deployment

Couchbase Server

18

SERVER AND CLUSTER SIZING(TIME FOR THE ARITHMETIC)


19

Size Couchbase Server

Sizing == performance• Serve reads out of RAM• Enough IO for writes• Mitigate inevitable failures

Reading Data Writing Data

Server

Give medocument A

Here is document A

Application Server

A

Server



Application Server

A

20

How many nodes?

4 Key Factors determine number of nodes needed:

1) RAM2) Disk3) Network4) Data Distribution/Safety

Couchbase Servers

Web application server

Application user

21

RAM sizing

1) RAM• Working set• Metadata• Buffer/overhead• Active+Replica(s)

Keep working set in RAM for best read performance

Server

Give medocument A Here is

document A

Application Server

A

A

A

Reading Data

23

Disk sizing: Space and I/O

2) Disk• Sustained write rate• Rebalance capacity• Backups • Total dataset• Active+Replicas

I/O

Space



Application Server

A

Server

A

A

Writing Data

24

Network sizing

3) Network• Client traffic• Replication (writes)• Rebalancing

Reads+Writes

Replication (multiply writes) and Rebalancing

25

Data Distribution

4) Data Distribution / Safety (assuming one replica):• 1 node = BAD• 2 nodes = …better…• 3+ nodes = BEST!

Note: Many applications will need more than 3 nodes

Servers fail, be prepared. The more nodes, the less impact a failure will have.

26

COUCHBASE CLIENT LIBRARY

Data Distribution

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Read/Write/Update


Read/Write/Update

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2

Replica Docs Replica Docs Replica Docs

Active Docs Active Docs Active Docs

CLUSTER MAP CLUSTER MAP

APP SERVER 1 APP SERVER 2

COUCHBASE SERVER CLUSTER

27

How many nodes? (recap)

4 Key Factors determine number of nodes needed:

1) RAM2) Disk3) Network4) Data Distribution

Couchbase Servers

Web application server

Application user

28

MONITORING


29Server

Key resources: RAM, Disk, Network

RAM

DISK

NETWORK

Server

RAM

DISK

Server

RAM

DISK

Application Server Application Server Application Server

30

Monitoring

Once in production, heart of operations is monitoring

-RAM Usage-Disk writes queues / read activity-Network bandwidth, replication queues-Data distribution (balance, replicas)

31

Monitoring

IMMENSE amount of information available

-Real-time traffic graphs-REST API accessible-Per bucket, per node and aggregate statistics-Application and inter-node traffic-RAM <-> Disk-Inter-system timing

32

How do you know when your working set is not in RAM?

Server

RAM

DISK

Application Server

Give me document A

A

Here is document A

If document A is in memoryreturn document A to the application

Elseadd document to read queuereader eventually loads document

from disk into memoryreturn document A to the application

A

Cache Miss Ratio

33

How do you know when you don’t have enough disk I/O?

Disk Write Queue

34

How do you know when you don’t have enough network I/O?

TAP Replication Queue

36

MANAGEMENT AND MAINTENANCE


37

Growth

Going from 5 million to 100 million users…– RAM usage is growing:

• Cache misses increasing• Resident item ratios decreasing• Disk fetches increasing

– Disk write queue growing higher than usual

Need to add a few more nodes...

…More RAM, disk and network without any downtime

38

Add Nodes

Read/Write/Update Read/Write/Update

Doc 7

Doc 9

Doc 3

Active Docs

Replica Docs

Doc 6


CLUSTER MAP

APP SERVER 1


CLUSTER MAP

APP SERVER 2

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2



SERVER 4 SERVER 5

Active Docs Active Docs

Replica Docs Replica Docs


39

As simple as running a packaged script (cbbackup)

Done on live system with minimal to no performance impact

Backup

40

1) Replace backup files, server will automatically “warmup” from disk files upon restart– Traditional RDBMS performance is acceptable while slowly

populating cache– Our applications demand a different level of performance– Couchbase Server pre-loads as much as possible into RAM

Restore

warmup

41

Restore

2) “cbrestore” used to restore data into live/different cluster

Data Files

cbrestore

42

1. Add nodes of new version, rebalance…2. Remove nodes of old version, rebalance…3. Done!

No disruption

General use for software upgrade, hardware refresh, planned maintenance

Upgrade existing Membase 1.7 to Couchbase Server 1.8

Upgrade

43

Current use of sqlite causes performance degradation as DB files get fragmented

-“vacuum” available (but not as online operation)- Best practice: Repeat rebalance to “clean” disk files

Under Development: “Maintenance mode” to allow for safely offlining of node to perform vacuuming in place.

Couchbase Server 2.0 has much improved behavior

Disk fragmentation

44

Failures Happen!

HardwareNetworkBugs

45

Easy to Manage failures with Couchbase

• Failover (automatic or manual):– Replica data promoted for immediate access– Replicas not recreated– Do NOT failover healthy node

46

Fail Over

Doc 7

Doc 9

Doc 3

Active Docs

Replica Docs

Doc 6


CLUSTER MAP

APP SERVER 1


CLUSTER MAP

APP SERVER 2

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7 Doc 8

Doc 6

Doc 3

DOC

DOC

DOCDOC

DOC

DOC

DOC DOC

DOC

DOC

DOC DOC

DOC

DOC

DOC

Doc 9

Doc 5DOC

DOC

DOC

Doc 1

Doc 8

Doc 2



SERVER 4 SERVER 5

Active Docs Active Docs

Replica Docs Replica Docs


47

Easy to maintain Couchbase

• Use remove+rebalance on “malfunctioning” node:– Protects data distribution and “safety”– Replicas recreated– Best to “swap” with new node to maintain capacity

48Dev/Test Size Deploy Monitor Manage

49

QUESTIONS?

50

DEMO

couchconf london: couchbase server in production

Technology

couchbase data couchbase

application elsea

memoryreturn document

document ahere

document aserverserverwell

couchbase cli

queuenetwork writing

disknetwork writing