developing with java and couchbase server

1

A Technical Overview of Couchbase using Java

Raghavan “Rags” Srinivas Developer Advocate

Simple. Fast. Elas.c.

2

•  Developer Advocate at Couchbase working with developers

•  Speaker at JavaOne, RSA conferences, Sun Tech Days, JUGs and other developer conferences

•  Taught undergrad and grad courses •  Technology Evangelist for Java and Sun Microsystems for 10+ years

•  SKll trying to understand and work more effecKvely on Java and distributed systems

•  Philosophy: “Be#er to have an unanswered ques1on than a unques1oned answer”

Speaker CredenKals

3

•  Distributed Systems and Performance •  Couchbase Server •  GePng Started •  DemonstraKon and Case Study •  Resources and Summary

Agenda

4

DISTRIBUTED SYSTEMS AND PERFORMANCE

5

Web Application Architecture and Performance

Application Scales Out Just add more commodity web servers

Database Scales Up Get a bigger, more complex server

www.facebook.com/animalparty

Web Servers

RelaKonal Database

Load Balancer

-‐ Expensive and disrupKve sharding -‐ Doesn’t perform at Web Scale

6

A Distributed Hash Table

ApplicaKon

set(key, value) get(key) returns value

DData

Cluster

Database

7

•  The Network is reliable •  Latency is zero •  Bandwidth is infinite •  The Network is secure •  Topology doesn’t change •  There is one administrator •  Transport cost is zero •  The Network is homogenous

Eight Fallacies of Distributed CompuKng

8

Couchbase data layer scales like application logic tier Data layer now scales with linear cost and constant performance.

Application Scales Out Just add more commodity web servers

Database Scales Out Just add more commodity data servers

Scaling out flattens the cost and performance curves.

Couchbase Servers

www.facebook.com/animalparty

Web Servers Load Balancer

Horizontally scalable, schema-‐less, auto-‐sharding, high-‐performance at Web Scale

9

COUCHBASE SERVER

10

Couchbase “write” Data Flow – applicaKon view

User acKon results in the need to change the VALUE of KEY

ApplicaKon updates key’s VALUE, performs SET operaKon

Couchbase client hashes KEY, idenKfies KEY’s master server SET request sent over

network to master server

Couchbase replicates KEY-‐VALUE pair, caches it in memory and stores it to disk

1

2

3 4

5

11

Couchbase Data Flow – under the hood

SET request arrives at KEY’s master server

Listener-‐Sender

Master server for KEY Replica Server 2 for KEY Replica Server 1 for KEY

2 2

1 SET acknowledgement returned to applicaKon 3

Disk Disk Disk

RAM*

Couchb

ase storage en

gine

Disk Disk Disk

2

4

12

Proven at Small, and Extra Large Scale

•  Leading cloud service (PAAS) provider

•  Over 150,000 hosted applications •  Couchbase Server serving over

6,200 Heroku customers

•  Social game leader – FarmVille, Mafia Wars, Empires and Allies, Café World, FishVille

•  Over 230 million monthly users •  Couchbase Server is the

primary database behind key Zynga properties

13

GETTING STARTED

14

Java Client SDK

.Net SDK

PHP SDK

Ruby SDK

…

spymemcached ConnecKon

HTTP couchDB connecKon

Java client API

User Code

Couchbase Server

List <URI> servers = new ArrayList<URI>(); servers.add(oneserver); servers.add(twoserver); CouchbaseClient cb = new CouchbaseClient(servers, "aBucket", "letmein"); // this is all the same as before cb.set("hello", 0, "world"); cb.get("hello"); Map<String, Object> manyThings = cb.getBulk(CollecKon<String> keys); // accessing a view View view = cb.getView("design_document", "my_view"); Query query = new Query(); query.getRange("abegin", "theend");

hvp://www.couchbase.com/develop

Couchbase Client Java SDK

15

Client at RunKme: Adding a node

15

Couchbase Client

Couchbase Server Node





Cluster ��Topology��Update

New node coming online

16

Asynchronous OperaKons

•  Set OperaKon // Do an asynchronous set! OperationFuture<Boolean> setOp = client.set(KEY, EXP_TIME, VALUE);!! // Block and check to see if our set succeeded! try {! if (setOp.get().booleanValue()) {! System.out.println("Set Succeeded");! } else {! System.err.println("Set failed: " + ! setOp.getStatus().getMessage());! }! } catch (Exception e) {! System.err.println("Exception while doing set: " +! e.getMessage());! }!

17

OperaKons Available to a Client of Couchbase Server

•  Store OperaKons –  Add: Store the document if it does not yet exist –  Set: Store the document, overwriKng exisKng if necessary

•  Retrieve OperaKons –  Get: Fetch the document –  Get and touch: Fetch the document and update the TTL –  MulKget: Fetch mulKple documents at the same Kme

•  Update operaKons –  Append/prepend: Add data in front of or on the end of a document –  Delete: Remove the document from the store –  Compare and Swap (CAS): Replace the current document, if CAS matches –  Replace: Replace the document if it exists, otherwise do not –  Touch: Update the TTL for a document

18

Document Metadata: •  TTL •  CAS value •  Flags

Q: What happens to a document persisted a<er it’s TTL?

A: A regular background job will remove expired documents.

Q: How do I use flags?

A: Frequently used to idenKfy data type or other avributes, such as compression. Behavior varies from client to client.

Metadata

19

Distributed System Design: Concurrency Controls

•  Compare and Swap OperaKons –  Owen referred to as “CAS” –  OpKmisKc concurrency control –  Available with many mutaKon

operaKons, depending on client.

•  Get with Lock –  Owen referred to as “GETL” –  PessimisKc concurrency control –  Locks have a short TTL –  Locks released with CAS

operaKons –  Useful when working with object

graphs

19

Actor 1 Actor 2

Couchbase Server

CAS mismatch Success

A

B

F

C D

E

20

DEMO

21

3 Objects (documents) within game: • Players • Monsters •  Items

Gameplay:

• Players fight monsters • Monsters drop items • Players own items

Demo: The next big social game

22

{ "_id": "Keith4540", "_rev": "1-‐ab354009ce09f198c555b693e057adce", "jsonType": "player", "uuid": "35767d02-‐a958-‐4b83-‐8179-‐616816692de1", "name": "Keith4540", "hitpoints": 75, "experience": 663, "level": 4, "loggedIn": false

}

Player Document

23

{ "_id": "Katana_e5890c94-‐11c6-‐48-‐65746ce6c560", "_rev": "1-‐d6bbd5e814c32c66e22db2918a2efcd9", "jsonType": "item", "name": "Katana_e5890c94-‐11c6-‐65746ce6c560", "uuid": "e5890c94-‐11c6-‐4856-‐a7a6-‐65746ce6c560", "ownerId": "Dale9887"

}

Item Document

Player “_id”

24

Monster Document

{ "_id": "Bauchan9932", "_rev": "1-‐5c90be58be58134a0fc5e7db77dab5f2", "jsonType": "monster", "name": "Bauchan9932", "uuid": "d10dfc1b-‐0412-‐4140-‐b4ec-‐affdbf2aa5ec", "hitpoints": 370, "experienceWhenKilled": 52,

}

25 25 25 thanks for the photo! hvp://flic.kr/p/88fCcH

GAME ON!

26

CASE STUDY: TRIBAL CROSSING

26

27

Use case -‐ simple farming game: •  A player can have a variety of plants on their farm. •  A player can add or remove plants from their farm. •  A Player can see what plants are on another player's farm.

Tribal Crossing: RepresenKng Game Data in Couchbase

28

Tribal Crossing: Challenges

●  Write-‐heavy requests –  Caching does not help –  MySQL / InnoDB limitaKon (Percona)

●  Need to scale drasKcally over night –  My Polls – 100 to 1m users over a weekend

●  Small team, no dedicated sysadmin –  Focus on what we do best – making games

●  Keeping cost down

29


RepresenKng Objects ●  Simply treat an object as a Map/dicKonary/JSON object ●  Determine the key for an object using the class name (or type) of the object and an unique ID

RepresenKng Object Lists

●  DenormalizaKon ●  Save a comma separated list or an array of object IDs

30


Player Object Key: 'Player1' JSON { “_id” : “Player1”, “nid” : 1, “name” : “Shawn” }

Plant Object Key: 'Plant201' JSON { “_id” : “Plant201”, “nid” : 201, “player_id” : 1 “name” : “Starflower” }

PlayerPlant List Key: 'Player1_PlantList' JSON { “_id” : “Player1_Plantlist”, “plants” : [201, 202, 204] }

31

Tribal Crossing: Schema-‐less Game Data

●  No need to “ALTER TABLE” ●  Add new “fields” all objects at any Kme

–  Specify default value for missing fields –  Increased development speed

●  Using JSON for data objects though, ●  Offers the ability to query and analyze arbitrary fields in Couchbase 2.0

32

Tribal Crossing: Accessing Game Data in Couchbase

Get all plants that belong to a given player Request: GET /player/1/farm // Create a new PlantList from players PlantList playersPlants = new PlantList(cbclient.get("Player1_PlantList"); for (Plant plant : playersPlants) { aPlayer.addPlant(plant);

33

Tribal Crossing: Modifying Game Data in Couchbase

Give a player a new plant // Create the new plant Plant givenPlant = new Plant(100, "Mushroom"); cbclient.set("Plant100", givenPlant); // Update the player plant list Player thePlayer = player.fetch(cbclient.get("Player1"); // Add the plant to the player thePlayer.receivePlant(givenPlant); // Store the player's new plant list cbclient.set("Player1_PlantList", thePlayer.getPlantsArray());

34

•  Significantly reduced the cost incurred by scaling up database servers and managing them.

•  Achieved significant improvements in various performance metrics (e.g., read, write, latency, etc.)

•  Allowed focus more on game development and opKmizing key metrics

•  Plan to use real-‐Kme Map-‐Reduce, querying, and indexing abiliKes provided by Couchbase Server 2.0

Tribal Crossing: Conclusion

35

ROADMAP

36

Couchbase Server 2.0

•  Next major release of Couchbase Server •  Currently in Developer Preview

What’s new: •  Indexing and Querying •  Incremental Map Reduce •  Cross Data Center ReplicaKon

•  Fully backwards compaKble with exisKng Couchbase Server

37

Accessing a View in Couchbase Server 2.0

Query query = new Query(); query.setReduce(false); query.setIncludeDocs(true); View view = cbclient.getView("dev_players", "player_by_level"); HttpFuture<ViewResponse> future = client.asyncQuery(view, query); ViewResponse response = future.get(); if (!future.getStatus().isSuccess()) { // do something to handle the errror } Iterator<ViewRow> itr = response.iterator(); while (itr.hasNext()) { ViewRow row = itr.next(); if (ITEMS.containsKey(row.getId())) { assert ITEMS.get(row.getId()).equals(row.getDocument()); } }

38

RESOURCES AND SUMMARY

39

•  Couchbase Server Downloads –  hvp://www.couchbase.com/downloads-‐all –  hvp://www.couchbase.com/couchbase-‐server/overview

•  Developing with Java Client libraries –  hvp://www.couchbase.com/develop/java/current

•  Couchbase forums –  hvp://www.couchbase.com/forums/

•  Sqoop-‐based Hadoop Connector for Couchbase –  hvp://www.couchbase.com/develop/connectors/hadoop

•  Couch Conference (coming to a city near you) –  hvp://www.couchbase.com/couchconf-‐world-‐tour

•  Tribal Crossing blog –  hvp://www.fablelabs.com/press/real-‐world-‐nosql-‐membase-‐at-‐tribal-‐crossing.html

Resources, Summary and Call For AcKon

40

THANKS -‐ Q&A

40

Raghavan “Rags” Srinivas, Couchbase Inc. ([email protected], @ragss)