couchbase 101 - installation of couchbase

Tuesday, October 8, 13

Technical Evangelist

twi0er: @scalabl3email: jasdeep@couchbase.com

Jasdeep Jaitla

Couchbase 101 -‐ The Server

Technical Evangelist

twi0er: @scalabl3email: jasdeep@couchbase.com

Jasdeep Jaitla

Couchbase 101 -‐ The Server

ARCHITECTURE

Evolution from memcached

•Founders were key contributors to memcached

•Evolved into Membase, a distributed and persisted key-‐value store

•Evolved into Couchbase Document Store with JSON support and Map-‐Reduce Indexes, ElasOc Search IntegraOon, and Cross-‐Data Center ReplicaOon

Couchbase Server Core Principles

Easy Scalability

Consistent High Performance

Always On 24x365

Grow cluster without applica9on changes, without down9me with a single click

Consistent sub-‐millisecond read and write response 9mes with consistent high throughput

No down9me for soBware upgrades, hardware maintenance, etc.

Flexible Data Model

JSON Anywhere document model with no fixed schema.

JSONJSONJSON

JSONJSON

PERFORMANCE

Couchbase Server 2.0 Architecture

Heartbeat

Process m

al singleton supe

gura9o

n manager

on each node

Rebalance orchestrator

alth m

one per cluster

vBucket state and

replica9

anager

anagem

HTTP8091

Erlang port mapper4369

Distributed Erlang21100 -‐ 21199

Erlang/OTP

storage interface

Couchbase EP Engine

11210Memcapable 2.0

11211Memcapable 1.0

Memcached

New Persistence Layer

8092Query API

ry Engine

Data Manager Cluster Manager

Couchbase Server 2.0 Architecture

New Persistence Layer

storage interface

Couchbase EP Engine

11210Memcapable 2.0

11211Memcapable 1.0

Object-‐level Cache

Disk Persistence

8092Query API

ry Engine

HTTP8091

Erlang port mapper4369

Distributed Erlang21100 -‐ 21199

Heartbeat

Process m

al singleton supe

gura9o

n manager

on each node

Rebalance orchestrator

alth m

one per cluster

vBucket state and

replica9

anager

REST m

anagem

Erlang/OTP

Server/Cluster Management & Communica6on

(Erlang)

RAM Cache, Indexing & Persistence Management

Couchbase OrganizaOon

• Couchbase operates like a Key-‐Value Document Store

• Key is a UTF-‐8 string up to 256 Bytes

• Values can be:

-‐ Simple Datatypes: strings, numbers, dateOme, boolean, and binary data can be stored -‐-‐ they are stored as Base64 encoded strings

-‐ Complex Datatypes: dicOonaries/hashes, arrays/lists, can be stored in JSON format (simple lists can be string based with delimiter)

-‐ JSON is a special class of string with a specific format for encoding simple and complex data structures

• Schema is unenforced and implicit, schema changes are programmaOc, done online, and can vary from Document to Document

Metadata and Documentsmeta{

“id”: “u::jasdeep@couchbase.com”,“rev”: “1-‐0002bce0000000000”,“flags”: 0,“expiraUon”: 0,“type”: “json”

document{

“uid”: 123456,“firstname”: “jasdeep”,“lastname”: “Jaitla”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “jasdeep@couchbase.com”

Meta Informa6on Including Key (id)

All Keys Unique and Kept in RAM

Document Value

Most Recent In RAM And Persisted To Disk

Retrieval Operations

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Replica Couchbase Cluster Machine

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Storage Operations

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Storage Operations

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

set/add/replace

Storage Operations

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

set/add/replace

Storage Operations

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

set/add/replace

Storage Operations

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

set/add/replace

Consistency

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Consistency

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Ejection, NRU, Cache Miss

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

set/add/replaceset/add/replaceset/add/replace

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

FULL (90%)

NRU Documents Ejected

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Couchbase Server

EP EngineRAM Cache

Disk Write Queue

Replication Queue

Application Server

Non-‐Resident Document

("Cache Miss")

Clients Connect Directly to Couchbase Nodes

Application Servers

8 GB RAMPartitions

3 IO Workers

Application Servers

8 GB RAMPartitions

3 IO Workers

Application Servers

8 GB RAMPartitions

3 IO Workers

Key Hash-Partitioning

Application Servers

8 GB RAMPartitions

3 IO Workers

ClientHashFunc6on("jasdeep@couchbase.com") => Par66on[0..1023] {25}ClusterMap[P(25)] => [x.x.x.x] => IP of Server Responsible for Par66on 25

Application Servers8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

Horizontal Scale-Rebalance

16 GB RAM

6 IO Workers

Partitions

8 GB RAM

3 IO Workers

Partitions

Application Servers8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

16 GB RAM

6 IO Workers

Partitions1024

Partitions

32 GB RAM

12 IO Workers

Partitions

Application Servers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

Partitions

16 GB RAM

6 IO Workers

Partitions

MAP MAP

32 GB RAM

12 IO Workers

Partitions

Application Servers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

Partitions

16 GB RAM

6 IO Workers

Partitions

32 GB RAM

12 IO Workers

Partitions

Application Servers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

8 GB RAM

3 IO Workers

Partitions

All Metadata for All Documents

(64 bytes + Key Length)

Document Values(NRU Ejected if RAM Quota

Used > 90%)

Also Leave RAM For OS:

[Filesystem Cache >> Views]

Document Indexing

Monitoring

Recommended:

minimum 4 Cores+ 1 core per design document+ 1 core per XDCR replicated

bucket

Persisted Documents

All Indexes for Design Documents/Views

Append-‐Only Disk Format& CompacUon

Performance:

MulUple EBS Volumes High IOPS Raid 0 on Amazon

RAM, CPU and IO Guidelines

RAM CPU Disk IO

Binary Socket Operations (11210/1)• get (key)

– Retrieve a document

• set (key, value)

– Store a document, overwrites if exists

• add (key, value)

– Store a document, error/exception if exists

• replace (key, value)

– Store a document, error/exception if doesn’t exist

• incr (key)

– Create/Increment Atomic Counter

• decr (key)

– Decrement Atomic Counter

• cas (key, value, cas)

– Compare and swap, set document only if it hasn’t changed (Optimistic Lock)

HTTP Operations (8092)

• View Querying

– Range Queries

– Index-Key Match Queries

– Set Match Queries

– Aggregate Reduces

– Group Level + Grouping Queries

INSTALL SERVER

Download Couchbasewww.couchbase.com/download

Install on Amazon via AMI

Run Couchbase and Open Browserlocalhost:8091 or ec2-‐xx-‐xx-‐xx-‐xx.compute-‐1.amazonaws.com:8091

Setup: Step 1 - Storage Paths & RAM Maximum

Setup: Step 2 - Sample Buckets

Setup: Step 3 - Configure Default Bucket

Setup: Step 4 - Register & Agree

Setup: Step 5 - Admin password

Setup: Complete

SETUP SDK

Supported SDK'swww.couchbase.com/communiUes

• Each supported SDK page has instrucOons for setup

• PHP, Ruby, NodeJS and Python clients are wrappers around libcouchbase C library, so libcouchbase must be installed first

• For other community clients, click on "All Clients" on lei nav, scroll down the page and you can see clients for Go, Erlang, Clojure, TCL, other nodejs and Perl.

Installing Libcouchbase

• Mac Tips before Libcouchbase & SDK Install

• Make sure you have XCode & Command Line Tools Installed

• Install Homebrew if you don't have it already

• Do a $ brew update, $ brew upgrade and $ brew doctor to be sure you're up to date

www.couchbase.com/communiUes/c/gekng-‐started

Installing Libcouchbase

• Mac Via Homebrew

• $ brew install libcouchbase

• PC-‐Windows

• Download appropriate Zip from website

• Redhat/CentOS

• wget the yum repositories

• $ sudo yum install -‐y libcouchbase2-‐libevent libcouchbase-‐devel

• Ubuntu

• wget ubuntu repositories

• $ sudo apt-‐get install libcouchbase2-‐libevent libcouchbase-‐dev

www.couchbase.com/communiUes/c/gekng-‐started

couchbase 101 - installation of couchbase

memcached newpersistencelayer

technicalevangelist

queryapi queryengine

hardware maintenance

schemachangesare programmaoc

documents meta

Technology

sizing your couchbase cluster: couchbase connect 2014

couchbase server vs. ncache - alachisoft€¦ · couchbase...

nosql 101: couchbase connect 2014

webinar: couchbase for mobile - introduction to couchbase...

couchbase in the digital economy – couchbase connect 2016

couchbase at linkedin: couchbase connect 2014

testing and deploying couchbase mobile – couchbase connect...

couchbase meetup - "introduzione a nosql e couchbase"

advanced development with couchbase lite: couchbase connect...

couchbase chennai meetup: developing with couchbase- made...

couchbase sydney meetup #1 couchbase architecture and...

couchbase chennai meetup #3 what's new in couchbase server...

online training - couchbase mobile 101

practical rxjava - jfokus · practical rxjava @couchbase...

the future of couchbase mobile: couchbase connect 2014

under the covers - couchbase server architecture: couchbase...

couchbase live europe 2015: couchbase 101

visual analytics with tableau & couchbase: couchbase connect...

couchbase server 101: couchbase connect 2015

accessing iot data with couchbase server, couchbase mobile...