cassandra at brighttag

Cassandra @ BrightTag Matthew Kemp

Our Problem(s)Initial Iteration

"We want to make a match table"

Second Iteration"We want to support flexible lookups"

Third Iteration"We want to support more attributes"

Cassandra Timeline: 2011

Evaluated Cassandra 0.8, Couch, Mongoand Riak

Summer 2011 Fall 2011 Winter 2011

Decided on Cassandra(1.0 just released)

Initial AWS deployment in two regions(8 total nodes)

Spring 2011

Our Very First Schemacreate column family brighttag_data

with comparator = 'UTF8Type'

and default_validation_class = 'UTF8Type'

and key_validation_class = 'UTF8Type'

and read_repair_chance = 0.0

and gc_grace = 864000;

Example Data RowsRow Key Columns ...

a111 wwww xxxx yyyy

w111 x111 y111

b222 xxxx zzzz

x222 z222

c333 wwww yyyy zzzz

w333 y333 z333


Attempted to use SuperColumns and started using Priam


Attempted to use Composite Columns and switched to m2.xlarges

Spring 2012

Upgraded to Cassandra 1.1 and start of production usage

Added index column family

Priam

Because nobody wants to deal with this

Super Column Failed

Row Key Columns ...

a111 wwww xxxx

id attr_1 attr_2

w111 value1 value2

id attr_3 attr_4

x111 value3 value4

To read a single value, you have to unpack the whole super column.

Composite Column AlmostRow Key Columns ...

a111 (wwww,id) (wwww,attr_1) (wwww,attr_2) (xxxx,id)

w111 value1 value2 x111

b222 (xxxx,id)

x222

c333 (wwww,id) (wwww,attr_2)

w333 something

Solution? Roll Your OwnRow Key Columns ...

a111 wwww:id wwww:attr_1 wwww:attr_2 xxxx:id

w111 value1 value2 x111

b222 xxxx:id

x222

c333 wwww:id wwww:attr_2

w333 something

Schema Additionscreate column family brighttag_index

with comparator = 'UTF8Type'

and default_validation_class = 'UTF8Type'

and key_validation_class = 'UTF8Type'

and read_repair_chance = 0.0

and gc_grace = 864000;

Example Index RowsRow Key Columns ... ... continued ...

wwww:w111 bt_id xxxx:x222 bt_id

a111 b222

xxxx:x111 bt_id zzzz:z222 bt_id

a111 b222

yyyy:y111 bt_id wwww:w333 bt_id

a111 c333


Performance testing, evaluating 1.2.x, switched to Agathon


Upgraded to Cassandra 1.2.9and schema redesign

Spring 2013

Upgraded to Virtual Nodes and started using OpsCenter

KeyspaceCREATE KEYSPACE bt WITH replication = {

'class': 'NetworkTopologyStrategy',

'eu-west': '2',

'ap-northeast': '2',

'us-west': '2',

'us-east': '2'

};

Data TableCREATE TABLE brighttag_data (

btid uuid,

identifier varchar,

ids map<varchar, varchar>,

data map<varchar, varchar>

PRIMARY KEY (btid, identifier)

);

Example Data Rowsbtid identifier ids data

a111 wwww {'id':'w111'} {'attr_1':'value1'}

a111 xxxx {'xid':'x111'} {'attr_2':'value-a'}

a111 yyyy {'id':'y111'} {}

b222 xxxx {'xid':'x222'} {'attr_2':'value-b'}

b222 zzzz {'zid':'z222'} {}

Index TableCREATE TABLE brighttag_index (

identifier varchar,

id_name varchar,

id_value varchar,

btid uuid

PRIMARY KEY (identifier, id_name, id_value));

Example Index Rowsidentifier id_name id_value btid

wwww id w111 a111

xxxx xid x111 a111

yyyy id y111 a111

xxxx xid x222 b222

zzzz zid z222 b222

Switching to Virtual Nodes

Old Ring(read only)

New Ring(read & write)

Data Access Service

"Migration" Script

AgathonSimilar to Priam, except not tied to AWS and does not have to run as a co-process

Seed Provider (drop in jar) & Java Webapp

Multi-ring support

Open Sourced (Github)

https://github.com/BrightTag/agathon

Agathon In PracticeAgathon"Manifest"

provider

SimpleDB

???


???

Summer 2014 Fall 2014 Winter 2014Spring 2014

Upgraded to i2.xlarges and Cassandra 1.2.16

Schema Improvements (planned)

AWS Instance Types

Instance Type ECUs Virtual CPUs Memory Disks

m2.xlarge 6.5 2 17.1 1x420GB

m2.2xlarge 13 4 34.2 4x420GB

i2.xlarge 14 4 30.5 1x800GB SSD

SSD Performance: WritesUpgrade to SSDs Upgrade to 1.2.16

SSD Performance: ReadsUpgrade to SSDs Upgrade to 1.2.16

Current JVM Settings-Xms16G

-Xmx16G

-Xmn1000M

-XX:+HeapDumpOnOutOfMemoryError

-Xss180k

-XX:+UseParNewGC

-XX:+UseConcMarkSweepGC

-XX:+CMSParallelRemarkEnabled

-XX:CMSInitiatingOccupancyFraction=75

-XX:+UseCMSInitiatingOccupancyOnly

-XX:SurvivorRatio=8

-XX:+UseTLAB

JVM Settings Being Tested# lets the CMS run in increments

-XX:+CMSIncrementalMode

# we probably want this, forces a minor GC as well

-XX:+CMSScavengeBeforeRemark

# CMS tunings

-XX:+CMSIncrementalPacing

-XX:CMSIncrementalDutyCycleMin=0

-XX:CMSIncrementalDutyCycle=10

# adjust tenuring thresholds

-XX:InitialTenuringThreshold=10

-XX:MaxTenuringThreshold=15

Some Ring Stats44 Nodes (i2.xlarge)

1B+ rows in data table

Estimated 3B+ rows in index table

~3.5 TB of data

New Use of CassandraWe're starting to use this pipeline for metrics

Storm + Trident + KairosDB + Cassandra

Different configurationOne ring per region

Each ring is still pretty small (~4 nodes)

Dealing With FailureWe've survived …

Amazon swallowing nodes

Disk Failures

Intra and inter region network issues

AdviceUse SSDs, don't skimp on hardware

Stay current, but not too current

Test various configurations for your use case

Simulating production load is important

The Best Advice"The best way to approach data modeling for Cassandra is to start with your queries and work backwards from there. Think about the actions your application needs to perform, how you want to access the data, and then design column families to support those access patterns."

Questions?