Honest Performance Testing withNDBenchNetflix Data Benchmark

• CDE, Cloud Database Engineering

• Providing data stores as a service○Cassandra

○Dynomite

○Elasticsearch and RDS

Who are we?

Vinay Chella

Cloud Data ArchitectCassandra MVPCloud Database Engineering @ Netflix

• 98% of streaming data is stored in Cassandra

• Data ranges from customer details to Viewing history / streaming bookmarks to billing and payment

Cassandra @ Netflix

Agenda

• Background

• Why NDBench?

• Architecture

• Usage

• Achievements

• Roadmap

• Take away

Perf testing persistence layer?

Capacity in my existing fleet?

Why?

Why not already existing Perf testing tools?

What is NDBench?

Netflix Data Benchmark (NDBench) is a Pluggable cloud-enabled benchmarking tool that can be used

across any data store system.

Side by Side comparison

Different driver/ software versions

Different instance types

Dynamically tunable configurations

Varying data models

Pluggable Patterns & Loads

Different Client APIs

Netflix homegrown

• Well integrated with netflix OSS infrastructure

Architecture

What is Pluggable?

• Load Patterns

• Load tests

Load Patterns

• Random

• Sliding Window

Load Tests

• Cassandra-JavaDriver

• Elastic Search

•Dynomite

• Cassandra-Astyanax

• In-Memory Test

What can be configured?

• ndbench.driver.numKeys - 1000000

• ndbench.driver.dataSize - 200 bytes

• ndbench.driver.numWriters - 1

• ndbench.driver.numReaders - 1

• ndbench.driver.writeRateLimit - 100

• ndbench.driver.readRateLimit - 200

• ndbench.driver.useVariableDataSize - false

• Many more….

Dynamic Script

How to use it

• REST API

•UI

REST API• Initialization

- Initialize: /pappy/driver/init/{DriverName}

- Init Script: /pappy/driver/initfromscript

• Perf API

- Start writes: /pappy/driver/startWrites

- Start reads: /pappy/driver/startReads

- Stop everything: /pappy/driver/stop

• Sanity check

- Verify Read: /pappy/driver/readSingle/key

- Verify Write: /pappy/driver/writeSingle/key

- Verify Read: /pappy/driver/readSingle/key

• Backfill

- Data Backfill: /pappy/driver/startDataFill

- DataBackfill Async: /pappy/driver/startDataFillAsync

• Status API

- /pappy/driver/{getRead/Write}Status

- /pappy/driver/getserverstatus

NDBench Demo...

NDBench @ Netflix

•As a

-Benchmarking Tool

- Integration Tests

-Deployment Validation

NDBench’s Achievements @ Netflix

N+1

C* 1.2 → C* 2.0, C*2.0 → C* 2.1

C* 2.0 vs C* 2.1 (Reads - Thrift)

C* 2.0 vs C* 2.1 (Writes - Thrift)

CentOS ---> Trusty

CentOS -> Trusty Migration

LCS on CentOS vs Trusty (writes)

LCS on CentOS vs Trusty (Reads)

Java 7 → Java 8

C* on Java 7 vs Java 8 (Writes)

C* on Java 7 vs Java 8 (Reads)

C* AMI Certification Pipeline

Dynomite benchmarking

• Generating Millions of Ops/ sec

Dynomite benchmarking

Roadmap

• Performance profile management

•Automated metrics analysis

•Dynamic load generation based on destination

schema

https://github.com/Netflix/ndbench

Take away

“Test the honesty of your data models, persistence layers in Cloud ecosystem using NDBench”

Q & A