case studies session 2

BlackbirdBillions of rows, couple of milliseconds away

Ishan ChhabraShrijeet PaliwalAbhijit Pol

$2.38965$0.6782$1.7234

$0.09$1.78964$1.6782$1.7234$0.809$2.421.25

$2.11$1.26

$2.178$2.056$0.809$2.421.25

$2.11$1.26$2.78$1.56

$1.809$2.421.25

$2.11$1.26$2.78$0.56$2.421.25

$2.11$1.26$2.78

$0.756$0.809$2.421.25

$2.11$1.26$2.78

$1.256$1.809$2.421.25

$2.11$1.26$2.78

$0.586$2.009

1.25$2.11$1.26$2.78$1.56

$0.00

Site/PageGeo/WeatherTime of DayBrand AffinityUser

[ + ][ + ]

User Segments

3. Bid Request

5. Rocket Fuel Winning Ad

2. Ad Request

6. Ad Served

1. Page Request

4. Bid & Ad

Browser

User Engagements

Publishers

Data Partners

Exchange Partners

Optimize

Simple View of Rocket Fuel Platform

Real-time Bidder

User Engagement

User DataStore

Model Scoring

So what is Blackbird?

Facebook likes

Searches on Google

Bid Requests Considered by Rocketfuel

5 B

6 B

45 B

Requests per day

Blink of an eye

SF to Tokyo network round trip

One beat of a hummindbird's wing

Look up in Blackbird

400

100

20

2

Time (ms)

Powered by

HBase, we have a problem..

Object NoSQL Mapper

List<KeyValue>

High Performance Collections

» Data loss on concurrent modification» Read per write» High amount of data per write» O(n)

» Significantly reduced flushes, compaction, network usage, GC.

» O(1)

Combined Column: 100 entries

c1:combined

1 entry

c1:rand1

2 entries

c1:rand2

1 entry

c1:rand3

Logical Collection

Append Only, the HBase view

Optimizing reads using normalization

Combined Column: 100 entries

c1:combined

1 entry

c1:rand1

2 entries

c1:rand2

1 entry

c1:rand3

Combined Column: 103 entries

c1:combined

.filter( ), .transform( ), ⋋ ⋋ ⨍

Secondary Indexes

High ThroughputLow Latency

Lookups

Not so easy!

HBase is designed for high throughput writes

Key Ideas

Read as little as possible

Stay stable, uniform, data local

Don’t go to disk

Even if you have to go

to disk, make it fast

Protobufs, Protobufs, everywhere

Stay stable, uniform, data local at all times

Good quality hardware

Properly designed row keys

Off peak daily major compaction

Give me all your Cache!

128 Gb machines with 50% block CacheHigh Cache hit ratio (90% +) by effective utilization

It’s time to disk(o)

15K SAS drivesLocal & Short circuit reads (20-30% improvement)

High throughput writes aresupported too!

Small Writes

• Append Only• Protobufs

Large Memstores

• 4 Gb• Avoids flushes,

memory churn, compaction

• Maintains read performance by avoiding multiple seeks

Tuned Compaction

• Avoid Minor compactions

• Off Peak Major compaction

Reliability & Availability

Organize the chaos or pay the cost..

» Blind writes can grow rows & table too big

» Newbie clients 'guess' a lot

» Simple queries such as row count can be hard on the fly

Be aware…

Web app Bid Serving

Ad serving Data augmentation

Batch data pipelines Ops Housekeeping

Real time data pipelines

Multitenant Blackbird

Multi tenancy makes it hard to find the defaulter

Use ACLs & client side metrics in all access paths

Draft guidelines for new clients, help them estimate the growth

Keep track of growth, row count, row size, column size etc.

Maintaining SLA Guarantees

It’s a delicate equilibrium that is hard to maintain

Shield it with aggressive alerting, dashboards & canary monitoring

1st region server dies after several hours of clogged RPC queue

Bad region moves to another region server & soon kills it too!

2jmj7l5rSw0yVb_vlWAYkK_Ybwk

stgLVlK_SsLMn4HoG82ymp-QlRtA

Clients can go rouge, it can get as bad as a DoS attack

Protection via dynamic blacklists & size limit filters

Surviving the failures

» In absence of proxy: ‘The client is part of the cluster’ [1]

» Client must report availability error to calling application thread in short time span

» Follow circuit breaker pattern for read calls (Anecdote)

» ‘pseudo’ puts (local file) for write calls

[1] Blog post from Lars Hofhansl http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html

Shoutouts!

Obligatory “we are hiring” slide!

http://rocketfuel.com/careers

ishan@rocketfuel.comshrijeet@rocketfuel.comapol@rocketfuel.com