HBase off heaping read – The Alibaba StoryRamkrishna S Vasudevan(Intel)Yu Li (Alibaba)

2

HBase Cache ▪HBase Cache for better read performance

• Keep data local to the process

▪L1 on heap LRU Cache• Larger Cache Sizes => Larger Java Heap => GC issues

▪L2 Bucket Cache• LRU• Backed by off heap memory / File• Can be larger than L1 allocation• Not constrained by Java Heap Size

▪ L2 offheap cache can give large cache size

▪ 4 MB physical memory buffers.

▪ Different sized buckets 5 KB, 9 KB,… 513 KB. Each bucket having at least 4 slots

▪ HFile blocks placed in appropriate sized bucket

▪ One Block may span across 2 ByteBuffers.

▪ Read path assumption of data being in a byte array.

▪ Cells having assumption of data parts being in byte array. (ie. Rowkey, family, value etc)

Bucket Cache overview4 MB

5 KB buckets

9 KB buckets

513 KB buckets

Bucket Cache - Reads

4

▪ Read block by block

▪ On heap block copy

▪ Less throughput vs L1

▪ GC Impacts

▪ Predictable latency problems

Region1

Region2

Read request

Read request

RegionServer

Scanner layers

Scanner layers

Copy

Copy

Get block

On heap Block

Off heap Bucket Cache

Java ByteBuffer

Bucket slot

Get block

On heap Block

5

Building the offheap read path

▪ Cellify the read path▪ Avoid the byte[] assumption and usage of KeyValue throughout the read path

▪ Add ByteBufferedCell extension in the server side

▪ Create NIO ByteBuffer wrapper called ByteBuff• SingleByteBuff, MultiByteBuff

§ Ref count based block cache and block eviction mechanism• Increment ref count when reader hits a block in L2 cache

• Decrement once response is created for RPC using a RPC callback

• Evict if/when ref count = 0

SingleByteBuff

MultiByteBuff

Complete Picture

Region1

Region2

Read request

Read request

HRegionServer

Off heap Bucket Cache

Refcount++

Read response

Read response

Scanner layers

Scanner layersRefcount++

callback

callback

Refcount--

Refcount--MultiByteBuff

SingleByteBuff

End to End off heap - from bucket cache till RPC

Bucket Cache – Off heap Reads

▪ Delivers data directly from off heap cache

▪ Lesser Heap needed

▪ Better throughput

▪ Predictable latency

▪ Available in HBase 2.0

▪ JIRA # HBASE-11425

8

HBase at Alibaba

▪ Core storage system in Search Infrastructure

▪ Query HBase to provide real time search index updates

▪ Data Store for online machine learning system

▪ Biggest HBase cluster is 1500+ nodes, 200,000+ regions, tens ofmillions QPS

▪ Importance of consistent Latency and Throughput

▪ Use HBase Cache for better read performance– L1 LRU Cache

– & L2 Off Heap Cache

Data Platform for Search Indexingl Data Storage for Batch and Streaming processing

Data Source

Hadoop cluster

HBase

Batch & Streaming

Event

Offline & Real Time Processing Exporting

Ali ODPS MySQLSearch Engines

HBase HBase

HDFS HDFS HDFS

Latency matters

Database and Queue for Machine Learning

UDF UDF UDFHQueue

Online log

Parsing Log Training

User Models

Training Item ModelsItem ID

User ID

HQueue

Aggregate

Updates

Machine Learning Models

Online System

Δw Export Model

Updates

Model

Flink+MR Processing over Yarn

Throughput matters

11

Old Performance Data

▪ From A/B test cluster with 400+ nodes and on simulated data

▪ 12 GB off heap L2 cache

▪ Inconsistent throughput. Dips in latency

▪ HBase read from L2 does data copy to on heap– More garbage and GC workload

12

Performance Data After Off heaping read path

▪ Backported patches to hbase-1.2 based codebase

▪ Average throughput improved from 18 to 25 million QPS – 30% improvement

▪ Linear throughput. Predictable latency.

▪ In production since Oct 2016

▪ Used in 2016 Singles’ Day, with peaks at 100K QPS per single RegionServer

13

Related features in HBase 2.0

▪ Off heaping write path (HBASE-11579)

▪ Compacting Memstore (HBASE-14920)§ References

● https://blogs.apache.org/hbase/● https://blogs.apache.org/hbase/entry/offheap-read-path-in-production

Questions?