real-time hbase: lessons from the cloud
DESCRIPTION
Speaker: Bryan Beaudreault (HubSpot) Running HBase in real time in the cloud provides an interesting and ever-changing set of challenges -- instance types are not ideal, neighbors can degrade your performance, and instances can randomly die in unanticipated ways. This talk will cover what HubSpot has learned about running in production on Amazon EC2, how it handle DR and redundancy, and the tooling the team has found to be the most helpful.TRANSCRIPT
Real-time HBase: Lessons from the Cloud
Bryan Beaudreault, @HubSpotDev
You’re doing it WRONG!
Instance types
Network,Neighbors,
PERSPECTIVE:
Consumer/Prospect
PERSPECTIVE:
Consumer/Prospect
PERSPECTIVE:
Consumer/Prospect
PERSPECTIVE:
Consumer/Prospect
Improve reads, limit impact
PERSPECTIVE: Consumer/
Prospect
PERSPECTIVE: Consumer/
Prospect
PERSPECTIVE: Consumer/
Prospect
Over-provision, fail fast
\PERSPECTIVE:
Consumer/Prospect
PERSPECTIVE: Consumer/
Prospect
PERSPECTIVE: Consumer/
Prospect
PERSPECTIVE: Consumer/
Prospect
PERSPECTIVE: Consumer/
Prospect
PERSPECTIVE: Consumer/
Prospect
PERSPECTIVE: Consumer/
Prospect
c1.xlarge
i2.4xlarge
CPU heavy workloadsReduce memory footprintAdd more servers
Excellent, but expensiveUse data encoding to reduce diskUse Java7 and G1 to reduce GCs
m1.xlarge Memory heavy workloadsBeware compactionsMore regions
PERSPECTIVE: Consumer/
Prospect
cpu fsWriteLatency
fsReadLatencyload
callQueueLen
compactionQueueSizeiowait
steal
heap
locality
requests
memstoreSizeMB
blockCacheHitCachingRatio
flushQueueSize
PERSPECTIVE: Consumer/
Prospect
Master HBase with us.
dev.hubspot.com
Bryan Beaudreault @HubSpotDev