high throughput analytics with cassandra & azure
DESCRIPTION
TRANSCRIPT
High Throughput Analytics with Cassandra & Azure
Charles LamannaPrincipal Dev Lead
@clamanna
MetricsHubkeep cloud services up and running for the lowest possible cost
Live Status
Cost Awareness
Alerts and Notifications
Actions and Scaling
$
2000+ customers in 6 months
10/18/2012 12/7/2012 1/26/2013 3/17/2013 5/6/2013 6/25/20130
500
1000
1500
2000
2500
storing data200M data points per hour80,000 data points per second (peak)
Planning for huge data ingestion ratesRequires high scale, real-time data
1,000 data points per minute per VM12 data points per endpoint per minute
Aggregate, analyze and take actions based on this data stream (in near real-time)
Must be cheap, scalable and reliable
Evaluated several technologiesAggregation in memory; good performance, bad COGs
Rolling tables for aggs; good tooling/support, hard to scale
Aggregation on write; easy to scale and good COGs
Cassandra UpsideScales fluidly Grows horizontally – double the nodes, double capacityAdd / remove capacity / nodes with no downtime
Highly availableNo single point of failureReplication factor (i.e. hot copies) is just a config switch
… and by the wayLittle-to-no operations cost
New nodes take minutes to setupNodes just keep running for months on end
“Aggregate on write” – no jobs required!Distributed counters make it easy to do aggregates on write
…and a nice kicker: has *great* perf / COGS in Azure
architecture68 virtual machines (PAAS and IAAS)
Table StorageJobs Worker Role (24 instances)
SQL Database
Portal Web Role
(3 instances)
Cassandra VM Cluster
(32 XL instances)
Web API Web Role
(8 instances)
End User Web Browsers
Monitored Customer Resources
(e.g. websites; SQL databases)
Monitored Virtual Machines
Endpoints Replicated datain multiple
datacenters
ClientsPaaS
IaaS
Services
Avoiding state
• Application logic / code all lives on stateless machines
• Keeps it simple: decreases human operations cost
• Use Azure PAAS offerings (Web and Worker roles)
Table Storage
Jobs Worker Role (24 instances)
SQL Database
Blob storage
Portal Web Role
(3 instances)
Cassandra VM Cluster
(32 XL instances)
Web API Web Role
(8 instances)
Endpoints Replicated datain multiple
datacenters
PaaS
Azure Cloud Services (PAAS)
• Scale horizontally (grew from 1 to 30+ instances)
• Managed by the platform (patched; coordinated recycling; failover; etc.)
• 1 click deployment from Visual Studio (with automatic load balancer swaps)
Web Role Worker Role
Table Storage
Jobs Worker Role (24 instances)
SQL Database
Blob storage
Web API Web Role
(8 instances)
Endpoints Replicated datain multiple
datacenters
Maintains all state for metrics / time series data
32 XL Linux Virtual Machines Portal Web
Role (3 instances)
Cassandra VM Cluster
(32 XL instances)
Cassandra Cluster
IaaS
32 nodes, 8 “pods” of 4 nodes
……..
……….
Exposed via a single endpoint
Exposed via a single endpoint
Exposing the pods• Each pod of 4 nodes
has a single load balanced endpoint
• Clients (on our stateless roles) treats the endpoints as a pool
• Blacklists and skips an endpoint if it starts producing a lot of errors
Where does the data go?
• Data files are on 16 mounted network backed disks (*not* ephemeral disks)
• Data disks are geo-replicated (3 copies local; 1 remote) for “free” DR
• Azure data disks offer great throughput (VMs end up CPU bound)
Our Column Families (CQL 3)
CREATE TABLE oneminute (
rk text, ck text, cnt counter, sum counter, PRIMARY KEY (rk, ck)
);
Updating values…Realtime “average” values at any granularity, for any time window
updateoneminute/tenminute/oneday
setsum = sum + {sample_value},cnt = cnt + 1
where rk = '{customer+metric}' and ck = '{tags_and_timestamp}'
Reading values…
*ONE* round trip to fetch a metric over time (e.g. CPU over past week)
select * from oneminutewhere rk = ‘{customer_name}' and ck < '{metric_path_start}' and ck >= '{metric_path_end}‘order by ck desc;
Some hard lessons…
• Static private IPs are a must; otherwise, reboots / outages can confuse the cluster when nodes come back up
• Monitor performance carefully; once you tip over, it is hard to rebalance the cluster and add new nodes
• Fit the cluster to the platform: in Azure, match the Upgrade Domains / Fault Domains to preserve uptime during service maintenance / hardware failure
Single node tests..• 4 disks, RAID 0, no read cache
Workload (%write)
Ops / sec Latency median
Latency 95th
Latency 99th
%100 20018 1.5 3.7 7.9%75 8361 85.9 376.6 584.8%25 5412 459.9 759.1 940.1
• 4 disks, RAID 0, read cacheWorkload (%write)
Ops / sec Latency median
Latency95th
Latency99th
%100 19208 1.5 3.8 7.9 18543 1.5 3.6 7.9 18563 1.4 3.6 8.2
%75 7112 195.9 595.8 1099.6 7581 168.9 589.5 985.2 5149 256.5 774.0 1402.9
%25 15358 23.0 110.2 309.1 3742 279.2 563.0 789.7 15376 22.1 98.8 293.3
jbod RAID00
1000
2000
3000
4000
5000
6000
7000
JBOD vs RAID 0 for read-heavy workload
Workload (%write)
Ops / sec
Latency Median
Latency 95th
Latency99th
%100 13638 1.9 4.9 24.0%75 3239 11.2 687.0 1099.3%25 1825 243.6 687.0 808.7
Multi-node load tests..
• 4 Nodes; RF = 3 (Quorom)
• 8 Disks, RAID 0
