beginning operations: 7 deadly sins for apache cassandra ops
TRANSCRIPT
![Page 1: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/1.jpg)
7 Deadly Sins for Cassandra OpsRachel Pedreschi
DSE Evangelist, Datastax
![Page 2: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/2.jpg)
2© 2015. All Rights Reserved.
![Page 3: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/3.jpg)
Lust
3© 2015. All Rights Reserved.
#1 USE THE SIMPLE SNITCH
![Page 4: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/4.jpg)
So many snitches…
4© 2015. All Rights Reserved.
• SimpleSnitch • RackInferringSnitch • PropertyFileSnitch • GossipingPropertyFileSnitch • Ec2Snitch • Ec2MultiRegionSnitch • GoogleCloudSnitch • CloudstackSnitch
![Page 5: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/5.jpg)
Switching snitches
5© 2015. All Rights Reserved.
If the topology of the network has changed: -Shut down all the nodes, then restart them. -Run a sequential repair and nodetool cleanup on each node.
DOWNTIME ALERT!!!!
![Page 6: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/6.jpg)
Greed
6© 2015. All Rights Reserved.
#3 NOT UNDERSTANDING
REPAIR
![Page 7: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/7.jpg)
Repair options
7© 2015. All Rights Reserved.
repair (default, check your version!) repair -pr (only repair the primary range) repair -inc (only new data that has not previously been repaired) sequential repair (creates snapshots) parallel repair uses replica not being repaired
![Page 8: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/8.jpg)
Envy
8© 2015. All Rights Reserved.
#3 CHOOSE THE WRONG
COMPACTION STRATEGY FOR
YOUR WORKLOAD
![Page 9: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/9.jpg)
Sized Tiered
9© 2015. All Rights Reserved.
SST1 SST1 SST2 SST1 SST2 SST3 SST1 SST2 SST3 SST4 SST5
FLUSHFLUSH
FLUSH FLUSH
COMPACT
SST5 SST6 SST5 SST6 SST7 SST5 SST6 SST7 SST8 SST5 SST10
FLUSH FLUSH FLUSH
SST5 SST6 SST7 SST8
FLUSH
SST9
COMPACT
Compacts a set number of SSTables into a single, larger SSTable
![Page 10: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/10.jpg)
Leveled
10© 2015. All Rights Reserved.
Level 0
Level 1
10
Level 2100
Level 3
1,000
Level 410,000
Level 5100,000
… Level 6, 7, etc.
![Page 11: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/11.jpg)
Date Tiered
11© 2015. All Rights Reserved. https://labs.spotify.com/2014/12/18/date-tiered-compaction/
![Page 12: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/12.jpg)
Gluttony
12© 2015. All Rights Reserved.
#4 CHOOSING THE WRONG
HARDWARE
![Page 13: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/13.jpg)
13© 2015. All Rights Reserved.
CPU
RAM
DISK
![Page 14: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/14.jpg)
14© 2015. All Rights Reserved.
• 2 socket, ECC memory • 16GiB minimum, prefer 32-64GiB, over 128GiB and Linux will need
serious tuning
• SSD where possible, Samsung 840 Pro is a good choice, any Intel is fine
• NO SAN/NAS, 20ms latency tops • if you MUST (and please, don’t) dedicate spindles to C* nodes, use
separate network
• Avoid disk configurations targeted at Hadoop, disks are too slow
![Page 15: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/15.jpg)
Sloth
15© 2015. All Rights Reserved.
#5 NOT TUNING YOUR OS
![Page 16: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/16.jpg)
/etc/rc.local
16© 2015. All Rights Reserved.
ra=$((2**14))# 16kss=$(blockdev --getss /dev/sda)blockdev --setra $(($ra / $ss)) /dev/sda
echo 128 > /sys/block/sda/queue/nr_requestsecho deadline > /sys/block/sda/queue/schedulerecho 16384 > /sys/block/md7/md/stripe_cache_size
![Page 17: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/17.jpg)
/etc/sysctl.conf
17© 2015. All Rights Reserved.
fs.file-max = 1048576vm.max_map_count = 1048576net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_rmem = 4096 65536 16777216net.ipv4.tcp_wmem = 4096 65536 16777216vm.swappiness = 1
![Page 18: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/18.jpg)
Wrath
18© 2015. All Rights Reserved.
#6 OVER OR UNDERSIZING
YOUR JVM
![Page 19: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/19.jpg)
G1 or CMS?
19© 2015. All Rights Reserved.
• Cassandra 8150 vs 7486 • Larger heap? Look into using G1 • Read the Docs • Test, test and did I mention, test?
![Page 20: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/20.jpg)
Pride
20© 2015. All Rights Reserved.
#7 NOT STRESS TESTING
![Page 21: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/21.jpg)
cassandra-stress .yaml (>= 2.1)
21© 2015. All Rights Reserved.
1. DDL – for defining your schema 2. Column Distributions – for defining the shape and size of each
column globally and within each partition 3. Insert Distributions – for defining how the data is written during the
stress test 4. DML – for defining how the data is queried during the stress test
![Page 22: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/22.jpg)
Recommended Sessions
22© 2015. All Rights Reserved.
DataStax Making Cassandra Fail (for effective testing) 3:30 Thursday Ballroom H
Pythian Manage your compactions before they manage you 4:20 Ballroom H
The Last Pickle Steady State Data Size with Compaction, Tombstones, and TTL 4:20 Great America #2 Wednesday
Crowdstrike, Inc. Real World DTCS For Operators
![Page 23: Beginning Operations: 7 Deadly Sins for Apache Cassandra Ops](https://reader031.vdocument.in/reader031/viewer/2022021922/58eca1c61a28ab7d258b4697/html5/thumbnails/23.jpg)
Thank you@RachelPedreschi