cassandra london march 2016 - lightening talk - introduction to incremental repair
TRANSCRIPT
![Page 1: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/1.jpg)
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
CASSANDRA LONDON FEBRUARY 2016
LIGHTENING TALK: INTRODUCTION TO
INCREMENTAL REPAIR
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
Aaron Morton@aaronmorton
Co-Founder & Team Member
![Page 2: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/2.jpg)
About The Last Pickle.
Work with clients to deliver and improve Apache Cassandra based
solutions.Based in New Zealand, USA,
Australia, France.
![Page 3: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/3.jpg)
Repair Incremental Repair
![Page 4: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/4.jpg)
Repair
:(
![Page 5: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/5.jpg)
Why Repair?
It is the way to ensure consistent data on disk.
![Page 6: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/6.jpg)
Why Repair?
Hinted Handoff and (non-blocking) Read Repair are
optimisations.
![Page 7: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/7.jpg)
Why Repair?
Consistency ensures Tombstones are distributed,
preventing deleted data returning.
![Page 8: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/8.jpg)
A Repair Of Two Halves
Step 1:Calculate a hash tree to
describe the data.(aka Merkle Tree)
![Page 9: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/9.jpg)
A Repair Of Two Halves
Step 2:Stream differences between
nodes.
![Page 10: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/10.jpg)
gc_grace_seconds
Repair will ignore Tombstones older than gc_grace_seconds.
So run repair every gc_grace_seconds.*
![Page 11: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/11.jpg)
RepairIncremental Repair
![Page 12: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/12.jpg)
Why Incremental Repairs?
SSTables are immutable, so repair was processing data
that had not changed.
![Page 13: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/13.jpg)
Why Incremental Repairs?
Repairing less data, more frequently, reduces the impact
on the cluster.
![Page 14: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/14.jpg)
repairedAt
Track repaired timestamp per SSTable.
Ignore SSTables that have been repaired.
![Page 15: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/15.jpg)
repairedAt
Set after when all data in the SSTable was repaired.
![Page 16: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/16.jpg)
repairedAt$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 0
$ bin/nodetool repair --incremental --parallel
$ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 1454500502306
![Page 17: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/17.jpg)
Anti Compaction
Splits partially repaired SSTables into Repaired and
Un-repaired SSTables.
![Page 18: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/18.jpg)
Anti Compaction$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-9-Data.db keyspace1-standard1-ka-10-Data.db
$ bin/nodetool repair --incremental --parallel -st 3074457345618258602 -et -9223372036854775808
$ ls -lah keyspace1/standard1 keyspace1-standard1-ka-11-Data.db keyspace1-standard1-ka-12-Data.db keyspace1-standard1-ka-13-Data.db keyspace1-standard1-ka-14-Data.db
![Page 19: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/19.jpg)
Anti Compaction$ tools/bin/sstablemetadata keyspace1-standard1-ka-11-Data.db SSTable: ... Repaired at: 1454504607623
$ tools/bin/sstablemetadata keyspace1-standard1-ka-12-Data.db SSTable: ... Repaired at: 0
$ tools/bin/sstablemetadata keyspace1-standard1-ka-13-Data.db SSTable: ... Repaired at: 1454504607623
$ tools/bin/sstablemetadata keyspace1-standard1-ka-14-Data.db SSTable: ... Repaired at: 0
![Page 20: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/20.jpg)
Compaction
Two sets of SSTables are maintained: Repaired and Un-
repaired.
![Page 21: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/21.jpg)
Compaction
Compaction Strategy runs in each SSTable set.
![Page 22: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/22.jpg)
Migration Option 1
1. Disable compaction.2. Run a classic full repair3. Stop the node4. stablerepairedset to mark all the SSTables from before step 1
![Page 23: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/23.jpg)
Migration Option 2
bin/nodetool repair --incremental --parallel
![Page 24: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/24.jpg)
More Information
http://www.datastax.com/dev/blog/more-efficient-repairs
![Page 25: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/25.jpg)
Thanks.
![Page 26: Cassandra London March 2016 - Lightening talk - introduction to incremental repair](https://reader031.vdocument.in/reader031/viewer/2022030402/58a20eb51a28ab40098b6539/html5/thumbnails/26.jpg)
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
Aaron Morton@aaronmorton
Co-Founder & Team Memberwww.thelastpickle.com
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License