reducing replication bandwidth for distributed …...reducing replication bandwidth for distributed...
TRANSCRIPT
![Page 1: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/1.jpg)
Reducing Replication Bandwidth for Distributed Document Databases
Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2
Jin Li2, Greg Ganger1 Carnegie Mellon University1, Microsoft Research2
![Page 2: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/2.jpg)
Document-oriented Databases { "_id" : "55ca4cf7bad4f75b8eb5c25c", "pageId" : "46780", "revId" : "41173", "timestamp" : "2002-03-30T20:06:22", "sha1" : "6i81h1zt22u1w4sfxoofyzmxd” "text" : “The Peer and the Peri is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts… just as predicting,…The fairy Queen, however, appears to … all live happily ever after. " }
{ "_id" : "55ca4cf7bad4f75b8eb5c25d”, "pageId" : "46780", "revId" : "128520", "timestamp" : "2002-03-30T20:11:12", "sha1" : "q08x58kbjmyljj4bow3e903uz” "text" : "The Peer and the Peri is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts… just as predicted, …The fairy Queen, on the other hand, is ''not'' happy, and appears to … all live happily ever after. " }
Update
Update: Reading a recent doc and writing back a similar one 2
![Page 3: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/3.jpg)
Replication Bandwidth { "_id" : "55ca4cf7bad4f75b8eb5c25c", "pageId" : "46780", "revId" : "41173", "timestamp" : "2002-‐03-‐30T20:06:22Z", "sha1" : "6i81h1zt22u1w4sfxoofyzmxd” "text" : "The Peer and the Peri” is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts… just as predicting,…The fairy Queen, however, appears to … all live happily ever after. " }
{ "_id" : "55ca4cf7bad4f75b8eb5c25d”, "pageId" : "46780", "revId" : "128520", "timestamp" : "2002-03-30T20:11:12Z", "sha1" : "q08x58kbjmyljj4bow3e903uz” "text" : "The Peer and the Peri” is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts… just as predicted, …The fairy Queen, on the other hand, is ''not'' happy, and appears to … all live happily ever after. " }
Operation logs
Operation logs
Secondary Secondary
WAN
Primary Database
Goal: Reduce bandwidth for WAN geo-replication
3
![Page 4: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/4.jpg)
Why Deduplication?
• Why not just compress? – Oplog batches are small and not enough overlap
• Why not just use diff? – Need application guidance to identify source
• Dedup finds and removes redundancies – In the entire data corpus
4
![Page 5: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/5.jpg)
Traditional Dedup: Ideal Modified Region Duplicate Region Chunk Boundary
Deduped Data
Incoming Data {BYTE STREAM }
1 2 4 5
1 2 4 5 3
Send dedup’ed data to replicas
5
![Page 6: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/6.jpg)
Traditional Dedup: Reality Modified Region Duplicate Region Chunk Boundary
Incoming Data
Deduped Data 4
1 2 4 5 3
Send almost the entire document.
6
![Page 7: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/7.jpg)
Similarity Dedup Modified Region Duplicate Region Chunk Boundary
Incoming Data
Dedup’ed Data
Delta!
Only send delta encoding.
7
![Page 8: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/8.jpg)
Compress vs. Dedup
20GB sampled Wikipedia dataset MongoDB v2.7 // 4MB Oplog batches 8
![Page 9: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/9.jpg)
Primary Node
Client
Secondary Node
sDedup: Similarity Dedup
Source documents
Insertion & Updates
Database
Oplog
sDedup Encoder
Unsynchronized oplog entries
Dedup’ed oplog entries Oplog
Re-constructed oplog entries
Replay
sDedup Decoder
Oplog syncer
Database
Source documents
Source Document
Cache
9
![Page 10: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/10.jpg)
sDedup Encoding Steps
• Identify Similar Documents • Select the Best Match • Delta Compression
10
![Page 11: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/11.jpg)
Identify Similar Documents Target Document
Consistent Sampling
Similarity Sketch
Rabin Chunking
32 17 25 41 12
41 32
Feature Index Table
Candidate Documents
41
32
32 25 38 41 12
32 17 38 41 12
39 32 22 15 Doc #1
Doc #2
Doc #3
32 25 38 41 12
32 17 38 41 12
Doc #2
Doc #3
1 Doc #1
2 Doc #2
2 Doc #3
Similarity Score
11
![Page 12: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/12.jpg)
Select the Best Match
Source Document Cache
Rank Candidates Score
1 2
1 2
2 1 Doc #1
Doc #2
Doc #3
Initial Ranking Final Ranking
Rank Candidates Cached? Score
1 Yes 4
1 Yes 3
2 No 2
Doc #1
Doc #3
Doc #2
Is doc cached? If yes, reward +2
12
![Page 13: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/13.jpg)
Evaluation
• MongoDB setup (v2.7) – 1 primary, 1 secondary node, 1 client – Node Config: 4 cores, 8GB RAM, 100GB HDD storage
• Datasets: – Wikipedia dump (20GB out of ~12TB) – Additional datasets evaluated in the paper
13
![Page 14: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/14.jpg)
Compression
9.9
26.3
38.4 38.9
2.3 4.6 9.1
15.2
0
10
20
30
40
50
4KB 1KB 256B 64B
Com
pres
sion
Rat
io
Chunk Size
sDedup trad-dedup
20GB sampled Wikipedia dataset 14
![Page 15: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/15.jpg)
Memory
34.1 47.9 57.3 61.0 80.2 133.0
272.5
780.5
0
200
400
600
800
4KB 1KB 256B 64B
Mem
ory
(MB)
Chunk Size
sDedup trad-dedup
20GB sampled Wikipedia dataset 15
![Page 16: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/16.jpg)
Other Results (See Paper)
• Negligible client performance overhead • Failure recovery is quick and easy • Sharding does not hurt compression rate • More datasets
– Microsoft Exchange, Stack Exchange
16
![Page 17: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/17.jpg)
Conclusion & Future Work
• sDedup: Similarity-based deduplication for replicated document databases. – Much greater data reduction than traditional dedup – Up to 38x compression ratio for Wikipedia – Resource-efficient design with negligible overhead
• Future work – More diverse datasets – Dedup for local database storage – Different similarity search schemes (e.g., super-fingerprints)
17
![Page 18: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/18.jpg)
Backup Slides
18
![Page 19: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/19.jpg)
Compression: StackExchange
1.0 1.2 1.3 1.8
1.0 1.0 1.1 1.2
0
1
2
3
4
5
4KB 1KB 256B 64B
Com
pres
sion
Rat
io
Chunk Size
sDedup trad-dedup
10GB sampled StackExchange dataset 19
![Page 20: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/20.jpg)
Memory: StackExchange
83.9 115.4 228.4 414.3 302.0 439.8 899.2
3,082.5
0 500
1000 1500 2000 2500 3000 3500
4KB 1KB 256B 64B
Mem
ory
(MB)
Chunk Size
sDedup trad-dedup
10GB sampled StackExchange dataset 20
![Page 21: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/21.jpg)
Throughput Overhead
21
![Page 22: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/22.jpg)
Failure Recovery
20GB sampled Wikipedia dataset.
Failure Point
22
![Page 23: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/23.jpg)
Dedup + Sharding
38.4 38.2 38.1 37.9
0
10
20
30
40
50
1 3 5 9
Com
pres
sion
Rat
io
Number of Shards 20GB sampled Wikipedia dataset 23
![Page 24: Reducing Replication Bandwidth for Distributed …...Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu1, Andy Pavlo1, Sudipta Sengupta2 Jin Li2, Greg Ganger1](https://reader031.vdocument.in/reader031/viewer/2022040607/5ebf25b629a6533782171a71/html5/thumbnails/24.jpg)
Delta Compression
• Byte-level diff between source and target docs: – Based on the xDelta algorithm – Improved speed with minimal loss of compression
• Encoding: – Descriptors about duplicate/unique regions + unique bytes
• Decoding: – Use source doc + encoded output – Concatenate byte regions in order
24