anti-entropy using crdts on ha datastores › system › files › presentation-slides ›...
TRANSCRIPT
![Page 1: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/1.jpg)
Anti-Entropy using CRDTs on HA Datastores
Sailesh MukilSenior Software Engineer, Netflix
![Page 2: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/2.jpg)
Timeline
NETFLIX
Cassandra adoption
2011 2013 2016
Multi-region Dynomite
![Page 3: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/3.jpg)
Dynomite
NETFLIX
Makes non-distributed datastores, distributed
![Page 4: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/4.jpg)
NETFLIX
Datastore
33% 33%
33%
Dynomite Overview
![Page 5: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/5.jpg)
NETFLIX
Replica 1 Replica 2 Replica 3
Dynomite Overview
![Page 6: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/6.jpg)
NETFLIX
Replica 1 Replica 2 Replica 3
Client
![Page 7: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/7.jpg)
NETFLIX
Replica 1 Replica 2 Replica 3
Client
![Page 8: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/8.jpg)
NETFLIX
Replica 1 Replica 2 Replica 3
Client
![Page 9: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/9.jpg)
NETFLIX
Dynomite overview
● Global replication● High availability● Shared nothing● Auto-sharding● Linear scale
● Pluggable datastores (Redis primarily)
● Multiple quorum levels
● Supports datastore API
![Page 10: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/10.jpg)
NETFLIX
Dynomite footprint @ Netflix
● ~1000 customer facing nodes● ~1M OPS/s● Largest cluster holds ~6 TB
![Page 11: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/11.jpg)
The problem
NETFLIX
Entropy in the system
![Page 12: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/12.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system SET K 123
![Page 13: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/13.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system SET K 123
K: 123
K: 123 K: 123
![Page 14: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/14.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system
K: 123
K: 123 K: 123
OK
![Page 15: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/15.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system
K: 123
K: 123 K: 123
SET K 456
![Page 16: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/16.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system
K: 123
K: 123 K: 123
SET K 456
K: 456
![Page 17: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/17.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system
K: 123
K: 123 K: 123K: 456
ERR
![Page 18: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/18.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system
K: 123
K: 123 K: 123K: 456
SET K 789
![Page 19: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/19.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system
K: 123
K: 123 K: 123K: 456
SET K 789
K: 789
![Page 20: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/20.jpg)
NETFLIX
R-2 R-3
R-1
Entropy in the system
K: 123
K: 123 K: 123K: 456K: 789
ERR
![Page 21: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/21.jpg)
NETFLIX
R-2 R-3
R-1
K: 123
K: 123 K: 123K: 456
GET K
K: 789
GET K
![Page 22: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/22.jpg)
NETFLIX
R-2 R-3
R-1
K: 123
K: 123 K: 123K: 456K: 789
789 456
![Page 23: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/23.jpg)
NETFLIX
R-2 R-3
R-1
K: 123
K: 123 K: 123K: 456
GET K (w/quorum)
K: 789
![Page 24: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/24.jpg)
NETFLIX
R-2 R-3
R-1
K: 123
K: 123 K: 123K: 456
GET K (w/quorum)
K: 789
![Page 25: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/25.jpg)
NETFLIX
R-2 R-3
R-1
K: 123
K: 123 K: 123K: 456
GET K (w/quorum)
K: 789
123
456
![Page 26: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/26.jpg)
NETFLIX
R-2 R-3
R-1
K: 123
K: 123 K: 123K: 456K: 789
ERR: QUORUM FAILED
![Page 27: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/27.jpg)
NETFLIX
R-2 R-3
R-1
K: 123
K: 123 K: 123K: 456K: 789
123
456
ERR: QUORUM FAILED
![Page 28: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/28.jpg)
NETFLIX
Replicas will go out of sync
![Page 29: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/29.jpg)
Timeline
NETFLIX
Cassandra adoption
2011 2013 2016
Multi-region DynomiteDynomite w/ CRDTs
2019
![Page 30: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/30.jpg)
NETFLIX
Last Writer Wins Vector Clocks
Achieving anti-entropy(traditionally)
● Uses Physical timestamps● Clock skew
● Shows causal relationships● But not for concurrent writes
![Page 31: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/31.jpg)
The solution
NETFLIX
Conflict free replicated data types
![Page 32: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/32.jpg)
Conflict free replicated data types
NETFLIX
SECTION DIVIDER
A CRDT is a data structure which can be replicated across the network, where the replicas can be updated independently and concurrently without coordination between the replicas, and where it is always mathematically possible to resolve inconsistencies which might result.
![Page 33: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/33.jpg)
NETFLIX
Associative Commutative Idempotent
Grouping of operations does not matter
(X + Y) + Z = X + (Y + Z)
Order of operations do not matter
X + Y = Y + X
Duplication of operations does not
matter
X + X = X
![Page 34: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/34.jpg)
NETFLIX
Update Merge
Types of operations on CRDTs
● Updates local state ● Converges replica states
![Page 35: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/35.jpg)
NETFLIX
When we write, we update
When we repair, we merge
Read repair = merge on read path
Introduction to CRDTs
![Page 36: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/36.jpg)
NETFLIX
CRDTs provide strong eventual consistency
Introduction to CRDTs
![Page 37: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/37.jpg)
NETFLIX
R-2 R-3
R-1
Naive distributed counter
CTR: 1
CTR: 1 CTR: 1
INCR CTR
![Page 38: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/38.jpg)
NETFLIX
R-2 R-3
R-1
Naive distributed counter
CTR: 1
CTR: 1 CTR: 1
DECR CTR INCR CTR
CTR: 0 CTR: 2
![Page 39: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/39.jpg)
NETFLIX
R-2 R-3
R-1
Naive distributed counter
CTR: 1
CTR: 1 CTR: 1CTR: 0 CTR: 2
Repair based on timestamp?
Latest value is 2, which is incorrect
![Page 40: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/40.jpg)
CRDT: PNCounters
NETFLIX
Each replica maintains 2 “local” counters● Positive counter: Tracks increments● Negative counter: Tracks decrements
Final counter value:(Sum of all PCounters - Sum of all NCounters)
![Page 41: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/41.jpg)
NETFLIX
R-2 R-3
R-1
CRDT: PNCounter INCR CTR
0 0 0
0 0 0CTR:
0 0 0
0 0 0CTR:
0 0 0
0 0 0CTR:
1
1 1
0 00 0 01 0 0
0 0 01
![Page 42: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/42.jpg)
NETFLIX
R-2 R-3
R-1
CRDT: PNCounter
0 0 0
0 0 0CTR:
0 0 0
00 0CTR:
0 0 0
0 0 0CTR:
1
1 1
DECR CTR INCR CTR
1
1
![Page 43: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/43.jpg)
NETFLIX
R-2 R-3
R-1
CRDT: PNCounter
0 0 0
0 0 0CTR:
0 0 0
00 0CTR:
0 0 1
0 0 0CTR:
1
1 1
1
1
CTR = 0
CTR = 1
CTR = 2
![Page 44: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/44.jpg)
NETFLIX
R-2 R-3
R-1
CRDT: PNCounter
0 0 0
0 0 0CTR:
0 0 0
00 0CTR:
0 0 1
0 0 0CTR:
1
1 1
1
1
GET CTR
0 00 01
1
0 10 0 01
![Page 45: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/45.jpg)
NETFLIX
R-2 R-3
R-1
CRDT: PNCounter
0 0 0
00 0CTR:
0 0 0
00 0CTR:
0 0 1
00 0CTR:
1
1 1
1
1
GET CTR1
1
repair(merge)
repair(merge)
repair(merge)
1
1
![Page 46: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/46.jpg)
NETFLIX
R-2 R-3
R-1
CRDT: PNCounter
0 0 0
00 0CTR:
0 0 0
00 0CTR:
0 0 1
00 0CTR:
1
1 1
1
1
1
1
1
1
CTR = 1
CTR = 1
CTR = 1
![Page 47: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/47.jpg)
CRDT: LWW-Element Set
NETFLIX
Used to maintain key metadata● Add set: Latest update timestamps for keys● Remove set: Timestamps at which keys were removed
Registers can take arbitrary values● Hence we still require LWW to resolve conflicts
Used for registers, hashmaps and sorted sets
![Page 48: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/48.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set SET K1 123 (t1)
add
rem
add
rem
add
rem
K1t1
K1t1
K1t1
K1: 123
K1: 123 K1: 123
![Page 49: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/49.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1t1
K1t1
K1t1
K1: 123
K1: 123 K1: 123
SET K1 456 (t2)
t2
K1: 456
![Page 50: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/50.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1t1
K1t1
K1t1
K1: 123
K1: 123
t2
SET K2 999 (t3)
K2t3
K2: 999
K2t3
K2: 999
K1: 456
![Page 51: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/51.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1t1
K1t1
K1t1
K1: 123
K1: 123
t2K2t3
K2: 999
K2t3
K2: 999
K1: 456
GET K1
K1 = 456 (t2)K1 = 123 (t1)
t2 > t1=> 456 latest value
t2
K1: 456
repair
![Page 52: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/52.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1
K1t1
K1t1
K1: 123
t2K2t3
K2: 999
K2t3
K2: 999
K1: 456
t2
K1: 456
“456”
repair
t2
K1: 456
![Page 53: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/53.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1
K1 K1t1t2
K2t3
K2: 999
K2t3
K2: 999
K1: 456
t2
K1: 456
t2
K1: 456
GET K2
(nil)K2 = 999 (t3)
![Page 54: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/54.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1
K1 K1t1t2
K2t3
K2: 999
K2t3
K2: 999
K1: 456
t2
K1: 456
t2
K1: 456
“999”
repair
K2t3
K2: 999
![Page 55: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/55.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1
K1 K1t1t2
K2t3
K2: 999
K2t3
K2: 999
K1: 456
t2
K1: 456
t2
K1: 456
K2t3
K2: 999
DEL K2 (t4)
K2t4
![Page 56: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/56.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1
K1 K1t1t2
K2t3
K2t3
K2: 999
K1: 456
t2
K1: 456
t2
K1: 456
K2t3
K2: 999
GET K2“999”
K2t4
![Page 57: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/57.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1
K1 K1t1t2
K2t3
K2t3
K2: 999
K1: 456
t2
K1: 456
t2
K1: 456
K2t3
K2: 999
GET K2
K2 del @t4
K2t4
K2 = 999 (t3)
K2t4
![Page 58: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/58.jpg)
NETFLIX
R-2 R-3
R-1
LWW-Element Set add
rem
add
rem
add
rem
K1
K1 K1t1t2
K2t3
K2t3
K2: 999
K1: 456
t2
K1: 456
t2
K1: 456
K2t3
(nil)
K2t4
DEL K2 (t4)
K2t4
K2t4
repair
![Page 59: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/59.jpg)
Implementation challenges (LWW-element set)
NETFLIX
Redis doesn’t maintain timestampsDynomite can track the timestamp of the client request
![Page 60: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/60.jpg)
Implementation challenges (LWW-element set)
NETFLIX
We’d like Dynomite to remain statelessStore the metadata inside Redis
![Page 61: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/61.jpg)
Implementation challenges (LWW-element set)
NETFLIX
Operations must modify data and metadata atomicallyRewrite operations into Redis Lua scripts (guarantees atomicity)
![Page 62: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/62.jpg)
Implementation challenges (LWW-element set)
NETFLIX
Does the remove set grow forever?Delete metadata ASAP from remove set if ALL replicas agreeBackground thread cleans restMaintain remove set as sorted set
![Page 63: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/63.jpg)
Implementation challenges (LWW-element set)
NETFLIX
What does an example Lua script look like?Check if update is oldDiscard if it isUpdate data + metadata otherwise
![Page 64: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/64.jpg)
NETFLIX
Repairs occur on read path in DynomiteRepairs for point reads only
![Page 65: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/65.jpg)
Background repairs
NETFLIX
(Note: Ongoing work)
![Page 66: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/66.jpg)
NETFLIX
Repairing on range reads is expensiveEg: Give me all members of a set
Return everything in this hashmapReturn me a range from this sorted set
Background repairs
![Page 67: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/67.jpg)
NETFLIX
How do we target keys that need repairing?Full key walk? (like Cassandra)
Background repairs
![Page 68: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/68.jpg)
NETFLIX
How do we target keys that need repairing?Maintain list of recently written to keys
Background repairs
Run merge operation on them (async)But, merge operation on large structures are expensive
![Page 69: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/69.jpg)
NETFLIX
Delta-state CRDTs
Maintain list of recent mutations done to keys
Background repairs
Ship only delta-state instead of entire data structure for mergeConfirm which replicas have received it
![Page 70: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/70.jpg)
NETFLIX
0
00CTR:
0 0
00CTR:
1 1
1
1
1
Background repairs What is a delta-state?
INCR CTR
2
0
0 1
2
Full state
R1 R2
![Page 71: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/71.jpg)
NETFLIX
0
00CTR:
0
00CTR:
1
1
1
1
Background repairs What is a delta-state?
INCR CTR
2R1 = 2
Delta state
2
R1 R2
![Page 72: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/72.jpg)
NETFLIX
Background repairs What is a delta-state?
R1 R3
R2
R2 R3Mutations
𝜹-1𝜹-2
𝜹-3
𝜹-4
![Page 73: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/73.jpg)
NETFLIX
Background repairs What is a delta-state?
R1 R3
R2
R2 R3Mutations
𝜹-1𝜹-2
𝜹-3
𝜹-4
ACK
ACK
![Page 74: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/74.jpg)
NETFLIX
Background repairs What is a delta-state?
R1 R3
R2
R2 R3Mutations
𝜹-1𝜹-2
𝜹-3
𝜹-4
ack ack
ackack
![Page 75: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/75.jpg)
NETFLIX
Background repairs What is a delta-state?
R1 R3
R2
R2 R3Mutations
𝜹-1𝜹-2
𝜹-3
𝜹-4
ack ack
ackack
ACK
![Page 76: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/76.jpg)
NETFLIX
Background repairs What is a delta-state?
R1 R3
R2
R2 R3Mutations
𝜹-1𝜹-2
𝜹-3
𝜹-4
ack ack
ackack
![Page 77: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/77.jpg)
NETFLIX
Challenge with Delta-state CRDTsDurability
Background repairs
Practical overhead of maintaining list
![Page 78: Anti-Entropy using CRDTs on HA Datastores › system › files › presentation-slides › ...Anti-Entropy using CRDTs on HA Datastores Sailesh Mukil Senior Software Engineer, Netflix](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f38e47e708231d44317cb/html5/thumbnails/78.jpg)
Sailesh Mukilsmukil@netflix
Thank You.