![Page 1: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/1.jpg)
Dynamo:Amazon'sHighlyAvailableKey-valueStore
Author:GiuseppeDeCandia,DenizHastorun,MadanJampani,GunavardhanKakulapati,AvinashLakshman,AlexPilchin,Swaminathan
Sivasubramanian,PeterVosshallandWernerVogels
Presentation:ShijieXu,YingWang
![Page 2: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/2.jpg)
WhyDynamo?
●FullyManaged●Fast,ConsistentPerformance●HighlyScalable●Flexible
![Page 3: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/3.jpg)
SystemAssumptionsandRequirements●QueryModel:simplereadandwriteoperationtoasmalldataitemthatisuniquelyidentifiedbyakey
●ACIDProperties:Atomicity,(Weaker)Consistency,(No)Isolation,Durability
●Efficiency:Latencyrequirementswhichareingeneralmeasuredatthe99.9thpercentileofthedistribution
●OtherAssumption:Onlydealwithbenignfailures
![Page 4: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/4.jpg)
ServiceLevelAgreements
●Applicationcandeliveritsfunctionalityinaboundedtime
Fig-1Service-orientedarchitectureofAmazon’splatform
![Page 5: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/5.jpg)
DesignConsideration
● Sacrificestrongconsistencyforavailability● Alwayswriteable● Conflictresolutionisexecutedduringreadinsteadofwrite● Otherprinciples:
○ Incrementalscalability○ Symmetry○ Decentralization○ Heterogeneity
![Page 6: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/6.jpg)
SystemArchitecture
Coretechniquesused:● Partitioning● Replication● Versioning● Membership● Failurehandling
![Page 7: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/7.jpg)
API● get(key)
ReturnsAsingleobjectoralistofobjects,andAcontext
● put(key,context,object)Useskey todeterminethewritereplicasWritesthereplicastodisk
● ContextSystemmetadataabouttheobject
![Page 8: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/8.jpg)
Partitioning● ConsistentHashing
○ Theoutputrangeofhashedvaluestreatasa“ring”
○ Pros: incrementallyscalable,addingasinglenodedoesnotaffectthesystemsignificantly
○ Cons:leadtotheunevendistributedload,andoblivioustotheheterogeneityintheperformanceofnodes.
![Page 9: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/9.jpg)
Partitioning
● “VirtualNode”○ Eachnodecanberesponsibleformorethanonevirtualnodes
○Workdistributionproportionaltothecapabilitiesoftheindividualnode
![Page 10: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/10.jpg)
Replication
● EachdataitemisreplicatedatNhosts● Preferencelist:Thelistofnodesthatisresponsiblefor
storingaparticularkey○ MaycontainmorethanNnodesduetofailures○ Containsonlydistinctphysicalnodes
![Page 11: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/11.jpg)
Replication
●Example:N=3○NodeBreplicatesthekeykatnodeCandDinadditiontostoringitlocally
○ NodeDwillstorethekeysintherange(A,B],(B,C],and(C,D]
Fig-4PartitioningareplicationofkeysinDynamoring.
![Page 12: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/12.jpg)
DataVersioning
● Systemiseventuallyconsistent,thusaget()callmayreturnmanyversionsofthesameobject
● Challenge:Anobjectcanhavedistinctversionsub-histories,thesystemneedstoreconcileinthefuture
● Solution:SyntacticreconciliationandSemanticreconciliation
![Page 13: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/13.jpg)
VectorClock● Avectorclockisalistof[node,counter]
pairs● Versionedobject->vectorclock● Updateanobject,put(key,context,
object)● “context”isobtainedfromanearlier
readoperation,whichcontainsthevectorclockinformation
![Page 14: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/14.jpg)
Syntacticreconciliation
![Page 15: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/15.jpg)
Whatif?
Source: Rick and Morty S02E01
![Page 16: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/16.jpg)
Semanticreconciliation● Failures+concurrent
updating=>versionbranching
● Collapse● Versionbranchingis
resolvedbydatastoreortheapplication○ Datastore:latestwrite
wins○ Application:merge
![Page 17: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/17.jpg)
VectorClockIssue● Vectorclockmaygrowwhenmanyserverscoordinatethe
writestooneobject● TruncationScheme
○ Deletetheoldest[node,counter]pairwhenthenumberofpairsreachesathreshold
● Moreissue:Inefficienciesinreconciliationbecauseofmissinginformation○ Notshowinproduction
![Page 18: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/18.jpg)
Clientrequestchoices
● Genericloadbalancer○ NocodespecifictoDynamo○ Extrarequestforwardingstep
● Partition-awareclientlibrary○ betterperformance
![Page 19: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/19.jpg)
Executegetandput:QuoruminDynamo
● Thefirstreachablenodeinthepreferencelististhecoordinator● R:minimumnumberofnodesthatmustparticipateinsuccessful
readoperation● W:minimumnumberofnodesthatmustparticipateinasuccessful
writeoperation● SettingR+W>Nyieldsaquorum-likesystem● Thelatencyofaget()(orput())operationisdictatedbytheslowest
oftheR(orW)replicas● RandWareusuallyconfiguredtobelessthanN,toprovidebetter
latency
![Page 20: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/20.jpg)
Executionofget() operationget()● CoordinatorrequestsreadingfromNnodes,waitsforRresponses● Iftheresponsesagree,returnstheobjectwithcontext● Iftheydisagree
○ Iftheyarecausallyrelated,returnsthemostrecentvalue○ Iftheyarecausallyunrelated,returnsallversions
![Page 21: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/21.jpg)
Executionofput() operationput():● Coordinatorgeneratesnewversionvectorclockandwritesnew
versionlocally● Forwardsmetadatatohighestrankedreachablenodesinthe
preferencelist● WaitsforW-1ormorewritestobesucceed
![Page 22: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/22.jpg)
HandlingFailures:HintedHandoff
●“Alwayswriteable”
●Avoidthereadandwriteoperationsfailure,duetotemporarynodeornetworkfailures
Fig-6PartitioningareplicationofkeysinDynamoring.
![Page 23: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/23.jpg)
HandlingpermanentFailures:ReplicaSynchronization●Merkletree:
○ Parentnodearehashesof(immediate)children
○ Comparisonofparentsatthesameleveltellsthedifferenceinchildren
○ Doesnotrequiretransferringentire(key,value)pairs
![Page 24: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/24.jpg)
Thepowerofgossip
![Page 25: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/25.jpg)
Thepowerofgossip
●RingMembership○Allnodesexchangetheirmembershiphistories○Eachnoderandomlycontactapeereverysecond○Eventuallyconsistent○Eachnodeforwardakey’sread/writeoperationsrightsetofnodesdirectly
![Page 26: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/26.jpg)
Thepowerofgossip
●ExternalDiscovery○Nodesmaynotknoweachother- logicalpartitions○SeedNodestoavoidlogicalpartitions
![Page 27: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/27.jpg)
Thepowerofgossip
●FailureDetection○Detectfailurelocallyissufficient○Periodicallyretryfailednode(s)○Noneedforadecentralizedfailuredetector
![Page 28: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/28.jpg)
Implementation
●Java●Localpersistencecomponentallowsfordifferentstorageenginestobepluggedin:
○BerkeleyDatabase(BDB)TransactionalDataStore:objectoftensofkilobytes
○MySQL:objectof>tensofkilobytes
![Page 29: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/29.jpg)
MainmodesofDynamo
●Businesslogicspecificreconciliation○Merge○Application-specificreconciliation
● Timestampbasedreconciliation○ Lastwritewins
●Highperformancereadengine○R=1,WisN,Dynamoprovidestheabilitytopartitionandreplicatetheirdataacrossmultiplenodestherebyofferingincrementalscalability
![Page 30: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/30.jpg)
Experiences
●N:durability●WandR:availability,durabilityandconsistency
○ IncreaseWcanincreasedurabilitybutreduceavailability● (N,R,W)=(3,2,2)providessatisfyingperformance,durability,consistency,andavailabilitySLAs
![Page 31: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/31.jpg)
Performance
●GuaranteeServiceLevelAgreements(SLA)○ Latencies:diurnalpattern(incomingrequestrate)
○Writelatencies>>Readlatencies
●Latenciesaround200ms
Fig-9Averageand99.9percentilesoflatenciesforreadandwriterequestduringpeakseasonofDec.2006
![Page 32: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/32.jpg)
BetterPerformance
●Tradedurabilityforbetterperformance
●Eachstoragenodemaintainsanobjectbufferinitsmainmemory
●Writeobjectsinbuffertodiskusingawriterthreadperiodically
●Readfrombufferinmemory
Fig-10Comparisonofperformanceof99.9th percentilelatenciesforbufferedvs.non-bufferedwritesoveraperiodof24hours
![Page 33: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/33.jpg)
Balance
●Outofbalance○ Ifthenode’srequestloaddeviatesfromtheaverageloadbyavaluemorethanacertainthreshold(hereis15%)
● Imbalanceratiodecreaseswithincreasingload●Underhighloads,alargenumberofpopularkeysareaccessedandtheloadisevenlydistributed
Fig-11Fractionofnodesthatareoutofbalance,andtherecorrespondingrequestload.Theintervalbetweenticksinx-axiscorrespondtoatimeperiodof30mins.
![Page 34: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/34.jpg)
Partitioningandplacementofkey
Strategy1:Trandomtokenspernodeandpartitionbytokenvalue● Slowbootstrappingprocess● RecalculationoftheMerkletree● Datapartitioninganddataplacementareintertwined
Strategy2andStrategy3● Equalsizepartitioningstrategiestodistributeloaduniformly
![Page 35: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/35.jpg)
Server-drivenandClient-drivenCoordination
●Useastatemachinetohandleincomingrequests
●Movethestatemachinetotheclientnodes
![Page 36: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/36.jpg)
Balancingbackground&foreground
●Eachnodeperformsbothbackgroundandforegroundoperation
●Backgroundtriggerresourcecontention
●Admissioncontroller:changetheruntimeslicesoftheresourceforbackground
![Page 37: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/37.jpg)
Conclusion
●Dynamoisahighlyavailable and scalabledatastoreforAmazon’se-commerceplatform.
●Techniques:○Gossiping formembershipandfailuredetection○Consistenthashing fornodeandkeydistribution○Objectversioning foreventually– consistentdataobjects○Quorums forpartition/failuretolerance○Merkletree forresynchronizationafterfailures/partitions
![Page 38: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/38.jpg)
Questions?
![Page 39: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/39.jpg)
HandlingpermanentFailures:ReplicaSynchronization●Comparingtwonodesthataresynchronized
○ Two(key,value)pairs:(k0,v0)&(k1,v1)
Fig-7ReplicaSynchronization
![Page 40: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/40.jpg)
HandlingpermanentFailures:ReplicaSynchronization●Comparingtwonodesthatarenotsynchronized
○ Two(key,value)pairs:(k0,v0)&(k1,v1)
Fig-8ReplicaNotSynchronization
![Page 41: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/41.jpg)
Partitioningandplacementofkey
Strategy1:● Trandomtokenspernodeand
partitionbytokenvalueProblems:● Slowbootstrappingprocess● RecalculationoftheMerkletree● Complicatedarchivalprocess
Fig-12Partitioningandplacementofkey,strategy1
![Page 42: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/42.jpg)
Partitioningandplacementofkey
Fig-13Partitioningandplacementofkey,strategy2
Strategy2:● Trandomtokenspernodeandequalsized
partitions
● DividesthehashspaceintoQequallysizedpartitions
![Page 43: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/43.jpg)
Partitioningandplacementofkey
Fig-14Partitioningandplacementofkey,strategy3
Strategy3:Q/Stokenspernode,equal-sizedpartitions
• DividesthehashspaceintoQequallysizedpartitions
• EachnodeisassignedQ/Stokens
![Page 44: Dynamo: Amazon's Highly Available Key-value Storemanosk/assets/slides/w18/dynamo.pdf · Service Level Agreements Application can deliver its functionality in a ... The power of gossip](https://reader033.vdocument.in/reader033/viewer/2022041714/5e4a3c8b2f130f797f6be8a8/html5/thumbnails/44.jpg)
Thankyou