locality-aware content distribution danny bickson, dahlia malkhi, david rabinowitz
Post on 22-Dec-2015
215 views
TRANSCRIPT
Locality-Aware Locality-Aware Content DistributionContent Distribution
Danny Bickson, Danny Bickson,
Dahlia Malkhi,Dahlia Malkhi,
David RabinowitzDavid Rabinowitz
Lecture outlineLecture outline
Problem statementProblem statement Previous workPrevious work The Julia algorithmThe Julia algorithm Algorithm deploymentAlgorithm deployment
Problem StatementProblem Statement Large scale content distributionLarge scale content distribution High speed data disseminationHigh speed data dissemination Application layer multicastApplication layer multicast
The problem: We would like to transfer a The problem: We would like to transfer a file file FF, from a source , from a source ss to a group of to a group of nn recipients.recipients.
MeasurementsMeasurements
Minimize worst completion timeMinimize worst completion time Minimize total work (the number of bits * Minimize total work (the number of bits *
link distance) over all linkslink distance) over all links Average fair sharing ratioAverage fair sharing ratio ConnectivityConnectivity
Naïve solutionsNaïve solutions
* * Client / ServerClient / Server* Mirroring / replication* Mirroring / replication* Multicast tree* Multicast tree
State-of-the-art solutionsState-of-the-art solutions
SRMSRM SplitStream / Coopnet / BayeuxSplitStream / Coopnet / Bayeux BulletBullet FastReplicaFastReplica Zigzag / TMesh / Narada Zigzag / TMesh / Narada BitTorrentBitTorrent
Julia AlgorithmJulia Algorithm
Load balanceLoad balance Fair sharingFair sharing Optimal finishing timeOptimal finishing time Local transfer of most parts of the fileLocal transfer of most parts of the file
Julia algorithmJulia algorithm
1 2 3 4 5 6 7 8
1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8
Round 1 –
Exchange 1 part along longest links
Julia algorithmJulia algorithm
1 2 3 4 5 6 7 8
1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8
Round 2
1,3,5,7 2,4,6,8 1,3,5,7 1,3,5,7 1,3,5,72,4,6,8 2,4,6,8 2,4,6,8
Julia algorithmJulia algorithm
1 2 3 4 5 6 7 8
1,5 2,6 3,7 4,8 1,5 2,6 3,7 4,8
Round logN – exchange half of the file along shortest links
1,3,5,7 2,4,6,8 1,3,5,7 1,3,5,7 1,3,5,72,4,6,8 2,4,6,8 2,4,6,8
1-8 1-8 1-8 1-8 1-8 1-8 1-8 1-8
Comparison SummaryComparison SummaryDownload time Download time (worst case)(worst case)
Number of Number of edgesedges
Total workTotal work
Application Application multicast treemulticast tree
K|F|logK|F|logkk(n)(n) n-1n-1 |F|D/2(n-1)|F|D/2(n-1)
SplitStream SplitStream protocolprotocol
|F|(log|F|(logkk(n)+1)(n)+1) (n-1)k(n-1)k (|D|n(|D|n1-1-
FastReplica FastReplica protocolprotocol
((2-12-1/k/k)|F|log)|F|logkk(n)(n) (n-1)*(n-1)*
(k+1)/2(k+1)/2
(|D|n(|D|n1-1-
Our protocolOur protocol |F||F| nlognlog22(n)(n) (|F|D log(|F|D log22(n))(n))
=1/log2(k)
The deploymentThe deployment
Distance estimationDistance estimation Level categorizationLevel categorization Node selection algorithmNode selection algorithm Chunk selection algorithmChunk selection algorithm
ImplementationImplementation
C++ client consisting of 15,000 lines of C++ client consisting of 15,000 lines of code.code.
Event queue model.Event queue model. Did both LAN and Planetlab experimentsDid both LAN and Planetlab experiments
Protocol MessagesProtocol Messages
Request file infoRequest file info Reply file infoReply file info Request chunkRequest chunk Reply chunkReply chunk Data / Error msgsData / Error msgs
Distance estimationDistance estimation
Distance Distance measurements are measurements are collected on the fly – collected on the fly – no spare bandwidth no spare bandwidth allocated.allocated.
Nodes are Nodes are categorized into 8 categorized into 8 levels levels
11 2Mbps2Mbps
22 1.5Mpbs1.5Mpbs
33 750Kbps750Kbps
44 500Kbps500Kbps
55 250Kbps250Kbps
66 100Kbps100Kbps
77 50Kbps50Kbps
88 UnknownUnknown
Node selection algorithmNode selection algorithm
Progress depended.Progress depended. Up to 25% progress – connect random Up to 25% progress – connect random
nodes.nodes. Up to 50% progress – connect close nodes Up to 50% progress – connect close nodes
with probability p1.with probability p1. Above 50% progress – connect close nodes Above 50% progress – connect close nodes
with probability p2.with probability p2.
Chunks selection Chunks selection algorithmalgorithm
Rarest firstRarest first RandomRandom Round robinRound robin MixedMixed
OptimizationsOptimizations
Pipelining of the Julia algorithm Pipelining of the Julia algorithm Message batchingMessage batching