stochastic analysis of file swarming systems the chinese university of hong kong john c.s. lui...
TRANSCRIPT
![Page 1: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/1.jpg)
Stochastic Analysis of File Stochastic Analysis of File Swarming SystemsSwarming Systems
The Chinese University of Hong KongJohn C.S. Lui
Collaborators: D.M. Chiu, M.H. Lin, B. Fan
![Page 2: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/2.jpg)
BackgroundTraditional Client/Server Sharing
Performance deteriorates rapidly as the number of clients increases
IP MulticastApplication Multicast (e.g., CDN, ESM)
reliability, unused resources at leaf nodesP2P (e.g., Naspter, Gnutella)
Free riders only download without contributing to the network.
BitTorrent P2P systems:Good scalabilityBuilt-in incentive mechanism to contribute
![Page 3: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/3.jpg)
BT ComponentsBT ComponentsOn a public domain site, obtain torrent file,
for example: http://bt.btchina.nethttp://bt.ydy.com/ Web Server
Harry Potter.torrentTransformer.torrent
The Lord of Ring.torrent
![Page 4: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/4.jpg)
BT ComponentsBT Components The .torrent file
Static “metainfo” file to contain necessary information :File name# of chunks, sizechecksumIP address of the TrackerTracker,…etc
A BitTorrent trackerA BitTorrent tracker Non-content-sharing node Track peers
File: File: Chunk size (256KB), has individual hash code in the torrent fileChunk size (256KB), has individual hash code in the torrent file
Types of peers:Types of peers: LeechersLeechers SeedersSeeders
€
F = C1 UC2 UL UCm,Ci I C j =∅
![Page 5: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/5.jpg)
BT: publishing a fileBT: publishing a file
Web ServerMoe
Tracker
Downloader:Larry
Seeder:John
Downloader:Curly
Harry Potter.torrent
![Page 6: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/6.jpg)
Simple exampleSimple example
Seeder:John
DownloaderMoe
{1,2,3,4,5,6,7,8,9,10}
{}{1,2,3}
Downloader Larry
{}{1,2,3}
{1,2,3,4}
{1,2,3,5}
{1,2,3,4,5}
![Page 7: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/7.jpg)
BT: internal Chunk Selection BT: internal Chunk Selection mechanismsmechanisms
Strict PriorityFirst Priority
Rarest FirstGeneral rules
Random First PieceSpecial case, at the beginning
Endgame ModeSpecial case
![Page 8: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/8.jpg)
BT: internal mechanismBT: internal mechanism
Built-in incentive mechanism (where all the magic happens):Choking AlgorithmOptimistic Unchoking
![Page 9: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/9.jpg)
BT: internal mechanismBT: internal mechanism
• Choking is a temporal refusal to upload• Each peer unchokes a fixed number of peers• Reasons for choking:
– Avoid free riders
– Network congestion
– Contribute to “useful” peers
Yaokun Wu
John C.S Lui
ChokedChoked
![Page 10: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/10.jpg)
BT: internal mechanism BT: internal mechanism (optimistic unchoking)(optimistic unchoking)
A BitTorrent peer has a single “optimistic unchoke” which uploads regardless of the current download rate from it. This peer rotates every 30s
Reasons:To discover currently unused connections are better
than the ones being usedTo provide minimal service to new peers
![Page 11: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/11.jpg)
Example: optimistic Example: optimistic unchokingunchoking
Andy Yao
Downloader:John Lui
Downloader:MelindaDownloader:
Larry Downloader:Curly
40kb/s
30kb/s10kb/s
100kb/s
20kb/s
70kb/s
15kb/s
10kb/s
70kb/s.
110kb/s
70kb/s
5kb/s
DownloaderMoe
![Page 12: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/12.jpg)
P2P content distribution P2P content distribution
BitTorrent
Sending a file to a large number of peers, with the help of peers
Producing the most Internet traffic today (over 50% of traffic, creates contention but ....)
What IP multicast tried to support
Modeling these systems => insights
![Page 13: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/13.jpg)
Why Study BitTorrent-like System?
BitTorrent is very efficient. Which features make it perform so well?
Motivating questions What is the effect of bandwidth constraints? Is the Rarest First policy really necessary? Must nodes perform seeding after file downloading? How serious is the Last Piece Problem? Is source coding useful? Does the incentive mechanism affect the performance much?
Our aim is to develop mathematical models of file swarming systems, Our aim is to develop mathematical models of file swarming systems,
allowing us to investigate these issues via analytical means.allowing us to investigate these issues via analytical means.
![Page 14: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/14.jpg)
Model for the File Swarming System A file has K non-overlapping chunks.
Peers arrive according to a Poisson process. Each peer is initialized with one random chunk.
Peers leave the system immediately when finish downloading.
The system is slotted: downlink bandwidth is one chunk per time slot for all peers. (download constraintdownload constraint)
In each time slot, each peer contacts m neighbors uniformly from the system to see whether they are useful. If some neighbors are useful, it randomly chooses one and requests a random useful chunk.
If a peer receives several requests, it will satisfy all / random one request(s). (without/with upload constraint)(without/with upload constraint)
![Page 15: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/15.jpg)
Model for the File Swarming System
peer A
peer B
peer C
peer D
peer E
Request C1
Request C5
C1
C5
Example: m=2
The case “m = 1 & no upload constraint” was studied by L.Massoulie et.al in ”Coupon replication systems”.
HELLO
HELLO
HELLO
HELLO
Bitmap
Bitmap
Bitmap
Bitmap
Without upload constraint
With upload constraint
![Page 16: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/16.jpg)
Model 1: Download Constraint Download Constraint OnlyOnly Classify peers into K−1 types. Peers holding i chunks are named
type i peers. Denote the number of type i peers, We are interested in the average sojourn time Ti for type i peers.
The average downloading time
For a type i peer, the probability that a type j peer is useful:
For a type i peer, the probability that a randomly picked peer is useful:
![Page 17: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/17.jpg)
Model 1: Download Constraint Model 1: Download Constraint OnlyOnly
Given the system state , is a Multi-dimensional infinite state-space Markov Process:
It is hard to solve this Markov Chain directlyTransform the Markov Chain to a “Density Dependent Density Dependent
jump Markov Processjump Markov Process”Focusing on its steady state and asymptotic behavior We derive tight boundstight bounds.
![Page 18: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/18.jpg)
Model 1: Download Constraint OnlyDownload Constraint Only
The case m=1 has been studied in [1], in which the authors gave a looser bound:
[[1] 1] L.Massoulie, M.VojnoviC, ”Coupon replication systems”, L.Massoulie, M.VojnoviC, ”Coupon replication systems”, SIGMETRICSSIGMETRICS, 2005, 2005..
The average downloading time .
![Page 19: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/19.jpg)
Lower bound v.s. Upper bound (K=200)
m=1 m=2
Last Piece ProblemIt takes a peer a longer time to download the last few chunks of the file, since it gets increasingly more difficult to find other peers that can help.
![Page 20: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/20.jpg)
Bounds v.s. Simulation (K=200)
m=1 m=2
The simulation shows the accuracy of our model.
How to relief the last piece problem?
![Page 21: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/21.jpg)
System with Source CodingSystem with Source Coding
K=4 Q=6
Source
peer A
peer B
peer C
peer D
peer E
C1
![Page 22: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/22.jpg)
System with Source CodingSystem with Source Coding
The source encodes the original K chunks into Q chunks, Any peer could reconstruct the original file after he receives any K distinct chunks.
![Page 23: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/23.jpg)
Source Coding vs. No Coding(K=200)
m=1, no coding
Source coding eliminates the Last Piece Problem !!!
m=1, source coding ( )
![Page 24: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/24.jpg)
Download constraint only
K=200; m=1 K=500; m=1
![Page 25: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/25.jpg)
Download Constraint
K=200; m=2 K=500; m=2
![Page 26: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/26.jpg)
Model 2: Download & Upload Model 2: Download & Upload ConstraintsConstraints —— m=1—— m=1
peer A
peer B
peer C
peer D
peer E
Request C1
Request C5
C1HELLO
HELLO
Bitmap
Bitmap
![Page 27: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/27.jpg)
Model 2: Download & Upload Model 2: Download & Upload ConstraintsConstraints —— m=1—— m=1Stage One: Requesting
The same as Model 1.
Stage Two: DownloadingThe distribution of the number of requests one peer
would receive (depending on its type).Only one request will be satisfied.
Still a density dependent jump Markov processThe transition rates are more complicated.
![Page 28: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/28.jpg)
Model 2: Download & Upload Model 2: Download & Upload ConstraintsConstraints —— m=1—— m=1
≈ 1.58
![Page 29: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/29.jpg)
Bounds v.s. Simulation (K=200, without source coding)
m=1 & satisfying one request
Ti is NOT close 1 any more, i.e. downloading time is far from being optimal.
![Page 30: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/30.jpg)
Model 3: Model 3: An Incentive MechanismAn Incentive Mechanism
peer A
peer B
peer C
peer D
peer ERequest C1
Request C5C5
Assuming peers are matched randomly at the beginning of each time slot. Each pair will perform chunk transfer iff both of them are useful to each other.
Request C2
C2
![Page 31: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/31.jpg)
Model 3: An Incentive Mechanism
![Page 32: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/32.jpg)
Bounds v.s. Simulation (K=200, without source coding)
First Piece Problem
It is not easy to download the first few chunks when a peer enters the system,
but one can solve this in various of ways….
![Page 33: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/33.jpg)
Incentive Mechanism
K=200; m=1 K=500; m=1
![Page 34: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/34.jpg)
ConclusionConclusionMany peers, steady state, certain mechanism to ensure fileavailability (e.g. some seeders), then The nature of swarming makes P2P systems very efficient. Rarest First policy is not necessary for performance. If peers are
cooperative, “random policy” is good enough, though it may be helpful to enhance file availability.
Peers are not necessary to perform seeding after file downloading. Simple strategies (everything is random) can make the downloading
time near optimal. Source coding is useful, to relief the last piece problem. With certain incentive mechanism, the downloading time can still
approach optimal.
Our mathematical models provide a basis for designing new BT-like protocol.
![Page 35: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/35.jpg)
Research QuestionsResearch Questions
What about fairness?How to extend file swarming to
multimedia streaming? For Joost?What about wide area network
exchange?What happen if there is ``network
congestion’’? What is the impact?Network Coding? Security?
![Page 36: Stochastic Analysis of File Swarming Systems The Chinese University of Hong Kong John C.S. Lui Collaborators: D.M. Chiu, M.H. Lin, B. Fan](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cd75503460f949a0294/html5/thumbnails/36.jpg)
Q & A
Thank You