peer-to-peer systems - tsn.hhu.de · e.g. sopcast, pplive and zattoo. two approaches for video...
TRANSCRIPT
Peer-to-Peer Systems
Winter semester 2014 Jun.-Prof. Dr.-Ing. Kalman Graffi Heinrich Heine University Düsseldorf
2 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Small Exercise: Bloom Filters 1
Consider a Bloom Filter consisting of § an array of size m and § k hash functions.
Initially all the bits of the array are set to zero. Assume that each position of the array can be selected by
each hash function with equal probability. Calculate the probability that a certain bit will be set to 1 after inserting n elements.
3 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Small Exercise: Bloom Filters 1 - Solution
The probability that a certain bit is not set to one by a certain hash function during the insertion of an element is: 1 − 1/m
The probability that it is not set by any of the hash functions
is: (1 − 1/m)^k If we have inserted n elements, the probability that a certain
bit is still 0 is: (1 − 1/m)^kn the probability that it is 1 is therefore: 1 − (1 − 1/m)^kn
4 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Small Exercise: Bloom Filters 2
Consider a Bloom Filter consisting of § an array of size m = 5 § K = 2 hash functions
H1(x) = x mod 5 H2(x) = (2 ∗ x + 3) mod 5. Initially the bit vector is unset (each element equal to 0). Insert 9 and 11 in the Bloom Filter. Show the state of each bit before and after each insertion.
5 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Small Exercise: Bloom Filters 2 - Solution
Calculate the outcome of each hash function for 9 and 11. H1(9) = 4 H2(9) = 1 H1(11) = 1 H2(11) = 0 Check the membership for the values 15 and 16. Calculate the outcome of each hash function for 9 and 15. 15: H1(15) = 0 H2(15) = 3 Bit at position 3 is set to 0, 15 is not a member. (correct) 16: H1(16) = 1 H2(16) = 0 Both bits are set to 1. (false positive)
H1(x) = xmod5H2(x) = (2 ⇤ x +3)mod5
H1(9) = 4H2(9) = 1H1(11) = 1H2(11) = 0
H1(15) = 0H2(15) = 3
H1(16) = 1H2(16) = 0
Peer-to-Peer Systems
Content Distribution and Multimedia Streaming – P2P Live and Video-on-Demand Streaming
This slide set is based on the lecture "Communication Networks 2" of Prof. Dr.-Ing. Ralf Steinmetz at TU Darmstadt
8 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
P2P Live and Video-on-Demand Streaming
Live P2P streaming has become § increasingly popular approach for
streaming live content § many receivers, similar to IPTV § e.g. SOPCAST, PPLive and Zattoo.
Two approaches for video streaming
§ Tree-based overlay • Push content delivery • Single or multiple tree
§ Mesh-shaped overlay • Pull content delivery (swarming) • Like BitTorrent with modified chunk selection
Research issues: § Streaming topology § Scalable video codecs (more bandwidth, better quality)
9 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
BBC iPlayer § P2P-based video streaming
platform
OctoShape § P2P live streaming
application § Peers receiving fragments of
live stream from other peers Results from ESC 2006
§ 140.000 viewers from 140 countries
§ High quality stream (700kbps)
§ Claim: Traffic savings of 97% compared to C/S system
à Estimation: 53% – 79% traffic savings
10 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Peer-to-Peer TV: Zattoo
Started at 2006 FIFA World Championship 2006
with 4 Swiss TV channels
11 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Tree-based Video Streaming
Each peer receives the video stream from his parent Each peer forwards the stream to its children Initial content streamed from server Content is pushed through the network Pros:
§ High efficiency § Low delay
Cons: § Not very resilient to churn § Upload requirements high
12 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Impact of Peer Churn on Tree based Structure
Churn destroys tree structure § Mechanisms for robustness needed § Can be complex and tricky
13 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Mesh-based Video Streaming
Idea Similar to BitTorrent, peers download the content from each other. Uses modified chunk selection
§ Higher priority for chunks that are about to be played.
Content is pulled from the network Pros
§ More resilient to churn § Upload requirements flexible
Cons
§ Redundant chunks § Lower efficiency than tree based topology § High delay, due to exchange of buffer maps.
14 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Data Pull in Mesh-based Video Streaming
Data Pull § Buffer map is a map of all available chunks at a certain peer. § Exchanged periodically to announce available chunks.
15 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Block Selection
High priority set § Sliding buffer window, updated § with playback position
Download task dispatching
§ Parallel download from multiple peers § Keep all peers as busy as possible
Priority calculation
§ High priority set: use greedy approach • Chunks close to playback position and base layer get highest priority
§ Low priority zone: use non-greedy approach • Download blocks “soon most wanted” by receiver set
Buffering Window
Playing Window
Low priority
t
Peer-to-Peer Systems
Content Distribution and Multimedia Streaming – Adaptive P2P Video Streaming
This slide set is based on the lecture "Communication Networks 2" of Prof. Dr.-Ing. Ralf Steinmetz at TU Darmstadt
17 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Motivation for Adaptive Video Streaming
Video is a major part of Internet traffic § By 2014 almost 90% of
Internet traffic Peer-assisted solution
§ Reduce server load by making use of client-side resources
§ System is more scalable
But how to overcome weak peer contributions and heterogeneity? § Scalable Video Coding
(SVC) can relieve and solve many issues and problems
Bandwidth Screen size CPU Home PC 10 Mbps 1920x1080 >2 GHz
Netbook 5 Mbps 1024 x 600 1.6 GHz
Smart phone 1 – 2 Mbps 800 x 480 900 MHz
18 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Main Challenges for (Adaptive) P2P Streaming
Peer selection § Select peers that can provide the required quality using the
available bandwidth § From which peers to retrieve a chunk
Coordinate delivery from the peers § Sender-Receiver coordination § Which peer to give chunks?
Chunk selection § Which chunks to request / to give?
Handle unpredictable variations in bandwidth § Using fast layer switching § What to do if chunks are missing at playback time?
Packet assignment § Based on network and route status from each sender
19 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Adaptive Video Streaming over P2P
Motivation § Client/Server: scalable only for low resolution content (e.g. YouTube) § P2P: works only when there are enough peers online
Solution and vision § Hybrid solution using adaptive video streaming
Adaptive video streaming § Multi-layer codec (e.g.: H264 Advanced Video Coding)
• Base layer always needed (streamed from server) • More layers à higher quality
– Support heterogeneity – Always on service and scalable
Underlay-aware overlay construction: § Awareness for delay, locality, bandwidth § How to gather the required information about the underlay
• Requirements for Future Internet
20 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
SVC: Block-based Quality Scalability
Three-dimensional scalability
SVC Cube-Model § Each GOP is modeled by a 3D-cube § Block-combinations form layers § Base layer is the most important
Must consider § Interdependencies of blocks § Deadline of blocks § User preference
Base Tier (0)
Tier 2 Tier 2
Tier 3 Tier 3
Tier 3
Tier 1 Tier 2
Tier 1 Tier 1 Tier 2
Tier 3 Tier 3 Tier 3
Spa1
al Resolu1
on (D
)
Temporal Resolu1on (T)
21 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Scalable Video Coding: Flavors of Scalability
Temporal Scalability: Different frame rate, e.g § 7.5 fps used in mobile phones. § 15 fps used in Netbooks, iPhone,... § 30 fps used in PCs
Spatial Scalability: Different resolution. § QCIF, CIF, TV
Quality scalability § Different quality levels.
Quality adaptation approaches Initial Adaptation
§ Support heterogeneous devices § Different memory, processing and bandwidth profiles
Progressive Adaptation § Support adaptation to changing underlay conditions § Bandwidth, delay,...
Picture sources: http://www.hhi.fraunhofer.de/index.php?id=2767&L=1
22 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Quality Adaptation Algorithms
Select SVC layer according to • Peer resources and network dynamics • Different strategy depending on the stage of the streaming session
Initial Quality Adaptation
Chunk/Block Selection
Progressive Quality Adaptation
Streaming
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
23 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Initial Quality Adaptation (IQA)
Determines stream-able SVC layers § Based on static peer resources § Invoked at the beginning of
video playback Goal
§ Avoid long startup times § Match resources at session start
Uses static peer resources
Stages § Spatial adaptation § Bit-rate adaptation § Complexity adaptation
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
24 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Progressive Quality Adaptation (PQA)
Adapt to real time changes of the network § Activated periodically (every T
seconds) § Based on real time network
information Goal
§ Predict possible stalls before they happen
§ Avoid stalls by temporary switching the layer
Uses real time information
Stages § Net-status adaptation § Bitrate adaptation § Complexity adaptation
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
25 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
PQA Stages
Net-status Adaptation § Only request layers that are available within local neighborhood
• Determine the highest supported SVC layer • Avoid waiting for rare blocks by temporary switching the layer
Bit-rate Adaptation § Adjust layer according to throughput of high priority set § Throughput observed through the fullness of the high priority set § Avoid buffer under run by switching down the layer when
throughput is low § Switch layer up in case throughput high enough
Complexity Adaptation § Use models that estimate required processing speed for decoding
each layer
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
26 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Peer Selection and Neighbor Management
Tracker manages: active peers and layers they currently stream § Important in order to connect correct providers and consumers
Peers advertise currently streamed SVC layers § After successful IQA/PQA to the tracker and neighbors § During connection establishment phase with neighbors § Buffer maps are extended to support SVC
The mechanism is bi-directional
§ Peers are eventually clustered according to their resources § Seeders/caches support both weak and strong peers.
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
27 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Evaluation Metrics
Session Quality § Relative playback delay
Video Quality
§ Number of layer changes § Relative received layer
§ Less layer changes or higher relative received layer à Better user experience
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
28 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Impact of Quality Adaptation: Session Quality
0 10 20 30 40 50 60 70 80 900
20
40
60
80
100
120
140
160
180
200
without IQA, PQAwith only IQAwith IQA and PQA
Weak Peers Medium Peers Strong Peers
Tota
l pla
ybac
k de
lay
(Sec
onds
)
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
29 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Impact of PQA Frequency: Visualization
§ The darker, the higher is the layer. White indicates a stall
PQA every 5 seconds PQA every 30 seconds
Stro
ng
M
ediu
m
W
eak
Slide based on Osama Abboud et al. , “Impact of Quality Adaptation in SVC-based P2P Video-on-Demand Systems"
30 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Small Exercise: Clustering Coefficient
Calculate the clustering coeffcient for the nodes: 3, 9 and 12. Calculate also the diameter of the network. CC = (Number of links between the vertices within its neighbourhood)/ (number of links that could possibly exist between them)
Ci vi
4 · 3/2 = 6 CC = 1/64 · 3/2 = 6 CC = 2/6 = 1/36 · 5/2 = 15 CC = 3/15 = 1/5
31 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Small Exercise: Clustering Coefficient - Solution
Calculate the clustering coeffcient for the nodes: 3, 9 and 12. For number 3: 4 neighbors, 1 link, 4·3/2 = 6 possible links, CC = 1/6 For number 9: 4 neighbors, 2 links, 4·3/2 = 6 possible links, CC = 2/6 = 1/3 For number 12: 6 neighbors, 3 links, 6·5/2 = 15 possible links,CC = 3/15 = 1/5 Diameter: maximum of all shortest routes between all nodes. In this case the diameter is 5, e.g. 1-3-7-9-15-16 Ci vi
4 · 3/2 = 6 CC = 1/64 · 3/2 = 6 CC = 2/6 = 1/36 · 5/2 = 15 CC = 3/15 = 1/5
Peer-to-Peer Systems
P2P-based Storage Systems – Overview on P2P Filesystem Approaches
This slide set is based on the lecture “Peer-to-Peer Networks" of Prof. Dr. Jussi Kangasharju at University of Helsinki
33 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
How to use a DHT?
Recall: DHT maps keys to values Applications based on DHTs must need this functionality
§ Or: Must be designed in this way! § Possible to design an application in several ways
Keys and values are application specific
§ For filesystem: Value = file § For email: Value = email message § For distributed DB: Value = contents of entry, etc. § For distributed data structures:
• Value = end item / file • Or value = structured set of further keys
Application stores values in DHT and uses them
§ Simple, but a powerful tool
34 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Problems Solved by DHT
DHT solves § The problem of mapping keys to values in the distributed hash
table
Implications
§ Efficient storage and retrieval of values / objects § Efficient routing
• Robust against many failures • Efficient in terms of network usage
Provides hash table-like abstraction to application
35 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Problems NOT Solved by DHT
Everything else except what is on previous slide… In particular, the following problems
§ Functionality • DHT is: Write-Once-Read-Many • No support for (multi-user) value modification, deletion
§ Robustness • No guarantees against big failures • Threat models for DHTs not well-understood yet
§ Availability • Data not guaranteed to be available • Only probabilistic guarantees (but possible to get high prob.)
§ Consistency • No support for consistency (=same view on object for every) • Data in DHT often highly replicated, consistency is a problem
§ Version management • No support for version management • Might be possible to support this to some degree
36 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
P2P Filesystems
P2P filesystems (FS) or P2P storage systems § File storage and retrieval § First applications of DHTs
Fundamental principle:
§ Key = filename, Value = file contents
Different kinds of systems
§ Storage for read-only objects § Read-write support § Stand-alone storage systems § Systems with links to standard filesystems (e.g., NFS)
37 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Design Space of P2P Storage
Slide based on Anwitaman Datta, “Peer�-to-Peer Storage Systems: Crowdsourcing the storage cloud“, ICDCN ‘10
38 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
P2P Filesystems: Basic Techniques
Fundamental basic techniques for building distributed storage systems § Splitting files into blocks § Replicating files (or blocks!) § Using logs to allow modifications
For now: Simple analysis of advantages and disadvantages and three examples
39 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Splitting Files into Blocks
Motivation § Files are of different sizes and peers storing large files have to serve
more data Benefits
§ Load balancing can be achieved by • Dividing files into equal-sized blocks • Storing blocks on different peers
§ If different files share blocks, we can save on storage Drawbacks:
§ Instead of needing one peer online, all peers with all blocks must be online
§ Need metadata about blocks to be stored somewhere § Granularity tradeoff:
• Small blocks à Good load balance, lots of overhead • Large blocks à Bad load balance, small overhead
40 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Replication
Motivation § If file (or block) is stored only on one peer and that peer is offline,
data is not available § Replicating content to multiple peers significantly increases
content availability Benefits
§ High availability and reliability § But only probabilistic guarantees
Con:
§ How to coordinate lots of replicas? • Especially important if content can change (or can be deleted)
§ Good availability in unreliable networks requires high degree of replication
• “Wastes” storage space
41 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Change Logs
Motivation § If we want to change the stored files, we need to modify every
stored replica § Keep a log for every file (user, …)
• Log gives information about the latest version
Benefits
§ Changes concentrated in one place § Anyone can figure out what is the latest version
Drawbacks
§ How to keep the log available? § By replicating it? ;-)
42 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
Unstructured Solutions
Motivation § Distributed data structures are harmed by churn § Idea: skip structure, have clever search mechanisms
Benefits § Easy to setup and implement § Good in small scenarios
• Example: former p2p collaboration tool Groove (up to 10 users)
Drawbacks § Large search overhead comes with full retrievability
• Turns solution not scalable § Too simple for many applications
43 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
P2P Filesystems: Open Issues
Capacity of hybrid architecture, and efficient schemes § How much centralized provisioning is necessary for a given
distribution of peer resources?
Redundancy placement issues § Cover 24 hours § Minimize storage cost & access latency
Adaptive redundancy management § Different data may need different QoS
Fine‐grained access control § Example scenarios from P2P-based online social networks
Heterogeneity, energy efficiency, load‐balance etc.
44 HHU – Technology of Social Networks – JProf. Dr. Kalman Graffi – Peer-to-Peer Systems – http://tsn.hhu.de/teaching/lectures/2014ws/p2p.html
P2P Filesystems: Real-World Examples
Microsoft has done lot of research in this area § Even built prototypes
Why no products on the market?
§ Below some wild speculation § P2P FS would compete with traditional file servers?
• P2P needs to be built in to the OS, would kill server market? • Who would build such an OS?
– No big companies. But small companies are to small? Academia? – Building “too much” software in academia: not research?
§ Impossible to build a “good enough” P2P FS? • Theoretically doable, but too slow and weak in practice?
§ P2P filesystem can be done, but has no advantages? • Solving one task (file system) opens many new requirements
– Monitoring, quality control, security and access control, privacy, ability to delete criminal content, accounting, quota enforcement, …
– Many tasks simple to solve centralized, but hard to solve decentralized