IEEE_ICPADS08_KG_SkyEye.ppt
KOM - Multimedia Communications LabProf. Dr.-Ing. Ralf Steinmetz (director)
Dept. of Electrical Engineering and Information TechnologyDept. of Computer Science (adjunct professor)
TUD – Technische Universität Darmstadt Merckstr. 25, D-64283 Darmstadt, Germany
Tel.+49 6151 166150, Fax. +49 6151 166152 www.KOM.tu-darmstadt.de
© author(s) of these slides 2008 including research results of the research network KOM and TU Darmstadt otherwise as specified at the respective slide
QuaP2P Improving the quality of P2P systems
DFG research group
Dipl.-Math., Dipl.-Inform. Kalman [email protected]
04/12/23
SkyEye.KOM: An Information Management Over-Overlay for Getting the Oracle View on
Structured P2P Systems
Kalman Graffi, Aleksandra Kovacevic,Song Xiao and Ralf Steinmetz
H(„my data“)= 3107
2207
7.31.10.25
peer-to-peer.info
12.5.7.31
95.7.6.10
86.8.10.18
planet-lab.orgberkeley.edu
29063485
201116221008709
611
89.11.20.15
?
? Does my p2p system work?
Underlay:The Internet
StructuredOverlay: DHT
Information Management Over-Overlay
Pick good peers
KOM – Multimedia Communications Lab 2
Outline
Motivation Information Management in P2P Systems Example
SkyEye.KOM Requirements Architecture
Evaluation Performance Costs
Conclusion
KOM – Multimedia Communications Lab 3
The Peer-to-Peer Paradigm
Peer-to-Peer Systems: Users of a system provide the infrastructure Service is provided from users/peers to users/peers Peer-to-Peer overlays:
virtual networks, providing new functionality E.g. Distributed Hash Tables, Keyword-based Search
Evolution of applications File sharing:
No QoS requirements Voice over IP
Real-time requirements Video-on-demand
Real-time and bandwidth requirements Online community platforms
Potential for high user interaction
Costs Security
Quality of P2P Systems
Retrievability
Coherence
Consistency
Correctness
PerformanceScalability
Flexibility
Stability
Dependability
Service Provisioning
Overlay Operations
Individual Node
Complete System
IP Infrastructure
Availability
Reliability
Robustness/ Fault tolerance
Integrity
Confidentiality
Authentication
Non-repudation
TrustValidityEfficiencyAdaptability
Costs Security
Quality of P2P Systems
Retrievability
Coherence
Consistency
Correctness
PerformanceScalability
Flexibility
Stability
Dependability
Service Provisioning
Overlay Operations
Individual Node
Complete System
IP Infrastructure
Availability
Reliability
Robustness/ Fault tolerance
Integrity
Confidentiality
Authentication
Non-repudation
TrustValidityEfficiencyAdaptability
KOM – Multimedia Communications Lab 4
Open Issues in P2P Research
Current P2P-applications do not consider sufficiently the heterogeneity of peers the state of the whole P2P-system
P2P systems and applications would benefit from monitoring the system generating system statistics determining the state of the P2P system accounting the capacities of peers
Possible results of the aforementioned actions Detection of limitations or failures in running P2P system Adaptation of the peers to the current state of the P2P system
KOM – Multimedia Communications Lab 5
Example: Let’s Build a new P2P Application!
Get ideas, make design, implement it…Cool hot product, lot of features!
A lot of modules involved All of them have some requirements, need specific peers See K.Graffi, “A Distributed Platform for Multimedia Communities”
at Proc. of IEEE Int. Symposium on Multimedia ’08, 15.-17.Dec.’08
Each functional layer requires specific peers Distribute Computing: Give me 4 peers with Java > 6.0 and
processor power > 10M flops Streaming: Give me 2 peers with sufficient processing power
for media file recoding Replication: Give me 1 peer with a lot of bandwidth
and 10 peers with specific life time and storage space Overlay: Give me 3 peers, with a node-degree > 50 Network wrapper: Give me 10 peers from my ISP Security: Find 5 trustworthy peers
Challenge No.1: Capacity-based Peer Search For identifying peer capacities in the network we can use
KOM – Multimedia Communications Lab 6
Hooray it Runs! Publish and Advertize!
P2P
P2P CompanyGive us money©
KOM – Multimedia Communications Lab 7
P2P App.
Deployment and Running a P2P System
Challenge No.1: Capacity-based Peer Search
More challenges?
Is everything running fine?
How to debug and to gain insight?
Challenge No. 2:
Live statistics on P2P systems
H(„my data“)= 3107
2207
7.31.10.25
peer-to-peer.info
12.5.7.31
95.7.6.10
86.8.10.18
planet-lab.orgberkeley.edu
29063485
201116221008709
611
89.11.20.15
?
? Does my p2p system work?
Underlay:The Internet
StructuredOverlay: DHT
KOM – Multimedia Communications Lab 8
Optimizing the RUNNING P2P System
Assume we have statistics on our P2P application We see that something is wrong / can be improved We know which parameters to change We change the parameter somehow (by hand?)How to change parameters in a running system?
Challenge No.3: Parameter optimization based on statistics
Our P2P application
KOM – Multimedia Communications Lab 9
Outline
Motivation Information Management in P2P Systems Example
SkyEye.KOM Requirements Architecture
Evaluation Performance Costs
Conclusion
KOM – Multimedia Communications Lab 10
Requirements on an Information Mgmt. System
For all structured P2P overlays Covered by DHT-function: route(msg, key), lookup(key) Usable by all functional layers/modules in the P2P system
Generating system statistics System wide average and absolute values, standard deviation, confidence intervals Num. of peers, online times, hops per lookup, hit rate, relative delay penalty, node degree Load (CPU, memory, storage space, bandwidth), additionally weighted with individual
peer capabilities
Capacity-based peer search Search for a specified number of peers with specific capacities E.g. give me 3 peers with:
Storage space > 20Mb Bandwidth > 100kb/s
Remote consistent parameter setting Reconfigure all peers during runtime
KOM – Multimedia Communications Lab 11
SkyEye: Information Management Over-overlay
For all structured P2P overlays Covered by DHT-function:
route(msg, key), lookup(key) Usable by all functional layers in the
P2P system
Features: Overlay-independency Robustness (double-churn) Load-balancing Supporting peer heterogeneity Scalability (# of info and peers) Lightweight Low overhead
Function: Generating system statistics Capacity-based peer search Enable parameter setting reconfiguration
Internet
DHT overlayoffering route(key,msg,nextHop), resp(key) .
API for DHT’s ID space and functions
Unified ID space and abstr. functions
Coordinator Support Peer Peer
KOM – Multimedia Communications Lab 12
SkyEye.KOM Architecture
Concept of Over-overlay Built on underlying structured overlay Communicates via common API (KBR)
route(msg, key), lookup(key) Unified ID space [0,1] decouples
from specific DHT implementation:e.g. divide ID by maxID
Information Domains: ID space separated in intervals (domains) Peer responsible for a specific ID (e.g. middle) is responsible for ID domain Peers in the domain send updates to this Coordinator Coordinators may choose Support Peers to share the load/responsibility
Coordinators define maximum load to carry Updates propagated upwards the tree
API for DHT’s ID space and functions
Unified ID space and abstr. functions
Coordinator Support Peer Peer
KOM – Multimedia Communications Lab 13
Example Figures
0 1
11050
2030
40
45
15
0,09 0,2 0,3 0,4 0,51 0,6 0,75 0,9
CoordinatorID 0,25
CoordinatorID 0,125
0D
CoordinatorID 0,375
CoordinatorID 0,751D
0 10,18 0,24 0,4 0,55 0,8
CoordinatorID 0,50C
1C 1C
2C 2C
KOM – Multimedia Communications Lab 14
SkyEye.KOM Architecture Details
Update strategy Each node
sends information up knows where to send updates to
Update consists of: Non-aggregatable peer capacity information Aggregatable systems statistics part
Updates are processed before forwarding Update information has a limited lifetime
Supporting Peers for Load Balancing Coordinator may chose Supporting Peers Good peers chosen by e.g. 50/50 ratio
Pick e.g. 20 best peers in the domain Best 10 peers in domain advertised one level up Second best 10 peers can be used as support
Workload can be delegated to supporting peers Tree depth / peer load adjustable
Coordinator Support Peer Peer
For SP: best 11-20
best 1-10 best 1-10
best 1-10best 1-10
For SP: best 11-20 from below
For SP: best 11-20
For SP: best 11-20 from below
KOM – Multimedia Communications Lab 15
Setting the Parameters
Capturing the state of every peer Periodically measuring and aggregating
statistics Forwarding of aggregated statistics
Evaluation of the collected information
Calculation of system statistics Enables the integration of data mining
tools for implementing self-organization Transmission of the new information
Adaption to the propagated information
Enabling the adaptive behavior of a peer
Evaluation of dataidentification ofcounteraction
0C
1C 1C
tC tC
Measuring and aggregatingmetrics
Measuring and aggregatingmetrics
Measuring and aggregatingmetrics
Measuring and aggregatingmetrics
Measuring and aggregatingmetrics
KOM – Multimedia Communications Lab 16
Queries in the Information Mgmt. System
Query Type: Capacity-based Peer Search Give me M peers Fulfilling specific requirements on
Bandwidth, storage space, computational capabilities, Online time, peer load, reputation … (wide set of requirements definable)
Query processing First sent to coordinator Query traverses bottom-up, until M matching peers found Result is sent then to requesting peer Tradeoff:
Upper peers in tree know more Load should be kept on lower levels of the tree
KOM – Multimedia Communications Lab 17
Very Simple Tree Maintenance
Join, Leave, Repair, Maintenance: Not needed Join: Just send an update to Coordinator After failure of “simple peer”: peer specific information times out After failure of Coordinator / Support Peer: route(msg, key) identifies new Coordinator
Coordinator picks a new Support Peer New responsible peer is updated in the next update interval
No additional maintenance needed (done by structured overlay)
KOM – Multimedia Communications Lab 18
Outline
Motivation Information Management in P2P Systems Example
SkyEye.KOM Requirements Architecture
Evaluation Performance Costs
Conclusion
KOM – Multimedia Communications Lab 19
Evaluation of SkyEye: Scalability and Information Freshness
Scalability Tree-structure of coordinators form
information architecture Supp. peers: Strong peers take the load Query performance: O(log N) hops
Freshness Freshness tightly related to tree depth Information freshness:
O(log N * updateInterval)
Conclusion: Information is fresh even under churn
Failing peers are quickly replaced It takes one update interval to recover
KOM – Multimedia Communications Lab 20
Evaluation of SkyEye: Quality of Information
Completeness of Knowledge Peers may define maximum load
Some information may be dropped Completeness of knowledge: Ratio of
considered peers in the corresp. domain All >90%, some decline at level 9
Quality of Information: Peers in the results: >98% are online Less useful peer information is dropped
Load on the Peers Constant update load (1 or 3 msg./intvl.) Either just leaf or Coordinator as well Most of the peers are at level 8-9 Information density is highest at level 8-9
Some information is dropped theredue to load limit of peers
KOM – Multimedia Communications Lab 21
Evaluation of SkyEye: Query Performance
Position of query resolving: Remember: most peers in level 8-9 Query takes 2-4 hops to be resolved Most of the queries resolved in level 4-5 Leaves room for optimization:
Injecting queries lower in the tree Adapting tree height to query load
Impact of query complexity Query complexity ranging from 5 to 15 Small impact on position of resolving peer Leaves room for optimization:
Resolving easier queries at less loaded peers
KOM – Multimedia Communications Lab 22
Outline
Motivation Information Management in P2P Systems Example
SkyEye.KOM Requirements Architecture
Evaluation Performance Costs
Conclusion
KOM – Multimedia Communications Lab 23
Conclusion and Future Work on SkyEye.KOM
Some functional requirements and features addressed: Gathering system statistics and capacity-based peer search solved Overlay-independency
Unified ID space Usage of DHT-function route(msg, key), lookup(key)
Supporting peer heterogeneity Load dispatching to Support Peers
Low overhead Relying on robust underlying DHT, no maintenance
Room for optimization Load-balancing
Several Support Peers per Coordinator: synchronization issues Scalability (# of info and peers)
Limitations on the information size Adapting to usage patterns
Injects queries and updates on various levels in the tree
KOM – Multimedia Communications Lab 24
Questions?