joint data streaming and sampling techniques for detection of super sources and destinations
DESCRIPTION
Joint Data Streaming and Sampling Techniques for Detection of Super Sources and Destinations. Qi (George) Zhao, Abhishek Kumar, Jun (Jim) Xu College of Computing, Georgia Institute of Technology Internet Measurement Conference 2005 Speaker: Yongming Chen on Dec. 07, 2006. Outline. - PowerPoint PPT PresentationTRANSCRIPT
Joint Data Streaming and Sampling Techniques for Detection of Super
Sources and DestinationsQi (George) Zhao, Abhishek Kumar, Jun (Jim) Xu
College of Computing, Georgia Institute of TechnologyInternet Measurement Conference 2005
Speaker: Yongming Chen on Dec. 07, 2006
Outline
Introduction Motivation AlgorithmsEvaluation Conclusion
Outline
Introduction Motivation AlgorithmsEvaluation Conclusion
Introduction
detect super sources/destinations at high link speeds (10 to 40 Gbps) in real-time super sources: source with a large fan-out fan-out: number of distinct destinations
data streaming: sequentially process every packet passing through a better alternative to sampling and monitoring
of high-speed links NO connection with multimedia streaming!
Outline
Introduction Motivation AlgorithmsEvaluation Conclusion
Motivation
detecting super sources and destinations is useful network monitoring and security hot-spot or flash crowds detection
traditional per-flow schemes cannot scale to high-speed links information lost in packet sampling FlowScan: maintain per-flow state with hash table
network data streaming with a small but well-organized data structure
Outline
Introduction Motivation Algorithms
Simple Scheme Advanced Scheme
Evaluation Conclusion
Simple Scheme
traditional hash-based flow sampling reduce the amount of incoming traffic,
e.g., a 10M pkt/s link can be processed in 400ns with 25% sampling rate
this approach will fail with traffic burst
filtering after sampling
Simple Scheme
Simple Scheme
at most one packet from each sampled flow need to be processed
need a umin to reduce estimation error typically set to w/2
extremely low storage complexity only 128KB SRAM is needed for OC-192 links
with 25% flow sampling
Advanced Scheme
Separation of counting and identity gathering: streaming module encodes fan-out information sampling module captures candidate source
On-line Streaming Module
Estimation Module
Fs : fan-out of source s
Ai, i=1 to k : the column indices we can obtain by hashing s with h1 to hk
Ti : the set of packets hashed into Ai
UTi : # ‘0’ bits in Ai
DTi : a fairly accurate estimator of |Ti|
Outline
Introduction Motivation AlgorithmsEvaluation Conclusion
Settings
packet header traces IPKS+ and IPKS- : OC192c USC : Los Nettos tracing facility at USC UNC : 1Gbps
flow label (<src_ip, src_port>, <dst_ip>) (<src_ip>, <dst_ip, dst_port>)
Simple Scheme
Advanced Scheme
Outline
Introduction Motivation AlgorithmsEvaluation Conclusion
Conclusion
combine the power of data streaming and sampling to perform efficient and accurate detection
but no comparison with other approaches in evaluation
Reference
Q. Zhao, A. Kumar, J. Xu. Joint Data Streaming and Sampling Techniques for Detection of Super Sources and Destinations. Internet Measurement Conference 2005.
Q. Zhao, A. Kumar, and J. Xu. Joint data streaming an sampling techniques for detection of super sources and destinations. In Technical Report, July 2005.
FlowScan: http://www.usenix.org/events/lisa2000/full_papers/plonka/plonka_html/