![Page 1: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/1.jpg)
Two Samples are Enough:Opportunistic Flow-level Latency Es-
timation using NetFlow
Myungjin Lee†, Nick Duffield‡, Ramana Rao Kompella†
†Purdue University, ‡AT&T Labs–Research
![Page 2: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/2.jpg)
2
Per-hop Measurements are important
R3R2
R1IPTV/VoIP/VoD
Server
Which router causes the prob-lem??
100 ms
AS 2
AS 1
Why 100 ms?!
![Page 3: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/3.jpg)
3
Aggregate vs. Per-Flow
R3R2
R1IPTV/VoIP/VoD
Server5 ms10 ms
AS 2
AS 1
Why 100 ms?!
flow-level latency measurements on a per-hop basis
Aggregate la-tencies look all right. Why?
![Page 4: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/4.jpg)
4
Existing Approaches Active probes and tomography
Chen et al. [SIGCOMM’04], Duffield et al. [IMC’03] Problems
Problem formulation is under-constrained No per-flow latency measurements
Lossy Difference Aggregator (LDA) Kompella et al. [SIGCOMM’09] Problems
Require hardware modification No per-flow latency measurements
![Page 5: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/5.jpg)
5
Basic Framework: NetFlow A measurement framework widely deployed in
routers Maintains per-flow state in the form of a flow record
Packet and byte counts Flow duration (flow start and end timestamps)
Usage Normally used for accounting, traffic matrix estimation,
etc. Does not support per-flow latency measurements
Goal: Enable per-flow latency estimation Harness flow start and end timestamps in NetFlow
framework
![Page 6: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/6.jpg)
6
Obtaining Two Delay SamplesFlow
IDStart
TSEnd TS
1 1
B
A
Flow ID
Start TS
End TS
2 2
12
Flow ID
Start TS
End TS
2 4−
−
=
= Delay2
Delay1
Two delay samples / flowEnd TS
4Start
TS2
R2R1
Flow ID
Start TS
End TS
1 2
End TS2
Start TS1
R3R2
R1IPTV/VoIP/VoD
Server
AS 2
AS 1
![Page 7: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/7.jpg)
7
Problem 1: Independent Packet Sam-pling
Flow ID
Start TS
End TS
1 1
B
A
Flow ID
Start TS
End TS
4 4
12
Only up-date
1st packet
Only up-date2nd
packet
R2R1No coordination between NetFlow instances
![Page 8: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/8.jpg)
8
Solution: Hash-based SamplingFlow
IDStart
TSEnd TS
1 1
B
A
Flow ID
Start TS
End TS
2 2
12R2R1
Hash Space
Sampling Space
Hash-based sampling achieves coordination
Sampled at both NetFlow
Instances
Not sampled at both Net-
Flow In-stances
1 2
![Page 9: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/9.jpg)
9
Problem 2: Packet LossFlow
IDStart
TSEnd TS
1 1
B
A
Flow ID
Start TS
End TS
2 2
12
Flow ID
Start TS
End TS
1 2
R2R1
X
Updateboth
packets
Only up-date
1st packet
Packet losses may cause inconsistencies
![Page 10: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/10.jpg)
10
Solution: Packet DigestsFlow
IDStart
TSEnd TS
1 1
B
A
Flow ID
Start TS
End TS
2 2
Flow ID
Start TS
End TS
1 2
Start PD
End PD
0x01 0x01
Start PD
End PD
0x01 0x01
Start PD
End PD
0x01 0x02
12
X Detect un-usable
timestamp
Detect unus-able
timestampR1 R2Packet digest achieves packet association
![Page 11: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/11.jpg)
11
Consistent NetFlow (CNF) Architecture Issue I: No coordination between NetFlow instances
Different packets are sampled on different NetFlow instances Solution: Hash-based sampling (filtering) in RFC 5475
Same packets are selected on different NetFlow instances IETF PSAMP working group
Issue II: Packet losses Discrepancy in selected packets due to packet losses
Solution: Maintaining packet digests Hash of the invariant packet contents Use timestamps iff packet digests match at the two routers
![Page 12: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/12.jpg)
12
Trivial Estimator: Endpoint Use two delay samples belonging to the same
flow Obtain accurate latency estimates for small flows
Problem: Accuracy penalty for large flows Solution: Multiflow estimator
Time
Endpoint estimator Avg ( , )=
Flow ID
![Page 13: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/13.jpg)
13
Better Estimator: Multiflow Key insight: Packets experiencing same queu-
ing busy periods will experience similar delays Use background delay samples from other flows Use only delay samples between the start and end
of a flow
Time
Multiflow estimator Avg ( , , , )=
Flow ID
![Page 14: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/14.jpg)
14
Evaluation Simulation Setting
Endpoint vs. Multiflow estimators
Comparison with Trajectory sampling
![Page 15: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/15.jpg)
15
Simulation Setting Modified YAF
Simulate a queuing model with RED active queue management policy
Dataset CHIC trace
1 min. trace collected from an OC-192 backbone link About 13M packets and 1M flows
![Page 16: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/16.jpg)
16
Multiflow vs. Endpoint Estimators
1 10 100 1000 100000
0.10.20.30.40.50.60.70.80.9
1MultiflowEndpoint
Flow size
Med
ian
rela
tive
err
or o
fde
lay
mea
n es
tim
ates
Endpoint obtainsgood accuracy
Multiflow performsbetter than Endpoint
![Page 17: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/17.jpg)
17
Trajectory Sampling Shares some similarity with CNF architecture
Routers use consistent hash function to sample packets
Facilitates direct observation of packet trajectories
Requires flow ID and timestamps for per-flow latency estimation Aggregate all sampled packets with same flow key Compute their average latency
![Page 18: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/18.jpg)
18
Comparison with Trajectory Sampling
0.001 0.01 0.1 1 100
0.10.20.30.40.50.60.70.80.9
1MultiflowTrajectory
Relative error of delay mean estimates
CDF
Packet sampling rate = 0.01
Multiflow is 2-3xbetter than Trajectory
![Page 19: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/19.jpg)
19
Summary Our approach retrofits per-flow latency esti-
mates in the NetFlow framework
Two main ideas Consistent NetFlow architecture ensures that dif -
ferent routers record the same set of flows
Multiflow estimator achieves significantly accurate estimates of per-flow latencies compared to prior approach
![Page 20: Two Samples are Enough: Opportunistic Flow-level Latency Estimation using NetFlow](https://reader036.vdocument.in/reader036/viewer/2022062811/56816133550346895dd08a7d/html5/thumbnails/20.jpg)
20
Questions?