State-of-the-Art ofInternet Traffic

Measurement and Analysis

The 31st APEC TEL WG MeetingApril 5th, 2005

Bangkok, Thailand

Sue B. MoonDivision of Computer Science

KAISTSouth Korea

2

Overview

• Brief Historical Overview• Evolution of Measurement Techniques• Status Quo of Measurement Techniques• Future Work

3

Brief Historical Overview

• 1970s and 1980s– Performance was not an issue– Very few papers about performance– “ping” and “traceroute” were only tools

• 1990s– Internet exploded– Lack of measurement/analysis/visualization tools sorely felt– Measurement became important in research

• 2000s– Competition between ISPs became intense– Service Level Agreements (SLAs) became critical– Security became sales point

4

Evolution of Measurement Techniques

• Internet Design Philosophy• Basic “ping” and “traceroute”• Vern Paxson’s Work• My Personal Perspective

5

Internet Design Philosophy

• Packet switching• Continued communcation around

failures• Support for diverse services and

protocols• Distributed management of resources• No access control• Simplicity at the core, complexity at

the edge

6

The Internet Hourglass (Deering@IETF)

IP

TCP, UDP, ...

SMTP, HTTP, RTP, ...

email, WWW, phone, ...

ethernet, PPP, ...

CSMA, Sonet, ...

Copper, fiber, radio, ...

7

What is the Internet today?

UUnet

Sprint (AS)

BBN

Dial-up ISP

Tier 2 ISP

BT

Peering point

8

Internet Users Different from PSTN Users

• ISPs– Too much & diverse traffic to monitor– Hard to get a complete picture– Routers barely keep up with core tasks

• End-users– More options than traditional telco customers

9

ping

• ICMP-based tool for host reachability • Algorithm

– Sends an ICMP echo request with:• Identifier for unique ping process• Sequence number per echo request

– Receiving host returns an ICMP echo reply– Prints out RTT, TTL, and seq. #.

• Issues– Many routers filter out ICMP packets– It goes thru slow path on routers– RTT includes end system processing time

10

traceroute

• Used to find out the forward path to a host• Algorithm

– Send an IP datagram with TTL=1– First router sends back ICMP time exceeded– Then send a datagram with TTL=2– Continue till destination is reached/TTL expired

• Issues– not suited for performance measurements

11

Vern Paxson’s PhD Thesis

• Many findings about Internet Performance– Delay– Loss– Unexpected routing behaviors

• route changes, flaps,

– Clock synchronization– Incomplete logging

12

Paxson’s Tools

• Instrumented “ping”s– Send packets between a set of nodes

• In today’s Internet– Active measurements for performance monitoring– Passive measurements for control-domain monito

ring

13

Passive Measurement

• No traffic injected for measurement purpose– Not invasive

• Only data collection increases traffic

– Access limited• Measurement about total traffic• Privacy/Security - serious concern

14

Passive Measurement Examples

• Packet monitors– Tcpdump for Unix-based hosts– Dedicated measurement systems

• DAGMON (up to 10GE)

• Router/switch traffic statistics– Network internal behavior– SNMP MIBs– Flow-level information

• Cisco’s NetFlow, Juniper’s Accounting, Arbor’s PickFlow

15

Packet-Level Measurements

• Pros :– very fine granularity

• Challenges :– link speeds are increasing!– Large volumes of data– system design issues:

• disk/PCI bus speeds• installation cost

16

Challenges in Data Collection

• On 1GE link– # of flows per sec = 100K ~ 1 mil– 1KB per flow => 1GB per sec

• On 10GE link– # of packets per sec = 10 mil ~ 200 mil– 2GHz processor => 10 cycles

• You need 10 GE link to monitor 10 GE link!

17

Why Sampling/Filtering?

• Problems with large volumes of data– feasibility of collection at high-speeds

• memory/bus/processor requirements

– storage limitations– complexity of analysis

18

State-of-the-Art

• Cisco– sampled netflow

• capture 1 in N • aggregate by five-tuple

• Juniper – filter on any combination of header fields– sample 1 in N

• recommends 1 in 1000 or less

• How much data do you collect when N = 1000?

19

Personal Experience at Sprint

• When I first arrived, I heard …– “No loss” on Sprint backbone network– “Almost no delay”– “Cadillac brand of IP service”

20

Min/Avg/Max Single-Hop Delay per Minute

21

Single-Hop Delay w/o Cisco Router Idiosyncracies

22

Multi-Hop Delay Distributions

Data Set 3

23

Three Paths Connectivity

• Data Set 3

28ms32ms34ms

Fiber prop.delay

24

Data Set 3

Identification of Constant Factors: Multi-Paths

• Equal Cost Multi Paths (ECMP)– Src/Dst addresses, Router ID

Min delay of src/dst flow (Data Set 3)

Path 1

Path 2

Path 3

25

Peaks in Variable Delay

26

Closer Look

• Queue Build up &Drain

27

Issues in "Good" Routing

• Misbehaving routing protocols– BGP misconfigurations– Pathological behaviors– Frequent changes

• Even under normal circumstances– Transient behaviors– Inter/intra-domain routing not well understood

28

Routing Across Internet

• Protocols– Interior Gateway Protocols (IS-IS, OSPF, RIP)– Exterior Gateway Protocols (BGP)

• How they work– IGP : find “best” (shortest) path across a domain– BGP : announce reachability between domains

• policy determines inter-domain paths

29

Routing Research Projects

• Routeviews– 50+ peering at route-views.oregon-ix.net– MRT format RIBs and BGP updates, “show ip bg

p” dumps, route dampening data– only E-BGP

• RIPE (Réseaux IP Européens)– routing updates from 9 mostly European IXs– “Looking Glass” services for BGP– Routing information service (RIS)

30

Scenario for a Transient Routing Loop

In Normal Operation

31

When a link fails, R1 is the first to detect.

32

R3 is updated before R2.

33

Finally R2 is updated, and the loop is resolved.

34

CDF of Routing Loop Duration in Time

35

VoIP experimental setup [Boutremans2002]

• Traffic injected in the network: – 200 byte UDP packets– every 5ms.

• Packets captured and timestamped at end-systems.

• Traceroute runs continuously during the experiment.

• Induced link failures on purpose to evalute convergence time and impact on e2e connections

36

Information Sources

• IS-IS & BGP listener logs • Router logs from both ends of

“failing” links• Controlled bi-directional VoIP traffic

between Reston and ATL• SNMP data

37

Delays (1 sec timescale)

2 links down

~3.4ms

2 links up

~2.6ms

3 links down

3 links up

38

When the two interfaces went down …

6.6 seconds

39

When three links came back up

For 30 secs packets follow a shorter path

Traffic “black-holed”for 1.745 seconds

Traffic “black-holed”for 0.975 seconds

40

Approaches To Fix It

• Fine-tuning parameters– Timer values [Alattinoglu2002]

• Modify Routing Protocols– Suppress advertisement and perform local rerouting using

a backwarding table [Lee04]– Centralized path computation [Feamster04,Rexford04]

• Exploit multi-path– Our approach to provide Value-Added Service

41

What I have learned …

• No loss, almost no delay– Almost. I gained insight into causes behind

• Debunking the myths [Odlyzko2005]– Streaming real-time traffic– QoS– Content is king– Usage-sensitive pricing

42

Other Issues Tackled

• Traffic Matrix Estimation– Inspired by tomography in other fields– Before arrival of efficient NetFlow

• Network Anomaly Detection– NIDS, IDS => PCA-based global monitoring

• Optimization– Cross-layer resource allocation

43

Taxonomy of Traffic Matrices

• Point-to-Point – demand btwn ingress and egress point

• Ingress/Egress : POP, link, router, BGP prefix

44

Scalability

• Example : 20 POPs, 500 routers, 3K links• Granularity/size tradeoff

– POP-to-POP : O(100) – router-to-router : O (104) – prefix-to-prefix : O (1010)

• Challenge - Collecting, storing and manipulating large TMs!

45

Usage Based Charging

• Feasible?– Where to measure?

• At last hop

– Scenario• A: “I want to download B’s webpage”• B: “That page is 1MB large”• A: “OK”

– Between ISPs• What do you do with retx, ack, delay?

46

Future Work

• In Measurement Technology– Keep up with increased link speed 40GE– Improve sampling techniques– Infer what we cannot measure– Pinpoint security holes

• Personal perspective– More into creating value-added services– MPLS/VPN performance issues

• “Sound” Measurement Infrastructure

47

Acknowledgements

• Thank D. Papagiannaki, B.-Y. Choi, U. Hengartner, C. Boutresmans, and G. Iannaccone for help with the slides.

BACKUP SLIDES