Internet Traffic Demand and Traffic Matrix Estimation
Challenges in directly measuring traffic demand or traffic matrix granularity and time scale of traffic demand matrix ? Focus mainly on two studies representing two approaches Partial (or sampled) measurement at ingress/egress points/links (optional material: will go over only briefly) Inference of traffic matrix based on link loads (aggregate SNMP link load measurement) gravity model tomogravity model (optional material) Readings: Please do the required readings CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Traffic Demands How to measure and model the traffic demands? Know where the traffic is coming from and going to Why do we care about traffic demands? Traffic engineering utilizes traffic demand matrices in balancing traffic loads and managing network congestion Support what-if questions about topology and routing changes Handle the large fraction of traffic crossing multiple domains Understanding traffic demand matrices are critical inputs to network design, capacity planning and business planning! How to populate the demand model? Typical measurements show only the impact of traffic demands Active probing of delay, loss, and throughput between hosts Passive monitoring of link utilization and packet loss Need network-wide direct measurements of traffic demands How to characterize the traffic dynamics? User behavior, time-of-day effects, and new applications Topology and routing changes within or outside your network 1. Model Considering aggregates of flows; not individual TCP flows We want to model traffic at a level where we can ask what if questions.What is the impact if topology changes?What if internal routing changes? Want to handle the large proportion of traffic that crosses domain boundaries. 2. Populating the Model What aboutthe production systems for network measurements (SNMP for utilization and other coarse statistics, active probes (e.g., ping) for delay and related statistics?No, measuring impact of demands, not demands. 3. Traffic characterization Of interest to understand the demands, and the dynamics of the demands.What are the sources of fluctuations?Inherent in the traffic?Inherent in the network? CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Traffic Demands Big Internet Web Site User Site [This slide is animated] Why the point to multipoint model? 1. Start simple! Consider a point to point flow from a web site to user site flow across the big internet. [Click to animate.] It would be intriguing to try to understand traffic at the host to host or network address to network address level. It might have been possible in the early days of the Internet, before # of hosts exploded, and before the complexity of the Internet cloud itself exploded. 2. Today: The situation is much more complex, and is heavily influencedby routing across the multiple domains. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Traffic Demands Interdomain Traffic Web Site User Site AS 1 AS 2 AS 3 AS 4 U AS 3, U What path will be taken between ASs to get to the User site? Next: What path will be taken within an AS to get to the User site? AS 4, AS 3, U [This slide is animated.] 1. AS = autonomous system. Internet today consists of 6 to 7K ASs. ASs representing the bigplayers are richly connected to one another. ASs share no internal views of topology with one another. ASs do share information needed to compute cross domain routes, via BGP. 2. In the example, BGP enables the flow from the web site to the user site to happen.[Click to animate this.In the example, AS 2 (not unwilling to provide transit service to U) drops out.AS 1 has two routes to U. All other things being equal (such as local preferences reflectingfinancial relationships between the parties), the shortest AS path will be taken, as shown in the example.] 3. Important point:After BGP does it thing: some of the border routers and corresponding links at AS 1 will be left standing as candidates to get to the user network address U.Which will be chosen? As well see next, this depends on the IGP, or internal routing protocol. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Traffic Demands Web Site User Site Zoom in on one AS 200 110 10 300 25 75 50 IN OUT3 OUT2 OUT1 Change in internal routing configuration changes flow exit point! 110 [This slide is animated.] 1.Terminology: IN is the input link.OUT1, OUT2, OUT3 are the border routers left standing by the BGP routing decision process:all are candidates for the flow out of the AS to U.In particular, the forwarding tables of each of these routers will contain next hops for U. Which route will be taken from IN1 to one of the three candidates to get to U? Depends on IGP: e.g., OSPF or IS-IS. Take shortest path, according to static weights. [Click to animate.] In the example: OUT3 is selected, and the traffic will follow a shortest path to OUT3. Suppose that as a result of TE logic (or whatever), change weight of the lower right link.Now, the traffic pops out at a completely different place: OUT2.] Something very non-linear has occurred. A change to the infernal routing configuration, changes the flow.Had a demand from this point (IN) to that point (OUT3).Now have a demand from this point (IN) to a different point (OUT2). However, the flow is still offered to the user site, which from the point of view of this AS means to one of OUT1, OUT2, or OUT3. From the point of view of the AS, the demand is point to multipoint CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Defining Traffic Demand Matrices
Granularity and time scale: Source/destination network prefix pairs, source/destination AS pairs ingress/egress routers, or ingress/egress PoP pairs? Finer granularity: traffic demands likely unstable or fluctuate too widely! 1. Methodology Measure at the edges (volumes, input link). Determine where the traffic leaves (reachability information: output links) Join the information to calculate the set of traffic demands, over a given time interval. 2. The basic measurement sources we used: Cisco NetFlow for flow level measurement, and forwarding tables for reachability information. These were readily available to us.Other options may be more appropriate for other settings; e.g., use of packet samplinginstead of NetFlow. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation 6 6 CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Traffic Matrix (TM) Point-to-Point Model: T: = [Ti,j ], where Ti,jfrom an ingress point i to an egress point j over a given time interval ingress/egress points: routers or PoPs an ingress-egress pair is often referred to as an O-D pair Point-to-Multipoint Model: Sometimes it may be difficult to determine egress points due to uncertainty in routingor route changes Definition: V(in, {out}, t) Entry link (in) Set of possible exit links ({out}) Time period (t) Volume of traffic (V(in,{out},t)) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Ideal Measurement Methodology
Measure traffic where it enters the network Input link, destination address, # bytes, and time Determine where traffic can leave the network Set of egress links associated with each network address (forwarding tables) Compute traffic demands Associate each measurement with a set of egress links Even at PoP-level, direct measurement can be too expensive! We either need to tap all ingress/egress links, or collect netflow records at all ingress/egress routers May lead to reduced performance at routers large amount of data: limited router disk space, export Netflow records consumes bandwidth! Either packet-level or flow-level data, need to map to ingress/egress points, and a lot of processing to generate TM! 1. Methodology Measure at the edges (volumes, input link). Determine where the traffic leaves (reachability information: output links) Join the information to calculate the set of traffic demands, over a given time interval. 2. The basic measurement sources we used: Cisco NetFlow for flow level measurement, and forwarding tables for reachability information. These were readily available to us.Other options may be more appropriate for other settings; e.g., use of packet samplinginstead of NetFlow. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation 8 8 Adapted Measurement Methodology Inter-domain Focus
[F+01] Paper (Optional Material): Driving traffic demands from netflow measurements based on selected links A large fraction of the traffic is interdomain Interdomain traffic is easiest to capture Large number of diverse access links to customers Small number of high speed links to peers Practical solution Flow level measurements at peering links (both directions!) Reachability information from all routers Interdomain traffic is tremendously important.A large fraction of the traffic is interdomain. Interdomain traffic is easiest in some sense to capture because it is conveniently concentrated on a relatively small number of high speed links, called peering links. Customer traffic is spread over a relatively large number of links with relatively diverse speeds and characteristics. Basic approach: flow level measurements at the peering links,inboth directions.In the example, see the flow from the web site to the user site, and from the user site to the web site. reachability information: forwarding tables from all routers.What we do is: identify links as output links (to customers or peers), and collect the network addressesassociated with those links. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Measuring Only at Peering Links
Why measure only at peering links? Measurement support directly in the interface cards Small number of routers (lower management overhead) Less frequent changes/additions to the network Smaller amount of measurement data Why is this enough? Large majority of traffic is interdomain Measurement enabled in both directions (in and out) Inference of ingress links for traffic from customers Necessary and sufficient to measure at peering links for our purposes. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Inbound & Outbound Flows on Peering Links
Peers Customers Inbound Outbound [This slide is animated.] Click to animate. Terminology: inbound = from peer to customer; outbound = from customer to peer. Classify traffic at peering links. Note: Ideal methodology applies for inbound flows. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Full Classification of Traffic Types at Peering Links
Peers Customers Internal Inbound Outbound Transit [This slide is animated.] More detail than used in the talk.Shows transit and internal traffic.Traffic classified at peering links, including: transit traffic (occurs because of special case business relationships) between peers internal traffic (which we did not measure) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Identifying Where the Traffic Can Leave
Traffic flows Each flow has a dest IP address (e.g., ) Each address belongs to a prefix (e.g., /24) Forwarding tables Each router has a table to forward a packet to next hop Forwarding table maps a prefix to a next hop link Process Dump the forwarding table from each edge router Identify entries where the next hop is an egress link Identify set of all egress links associated with a prefix Some algorithmic details. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Flows Leaving at Peer Links
Single-hop transit Flow enters and leaves the network at the same router Keep the single flow record measured at ingress point Multi-hop transit Flow measured twice as it enters and leaves the network Avoid double counting by omitting second flow record Discard flow record if source does not match a customer Outbound Flow measured only as it leaves the network Keep flow record if source address matches a customer Identify ingress link(s) that could have sent the traffic More detail than in the slide given during the talk.Transit flows are seen twice on peering links.Do not want to double count, and do not want to match same flow seen at different links.So this requires some care to count, without double counting. Outbound: hardest case. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Most Challenging Part: Inferring Ingress Links for Outbound Flows
Outbound traffic flow measured at peering link Example output Use outing simulation to trace back to the ingress links! ? input Customers destination Backbone topology and the intradomain routing configuration at the time the flow was measured. [This slide is animated.] 1.For outbound traffic, measure traffic where it leaves.Where does it enter?Compute this in several steps:Consider a traffic flow measured flowing out ofa given peering link. A set of possible inputs might be responsible for the flow.Key component in computing this are the customer address blocks, obtained from configuration information (reflected in ingress filters). Which input is it?In the paper, we discuss disambiguation.Key idea is to use a routing model.See where traffic headed for the given destination will leave. Click to animate.In the example, blue flow leaves at a different destination than the one where the flow was measured, so we can rule out the lower input.In this way, we can narrow the set of possible inputs.If we get a single input, we have successfully identified the traffic demand. Left with the flow. Click to animate the flow icons (the two marked outputs (there may be others), and the upper input.) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Computing the Demands Forwarding Tables Configuration Files NetFlow SNMP NETWORK researcher in data mining gear Data Large, diverse, lossy Collected at slightly different, overlapping time intervals, across the network. Subject to network and operational dynamics.Anomalies explained and fixed via understanding of these dynamics Algorithms, details and anecdotes in paper! In practice, there are several other challenges, associated with collecting, cleaning big data in a big IP network. Issues include: The collection was in some cases not supported by production systems particularly true for NetFlow. Fluctuations in information over the interval in which it is collected.Intrinsic fluctuations in reachability occur (as the paper by Labovitz et al. illustrates).Second, the network itself changes; e.g., new interfaces are added, causing various measurement glitches. On inspection of the data where we saw anomalous results we were able to identify and in some cases correct for these problems. This buttressed our confidence in the methodology.See the paper! The guy with the hard hat is a composite view of the coauthors, in their data mining gear.Just kidding.. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Experience with Populating the Model
Largely successful 98% of all traffic (bytes) associated with a set of egress links 95-99% of traffic consistent with an OSPF simulator Disambiguating outbound traffic 67% of traffic associated with a single ingress link 33% of traffic split across multiple ingress (typically, same city!) Inbound and transit traffic (uses input measurement) Results are good Outbound traffic (uses input disambiguation) Results are pretty good, for traffic engineering applications, but there are limitations To improve results, may want to measure at selected or sampled customer links; e.g., links to, hosting or data centers. For 98% of all traffic (in bytes) successfully inferred a demand.Promising but does not necessarily mean we have the right demands.To check this, we used our routing model.Ofthe 98%, 95 to 99% consistentwith our OSPF simulation. This is for all traffic, inbound, output, and a third class of transit traffic between peers.Other consistency tests also successful. Outbound: via the disambiguation procedure, 67% disambiguated.Of the 33% remaining, we found about 15% disambiguated to border routers in the same physical location (city). Traffic from that set of routers takes essentially the same paths. Inboundtraffic: all contributed to the first bullet.Results are good. Outbound traffic: where cannot disambiguate consider measurements at certain customer-side links (e.g., hosting centers), responsible for heavy traffic. Otherwise, to get the last 15% accounted for in TE logic, would have to model distribution among the possible inputs. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Proportion of Traffic in Top Demands (Log Scale)
Zipf-like distribution. kth most popular demand contributes 1/k^c to the distribution for some c>=1. Saw this behavior across our data sets. Has been observed before at different levels of granularity: e.g., AS traffic matrices, or web content (e.g., by Mark Crovella). Zipf-like distribution. Relatively small number of heavy demands dominate. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Time-of-Day Effects (San Francisco)
Had a look at how traffic to the heavy demands fluctuates. Focus on three of the largest demands, which happened to be measured at the same router in San Francisco. 3 different time of day patterns, among three heavy demands, which we termed domestic consumer, domestic business, and international (consistent with Asia Pacific day). midnight EST midnight CST Heavy demands at same site may show different time of day behavior CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Discussion Distribution of traffic volume across demands Small number of heavy demands (Zipfs Law!) Optimize routing based on the heavy demands Measure a small fraction of the traffic (sample) Watch out for changes in load and egress links Time-of-day fluctuations in traffic volumes U.S. business, U.S. residential, & International traffic Depends on the time-of-day for human end-point(s) Reoptimize the routes a few times a day (three?) Stability? No and Yes Implications of Zipf-like distributions: Watch and plan for the relatively small number ofheavy demands. Suffices to sample traffic.In particular, pseudo-random or 1 in N packet sampling should suffice to pick these large demands up, in a scalable way. than todays NetFlow. Implications of time-of-day fluctuations. Time zone of network equipment isnot necessarily relevant. A given ISP is in the middle of the Internet.What matters is the time-zone of users whose traffic happens to flow over the link.Understanding this may suggest a time scale for reoptimizing routing. What about stability?Saw significant stability in the rankings of demands.After binning the demands by quantile saw little quantile jumping among the more popular demands between adjacent half hours, or corresponding half hours in consecutive days.Have just scratched the surface here.See the paper! CSci5221:Internet Traffic Demand and Traffic Matrix Estimation TM Estimation Using Link Loads
[M+02] Paper:TM estimation using SNMP link loads Available information: Link counts from SNMP data. Routing information. (Weights of links) Additional topological information. ( Peerings, access links) Assumption on the distribution of demands. TM Estimation => using indirect measurements (here link loads), solving an inference problem! Y: link load measurements, A routing matrix Given Y, solving for X, where Y=AX CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Terminology Y=AX c=n*(n-1) origin-destination (OD) pairs.
X: Traffic matrix. (Xj data transmitted by OD pair j) Y=(y1,y2,,yr ) : vector of link counts. A: r-by-c routing matrix (aij=1, if link i belongs to the path associated to OD pair j) Y=AX r< Infinitely many solutions! CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Three Existing Techniques
Key issue: linear equations under-strained! More (N^2) unknowns (X_{ij}s) than # of knowns Y_{l}s Linear Programming (LP) approach. O. Goldschmidt - ISMA Workshop 2000 Bayesian estimation. C. Tebaldi, M. West -J. of American Statistical Association, June 1998. Expectation Maximization (EM) approach. J. Cao, D. Davis, S. Vander Weil, B. Yu - J. of American Statistical Association, 2000 CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Linear Programming Objective: Constraints: CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Statistical Approaches
CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Bayesian Approach Assumes P(Xj) follows a Poisson distribution with mean j. (independently dist.) needs to be estimated. (a prior is needed) Conditioning on link counts: P(X,|Y) Uses Markov Chain Monte Carlo (MCMC) simulation method to get posterior distributions. Ultimate goal: compute P(X|Y) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Expectation Maximization (EM)
Assumes Xj are ind. dist. Gaussian. Y=AX implies: Requires a prior for initialization. Incorporates multiple sets of link measurements. Uses EM algorithm to compute MLE. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Comparison of Methodologies
Considers PoP-PoP traffic demands. Two different topologies (4-node, 14-node). Synthetic TMs. (constant, Poisson, Gaussian, Uniform, Bimodal) Comparison criteria: Estimation errors yielded. Sensitivity to prior. Sensitivity to distribution assumptions. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
4-node Topology CSci5221:Internet Traffic Demand and Traffic Matrix Estimation 4-node Topology Results
CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
14-node Topology CSci5221:Internet Traffic Demand and Traffic Matrix Estimation 14-node Topology Results
CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Marginal Gains of Known Rows
CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
New Directions Lessons learned: Model assumptions do not reflect the true nature of traffic(multimodal behavior) Dependence on priors Link count is not sufficient (Generally more data is available to network operators.) Proposed Solutions: Use choice models to incorporate additional information. Generate a good prior solution. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation New Statement of the Problem
Xij= Oi.ij Oi : outflow from node (PoP) i. ij : fraction Oigoing to PoP j. Equivalent problem: estimating ij . Solution via Discrete Choice Models (DCM). User choices. ISP choices. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Choice Models Decision makers: PoPs Set of alternatives: egress PoPs. Attributes of decision makers and alternatives: attractiveness (capacity, number of attached customers, peering links). Utility maximization with random utility models. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Random Utility Model Uij= Vij + ij : Utility of PoP i choosing to send packet to PoP j. Choice problem: Deterministic component: Random component: mlogit model used. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Gravity Modeling General formula: Simple gravity model: Try to estimate the amount of traffic between edge links. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Results Two different models (Model 1:attractiveness, Model 2: attractiveness + repulsion ) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Further Improvement: Tomogravity Model (Optional Material)
Two step modeling. Gravity Model: Initial solution obtained using edge link load data and ISP routing policy. Tomographic Estimation: Initial solution is refined by applying quadratic programming to minimize distance to initial solution subject to tomographic constraints (link counts). CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Highlights Router to router traffic matrix is computed instead of PoP to PoP. Performance evaluation with real traffic matrices. Tomogravity method (Gravity + Tomography) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Recall: Gravity Model General formula: Simple gravity model: Try to estimate the amount of traffic between edge links. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Generalized Gravity Model
Four traffic categories Transit Outbound Inbound Internal Peers: P1, P2, Access links: a1, a2, ... Peering links: p1,p2, CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Generalized Gravity Model
CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Tomography Solution should be consistent with the link counts. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Reducing the Computational Complexity
Hundreds of backbone routers, ten thousands of unknowns. Observations: Some elements of the BR to BR matrix are empty. (Multiple BRs in each PoP, shortest paths) Topological equivalence. (Reduce the number of IGP simulations) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Quadratic Programming
Problem Definition: Use SVD (singular value decomposition) to solve the inverse problem. Use Iterative Proportional Fitting (IPF) to ensure non-negativity. CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Evaluation of Gravity Models
CSci5221:Internet Traffic Demand and Traffic Matrix Estimation Performance of Proposed Algorithm
CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Comparison CSci5221:Internet Traffic Demand and Traffic Matrix Estimation CSci5221: Internet Traffic Demand and Traffic Matrix Estimation
Robustness Measurement errors x=At+ =x*N(0,) CSci5221:Internet Traffic Demand and Traffic Matrix Estimation