PoiRoot: Investigating the Root Causeof Interdomain Path Changes

ABSTRACTInterdomain path changes occur frequently. Because routingprotocols expose insufficient information to reason about allchanges, the general problem of identifying the root causeremains unsolved. This has implications for large serviceproviders, cloud operators, ISPs, networking researchers andprotocol design.

In this work, we design and evaluate PoiRoot, a real-timesystem that allows a provider to accurately isolate the rootcause (the network responsible) of path changes affectingtheir prefixes. First, we develop a new model describingpath changes and use it to provably identify the set of allpotentially responsible networks. We then use empiricallyvalidated assumptions about routing policy to further restrictthis set. Next, we develop a recursive algorithm that accu-rately isolates the root cause of any path change. We observethat the algorithm requires monitoring available-but-inactivepaths that are generally not visible using standard measure-ment tools. To address this limitation, we combine exist-ing measurement tools in new ways to acquire path informa-tion required for isolating the root cause of a path change.We evaluate PoiRoot on path changes obtained through con-trolled Internet experiments, simulations and ‘in-the-wild’measurements. We demonstrate that PoiRoot is highly ac-curate, works well even with partial information, and gen-erally narrows down the root cause to a single network ortwo neighboring ones. On controlled experiments PoiRootis 100% accurate, as opposed to prior work which is accu-rate only 61.7% of the time.

1. INTRODUCTIONInternet paths change frequently, as links fail or become

available and as traffic engineering and policies evolve. Al-though many of these changes are benign, some can seri-ously disrupt the performance or availability of distant net-works and services. This can manifest as unwelcome changesin flow volumes over ingress links, longer round-trip times,less available bandwidth, and loss of connectivity [17, 41].

These changes in performance and availability can be costlyfor service providers. Amazon found that every additional100ms of delay in loading a web page costs them 1% of theirsales [22]. Similarly, a Yahoo! study found that, when a page

took an additional 400ms to load, the latency caused 5-9%of users to browse away, rather than letting the page load tocompletion [36]. Outages at large data centers cost an aver-age of $5,000 per minute [33]. Previous work demonstratesthat these providers can work with remote networks to re-solve problems [19], but doing so requires knowing whichnetwork to contact. Taken together, these facts suggest thatcontent and service providers would like to quickly under-stand what caused a path change that impacts their clients’performance.

There are other reasons to understand the root cause ofa path change. Research has established that Internet rout-ing policies can create unstable configurations [12, 13] andthat some paths experience frequent updates [21], makingit important to understand the underlying causes of theseupdates. Networks may want to avoid providers or pathsbased on an historical understanding of which ISPs causemany changes [2]. Routing policies have a significant im-pact on performance [35], and understanding the causes ofpath changes and selections provides some visibility into thepolicies in use [23]. An understanding of how path changesripple through the Internet could prove helpful in develop-ing new protocols for routing, for distributing routing up-dates [14], or for moving toward a more stable Internet.

The goal of this work is to accurately isolate the root causeof a path change within minutes of it happening. For ourpurposes, the root cause is the network or router whose linkavailability or policy adjustment initially triggered the se-quence of route updates leading to the path change in ques-tion. Operators can use this information to debug and ad-dress performance problems as they occur, and it could even-tually drive a system that automatically reroutes traffic toimprove performance (similar to LIFEGUARD [18]).

Previous work provided initial inroads towards our goal [2,9], but opaque policies and poor network visibility limit ourability to understand the root cause of path changes and howthey propagate through the Internet. In particular, we iden-tify the following open challenges that these limitations present.

First, interdomain routing policies can interact in complexways, so the network responsible for a change may appearneither on the old path nor the new path [9]. Existing algo-rithms assume that the change is on the old or new path [2,

1

9], and they cannot properly account for these induced pathchanges. Second, because path changes can have cascadingeffects, proper root cause identification can require monitor-ing a potentially large number of paths, many of which areinvisible to traditional measurement vantage points such aspublic BGP feeds and traceroute servers. Third, little to noground truth information is available to validate assumptionsabout Internet routing made by root cause identification al-gorithms. Our results from controlled experiments on realInternet paths demonstrate that these challenges and otherscause an existing approach [9] to correctly identify the rootcause in only 62% of path changes.

This paper describes how we address these issues to builda system, which we call PoiRoot [4], that allows a provider toquickly and accurately isolate the root cause of path changesaffecting their clients. We derive new constraints on how in-terdomain path changes propagate and develop an algorithmto accurately isolate the root cause. We then deploy an exten-sive measurement system and demonstrate that our approachis both precise (i.e., produces a small set of suspected rootcauses) and accurate (i.e., the suspect set always includes theroot cause of a path change) for real Internet paths. Our maincontributions are as follows.

First, we identify a provable upper bound for the set ofpaths that must be monitored to identify the root cause ofan interdomain path change. Because this set might be pro-hibitively large for an arbitrary change, we use a simple, yeteffective, routing model to refine the set to improve scalabil-ity. Our bound reduces the average number of ASes we needto monitor to perform root cause analysis by up to 65%.

Second, we combine existing measurement tools in newways to acquire path information required to isolate the rootcause of a path change. In addition to public BGP feeds, weuse BGP prepending on prefixes we control to reveal alter-native, less preferred, paths toward our prefixes from distantASes. We then use data-plane active measurements (tracer-outes and reverse traceroutes [16]) toward our prefixes toidentify the paths before and after a given path change. Inour experiments, we are able to collect the necessary pathinformation within minutes of a path change, allowing ourapproach to quickly perform root cause analysis.

Third, we develop an algorithm – based on our model andusing our measurements – that allows us to isolate the rootcause of a path change, even with partial routing informa-tion. We use controlled experiments on real Internet pathsto demonstrate the effectiveness of our approach. In partic-ular, we use BGP announcements to cause path changes forprefixes we control, then validate our algorithm using thisground-truth information. We show that PoiRoot accuratelyidentifies the root cause of a path change in every case inour experiments, including induced path changes that previ-ous approaches do not address. Finally, we show that pathchanges in the wild cause significant performance problems,and that PoiRoot is able to identify small suspect sets con-taining the network responsible for them.

The rest of the paper is organized as follows. In the fol-lowing section, we provide background related to the prob-lem of root cause analysis and motivate PoiRoot by describ-ing limitations of previous work. In §3, we develop a generalalgorithm for identifying the root cause of an arbitrary pathchange, then use a simple model of BGP path changes to de-rive the set of paths that must be monitored to identify theroot cause. In §4, we describe the design of a measurementsystem to gather this path information. §5 uses experimentson real Internet paths, simulations, and “in the wild” pathchanges to demonstrate the effectiveness of our approach.We discuss related work in §6 and conclude in §7.

2. BACKGROUND AND CHALLENGESIn this section, we discuss closely related work and how it

fails to address several open challenges for root cause anal-ysis. We discuss other related work in §6.

2.1 NOOR ApproachThe first general approach to identify the AS that origi-

nates a routing change in the Internet, i.e., the root cause,was proposed by Feldmann et al. [9]. The key idea behindthis algorithm is that the root cause of a path change ob-served at an AS is either on the old path used before thechange or the new path used after the change. We refer tothis approach as NOOR, because it identifies changes on theNew Or Old Route. (We use the terms ’route’ and ’path’ in-terchangeably in this paper.) The algorithm then combinessimultaneous path change observations from different ASesand builds a suspect set containing the ASes that appear inall paths that changed and that never appear in paths that didnot change. Although not evaluated explicitly, to reduce theset of suspect ASes further, the algorithm assumes that theroot cause is on the most preferred path (otherwise AS Awould keep using the preferred path and there would be nochange).

The authors use simulation to inform their design, thenevaluate their approach using experiments on real-world datawith BGP beacons announcements for ground truth. In thiscontext, Feldmann et al. were able to isolate the origin of aBGP path change to a single AS or inter-AS link for 76% ofthe cases they study. Applying further heuristics, this is thecase for 96% of the cases. While these results may suggestthat we can close the book on root cause analysis, we list sev-eral challenges below that may impact the their algorithm’saccuracy and precision.

2.2 Open Challenges

Induced Path Changes. The root cause of a path changeobserved at an AS v may lie outside the old and new pathsused by v. Feldmann et al. [9] recognize this challenge butdid not have an approach to address it. Fig. 1 shows an ex-ample network topology where an induced change happens.The solid arrows show the paths ASes chose toward s beforethe link between ASes d and z failed. The dashed arrows

2

S VY

X

Z

E

AS with path

change

old path

new pathD

A B

C

Ø

2

22

1

MN

unused link

1

Figure 1: Example of an induced path change at v afterthe link d–z fails. Numbers on edges show the local pref-erence (LocalPref) of a peer, higher is more preferred.

show the path ASes chose to AS s after the failure. ASeswith one outgoing arrow (solid circles) did not change pathsdue to the failure. Note that, before the failure, AS y prefersthe longer route through z (LocalPref 2, higher is more pre-ferred) than the shorter route through e (LocalPref 1). Whenthe failure happens, AS z cannot find a path to s and with-draws its path. AS y then changes to the shorter, less pre-ferred route through e. When y announces the new routeto v, AS v changes to the new route [y, e, s] because it isshorter than the route through x. Note that the root cause,i.e., the link between ASes d and z, does not lie in either theold or the new path used by AS v. NOOR cannot identifythe root cause of induced path changes; at best, it would findthe AS where the change was induced, e.g., AS y. More-over, because NOOR correlates changes across all observa-tion points, it suffices that one observation point experiencesan induced path change to make identification impossible.We address such cascading path changes in §3.1.

Extensive Path Monitoring. BGP information about pathchanges collected passively [28, 38] does not propagate farin the Internet. An AS that receives updated paths fromdifferent downstream neighbors will forward only its bestpath upstream. One consequence is that BGP and traceroutemeasurements from a small number of vantage points areinsufficient. We potentially need BGP measurements fromeach AS in the network [37], or techniques such as ReverseTraceroute [16] that provide path measurements from arbi-trary ASes, to have complete information to perform rootcause analysis. We determine the set of required measure-ments in §3.2 then use a simple routing model to bound thisset in §3.3.

Inferring Route Preferences. When an AS switches frompath P to P ′, it may be unclear which of the two paths itprefers. Feldmann et al. proposed a heuristic that consid-ers the shorter path the preferred path; however, in prac-tice inter-AS business relationships (i.e., policy routing) takeprecedence over path length in the route selection process.Incorrect assumptions about routing policies can prevent rootcause analysis from identifying the correct AS responsiblefor a path change. We describe our preference inferencemechanism in §3.3.

Measurement Granularity. Most previous work focuseson isolating the root cause of path changes to an AS; how-ever, this coarse grained analysis can be insufficient for two

reasons. First, different routers in the same AS may use dif-ferent routes toward the same prefix. For example, a router inEurope is likely to choose different routes than a router in theUS, even if they belong to the same AS. Second, when de-bugging performance or connectivity problems, fine-grainedisolation of the root cause (e.g., at the level of point-of-presence or router IP) can speed diagnosis and repair by fo-cusing limited operator resources on a narrow region of thenetwork.

3. GENERAL ROOT CAUSE ANALYSISIn this section we frame the general problem of finding

the root cause of an arbitrary interdomain path change. Forthe sake of exposition, we use an AS as the basic routingelement. This can be easily generalized to finer granularity(e.g., PoPs). First we develop a simple, yet generic, set ofrules for reasoning about a path change and present a recur-sive algorithm for isolating the root cause when all availablerouting paths toward a prefix are known (§3.1). Because wegenerally have only an incomplete view of those paths, weprovide a proof as to what is the upper bound on the setof ASes that can be the root cause (§3.2). We observe thatthe number of paths to monitor to obtain this subgraph canbe large for arbitrary path changes. To address this, we usecommonly accepted assumptions about routing decisions tofurther restrict the set of monitored paths (§3.3).

3.1 Root Cause Analysis Rules and AlgorithmWe now consider root cause identification for a change

observed on the path from an AS v towards an AS s thatoriginates a prefix. In the following discussion we isolatethe root cause at the granularity of ASes, but our approachmakes few assumptions about routing and can equally ap-ply to other grouping such as quasi-routers [29]. We assumethat at any time there is only one routing event causing apath change, and that we have complete and accurate pathinformation. We revisit these assumptions in later sections.We also make the reasonable assumptions that (i) an AS vchanges paths either due to modifications in its route pref-erence or due to a change in one of its neighbor’s paths orexport policies; and that (ii) if v did not change its next hopto s nor its export policy, it is not responsible for the routechange. Let O(v) and N(v) denote the old and the new pathsAS v used before and after the event, respectively. We iden-tify the following four rules for root cause analysis.1

Rule 1 Let O(v) = N(v) = [x, . . . , s]. Then x and down-stream ASes are not the root cause of the event. AS v canstill be the root cause if it changed its export policy (i.e.,started or stopped announcing the route upstream).

Rule 2 Let O(v) = [x, . . . , s] 6= N(v) = [y, . . . , s], O(x) =N(x), and O(y) = N(y). Then the root cause is the set

1Even though the text focuses on the sequence of ASes in a path,comparison of paths includes other relevant data such as BGP com-munities and multi-exit discriminators when available.

3

Algorithm 1 Recursive algorithm for general root causeanalysis using rules 1–4.Precondition: A path change observed on the path from vto s. Let O(v) = [v, x, . . . , s] and N(v) = [v, y, . . . , s].RootCause(v, s):1 if x = y: // Rule 3 applies2 return RootCause(x, s)3 if O(x) = N(x) and O(y) = N(y): // Rule 2 applies4 return {v, x, y}5 if O(x) 6= N(x): // Rule 4 applies7 return RootCause(x, s)8 if O(y) 6= N(y): // Rule 4 applies9 return RootCause(y, s)

{v, x, y}. Either v changed its route preference or either x ory changed their export policy (Rule 1). If there is no workingpath before or after a change, either x or y is null.

Rule 3 Let O(v) = [x, . . . , s] 6= N(v) = [x, . . . , s] (i.e., theold and new paths share the first hop but downstream hopsare different). Then v is not the root cause of the event.

Rule 4 Let O(v) = [x, . . . , s] 6= N(v) = [y, . . . , s], andeither O(x) 6= N(x) or O(y) 6= N(y). Then the change atv was induced by the change in its next hop neighbor and vis not the root cause of the event.

The rules above can be combined to build a generic rootcause identification algorithm as shown in Alg. 1. The al-gorithm starts from an AS v where a change was observed.If v uses the same next hop AS toward s before and afterthe change, v is not the root cause and we search the rootcause downstream (Rule 3). If v uses different next hopASes before and after the change and the next hops did notchange path, then we have found the root cause (Rule 2). If vuses different next hop ASes and one of them changed paths,then we search downstream the next hop that changed paths(Rule 4). If both next hops of an AS v change paths, thechanges are due to the same (unique) root cause and Alg. 1finds the root cause regardless of the next hop chosen in therecursive call.

We can simplify Rules 1 and 2 if we assume that ASesdo not change their export policy (i.e., ASes either alwaysexport their best route or never export). Under this assump-tion, AS x cannot be the root cause in Rule 1 and ASes yand z cannot be the root cause in Rule 2. As a result, line 4of Alg. 1 would return only {x} as the root cause.

As an example, Tab. 1 shows the steps taken by Alg. 1when applied to the path change in Fig. 1. The algorithmstarts at v, applies Rule 4, and calls itself recursively tosearch the root cause downstream from y. At y, the al-gorithm applies Rule 4 again and searches the root causedownstream from z. At z, the algorithm applies Rule 2 andidentifies the {z, d, ∅} as the root cause as both neighbors(d and ∅) did not change paths. Under the assumption thatexport policies are fixed, we would get {z} as the root causeinstead.

Recursion TargetO(·) N(·) Rule

Depth AS Applied1 v [x, b, a, s] [y, e, s] 42 y [z, d, c, s] [e, s] 43 z [d, c, s] ∅ 2

Table 1: An example of Alg. 1 applied to the path changeshown in Fig. 1.

3.2 General Candidate and Monitored SetsWe now consider the size of the set of ASes that Alg. 1

could traverse for an arbitrary path change observed at v.We then use this to determine a lower bound for the pathinformation required to identify the root cause of the change.

We define the general candidate set of an AS v, denotedC(v), as the set of ASes Alg. 1 may identify as the root causeof a path change observed at v. For a given path change forprefix p observed at AS v, C(v) is a set containing v itselfand the ASes in the general candidate sets of the downstreamneighbors On(v) (short for Oneighbor(v)) and Nn(v) that ASv chose toward p before and after the change, respectively.Then C(v) = v ∪ C(On(v)) ∪ C(Nn(v)).

As an example, in Fig. 1, the general candidate set of vcontains the ASes in v’s old and new paths (v, x, b, a, y, e, s)as well as the ASes in y’s old path (z, d, c). If z had a newpath, they would also be in C(v). ASes like m and n that arenot involved in the change are not in v’s general candidateset. We can prove the following result.

THEOREM 1. If there is only one routing event in the net-work, the root cause of a path change at an AS v lies in C(v).Proof If there is another AS x in O(v) or N(v) that changedpaths, the change at v was induced by the downstream changeat x (Rules 3 and 4). Such an AS x is in C(v). Similarly, ifthere is another AS y in O(x) or N(x) that changed paths,the change at x was induced by the change at y. Such anAS y is in C(v). Let c be the AS closest to the sourceAS that changed paths (i.e., there is no AS downstream cthat changed paths). Without loss of generality, we need toprove that the root cause of the change at c is in C(v). Anetwork event at an AS not in O(c) ∪ N(c) cannot be thecause of the change at c because it cannot change the rel-ative preference between the (unchanged) paths c receivesfrom its downstream neighbors On(c) and Nn(c). Thus, ei-ther c changed its route preference and is the root cause, orone of its downstream neighbors changed their export policyand are the root cause (Rule 2). ASes c, On(c), and Nn(c)are in C(v). This completes the proof.

Alg. 1 is guaranteed to identify the root cause of pathchanges; however, it requires path information from a largeset of ASes in the network. Identifying the root cause of achange at an AS v requires information about the old pathused by an AS x in the new path chosen by v, i.e., we needto know O(x), where x ∈ N(v). Because we cannot knowthe ASes in N(v) until after the path change, we need to

4

S

V

F

A

B AS with path

change

old path

new pathC

E

DØ

Figure 2: General topology subgraph demonstrating athree-level induced path change starting at c and prop-agating up to v (impossible under the simplified routingmodel). The ASes shown can be anywhere in the path.

track the old paths from all ASes in all possible paths v canchange into. For example, in Fig. 1, we need to monitorpaths from x, y, and m, as we do not know a priori whatneighbor v will choose after a change. Given the recursivecalls in Alg. 1, the set of ASes we need to track paths frommay grow exponentially with the path length, subject to theupper bound of the size of the AS graph.

Formally, we define the general monitored set of an AS v,denoted M(v), as the set of ASes whose old paths we needto track to run Alg. 1 on path changes observed at AS v. LetL(v) denote the set of neighbor ASes v can use to reach anAS s. The set M(v) includes the neighbors v can use toreach s and all ASes in the general monitored sets of eachof v’s neighbors. This gives M(v) = L(v) ∪ {M(x) | x ∈L(v)}. Note that C(v) ⊂ M(v): at an AS v, C(v) includesthe candidate set of the neighbors v chose before and afterthe change (e.g., C(x) and C(y) in Fig. 1), while M(v) in-cludes the monitored sets of all of v’s neighbors (e.g., M(x),M(y), and M(m) in Fig. 1).

3.3 Bounding Candidate and Monitored SetsTo address the challenge of monitoring a prohibitively

large set of ASes, we provide a bound on the general can-didate set C(v). We then use this bound to reduce the set ofASes we need to monitor to perform root cause analysis.

Our bound on the candidate set is derived from the sim-ple and widely used Gao-Rexford routing model [10]. Whenselecting among paths announced by multiple neighbors to-ward the same prefix, an AS considers its LocalPref (whichcan encode policies such as business relationships and/ortraffic engineering) for each neighbor before AS path length.For example, an AS typically prefers a path learned froma customer over another learned from a peer or provider,and a path learned from a peer over another learned from aprovider. We assume that path selection involves only peer-ing ASes. In other words, path preference at an AS dependson only the next hop ASes in each available path, a prop-erty that holds empirically the vast majority of the time [11].The shorter path is picked only if two or more paths havethe same LocalPref (e.g., both are customer paths). Basedon this routing model we identify the ASes from the generalcandidate set most likely to have caused the path change.

We overload our notation for the old and new paths usedby an AS v, O(v) and N(v), as follows. N(O(v)) is the set

of ASes that appear on the new paths originating from theASes in O(v), and O(N(v)) is the set of ASes that appear onthe old paths originating from ASes in N(v). For example,in Fig. 2, O(v) = [a, e, . . . , s] and N(O(v)) = O(v) ∪N(a) ∪N(e) ∪ · · · ∪N(s) = {a, e, b, d, . . . , s}.

THEOREM 2. Under the simplified routing model, the rootcause of a path change observed at an AS v is contained inthe bounded candidate set B(v) = N(O(v)) ∪O(N(v)).

Proof Observe that B(v) = N(O(v)) ∪ O(N(v)) is con-tained in v’s general candidate set C(v). Hence, if the rootcause lies in B(v), it also lies in C(v), as per Theorem 1.

We first show that the root cause AS may appear in B(v).Using Fig. 2 as an example, it is obvious that the root causemay lie either the old or the new path, i.e., ASes like f anda. ASes in the set N(O(v)), like b, can be the root cause aswell. For example, suppose a failed link b–a comes back upand b starts announcing a path to a. AS a may start routingthrough b if it prefers paths from b over paths from e. AS vmay then change its route from a to f if the new path through[a, b, . . . , s] is longer. The case of O(N(v)) is similar.

Now we show that ASes in the set C(v) − B(v) cannotbe the root cause. These include ASes like c, which liesin O(N(O(v))). Proceeding by contradiction, assume thatthe path change shown in Fig. 2 happens and that the rootcause is c, causing b to switch to a different path throughd and causing a to switch to b’s new path. This impliesthat a’s criterion for path selection between e and b is pathlength. If a had a a policy favorable to b (e.g., a higher Lo-calPref for b) then it would have picked the path through bin the first place. If a had a policy favorable to e it wouldnot change paths. Shortest path routing at a implies that[a, b, d, . . . , s] ≤ [a, e, . . . , s] ≤ [a, b, c, . . . , s], where the‘≤’ operator compares path lengths. This is because a prefersthe path through e before the routing event and the paththrough b–d after. Now, if v switches to the new path throughf , then it implies that v is using shortest path routing tochoose between a and f . As v prefers the path through a–e over the path through f before the routing event and thepath through f over the path through a–b after, we have that[v, a, e, . . . , s] ≤ [v, f, . . . , s] ≤ [v, a, b, d, . . . , s]. Lookingat the paths from v we have that [a, e, . . . , s] ≤ [a, b, d, . . . , s].This contradicts the earlier result from a that [a, b, d, . . . , s] ≤[a, e, . . . , s]. Hence, c or any other AS in O(N(O(v))) can-not be the root cause. The argument is similar for N(O(N(v))).wrapwrap

Note that if our assumptions and routing model are wrong,the root cause of a path change at v can lie outside B(v).However, we can still detect violations of the assumptionschecking that the root cause is outside B(v).

Even if we limit our search for the root cause in the boundedcandidate set B(v) = N(O(v)) ∪ O(N(v)), the O(N(v))term implies we need to track the old paths from all ASesthat may appear in any new path v can choose. This makes

5

the set of ASes whose old paths we need to track the sameas the general monitored set M(v). To bound the monitoredset, we only track the old paths from ASes in the most pre-ferred paths v can change to.

We define the bounded monitored set, denoted T (v), asthe set of ASes in all paths that v prefers over O(v) plus theASes from the next preferred path from each AS in O(v).AS v may change away from O(v) if a more preferred pathbecomes available. In this case we have complete informa-tion to run root cause analysis as the monitored set T (v) in-cludes all ASes in paths that are more preferred than O(v).Conversely, AS v may change away from O(v) to a less pre-ferred path if O(v) becomes unavailable. In these cases, weonly miss information when an AS x ∈ O(v) changes to apath that is not its next preferred path (which may happen,e.g., when the network lacks redundancy and an event im-pacts both of an AS’s current and next preferred path). Ifpath preference information is unavailable for an AS v, wetake a conservative approach and add the best path from eachof v’s downstream neighbors to T (v).

Tracking paths from ASes in the most preferred paths re-duces the monitored set when identifying the root cause inthe bounded candidate set B(v) because we need to knowonly O(N(v)). The heuristic would not be as effective toreduce the monitored set when identifying the root cause inthe general candidate set C(v). The reason is that we wouldneed to monitor ASes in the most preferred paths for any ASthat can show up in new paths, recursively. For example, wewould need to track ASes that can show up in the most pre-ferred paths of all ASes in N(O(v)) to know O(N(O(v))).

4. DESIGN AND IMPLEMENTATIONOur objective is to build a system able to identify the root

cause of a path change in the Internet. We now state thegoals of the system and how we achieve them.

4.1 GoalsOur work is motivated by the fact that Internet path changes

can have a significant impact on performance and availabil-ity, affecting service providers, operators and customers. Whensuch changes occur, we would like to to be able to quicklyand accurately identify the network responsible for the change.This allows operators to more quickly debug problems andtake corrective action (e.g., by contacting the responsiblenetwork). To enable such a system, we identify the follow-ing system goals.

• Accuracy: The output candidate set for root cause shouldbe as close to the ground truth as possible.

• Precision: The system should identify a small set ofASes as candidates for being the root cause of a pathchange, ideally a single AS or single AS link.

• Robustness: The system should be able to deal reason-ably well with uncertainty or absence of information.

Figure 3: PoiRoot operation modes.

• Scalability: The system needs to be selective in itschoice of ASes to monitor and should introduce as lit-tle measurement overhead as possible. It should con-duct measurements and quickly identify root causes ofpath changes (e.g., within minutes), enabling operatorsto quickly react when an unexpected change happens.

4.2 System OverviewPoiRoot employs a number of measurement components

(§4.3 and §4.4) to identify the root cause of path changes ob-served on paths from a set of target ASes V to a set of mon-itored prefixes P . At a high level, PoiRoot operates in twomodes as shown in Fig. 3. In the absence of path changes,the system stays in monitoring mode. The first task of mon-itoring mode is to estimate the set of ASes we need to mon-itor to perform root cause for all target ASes, i.e.,

⋃T (v)

for all v ∈ V . The second task is to measure paths from themonitored ASes toward the monitored prefixes in P .

PoiRoot enters identification mode whenever the path froma target AS v to a monitored prefix changes. Identificationuses information collected while in monitoring mode aboutthe old paths used before the change. Identification startsby collecting missing information about new paths used byASes in v’s old path, i.e., measure N(O(v)). PoiRoot thenruns the identification algorithm to identify the root cause ofthe change. Finally, PoiRoot goes back to monitoring modeafter identification is complete.

4.3 Path MeasurementPoiRoot maintains an atlas of paths from each AS v ∈ V

to each monitored prefix. To this end, we combine publicly-available BGP feeds and active traceroute measurements. Tosatisfy the assumption that there is only one routing event inthe network, we need path measurements from each AS athigh enough frequency such that the path is available be-fore and after each routing event. We also need to collect allnecessary information and identify the root cause before thenext routing event happens. We describe both control-planeand data-plane monitoring measurements we use below.

Passive BGP Monitoring. PoiRoot uses BGP paths madepublic by over 100 RouteViews peers [28]. We downloaddata from RouteViews every 15 minutes. For the set of pre-fixes being monitored, we store historical and current BGPpath information. We group updates into convergence pe-riods if they are less than 4 minutes apart to identify stable

6

routes. Previous work has shown that varying the groupingthreshold between 2 and 8 minutes has negligible impact onthe resulting convergence periods and stable paths [18, 41].

Traceroute Monitoring. We supplement our view of pathstoward a monitored prefix using traceroute measurementsthat we translate to AS paths using IP-to-AS translation. Specif-ically, we use forward traceroutes from 175 geographicallydistributed PlanetLab sites toward the monitored prefixes.The forward traceroute measurements are issued every 10minutes. Forward traceroutes allow us to find paths fromASes not present in route collector feeds, but are still limitedto the locations where PlanetLab nodes are located.

Reverse Traceroute [16] allows us to address this by po-tentially measuring paths to monitored prefixes from routersin arbitrary ASes. In particular, we use Reverse Tracerouteto measure paths from three IPs in each AS discovered withBGP feeds and forward traceroutes toward the monitoredprefixes. We refresh reverse traceroutes every 15 minutes.Note that for a variety of reasons (e.g., routers not respond-ing to IP Options probes), Reverse Traceroute may not beable to collect measurements from all ASes.

Combining Data Sources. In some cases, a path betweentwo ASes is obtained via traceroute measurements and BGPfeeds. Because default routes are a common phenomenon [1]and because IP-to-AS mappings can be inaccurate [3, 27],PoiRoot uses the control-plane paths from BGP feeds insteadof data-plane paths in this case.

4.4 Identifying the Set of ASes to MonitorPoiRoot identifies the set of ASes whose paths we need to

monitor to run root cause analysis,⋃T (v), while in moni-

toring mode. We consider the following data sources.

Passive Monitoring of Paths to Prefixes. One way to iden-tify the ASes in the monitored set T (v) is to continuouslymonitor public BGP feeds [28]. Monitoring changes for aprolonged period of time allows us to discover a set of pathsan AS may use to reach a prefix and build the set of ASes weneed to monitor to perform root cause analysis. Althougha passive approach reveals actual paths used in the Inter-net and is minimally invasive, not all paths appear in BGPfeeds because ASes only propagate their best path, apply ex-port filters, and perform prefix aggregation. Waiting for pathchanges to take place naturally is likely to be slow (on the or-der of months or years [30]) for the purposes of identifyingthe ASes we need to monitor.

Active Monitoring of Paths to Prefixes. We complementpassive BGP data with active periodic traceroute measure-ments. We store all paths observed with passive and activemonitoring and build a database of historical paths towardthe monitored prefixes.

Revealing Alternative Paths via BGP Announcements.PoiRoot reveals alternative paths in the Internet using theBGP-Mux platform [40], which lets us announce prefixesusing five different US universities as our providers (Univer-

sity of Washington, University of Wisconsin, Georgia Tech,Princeton University, and Clemson University). Suppose ASv has AS x in its path toward a prefix we control in AS s. Inorder to reveal an alternative path from v to s that does nottraverse x, we can prepend our BGP announcements with[s, x, s]. This approach is generally referred to as “BGP poi-soning” [1, 5].2

When a poisoned announcement propagates and arrives atx, the BGP loop-prevention mechanism triggers and x willdiscard all updates for our prefix. AS x will then withdrawits path to our prefix from its upstream neighbors (includingv), forcing them to choose new paths. For example, in Fig. 2poisoning c emulates a link failure between c and b. Thiscauses c to withdraw the route to b, making b use the routethrough d.

To discover all possible paths to our prefixes from an ASx, we first discover all neighbors L(x) that x can use to reachour prefix and then call the discovery process recursively oneach AS in L(x). To discover the neighbors L(x) that an ASx can use, we first poison the neighbor On(x) that x is usingto reach our prefix. When x changes to a new path, we poi-son its new neighbor. We repeat this process until x has nonew path to our prefix. Note that we include multiple ASesin our announcements as the discovery process proceeds. Inparticular, to call the discovery process recursively on an ASy ∈ L(x), we need to poison all other ASes in L(x) thatx prefers over y. For example, suppose our measurementsgo through a provider x that has y and z as the two mostpreferred neighbors to reach our prefixes. We will keep ypoisoned throughout the discovery of the paths z can use to-ward our prefixes, otherwise x would change back to y andour measurements would not go through z.

We argue that this approach to learning alternative avail-able paths is reasonable for the context of a large cloud provideror ISP. Such networks already are BGP speakers, and we ex-pect they can dedicate a small (e.g.,/24) prefix for the pur-poses of exploring alternate paths and inferring route prefer-ences of different ISPs routing towards them.

Path Preference Estimation. We use the order in whichx chooses neighbors in the path discovery procedure aboveto rank paths by preference. When the discovery procedureprovides insufficient information to estimate preference, weuse path usage time as in indicative of preference as pro-posed by Oliveira et al. [31].

4.5 Root Cause Analysis AlgorithmThe root cause analysis algorithm in Alg. 1 requires com-

plete and accurate information. PoiRoot uses a modified ver-

2Similar to the study conducted for LIFEGUARD [18], these an-nouncements affect only paths toward our experimental prefixesand we impose a conservative rate limit for announcements to en-sure our experiments do not generate unduly large numbers of up-dates. Operators were given an opportunity to opt out and we re-spected those requests. We have received zero complaints duringseveral months of experiments.

7

sion, shown in Alg. 2, that is more robust to missing and in-consistent measurements. Because it is impossible to deter-mine whether an AS changed its export policy without BGPfeeds from its neighbors, we assume that ASes do not changetheir export policy (i.e., ASes either always export their bestroute or never export). If an AS changes its export policyin practice, our system will blame the root cause’s upstream.We believe this case is uncommon; further, this informationis useful in that the upstream AS is the only one that coulddetect the change in export policy – information necessaryto debug the path change.

We now describe how our practical root cause analysis al-gorithm works. Suppose a path change happens on the pathfrom an AS v to another AS s that originates a prefix. Thealgorithm starts adding the ASes in O(v) and N(v) to thesuspect set S. At each step, the algorithm visits the unvis-ited AS x in the suspect set that is closest to v (order is notimportant if multiple ASes are at tied for closest). If a mea-surement from x is missing (lines 2–3), the algorithm usesseveral heuristics to resolve uncertainty, described in § 4.6.If the visited AS x did not change paths, then x cannot be theroot cause and we remove x and all downstream ASes fromthe suspect set (Rules 1 and 2, lines 4–6). If x changed paths,we know that either it is the root cause or the root causeis downstream x (Rules 3 and 4). We remove all upstreamASes x from the suspect set (line 8). If x is on O(v) or N(v),then we add any new downstream ASes x to the suspect setand visit them to determine whether they are the root causeof the path change. Note that this algorithm has no recur-sive call and that lines 10 and 12 only execute if x ∈ O(v)or x ∈ N(v). Only ASes in N(O(v)) ∪ O(N(v)) may beadded to the suspect set. The algorithm finishes when allASes in the suspect set have been visited.

Our algorithm will misidentify the root cause if our sim-plified routing model is wrong; i.e., the root cause AS isnot in N(O(v)) ∪ O(N(v)). Say Alg. 2 identified x as theroot cause. We can check for violations of the model (andmisidentifications) by extending Alg. 2 to check if anotherdownstream AS in O(x) ∪N(x) changed paths.

4.6 Dealing with UncertaintyAlg. 2 can find the root cause of a path change by querying

just a small subset of all monitored ASes in T (v). However,it may happen that measurements are missing. In particular,measurements for the new path of an AS x ∈ O(v) may bemissing because no vantage point has a path that traverses xafter the change. Similarly, measurements for the old path ofan AS y ∈ N(v) may be missing. We employ the followingheuristics to deal with missing measurements in Alg. 2.

FailedPath. If AS x experiences an outage, i.e., O(x) 6=N(x) = ∅, then the root cause is downstream of x andall ASes upstream of x cannot be the root cause (Rule 4,§3.1) and we remove ASes upstream of x from the suspectset. Similarly, FailedPath removes x’s upstream ASes in thecase of a link restoration, i.e., N(x) 6= O(x) = ∅. We keep

Algorithm 2 Practical algorithm for root cause analysis of achange on the path from v to s with potentially incompletepath information.PracticalRootCause(v, s):1 S ← O(v) ∪N(v)2 for each unvisited x ∈ S in order of distance from v:3 if O(x) not found or N(x) not found:4 // resolve uncertainty5 if O(x) = N(x):6 remove x from S7 remove ASes downstream of x from S8 else:9 remove ASes upstream of x from S10 if x ∈ O(v):11 add ASes in N(x)−O(x) to S12 else if x ∈ N(v):13 add ASes in O(x)−N(x) to S14 return Strack of reachability from ASes using pings to historicallyreachable routers inside ASes.

MissingPath. If we are missing a path from AS x and xis reachable, we use information from our database of his-torical paths (§4.4) to predict the missing path. When eitherO(x) or N(x) is missing, MissingPath adds all ASes in x’sbounded monitored set, T (x), to the suspect set, as these arethe ASes that x is most likely to be using after the change.

Correlation. When Alg. 2 runs and encounters missingmeasurements, the resulting candidate sets may contain morethan one AS. Because routing is generally destination-based,the same root cause can appear in multiple paths from van-tage points toward the monitored prefix. We use this obser-vation to reduce the suspect set by correlating suspect setsacross path changes from the same event observed at differ-ent vantage points.

Note that the candidate sets from different vantage pointsmay be disjoint for two reasons. First, our information aboutrouting preferences for each AS in the candidate set maybe incorrect (e.g., because it is stale) leading the system tomonitor the wrong set T (v) of ASes. Second, there may bemultiple concurrent and independent path changes. Thus,taking the intersection of suspect sets, as done by Feldmannet al. [9], would filter out a potential root cause.

To avoid this, we compute the suspect set for each pathchange resulting from a network event, then count the num-ber of occurrences of each suspect AS over all suspect sets.Then, each suspect set is pruned to only those ASes with thehighest number of occurrences.

5. EVALUATIONIn this section, we evaluate PoiRoot algorithm on a large

set of path changes. Our evaluation goals are to demonstratethat our approach is 1) accurate in terms of always includingthe root cause in the suspect set; 2) precise in that it identi-fies a small suspect set (ideally of size one); 3) robust to

8

missing measurement from vantage points; and 4) scalablein that it does not require monitoring a prohibitive number ofpaths and it can identify root causes within minutes of pathschanging. To evaluate these goals, we use controlled pathchanges on real Internet paths, simulations for larger topolo-gies, and path changes “in the wild.” In contrast to previ-ous work, our approach is both precise and accurate, largelybecause it accounts for induced path changes. Further, weshow that our results are similar even with only partial infor-mation from paths in the monitoring set for a prefix.

5.1 Controlled Internet ExperimentsIn this section, we use controlled experiments on real In-

ternet paths to demonstrate the accuracy and precision ofPoiRoot for identifying the root cause of a path change. Webegin by describing our experiment methodology, then presenta case study of how our algorithm works on a real inducedpath change. Last, we use a large set of AS-link failures toevaluate our approach and compare it with NOOR [9].

5.1.1 MethodologyA key challenge for evaluating root cause analysis algo-

rithms is the lack of control or ground truth for interdomainpath changes. In this work, we address this using the BGP-Mux platform [40] to craft announcements that cause con-trollable path changes for real Internet paths. Specifically,we use BGP poisoning [1] to induce changes for paths to-ward a prefix we control.3 Because such announcementscause the targeted AS to withdraw routes from its upstreamneighbors, we know that the “root cause” for the correspond-ing path change is the targeted AS. Likewise, when unpoi-soning a prefix, we effectively emulate a “link up” event forthe targeted AS.

We use five /24 poisoning prefixes to cause path changes.Each prefix is announced from one of the five muxes locatedat these locations: University of Washington, University ofWisconsin, Georgia Tech, Princeton University, and Clem-son University. We perform a sequence of poisonings usingthe same topology discovery scheme described in §4.4, start-ing the discovery process from the ASes where we have aPlanetLab vantage point. We change announcements (add orremove poisonings) at most once every 90 minutes to allowfor BGP convergence and to avoid flap dampening effects.In addition to changes caused by our poisoned announce-ments, paths may change in response to exogenous events.To account for this, we announce a separate sentinel /24 pre-fix from each Mux location and do not vary its AS-Path. Ifthe sentinel and poisoning prefixes change at the same time,then it is likely caused by an exogenous event and we filterthe change from our dataset.

We let the experiment run for one week starting Novem-ber 25th, 2012, yielding a total of 105 distinct poisoningsfor each prefix. The BGP updates and traceroutes collected

3We use the same conservative approach to poisoning as describedin §4.4.

NTU

TANET2 old pathnew path

ØASNETAPAN

TRANSPACI2

GENI

UWisc

Figure 4: Case study for an induced path change. Thepath change is caused by a poisoned announcement thatemulates a link failure at TRANSPAC (AS: 22388).

throughout the experiment traverse 351 ASes and a total of2572 path changes. Many of these path changes are dupli-cates, and are therefore ignored. We also ignore all experi-ments where the path remains stable even though it traversesthe poisoned AS (i.e., the poisoned announcement did nottrigger BGP loop prevention). Similarly, we also ignore ex-periments where we detect that one of the poisoned AS’speers filtered the poisoned announcement. After applyingthe filters, our experiments have a total of 638 unique pathchanges.

5.1.2 Induced Path Change Case StudyWe begin by describing how PoiRoot locates the root cause

AS for an induced path change via a case study. Note thatsince the root cause AS lies neither on the old nor the newpath, NOOR will not be able to find it. The path change takesplace between the National Taiwan University (NTU, ASN:17716) and our experimental AS, GENI (ASN: 47065). Weannounced prefix 184.164.249.0/24 from the University ofWisconsin BGP-Mux (ASN: 2381) and poisoned TRANSPAC(ASN: 22388) at 2:22 AM PST on November 27th, 2012.After convergence a path change was observed at a Planet-Lab node located in NTU at 2:35 AM. The AS-level pathchange is shown in Figure 4.

Upon observing the path change at NTU, PoiRoot runsAlg. 2 to identify the root cause. It begins by adding ASesin NTU’s old and new paths to the suspect set. Then, itsearches for the root cause in order of distance from NTU.It first checks that TANET2 uses the same path before andafter the change, so TANET2 and all its downstream ASescannot be the root cause of the change (Rule 2, lines 5–7).The sequence of visited ASes and the resulting suspect setsare shown in Tab. 3. Each row shows the AS being currentlyvisited and the suspect set from the previous step. For lack ofspace, we do not show GENI and UWisc, which are triviallyeliminated as root causes. When PoiRoot visits ASNET, it (i)sees ASNET changed paths and removes NTU from the sus-pect set (line 9), and (ii) adds TRANSPAC from ASNET’sold path to the suspect set (line 13). Similarly when it vis-its APAN, it removes ASNET from the suspect set sinceAPAN experienced a path change. Finally, the algorithmvisits TRANSPAC and removes APAN from the suspect set(because TRANSPAC changed paths) and terminates return-ing a suspect set containing only TRANSPAC, the correct

9

accuracy of suspect set suspect set size when accurate

ground truth in set empty inaccurate mean median 25th perc. 75th perc.

PoiRoot 100.0% 0.0% 0.0% 1.66 1 1 2PoiRoot-correlation across vps 100.0% 0.0% 0.0% 1.32 1 1 1NOOR-standard 61.7% 38.3% 0.0% 1.20 1 1 1NOOR-individualPath 91.1% 0.0% 8.9% 2.70 2 1 3

Table 2: Comparison of PoiRoot with NOOR, the approach used in previous work [9]. PoiRoot is both accurate andprecise: it never misses the root cause and nearly always produces a suspect set size of two or smaller. NOOR tradesoff precision for accuracy in the face of induced path changes: NOOR-standard misses the root cause in many cases buthas small suspect set sizes; NOOR-individualPath misses fewer root causes but exhibits larger suspect set sizes.

Step Visit Suspect Set1 TANET2 NTU, TANET2, I2, ASNET, APAN2 ASNET NTU, ASNET, APAN3 APAN APAN, TRANSPAC, ASNET4 TRANSPAC TRANSPAC, APAN5 ∅ TRANSPAC

Table 3: Illustration of the suspect set when recursingthrough Alg. 2 for the induced path change shown inFig. 4.

identification. In addition to the knowledge that we poisonedTRANSPAC, we have also checked with TRANSPAC’s NOCthat it would withdraw its announcement for the prefix as aresult of this announcement.

5.1.3 Controlled Internet ExperimentsIn this section, we use controlled experiments to show that

PoiRoot is accurate (always includes the root cause), precise(identifies suspect sets of size 2 or smaller 96% of the time)and robust to missing path information (more than 50% ofpaths must be missing to significantly impact suspect setsize). By comparison, previous work cannot be both accu-rate and precise in the face of induced path changes, whichoccur for a significant fraction of our evaluation data. Fur-ther, we show that our approach is scalable in that it requiresmonitoring a reasonable number of paths and provides re-sults within minutes of path changes.

Accuracy and Precision. We use the path changes col-lected in the previous section to evaluate PoiRoot. Tab. 2presents our results and compares them with NOOR, whichwe discuss in the next section.

The table shows the fraction of paths where the suspectset (i) contains the ground truth (i.e., the poisoned or un-poisoned AS), (ii) is empty, and (iii) is nonempty but doesnot contain the ground truth. We also show the mean andthe quartiles of the suspect set size for correctly identifiedchanges.

PoiRoot always includes the root cause in the suspect setand builds suspect sets with average size of 1.66 ASes andmedian size of 1 AS. When PoiRoot identifies a suspect setof size 2, the ASes are adjacent, meaning the algorithm still

0

20

40

60

80

100

1 2 3 4 5cum

ulat

ive

% o

f pat

h ch

ange

s

size of the final suspect set containing ground truth

PoiRootPoiRoot-correlation

Figure 5: CDFs of the size of the final suspect set. Poi-Root, combined with an optimization that correlates theresults across sources, results in a small suspect set onaverage as well as high accuracy.

isolates the root cause to the endpoints of an AS link. Themedian of the maximum candidate set size observed duringexecutions of Alg. 2 is 5, and the quartiles are 4 and 6, re-spectively.

PoiRoot may return a suspect set with more than one ASwhen we have incomplete path information from some ASes.To reduce the impact of this issue we use the Correlationheuristic described in §4.6. Fig. 5 compares Correlation’sperformance with the standard version of PoiRoot. For thestandard algorithm, the suspect set contains one AS for 51%and two ASes for 84% of the path changes. Correlationacross vantage points improves suspect set size: it is a singleAS for 73%, and two ASes for more than 95% of the pathchanges. Again, in all cases of suspect sets larger than one,the ASes lie adjacent to each other on a path. From here onwhenever we discuss our algorithm’s performance we use itsCorrelation version.

Comparison with NOOR. We now compare our results withthose from using NOOR, which is based on the approachproposed by Feldmann et al. [9].

The standard NOOR algorithm (see §2) builds a suspectset that contains the root cause for only 61.7% of the pathchanges, but with an average suspect set size smaller than

10

that of our algorithm. This suggests an inclusion policy thatis too conservative and an elimination policy that is too ag-gressive. We now analyze in detail the reasons behind theseissues.

A key reason for inaccuracy is that the standard NOOR al-gorithm may outputs an empty set for any path change whosenetwork event also caused an induced path change. For ex-ample, when AS x causes an induced path change, x may beidentified as the root cause when considering paths from x,but it does not appear on the new or old path for an AS y ex-periencing an induced path change. Because the suspect setsfrom x and y may not overlap, the intersection may producean empty set. The impact of this limitation is significant: inour experiments, 38.3% of the path changes were due to anevent that generated an induced change. All such events leadto NOOR outputting an empty suspect set.

The existence of even a small number of induced pathchanges can significantly decrease NOOR’s accuracy. Infact, a larger set of available vantage points leads to a higherprobability that at least one induced change will be observedfor a network event. Out of the 292 RouteViews peers andPlanetLab nodes used in our system, 15.8% observed at leastone induced path change over the course of the experiment.

One way to address the induced path change problem isto modify the standard NOOR algorithm so that every pathchange is treated on an individual basis instead of groupingall changes for an event (NOOR-individualPath). This ap-proach avoids taking the intersection of the individual can-didate sets. It still fails to deal with an induced change dueto the assumption that the root cause lies on either the oldor new path. However, it improves results for non-inducedchanges, increasing accuracy to 91.1%. This comes at thecost of precision: the elimination policy is less aggressivewithout correlation, so the median and mean suspect set sizefor correct identification is larger (2 and 2.7, respectively).This highlights a key limitation of NOOR: the algorithm iseither precise but not accurate; or in this case, accurate butnot precise.

Robustness to Missing Information. We now address thequestion of whether PoiRoot is robust to missing path infor-mation. Note that PoiRoot always includes the root causein our suspect set unless a new path from a monitored AS auses an AS b that never appeared in previous measurements.This case never occurred in our experiments. Thus we quan-tify the robustness of our approach by evaluating the preci-sion of PoiRoot when previously available paths are removedfrom the dataset.

In particular, consider AS x in our suspect set first dis-covered on the old path originating from AS v, hence O(x)is known. Now our system measures N(x). To determinehow PoiRoot would perform with missing information, weset N(x) = missing with probability p. Similarly if x wereoriginally discovered on the new path from v, i.e., N(x) isknown, we set O(x) = missing with probability p. Increas-ing p causes PoiRoot to use the MissingPath heuristic (from

0 0.5

1 1.5

2 2.5

3 3.5

4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

mea

n su

spec

t set

siz

e

probability p of missing path from an AS

Figure 6: Mean suspect set size (and standard deviation)as we vary the probability of a missing path from an AS.PoiRoot is reasonably robust to missing path informa-tion: on average the suspect set size is only 2 when 70%of paths are missing.

§4.6) more frequently; this in turn can increase the suspectset size.

In Fig. 6 we plot the average suspect set sizes (and stan-dard deviations) as we vary p. For each value of p, 0 ≤ p < 1we run the algorithm ten times on each path change and takethe average suspect size. The final average for p is takenover all these averages. We notice that although the meansuspect set size goes up, it goes up only slightly as p is in-creased. In fact the mean suspect size is less than 2 for amissing probability as high as 0.7.

The reason why PoiRoot is so robust to missing measure-ments is that paths from multiple vantage points tend to over-lap as they converge toward the destination prefix. A singlepath measurement can remove multiple ASes from the sus-pect set, even ASes added to the suspect set due to missingmeasurements. For example, if Rule 2 from §3.1 applies,i.e., O(x) 6= N(x), all ASes upstream from x are removedsince they cannot be the root cause.

Prevalence of Induced Path Changes. One of the keystrengths of PoiRoot is its success in dealing with inducedpath changes. We showed earlier that even a small numberof induced changes can significantly decrease the effective-ness of previous approaches, such as NOOR. Now using acombination of our measurement data and the UCLA Inter-net graph [39], we argue that a larger set of vantage pointswould cause even more widespread occurrence of inducedchanges in the Internet.

To simulate the scenario where we have more vantagepoints, and thus more paths, available to PoiRoot, we ex-tend the empirically derived AS topology used in the pre-vious section. Specifically, for each AS x in our measuredpaths, we find its set of upstream neighbors from the UCLAInternet graph. We use the UCLA dataset’s annotated busi-ness relationships to determine whether these neighbors areupstream. Then, for each upstream neighbor y, we calcu-late the shortest valley-free path from the neighbor toward aBGP-Mux prefix. AS y will use this path only if it is shorter

11

than the path exported by x. We use path changes from ourpoisonings to determine which ASes x choose a longer pathover a short one. These ASes may cause an induced changeat an upstream AS y if a failure occurs on x’s old path andx selects a new shorter path, i.e., N(x) < O(x). This is ex-actly the same as the example in Fig. 1, where y chooses alonger path and induces changes at v.

We find 302 cases where AS x prefers a longer BGP path.This leads to a total of 7555 induced path changes causedby just 55 simulated link failures. Hence with a large set ofpaths being monitored, the number of induced path changesincreases. Therefore an approach that does not account forinduced changes is more likely to be inaccurate as the num-ber of vantage points increases.

We also run PoiRoot on this set of simulated induced pathchanges. PoiRoot continues to always include the root causein its suspect set. With perfect information, i.e., assumingpaths are available both before and after the failure for everyAS, the precision is also perfect, i.e., the suspect set containsonly the failed AS. We also model missing paths with theparameter p similar to Fig. 6. The results are very similar toFig. 6. For example, with p = 0.7 we get a mean suspectsize of 1.65.

Candidate and Monitored Set Bounds. We now quantifythe scalability gains obtained with our bound on the candi-date set. We compute the sizes of the general and boundedcandidate sets C(v) and B(v) for all path changes in ourpoisoning experiments and also the sizes of the general andbounded monitored sets M(v) and T (v). We use exhaustivepoisoning experiments as we need topology information ascomplete as possible to compute C(v) and M(v)

The further an AS is from the origin AS (i.e., our BGP-Mux providers), the larger its candidate and monitored sets.We compute candidate and monitored sets for the ASes wherewe have vantage points as they are the furthest from the ori-gin AS in our dataset. We also note that the bounded can-didate and monitored sets B(v) and T (v) depend on v’s oldpath O(v); in particular, the size of T (v) depends heavilyon the preference rank of its old path. The general candidateand monitored sets C(v) and M(v) are independent of O(v).

The solid line in Fig. 7 shows the distribution of the re-duction in monitored set size (i.e., |M(v)−T (v)|÷ |M(v)|)across all path changes in the data set. Overall, we reducethe monitored set size by more than 50% for 38% of pathchanges. The average size of the general and bounded moni-tored sets are E[M(v)] = 38.9 and E[T (v)] = 27.0, and theaverage reduction in size is 31%. The bounded monitored setis identical to the general monitored set for 38% of the pathchanges (where the solid line intersects the y-axis). Thishappens whenever an AS is using its least preferred path, aswe have to monitor all other paths that are more preferred.Even if an AS is not using its least preferred path, the setof ASes to monitor quickly converges to the general moni-tored set as the number of more preferred paths we need tomonitor increases.

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80 100cum

ula

tive %

of path

changes

monitored set size reduction (%)

all changeschanges from

preferred paths

Figure 7: Our bound on the candidate set size allows usto reduce the number of ASes we have to monitor to per-form root cause analysis.

The dashed line in Fig. 7 shows the reduction in moni-tored set size across the subset of path changes when an ASv changes from its most preferred path to an arbitrarily lesspreferable path. Because we only need to monitor a few lesspreferable paths when v is using its most preferred path, thereduction in monitored set size is significantly higher. Theaverage reduction in this case is 65%.

The reduction in candidate set sizes (i.e., |C(v)−B(v)| ÷|C(v)|) is much smaller (not shown), as induced path changesare rare in practice and we do not observe any 3-level in-duced path change.

Detection Speed. To allow providers and operators to de-bug problems caused by path changes in real time, we wouldlike PoiRoot to provide root cause analysis as changes occur.Our current system refreshes paths from traceroutes every10 minutes and from BGP feeds every 15 minutes. The al-gorithm for generating the suspect set takes seconds to run,so our system currently can provide results at the same ratethat paths are refreshed (i.e., 15 minutes or less). We notethat we can further reduce detection time by monitoring real-time BGP feeds (e.g., via BGPMon [42]) and triggering pathmeasurements on demand in response to issues such as per-formance problems.

5.2 In-the-Wild Path ChangesHaving shown that our approach is accurate and precise

for path changes obtained by controlled BGP poisonings andsimulations, we now demonstrate its effectiveness for pathchanges seen “in-the-wild”. In particular, we focus on pathchanges that significantly worsen end-to-end latency, as theyare most likely to warrant further debugging from serviceproviders and operators that optimize for performance.

5.2.1 MethodologyWe compute path changes as follows. We issued tracer-

outes between all PlanetLab sites every ten minutes for aperiod of three days starting November 1st, 2012, and per-form IP-to-AS translation to convert traceroutes to AS-levelpaths. We extract from this dataset AS-path changes that last

12

0

20

40

60

80

100

1 2 3 4 5cum

ulat

ive

% o

f pat

h ch

ange

s

final suspect set size

controlled experimentsin-the-wild

Figure 8: Distribution of suspect set sizes built by Poi-Root running on path changes observed ‘in-the-wild’.The corresponding CDF from §5.1.3 is shown as well.Our algorithm narrows down the root cause to one ortwo ASes for more than 84% of the cases.

for at least an hour. In addition we require that the path be-fore the change is stable for at least 30 minutes to allow forconvergence.

Since network operators will be most interested in furtherinvestigating those path changes that degrade performancesignificantly, we use the relative change in RTT as our per-formance metric and consider only those path changes thatexperience an increase in RTT larger than 10ms and whoserelative increase in RTT is greater than 25%. Each RTTvalue is averaged over at least three samples. This gives us aset of 643 path changes for the three day period.

5.2.2 ResultsFig. 8 shows the distribution of the size of the suspect set

built by PoiRoot with the Correlation heuristic for the high-impact path changes identified above, as well as the cor-responding distribution from controlled experiments §5.1.3.The results are very similar, with the suspect set narroweddown to one AS for more than 64%, and to two ASes formore than 83% of the path changes. We find that 96% of thefinal suspect sets with more than one AS include a sequenceof consecutive ASes (i.e., ASes are adjacent).

6. RELATED WORK

Measuring path changes. Several previous studies have in-troduced new path measurement techniques [7,16,17,24,31,34, 43], e.g., to discover inter-AS links, quantify BGP pathexploration, or infer an AS’s path preferences. Our systemcombines multiple sources of data to collect path measure-ments in the Internet, including public BGP feeds [28] andtraceroute measurements collected from PlanetLab. Lesswidespread is our use of Reverse Traceroute [16] to collectpath measurements from arbitrary ASes back to monitoredprefixes. Reverse Traceroute sends IP Record Route probesto a target router from distributed vantage points spoofing asthe source, so that the source receives a sequence of routers

on the path back from the target router to itself. Even thoughReverse Traceroute does not work on paths that filter IP RecordRoute probes or spoofed packets, it helps us cover ASes in-visible from BGP feeds or traceroutes. Previous work hasused BGP poisoning to quantify the use of default routes inthe Internet [1]. We use BGP poisoning to perform topol-ogy discovery in a way equivalent to that of Colitti et al. [5].However, ours is the first work we are aware of that usesBGP poisoning to obtain ground truth for evaluating rootcause analysis in the Internet.

Impact of path changes. Several previous studies highlightthe impact of route changes [8, 15, 20, 25, 26], e.g., caus-ing increased delays and transient packet loss due to con-vergence, in addition to outages due to misconfiguration. Inthis work, we identify the root cause of such pathologicalchanges, which can assist in debugging and correcting theproblem. Previous work also studied how paths propagate inthe Internet [12, 13]. Our work uses a basic routing modelto understand how path changes propagate in the Internet,building upon previous work that models the BGP decisionprocess [10].

Root cause analysis. As described in §2, the work by Feld-mann et al. [9] and Caesar et al. [2] are most closely re-lated to ours. We demonstrated that the NOOR assumptionprevents their approach from achieving both accuracy andprecision. Several other papers have looked at the problemof root cause analysis from different perspectives. Wu etal. [41] propose a system to identify high-impact networkevents affecting an AS, using routing changes observed at anAS’s border routers and correlating it with traffic changes.The system helps operators identify important events, butis limited to one network and cannot identify the cause ofevents. In particular, if an event is caused by a distant AS,their system can only indicate the border routers involved.Pei et al. [32] propose a modification to BGP that includesinformation about the networks responsible for a path change.Our work demonstrates that one can identify root causeswithout requiring protocol modification. LIFEGUARD [18]and NetDiagnoser [6] identified a set of routers responsiblefor packet-forwarding outages, but did not focus on whichnetwork’s path change (if any) caused the outage.

7. CONCLUSIONIn this paper, we described an algorithm and a scalable

system, PoiRoot, for accurately and precisely identifying thenetwork responsible for Internet path changes, even in theface of induced path changes and incomplete information.We developed a general algorithm for identifying the rootcause of path changes, then used simple, validated assump-tions about routing behavior to derive new constraints on thenetworks that can possibly be responsible for a path change.We then designed a system to gather the requisite path in-formation and produced an algorithm that identifies the rootcause of a path change even with missing paths. Finally,

13

we evaluated PoiRoot using ground-truth from path changesinduced on real Internet paths, empirically informed simula-tions and natural path changes “in the wild.” We showed thatprevious approaches to root cause analysis can either be ac-curate or precise, while our approach achieves both. As partof our future work, we are investigating ways to use this rootcause information to automatically address routing changesthat negatively impact performance.

8. REFERENCES[1] R. Bush, O. Maennel, M. Roughan, and S. Uhlig. Internet

optometry: assessing the broken glasses in Internetreachability. In IMC, 2009.

[2] M. Caesar, L. Subramanian, and R. H. Katz. Towardslocalizing root causes of BGP dynamics. Technical report,University of California, Berkeley, 2003.

[3] K. Chen, D. R. Choffnes, R. Potharaju, Y. Chen, F. E.Bustamante, D. Pei, and Y. Zhao. Where the sidewalk ends:Extending the Internet AS graph using traceroutes from P2Pusers. In CoNEXT, 2009.

[4] A. Christie. Murder on the Links. The Bodley Head, 1923.[5] L. Colitti. Internet Topology Discovery Using Active

Probing. PhD thesis, University di Roma Tre, 2006.[6] A. Dhamdhere, R. Teixeira, C. Dovrolis, and C. Diot.

NetDiagnoser: Troubleshooting network unreachabilitiesusing end-to-end probes and routing data. In CoNEXT, 2007.

[7] X. Dimitropoulos, D. Krioukov, M. Fomenkov, B. Huffaker,Y. Hyun, kc claffy, and G. Riley. As relationships: inferenceand validation. SIGCOMM Comput. Commun. Rev.,37(1):29–40, 2007.

[8] N. Feamster, D. G. Andersen, H. Balakrishnan, and M. F.Kaashoek. Measuring the effects of Internet path faults onreactive routing. In SIGMETRICS, 2003.

[9] A. Feldmann, O. Maennel, Z. M. Mao, A. Berger, andB. Maggs. Locating Internet routing instabilities. InSIGCOMM, 2004.

[10] L. Gao. On inferring autonomous system relationships in theInternet. IEEE/ACM TON, 9(6):733–745, 2001.

[11] P. Gill, S. Goldberg, and M. Schapira. A survey ofinterdomain routing policies. NANOG 56, 2012.http://www.nanog.org/meetings/nanog56/presentations/Monday/mon.general.gill.11.pdf.

[12] T. Griffin and G. Huston. BGP wedgies. Network WorkingGroup, RFC 4264, Nov. 2005.

[13] T. G. Griffin, B. F. Shepherd, and G. Wilfong. The stablepaths problem and interdomain routing. IEEE/ACM ToN,10(2):232–243, 2002.

[14] N. Gvozdiev, B. Karp, and M. Handley. LOUP: Theprinciples and practice of intra-domain route dissemination.In NSDI, 2013.

[15] Y. Huang, N. Feamster, A. Lakhina, and J. J. Xu. Diagnosingnetwork disruptions with network-wide analysis. InSIGMETRICS, 2007.

[16] E. Katz-Bassett, H. V. Madhyastha, V. K. Adhikari, C. Scott,J. Sherry, P. van Wesep, A. Krishnamurthy, and T. Anderson.Reverse traceroute. In NSDI, 2010.

[17] E. Katz-Bassett, H. V. Madhyastha, J. P. John,A. Krishnamurthy, D. Wetherall, and T. Anderson. Studyingblack holes in the Internet with Hubble. In NSDI, 2008.

[18] E. Katz-Bassett, C. Scott, D. R. Choffnes, I. Cunha,V. Valancius, N. Feamster, H. V. Madhyastha, T. Anderson,and A. Krishnamurthy. LIFEGUARD: Practical repair ofpersistent route failures. In SIGCOMM, 2012.

[19] R. Krishnan, H. V. Madhyastha, S. Srinivasan, S. Jain,A. Krishnamurthy, T. Anderson, and J. Gao. Moving beyondend-to-end path information to optimize CDN performance.In IMC, 2009.

[20] C. Labovitz, A. Ahuja, A. Bose, and F. Jahanian. DelayedInternet routing convergence. In SIGCOMM, 2000.

[21] C. Labovitz, G. R. Malan, and F. Jahanian. Internet routinginstability. IEEE/ACM TON, 6(5):515–528, 1998.

[22] G. Linden. Make data useful. http://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt, 2006.

[23] H. Madhyastha, E. Katz-Bassett, T. Anderson,A. Krishnamurthy, and A. Venkataramani. iPlane Nano: PathPrediction for Peer-to-Peer Applications. In NSDI, 2009.

[24] H. V. Madhyastha, T. Isdal, M. Piatek, C. Dixon,T. Anderson, A. Krishnamurthy, and A. Venkataramani.iPlane: An information plane for distributed services. InOSDI, 2006.

[25] R. Mahajan, D. Wetherall, and T. Anderson. UnderstandingBGP misconfiguration. In SIGCOMM, 2002.

[26] Z. M. Mao, R. Bush, T. G. Griffin, and M. Roughan. BGPbeacons. In IMC, 2003.

[27] Z. M. Mao, J. Rexford, J. Wang, and R. H. Katz. Towards anaccurate AS-level traceroute tool. In SIGCOMM, 2003.

[28] D. Meyer. RouteViews. http://www.routeviews.org.[29] W. Muhlbauer, A. Feldmann, O. Maennel, M. Roughan, and

S. Uhlig. Building an AS-topology model that captures routediversity. In SIGCOMM, 2006.

[30] R. Oliveira, D. Pei, W. Willinger, B. Zhang, and L. Zhang.The (in)completeness of the observed internet as-levelstructure. IEEE/ACM Trans. Netw., 18(1):109–122, 2010.

[31] R. Oliveira, B. Zhang, D. Pei, and L. Zhang. Quantifyingpath exploration in the internet. IEEE/ACM Trans. Netw.,17(2):445–458, 2009.

[32] D. Pei, M. Azuma, D. Massey, and L. Zhang. BGP-RCN:Improving BGP convergence through root cause notification.Computer Networks, 48(2):175–194, 2005.

[33] Ponemon Institute. Calculating the cost of data centeroutages, 2011. http://www.emersonnetworkpower.com/en-US/Brands/Liebert/Documents/White%20Papers/data-center-costs 24659-R02-11.pdf.

[34] A. Shaikh and A. Greenberg. OSPF monitoring:Architecture, design and deployment experience. In NSDI,2004.

[35] N. Spring, R. Mahajan, and T. Anderson. Quantifying thecauses of path inflation. In SIGCOMM, 2003.

[36] S. Stefanov. Yslow 2.0. In CSDN SD2C, 2008.[37] R. Teixeira and J. Rexford. A measurement framework for

pin-pointing routing changes. In ACM SIGCOMM Workshopon Network Troubleshooting, 2004.

[38] A. Toonk. BGPmon. In NANOG 45, 2009.http://www.nanog.org/meetings/nanog45/presentations/Sunday/Toonk bgpmon N45.pdf.

[39] UCLA Internet topology collection.http://irl.cs.ucla.edu/topology/.

[40] V. Valancius, N. Feamster, J. Rexford, and A. Nakao.Wide-area route control for distributed services. In ATC,2010.

[41] J. Wu, Z. M. Mao, J. Rexford, and J. Wang. Finding a needlein a haystack: Pinpointing significant BGP routing changesin an IP network. In NSDI, 2005.

[42] H. Yan, D. Matthews, R. Oliveira, L. Zhang, K. Burnett, andD. Massey. Bgpmon: A real-time, scalable, extensiblemonitoring system. In In Cybersecurity Applications andTechnologies Conference for Homeland Security(CATCH,2009.

[43] M. Zhang, C. Zhang, V. Pai, L. Peterson, and R. Wang.PlanetSeer: Internet path failure monitoring andcharacterization in wide-area services. In OSDI, 2004.

14