1. PEMP: Peering Equilibrium MultiPath routingStefano Seccia,b , Jean-Louis Rougiera , Achille Pattavinab , Fioravante Patronec , Guido Maierb a Institut Telecom, Telecom ParisTech, LTCI CNRS, France. E-mail: {secci, rougier}@telecom-paristech.fr b Politecnico di Milano, Italy. E-mail: {secci, pattavina, maier}@elet.polimi.itc Universit` di Genova, Italy. E-mail: patrone@diptem.unige.it a AbstractIt is generally admitted that Inter-domain peering We propose to re-use the Multiple Exit Discriminatorlinks represent nowadays the main bottleneck of the Internet, (MED) attribute of BGP as the simple medium to convey co-particularly because of lack of coordination between providers, ordination costs between carriers. A potential non-cooperativewhich use independent and selsh routing policies. We are game that arises from load balancing based upon this data isinterested in identifying possible light coordination strategies, then proposed. Pareto-efcient equilibrium solutions can bethat would allow carriers to better control their peering links,coordinatively selected by carriers. We show by simulationswhile preserving their independence and respective interests. Wepropose a robust multi-path routing coordination framework forthat this choice prevents congestion on peering links, decreasespeering carriers, which relies on the MED attribute of BGP as the global routing cost while increasing the route stability.signalling medium. Our scheme relies on a game theoretic mo- Sect. II presents the inter-carrier routing issues that wedelling, with a non-cooperative potential game considering both tackle. Sect. III present the ClubMED (Coordinated MED)routing and congestions costs. Peering Equilibrium MultiPathframework for inter-domain multipath routing over peering(PEMP) coordination policies can be implemented by selectinglinks. We explain how load balancing shall be implementedPareto-superior Nash equilibria at each carrier. We compare over efcient equilibrium strategies. Sect. IV denes thedifferent PEMP policies to BGP Multipath schemes by emulating Peering Equilibrium MultiPath (PEMP) routing coordinationa realistic peering scenario. Our results show that the routing policies and discuss their possible benets and technicalcost can be decreased by roughly 10% with PEMP. We also showimplementation issues. Sect. V presents results from realisticthat the stability of routes can be signicantly improved and thatcongestion can be practically avoided on the peering links1 . simulations assessing the PEMP policy performance. We showhow our approach can outperform BGP Multipath in termsof routing cost, route stability and peering link congestion.I. I NTRODUCTIONSect. VI concludes the paper and discusses further work. Multipath routing has received interest for a long time, as II. I NTER - CARRIER ROUTING ISSUESit is considered as a very efcient solution providing morerobustness and better load distribution on the network. Intra-A. BGP and selsh routingdomain multipath routing is commonly performed in Interior It is worth briey reminding how the route selection isGateway Protocol (IGP) networks, by balancing the load over performed via BGP [5]. When multiple paths to a destinationEqual Cost Multiple Paths (ECMP) [1]. In the multi-domain network are available, a cascade of criteria is employed tocontext, multi-path routing is generally not implemented, compare them. The rst is the local preference throughits introduction raising important scalability and complexity which local policies with neighbor Autonomous Systemsissues (see eg. [2]). Multipath interdomain routing is, to our(ASs), mainly guided by economic issues, can be applied:knowledge, still an open issue (and a target for future internete.g., a peering link (i.e., free transit) is preferred to a transitarchitectures). However, some limited solutions based on thelink (transit fees). The subsequent criteria incorporate purelyBorder Gateway Protocol (BGP) have been introduced, at leastoperational network issues to select the best route: (i) the routewith some vendors routers (see e.g. [3] [4]). Multipath BGPwith a smaller AS hop count; (ii) if the routes are received bycan then be used to balance load on different routes underthe same neighbor AS, the route with a smaller MED; (iii) thespecic conditions (detailed in the next section), in particularroute with the closer egress point (hot-potato rule), using ason several peering links between two adjacent carriers. distance metric the IGP path cost; (iv) the more recent route; Nevertheless, the lack of routing collaboration among neigh- (v) the AS path learned by the router with the smaller IPboring carriers causes BGP Multipath to produce unilateral(tie-breaking rule). Considering these criteria, BGP selectsrouting choices that, even if potentially efcient for the ups- the best route. This best route is then eventually advertised totream carrier w.r.t. load distribution, may lead to an inefcient its peers (if not ltered by local policies).situation for the downstream carrier. In this paper we propose a Two peering ASs have usually many links in several dis-framework that allow carriers to select efcient load balancing tributed locations and can thus dispose of many routes tostrategies in a coordinated manner, while preserving theirthe same network through the same AS. By default, theseindependence and respective interests. Our proposal is basedroutes have equal local preferences and AS hop counts. Hence,on a game theoretical model, as a natural tool to study possiblethe best route is chosen w.r.t. either the smaller MED ortrade-offs between selshness and cooperation. Possible coor- (if the MED is disabled) the smaller IGP path cost. Thedination policies can be highlighted, from quite selsh to more decision is taken minimizing the routing cost of a single peer:cooperative ones, with different degrees of Pareto-efciency. either the upstreaming ASs IGP path cost (hot-potato), or thedownstreaming ASs weight (smaller MED). The challenge is1 Work funded by the European ICT FP7 Euro-NF Network of Excellence thus the denition of methods that consider both the routingand the I-GATE project of the Institut Telecom, ICF Networks of the Future. costs when taking the peering routing decision. 978-1-4244-4148-8/09/$25.00 2009This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE "GLOBECOM" 2009 proceedings.

2. The Multi-Exit Discriminator (MED): The MED is a metricC. Peering link congestionthat an AS can attach to route advertisements toward a poten-It should also be noted that the incentives for increasingtial upstream AS, to suggest an entry point when many exist.the capacities of peering links are not straightforward. Indeed,In this way, the upstream AS can prefer an entry point toward peering agreements do not relay on any payment, as opposedthe advertised network. By default, the MED is set to the to transit agreement. Controlling the load on the peering linkscorresponding intra-AS IGP path cost (from the downstream is thus essential. However, this is difcult, as it requires to set-border router to the egress router). On transit links, subjectup very complex routing policies [2]. Furthermore, the currentto provider/customer agreements, the provider should always inability to estimate possible IGP weight variations, and thusfollow MED-icated routes suggesting preferred entry pointsto foresee the associated inter-domain route deections theybecause the customers pay for. This is not the case for peering might cause, prevents carriers from controlling the inter-ASsettlements, and this can be considered as the main reason whylink congestion precisely. Whenever available, Multipath BGPthe MED is often disabled between peers [6].is expected to reduce congestion, by better distributing the BGP Multipath: If the MEDs and/or the IGP path costs load over the different available routes (through the differentare equal, to avoid tie-breaking the load may be balanced peering links) with the same IGP costs. However, the choiceon the equivalent routes. At the time being, such multipath of routes on which to distribute the load is based on internalextensions for BGP did not nd consensus at the IETF, and for costs, which might lead to inefcient trafc distribution for thethis reason there is no standard specication. However, somepeers network. The challenge is thus the denition of scalablesuggestions are indicated in [7]. As of our knowledge, thepeering link control methods, with some collaboration.only implemented method carriers can use for multipath inter-domain routing is the BGP Multipath mode that some routerIII. T HE C LUB MED F RAMEWORKvendors now provide (e.g., Juniper [3] and Cisco [4]), with We present the ClubMED (Coordinated MED) framework,some little variations on the routing decision. Therefore, BGPcharacterized in detail in [14]. Within it the MED signallingMultipath allows adding multiple paths to the same destinationbetween peering ASs is modeled as a non-cooperative peeringin the routing table. This does not affect the best path selection: game that can allows the peers to coordinate towards rational,a router still designates a single best path and advertises it to its efcient and stable multipath routing solutions.neighbors. More precisely, BGP Multipath can be used whenmore than one IBGP (Internal BGP) routers have equivalent A. The ClubMED peering gameroutes to a destination through many border routers, or when The idea is to re-use the MED as the means to exchangeall of the candidates routes are learned via EBGP (External loose routing and link congestion costs between peer networksBGP). As stated in [7], other cases, with a combination offor a subset of destination prexes, in order to help carriers toroutes learned from IBGP and EBGP peers, should be avoided, better collaborate in the load sharing decisions. The schemeas they may lead to routing loops for instance. relies on a game theoritical modeling of the load sharingproblem. Each peer is represented as a rational player thatB. BGP route deectioncan take benet by routing accordingly to a cost game built The peering routing decision with BGP thus relies on IGP upon routing and congestion costs. The principle is to takerouting costs. Nowadays, the interaction between IGP routingthe peering routing decision following efcient equilibriumand inter-AS routing represents a major issue because IGP strategy proles of the game - in its one-shot form or repeatedweights are optimized and recongured automatically. To react form - thus allowing better collaboration between carriers.to non-transient network events, a carrier may re-optimize the We can introduce the game on a simple example, depictedIGP weights, inducing changes in the BGP routing decision,in Fig. 1, with two peers, AS I and AS II. Let us rst deneso that congestions might appear where not expected.a destination cone as a set of customers destination prexes. Many works concern BGP route deection control methods.On Fig. 1, Community A and Community B represent two[8] reformulates the egress routing problem and proposescritical destination cones that may deserve careful peer routing,to replace the hot-potato rule with a more expressive and e.g. because they produce high bit-rate ow aggregates. Theefcient rule. [9] presents a comprehensive yet hard IGPinter-cone ows are supposed to be equivalent, for instanceWeight Optimization (IGP-WO) method aware of possible hot-w.r.t. their bandwidth, so that their path cost can be fairlypotato route deections to bound them (they report that 70% ofcompared and their routing coordinated. We assume that thetrafc can be affected in a real network). [10] presents a similarcones represent direct customers or stub ASs, which wouldproposition relying on graph expansion tricks. However, while often assure that their entry point in a peer network is uniqueeffective, a problem seems to persist with the latter proposi-(this would reinforce the equivalence condition of the twotions: each time the BGP routes change, the BGP-aware IGP-ows, but is not, however, a strict requirement).WO is to be triggered. The scalability may be thus a practical We propose that the two ASs coordinate the choice on theissue: the occurrence of IGP-WOs, normally triggered only for egress peering link for each outgoing ow, from Community Aintra-AS issues, would drastically increase. To better assess to Community B and vice-versa. A ClubMED peering gamethis issue, we worked at the detection of deections usingis built at Ra and Rb routers, called ClubMED nodes, usingTRACETREE radar data [11]. Preliminary results conrm thatthe egress IGP path cost, the ingress IGP path cost, the sametop-tier AS interconnections suffer from frequent deections, costs for the peer announced via the MED, and endogenously-and some periodic oscillations [12]. The challenge is thus theset peering link congestion costs. At ClubMED nodes, efcientdenition of methods to control the coupling between inter- equilibria can be selected, accordingly to the different policiesAS and intra-AS routing, as the authors in [13] conclude afterdetailed in the next section, so as to decide the egress route(s)studying these interactions.for each inter-community ow.978-1-4244-4148-8/09/$25.00 2009This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE "GLOBECOM" 2009 proceedings. 3. Fig. 2. Multi-pair 2-link ClubMED game composition example.Fig. 1. Single-pair ClubMED interaction example.gs (x+ , y + ) = s (y + ) = 2cII + cII + cII . 112122In order to take broader decisions, many pairs of inter-cone Gd = (X, Y ; fd , gd ), is a game of pure externality, whereows shall be considered in a same ClubMED game. In thisfd , gd : X Y N, fd (x, y) = d (y) and d : Y N,way, the equivalence condition (e.g., on the bandwidth) can gd (x, y) = d (x) and d : X N. For the above example: be extended to all the pairs together, not necessarily relatedfd (x+ , y + ) = d (y + ) = 2cI + cI + cI 11 12 22 to a same couple of ClubMED nodes. Therefore, the nalgd (x+ , y + ) = d (x+ ) = cII + 2cII + cII .1112 21ClubMED game derives from the superposition of many inter- Gc = (X, Y ; fc , gc ) is an endogenous game too, wherecommunity ows (e.g. in Fig.2 we have 4 pairs and 8 ows).fc , gc : X Y N. fc (x, y) = c (x) and gc (x, y) = c (y).With multiple pairs of cones, carriers shall control theIn order to build the congestion game, the ow bit-rates havecongestion on inter-peer links. The more egress ows areto be known. Let H be the set of inter-peer ow pairs, h therouted on a peering link, the more loaded the link and theoutgoing ow bitrate of the pair h H, and Ci the egresscongestion risk, and the higher the routing cost. Hence, we aim available capacity of li . With multipath, h can be partioned,at weighting the inter-carrier links with congestion costs when and i is the fraction routed towards li . Gc should not count hcongestion may arise. This could be alternatively done by when hH hminiE {Ci }, otherwise it would affectmodeling the inter-peer link in IGP-WO operations (e.g. [10]),the G equilibrium selection. The congestion cost is to bebut this would violate, however, the requirement of decouplingmonotonically increasing with the number of ows routed onintra-domain from inter-domain routing [13].a link [18]; one can use (idem for c (y)):1) Notations: The ClubMED game can be described asG = Gs + Gd + Gc , sum of a selsh game, a dummy game1c (x) = Ki(1)and a congestion game, respectively, as depicted in Fig. 2. Ci hH ih iE|li xLet X and Y be the set of strategies available to AS I andAS II (resp.): each strategy indicates the peering link where If Ci < hH i , Ki = . Otherwise, Ki are constants tohto route each inter-community ow. Let ((x, y), (x, y)) bebe scaled to make the cost comparable to IGP costs, e.g., suchthe strategy cost vector for the the strategy prole (x, y),that it is 1 when the idle capacity is maximum, i.e., Ki = Ci .x X, y Y . E.g., in Fig. 2, we have 4 pairs2) Peering Nash equilibrium: Gs + Gc is a cardinal poten-(A1B1,A1B2,A2B1,A2B2) and 2 links (l1 ,l2 ), andtial game [17], i.e., the incentive to change players strategyX and Y become {l1 l1 l1 l1 , l1 l1 l1 l2 , ..., l2 l2 l2 l2 }. For m pairs can be expressed in one potential function, and the differenceand n links, the game is the repeated permutation of m single-in individual costs by an individual strategy move has thepair n-link games, thus with |X|=|Y |=nm . Gs considers egresssame value as the potential difference. Gd can be seen asIGP weights only, modeling a sort of extended hot-potato rule.a potential game too, but with null potential. Hence, the GGd considers ingress IGP weights only, impacted by the otherpotential P : X Y N depends on Gs and Gc only. Aspeers routing decision (not taken into account in the legacy property of potential games [17], the P minimum correspondsBGP decision process). Gc considers peering link congestion to a Nash equilibrium and always exists. The inverse is notcosts computed as explained hereafter.necessarily true, but it can be easily proven that for G it isLet cI and cII be the egress IGP weight from the j thjijithanks to the endogenous nature of Gs and Gc . The ClubMEDClubMED node of AS I and AS II to the ith peering link li , peering Nash equilibrium is thus guided by the egress IGP i E, |E|=n. Let cI and cII be the corresponding ingressij ij weights and the congestion costs, and may be not unique whenweights, from the i link to the j th ClubMED node. th their sum is equal over different strategies.Gs = (X, Y ; fs , gs ), is a purely endogenous game, where The opportunity of minimizing of the potential functionfs , gs : X Y N are the cost functions for AS I and AS II to catch all the peering Nash equilibria represents a key ad-(resp.). In particular, fs (x, y) = s (x), where s : X N, vantage. It decreases the equilibrium computation complexity,and gs (x, y) = s (y), where s : Y N. E.g., for the which would have been very high for instances with manytopology in Fig. 2, consider the prole (x+ , y + ) with x+ = links and pairs. When there are multiple equilibria, Gd canl1 l2 l1 l1 and y + = l1 l1 l1 l2 ; we have:help in avoiding inefcient solutions (e.g. due to tie-breaking)fs (x+ , y + ) = s (x+ ) = cI + cI + 2cI111221by the selection of an efcient equilibrium in the Pareto-sense.978-1-4244-4148-8/09/$25.00 2009This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE "GLOBECOM" 2009 proceedings. 4. 3) Pareto-efciency: A strategy prole p is Pareto-superiorIIIl1l2l3to another prole p if a players cost can be decreasedfrom p to p without increasing the other players costs. l1 (17,36) 6(19,32)2 (16,38)8The Pareto-frontier contains the Pareto-efcient proles, i.e. l2 (15,23)4(17,19)0 (14,25)6those not Pareto-inferior to any other. In the ClubMED game,ingress costs affect the Pareto-efciency (because of the Gd l3 (18,18)7(20,14)3 (17,20)9pure externality). In particular, given many Nash equilibria,the Pareto-superiority strictly depends on Gd . E.g., Fig. 3IIIl1l2l3depicts two cases with 3 links and their strategic forms (Gcis not considered). The exponent indicates the corresponding l1 (16,10)2(19,10)2 (13,18)8potential value. Egress costs are close to the egress points,while ingress costs to the communities. For the upper case,l2 (14,19)0(17,19)0 (11,25)6there is a single equilibrium, (l2 , l2 ). For the lower one, therel3 (14,18)0(17,18)0 (11,24)6are four equilibria, and (l3 , l1 ) is the single Pareto-superiorone; however, it is not Pareto-efcient, but Pareto-inferior to Fig. 3. 3-link examples.(l1 ,l3 ) that is not an equilibrium because AS I will alwaysprefer l2 or l3 to l1 (11 < 13). This is due to the external effectof Gd . Indeed, it is possible that, after an iterated reduction of take into account these error margins. They dene a potentialstrategies, G assumes the form of a Prisoner-dilemma game,threshold under which a prole becomes an equilibrium. Morein which equilibria are Pareto-inferior to other proles. precisely, the minimum potential strategies are found, thenNote 1: To explicate P in calculus, we use a form in which we the other proles that have a potential within the minimumset to 0 the minimum of s and s , i.e., Ps (x0 , y0 ) = 0 where:plus the threshold (TP ) are considered as equilibria too. Eachs (x0 ) s (x) x X, and s (y0 ) s (y) y Y .potential difference P from (x1 , y1 ) to (x2 , y2 ) can beNote 2: In the simple example of Fig. 3, all the Nash equilibriaincreased of aI (x1 , x2 ) + aII (y1 , y2 ), where aI (x1 , x2 ) =have a null potential value, but this is not the case in general.I (s (x1 ) + s (x2 )) and aII (y1 , y2 ) = II (s (y1 ) + s (y2 )).An optimistic threshold can be:B. Modeling of IGP-WO operations Nowadays, IGP weights are frequently optimized and auto-TP =min {a(x1 , x2 )} + min {a(y1 , y2 )} (3) x1 ,x2 Xy1 ,y2 Ymatically updated rather than being manually congured. Inthis sense, we should assume that the ClubMED costs are Indicating with P (x0 , y0 ) the potential minimum, all strategysubject to changes when the ingress/egress ow directions proles (x, y) such that P (x, y) P (x0 , y0 ) + TP willchanges. In the following we explain how, in the ClubMEDbe considered as equilibria. This operation can also allowframework, the coupling among IGP and BGP routing can beescaping selsh (endogenous) solutions mainly guided by Gsmodeled to anticipate the route deection issue presented in+ Gc , introducing Pareto-superior proles in the Nash set.Sect. II-B. We aim at selecting a robust peering equilibriumwith an approach that is vaguely related somehow to the ideaIV. P EERING E QUILIBRIUM M ULTI PATH (PEMP) POLICIESpresented in [15] to stabilize intra-domain routing w.r.t. trafcpattern variations.Peers would route accordingly to an equilibrium because it At a given ClubMED node i of AS I, let s i,j,Ij,i,I and s grants a rational stability to the routing decision. The Nashbe the (i, j) path cost variations in the egress and ingressset and the Pareto-frontier may be quite broad, especiallydirections (resp.) when passing from the current routing to the considering IGP path cost errors. This leads to different pos-routing prole s X (idem si,j,IIj,i,II and sfor AS II). sible Peering Equilibrium MultiPath (PEMP) load balancingvariations could be used to extend the G Nash set and Pareto- policies (upon these prole sets), which are presented below.frontier. However, the should not be announced via the MEDA. Nash Equilibrium MultiPath (NEMP) implicit coordinationto avoid a large overhead and an excessive insight in a carriersoperations. Each peer can announce just a directional path costAssuming thus that ClubMED remains a fully non-error. Let I and II be these egress cost errors for AS I andcooperative framework, its implicit solution strategy to whichAS II (resp.). Being aware that IGP weights may signicantlyto coordinate without any signaling message is: play theincrease, an optimistic min-max computation can be: equilibria of the Nash set. Hence, it is feasible to nativelyimplement a Nash Equilibrium MultiPath (NEMP) routingI= min max s i,j,I /cI i,j (2)policy. E.g, in the bottom of Fig. 3 AS I may balance the(i,j) sX load on l2 and l3 , being aware that AS II may balance its load Similarly for II , I and II . The cost errors represent on l1 and l2 . However, the set of equilibria can be shrinked toa good trade-offs between network information hiding andthe Pareto-superior ones; but many Pareto-superior equilibriacoordination requirement: not announcing per-link errors avoidcan exist, so the NEMP policy is to be used in this case too.revealing the variations; announcing directed errors (ingress Please note that there may not exist Pareto-superior equilibria:and egress) allows reecting the fact that upstream and downs-in this case, NEMP is performed over all the equilibria.tream availability is likely to be unbalanced because of thebottleneck asymmetry in inter-AS links. B. Repeated coordination The errors induce a larger number of equilibria for the Given that the the G Pareto-frontier may not contain equili-multipath routing solution. The game can be easily extended tobria, in a repeated ClubMED context, an explicit coordination978-1-4244-4148-8/09/$25.00 2009This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE "GLOBECOM" 2009 proceedings. 5. strategy is: play the proles of the Pareto-frontier. The Club-MED game would be repeated an indenite number of times,indeed. From folk-theorem-like results [16], this strategy isan equilibrium of the repeated game and grants a maximumgain for the players in the long-run. Nevertheless, the unilateraltrust for such a strategy could decrease whether in a shortperiod of analysis the gains reveal to be unbalanced and infavor of a single peer. The reciprocal trust among peers canthus affect the reliability of such a Pareto coordination. Unself-Jump: Another strategy is conceivable to guaranteebalancedness in gains in the short term, and thus helping tokeep a high level of reciprocal trust. After shrinking the Nashset w.r.t. the Pareto-efciency, for each equilibrium the ASsmight agree to make both a further step towards the best Fig. 4. IGP routing cost Boxplot statistics: NEMP vs BGP Multipath.available strategy prole (xj , y j ) such that: (xj , y j ) (x0 , y0 ) + (xj , y j ) (x0 , y0 ) < 0 (4)where (x0 ,y0 ) is the starting equilibrium. One AS may un-selshly sacrice for a better bilateral solution: the lossthat one may have moving from the selected equilibrium iscompensated by the improvement upon the other AS. Thisstrategy makes sense only if the other AS is compensated witha bigger improvement, and returns the favor the next times. Pareto-Jump: Instead, with the addition of the constraint:(xj , y j ) (x0 , y0 ) 0 (xj , y j ) (x0 , y0 ) 0 (5)we select a Pareto-superior prole (not necessarily in thePareto-frontier), without unselshly sacrices. If at least one(xj , y j ) is found we obtain a new prole set that is to beshrinked w.r.t. the Pareto-superiority for the nal solution. E.g., in the bottom example of Fig. 3, we would jump fromFig. 5. IGP routing cost Boxplot statistics: PEMP strategies.the Pareto-superior Nash equilibrium (l3 , l1 ) to the Pareto-superior prole (l1 , l3 ). We would not have this jump for theUnself-Jump policy, that would prefer instead (l1 , l1 ) with aWe compare the PEMP routing policies (NEMP, Pareto-global gain of 6 instead of just 3 with (l1 , l3 ). Frontier, Pareto-Jump, Unself-Jump) to the BGP Mul- Finally, note the last two policies are not binding: it wouldtipath solution without and with (...+MED) classical MEDbe enough to associate the policy with the menace to pass tosignalling enabled at both sides, and to a Full BGP Multipathone of the more selsh choices. Also note that MEDs fromsolution in which all the peering links (i.e., the availabledifferent ASs should be normalized to the same IGP weight routes) are used for the multipath solution.scale in order to be comparable.A. Routing cost V. P ERFORMANCE E VALUATION Fig. 4 reports the IGP routing costs statistics in BoxPlot We evaluated the performance of the three PEMP routing format (minimum; box with lower quartile, median, upperpolicies with realistic simulations. We created a virtual in- quartile; maximum; outliers). We show four solutions: Fullterconnection scenario among the Geant2 and the Internet2 BGP Multipath, BGP Multipath, the NEMP policy without andASs, depicted in Fig. 6, emulating their existing peering withwith the congestion game Gc . For each method, we displaythree cross-atlantic links. We considered six pairs of inter-cone the Internet2, the Geant2 and the global routing costs. Weows among the routers depicted with crossed circles. The considered two ClubMED solutions, with and without theTOTEM toolbox [19] was used to run a IGP-WO heuristic,congestion game Gc (for the rst two gures only).with a maximum IGP weigth of 50 for both ASs. We used 252The full BGP multipath solution obviously guarantees ansuccessive trafc samples, oversampling the datasets from [20]even load on all the peering links. However, its routingfor Geant2 and from [21] for Internet2 on a 8h basis (to covercost almost doubles than with normal BGP multipath, whichall the day times). The original link capacity was scaled bybalances the load only on equal cost paths (egress IGPs or10 to create an intra-domain congestion risk. The inter-coneMEDs). Simple MED usage decreases the cost of the BGProuting generates additional trafc for the trafc matrices. We case without MED, due to one network that is more loadedused a random inter-cone trafc matrix such that ows are (hence, higher IGP weights), and to the fact that with the MEDbalanced with 200 Mb/s per direction, which corresponds tothe chance of ECMP is higher (not only on equal IGP path2/3 of the total available peering capacity. To evaluate thecost routes, but also on equal MED routes). The ClubMEDeffectiveness of the congestion game we considered peeringsolution, instead, outperforms BGP with a median cost lowerlinks with 100 Mb/s available per direction.by 10% without Gc , and by 6,6% in its complete form.978-1-4244-4148-8/09/$25.00 2009This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE "GLOBECOM" 2009 proceedings. 6. Fig. 6. Internet2 - Geant2 peering scenario with 3 peering links. Fig. 7. Number of route deections.Fig. 8. Maximum peering link utilization boxplot statistics. Fig. 5 compares the four PEMP policies. With respect toNEMP, the Pareto policies give statistically very close results.This may sound disappointing: one may expect more from3 route deections against 5, and the upper quartile andthe Pareto-frontier and the Pareto-Jump policies. By analyzingthe maximum much lower. Interestingly, among the PEMPthe results in detail, we veried that the reason of this poorpolicies, the Pareto-frontier one statistically behaves betterperformance is that the Pareto-frontier often contains strategy than the other policies for all the criteria but for the minimum.proles with the least cost for a peer and very high cost for The reason may be that the Pareto-superiority condition -the other peer. Such strategy proles are not marked as Pareto- applied on a very large set of candidate proles (in fact,inferior because of the single peers least cost and thus belongn2m = 531441) - offers a ner selection than the approximateto the Pareto-frontier. Such situations are likely to be frequent potential threshold one. Finally, the Jump policies present asince an uncongested intra-domain link may produce a IGPlower route stability w.r.t. all the statistical criteria. This isweight much lower than the others thus affecting the G prole reasonably due to the fact that the jump from the Nash set,cost components. This risk is augmented in the Pareto-Jumpi.e., the unself and Pareto-superior conditions, are computedpolicy since the new selected proles can just be Pareto- in the simulations without considering the cost errors.superior: they do not necessarily belong to the Pareto-frontier.However, for the Pareto-jump policy the median, the minimum C. Peering link congestionand the upper and lower quartiles outperform the NEMP result;Fig. 8 reports the Boxplot statistics maximum link utiliza-in fact, the starting Nash set for its Pareto-improvement is thetion as seen by each peer, with all the methods. All the PEMPNEMP one (see Sect. IV-B). Finally, the Unself-Jump policystrategies but the Pareto-frontier one never caused congestionshould outperform or equalize the Pareto-Jump one w.r.t. theon peering links (utilization above 100%). The enabling ofrouting cost since, without (5), it can be see as its relaxation. the Multipath mode in BGP does not have a signicant effectIndeed, as reported in Fig. 5, the Unself-Jump gives a median on the peering link congestion. With ClubMED, instead, thecost roughly 3% inferior than the NEMP cost.multipath routing choice is carefully guided toward efcientsolutions. The NEMP, Pareto-Jump and Unself-Jump policiesB. Route deections show the median, the upper and lower quartiles always above Fig. 7 reports the statistics of routing changes with respect85%, remembering that with full BGP Multipath one wouldto the previous round (with an upper bound equal to the total have a 200/300 = 66, 7% utilization. The Pareto-frontiernumber of ows). The PEMP policies behave signicantlystrategy does not guarantee, however, a congestion-free so-better than BGP Multipath: they have a median of around lution, with a median close to 100% utilization. The reason 978-1-4244-4148-8/09/$25.00 2009This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE "GLOBECOM" 2009 proceedings. 7. game. The game components can be adapted to consider IGPcost variations due to IGP-WO re-optimizations. We proposed a low-computational way to compute the Nashequilibria, and four possible Peering Equilibrium MultiPath(PEMP) routing coordination policies. The rst twos cor-responds to balance the load on the Pareto-superior Nashequilibria of the one-shot game, and on the Pareto-frontier(equilibrium of the repeated game), respectively. The lattertwo policies correspond to improve the rst strategy movingfrom the Pareto-superior Nash set renement toward exteriorPareto-superior and unselsh routing proles, respectively. We simulated the PEMP policies with a realistic emula-tion, comparing them to BGP Multipath. The results show Fig. 9. PEMP strategies execution time.they outperforms BGP Multipath in terms of routing cost,route stability and peering link congestion. In particular, theroute stability is signicantly improved and the peering linkcongestion can be practically avoided. Some differences existbetween the PEMP policies. Namely, the Pareto-frontier oneis extremely complex and shall not be implemented. Theother ones present some trade-offs but represent all promisingsolutions to perform an efcient and rational routing acrosspeering links. In particular, the Unself-Jump policy representsthe best trade-off between peering trust insurance, routing cost,congestion control, routing stability and execution time. We are currently working on the denition of an extendedpeering framework, modeling the border with multiple ASs as Fig. 10. Nash set dynamics.the single border of a classical peering.R EFERENCESfor this behavior are still the highly asymmetric cost proles [1] R. Teixera et al., In Search of Path Diversity in ISP Networks, inintroduced by the Pareto-superiority condition in the solution.Proc. of ACM Internet Measurement Conference, Oct. 2003. [2] Jiayue He, J. Rexford, Towards Internet-wide multipath routing inD. Time complexity IEEE Network magazine, March 2008. [3] Conguring BGP to Select Multiple BGP Paths, JUNOS document. Fig. 9 reports the PEMP execution time. We know that the[4] BGP Best Path Selection Algorithm, Cisco documentation.Pareto-frontier computation is cumbersome, with a O(n2m )[5] Y. Rekhter, T. Li, A Border Gateway Protocol 4 (BGP-4), RFC 1771.time complexity, while the other policies have a polynomial[6] D. McPherson, V. Gill, BGP MED considerations, RFC 4451.complexity that asymptotically depends on the minimization [7] A. Lange, Issues in Revising BGP-4, draft-ietf-idr-bgp-issues (2003) [8] R. Teixeira et al., TIE Breaking: Tunable Interdomain Egress Selec-of a (mono-dimension) potential function to populate the Nashtion, in Proc. of CoNEXT 2005.set. In fact, the other policies have an average computation [9] S. Agarwal, A. Nucci, S. Bhattacharyya, Controlling Hot Potatoes intime below the 2 seconds (however, rare peaks of a few moreIntradomain Trafc Engineering, SPRINT RR04-ATL-070677, 2004.seconds appear, probably due to the cases with very large Nash[10] S. Balon, G. Leduc,Combined Intra- and inter-domain traf-set, as it can be see cross-checking with Fig. (10). Hence, only c engineering using hot-potato aware link weights optimization, arXiv:0803.2824.the NEMP, Pareto-Jump and Unself-Jump policies shall be [11] M. Latapy, C. Magnien, F. Ou draogo, A Radar for the Internet, ineconsidered for a practical implementation. We have, however, Proc. of ADN 2008.introduced the Pareto-frontier case for a thorough comparison.[12] S. Secci et al., Detection of route deections across top-tier intercon- nections, working paper, http://perso.enst.fr/secci/papers/radar.pdfE. Nash equilibrium dynamics[13] R. Teixeira et al., Impact of Hot-Potato Routing Changes in IP Networks, IEEE/ACM Trans. on Networking, Vol. 16, Dec. 2008. Fig. 10 reports the number of equilibria and those Pareto- [14] S. Secci et al., ClubMED: Coordinated Multi-Exit Discriminatorsuperior in a log-scale for all the rounds. The Pareto-Strategies for Peering Carriers, in Proc. of NGI 2009.superiority condition permits to pick a few efcient Nash equi- [15] P. Casas, L. Fillatre, S. Vaton, Multi hour robust routing and fast load change detection for trafc engineering, in Proc. of IEEE ICC 2008.libria over broad sets, whose dimension varies signicantly in[16] R.B. Myerson, Game Theory: Analysis of Conict, Harvard Univ. Press.time. This reveals a high sensibility to the routing costs due[17] D. Monderer, L.S. Shapley, Potential Games, Games and Economicto the endogenous effect of Gc with high congestion costs. Behavior, Vol. 14, No. 1, May 1996, Pp: 124-143.[18] F. Larroca, J.-L. Rougier, Routing Games for Trafc Engineering, inVI. S UMMARY Proc. of IEEE ICC 2009.[19] J. Lepropre, S. Balon, G. Leduc, Totem: a toolbox for trafc enginee- We modeled the routing on peering links as a non- ring methods, in Proc. of INFOCOM 2006.cooperative game with the aim to allow carriers ne-selecting [20] S. Uhlig et al., Providing public intradomain trafc matrices to theroutes for critical ows by following efcient equilibrium research community, Computer Communication Review 36 (1) (2006).multipath solutions. We presented the mathematical model of [21] By courtesy of Y. Zhang. Abilene topology and trafc dataset. http: //www.cs.utexas.edu/yzhang/research/AbileneTM.the game, composed of a selsh game (with egress IGP costs),a dummy game (with ingress IGP costs) and a congestion978-1-4244-4148-8/09/$25.00 2009This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE "GLOBECOM" 2009 proceedings.