crafting confederations an overview of the confederation pop approach to network architecture

Click here to load reader

Download Crafting Confederations An overview of the Confederation POP Approach to Network Architecture

Post on 20-Mar-2016




0 download

Embed Size (px)


Crafting Confederations An overview of the Confederation POP Approach to Network Architecture. Dan Golding NetRail, Inc. [email protected] Miguel Dimayuga Earthlink, Inc. [email protected] The Old Way…. Conventional Network Routing Architectures…. - PowerPoint PPT Presentation


  • Crafting Confederations

    An overview of the Confederation POP Approachto Network ArchitectureDan Golding NetRail, [email protected] DimayugaEarthlink, [email protected]

  • The Old WayConventional Network Routing Architectures.

    Full Mesh iBGP or Route Reflectors

    A fully meshed Network via ATM PVCs.

  • Its not adapted to the New Optical Network!POS is here in force, ATMs value in the core is receding.It is far more fragile, and far less agile than newer methods of Inter-domain Routing.The Old Way was prone to user-error. The E-Commerce Revolution demands a New Way! Whats Wrong With The Old Way?

  • A Better WayEmphasizes Large Scale, IP Based, Fiber Ring Networks

    Optimized for Service Provider Needs

    Utilizes cutting edge routing technologies to provide far greater fault tolerance and usable traffic engineering.

    Implemented via advanced BGP techniques: Communities and Confederations.

  • How the Old worked(Full Mesh iBGP)Every router must be fully meshed with all others.Works well in small systemsGrows exponentiallyEventually consumes all CPU, memory, and engineering resources.

    Full iBGP MeshExponential growth!

  • How the Old Way worked(Route Reflectors)Scaled WellWell suited to fully meshed ATM Networks Star Topology.but...Not Survivable in a Fiber Ring Network.

    Peer Isolation withBGP Route ReflectionPeersRR ServerPeersRR Client

  • How the Old Way worked(Filtering)List of IP Prefixes and/or AS numbers set on all border routers to other ISPs. Only the access-list contents would be advertised.

    Worked well when most customers were single-homed and didnt run BGP.

    Changes were VERY manpower intensive.

    With multi-homed e-commerce shops, no longer feasible.

  • How the New Way works(Confederations)Routers peer with neighborsHighly SurvivableVery ScalableEasily ConfiguredAids TroubleshootingPeersPeersRouters Peerwith NeighborsBGP Confederations

  • Confederation OverviewBGP allows three types of peer relationships:iBGP (Full iBGP mesh)eBGP (External Peering or Transit)Confederation eBGP (its an iBGP with an eBGP look!)

    Confederation eBGP is like regular eBGP, exceptNext Hop, Local Preference and MEDs are preservedConfederation elements in the AS-PATH are not counted for route selection purposes

  • Confederation OverviewConfederations allow groups of routers to form sub-autonomous systems to eliminate scaling problems with full mesh iBGP

    All Routers within a sub-AS must be fully meshed (or optionally in a route reflector cluster configuration)

    Confederations are most advantageous when there are few routers per sub-AS. There is no reason to limit the number of sub-ASs you have nothing is gained.

  • Confederation OverviewMost confederation designs start out with only two or three sub-ASes. This offers few advantages over full mesh iBGP in a ring network topology.

    The more sub-ASes you add, the greater the advantage

    The final result: One sub-AS per POP

    The upper limit on this is 1000 sub-ASs per RFC

  • The Advantages of a Confederation of POPsThe routers within each POP need only peer with each other, utilizing iBGP

    Neighboring POPs are peered with via POP border routers speaking confederation eBGP

    Next Hop, Local Pref and MEDs are preservedMore survivable than Route ReflectorsFar more scalable than full iBGP mesh

  • How to Make It WorkThoughtful use of sub-AS numbersLocal Preference HierarchyUseful and Descriptive Community StringsMeaningful MEDsUse of various policies via access lists, community lists, etc as building blocksUse of Peer Groups whenever implementation allows.

  • Sub-AS AssignmentSub-ASs become useful tools for debugging show ip bgp, show route Suggested assignment is geographicalAlways remember to keep room for expansion!Put plenty of extra sub-ASs in your configs dont count on adding them later!

  • Geographical Region as sub-ASSoutheast 65000-65099Northeast65100-65199Northcentral65200-65299Southcentral65300-65399Western65400-65499Canadian65500-65535Latin/South American64512-64599European64600-64699Asian64700-64799Reserved64800-64999

  • Sample Community Assignments

  • Community Strings are the KeyCommunities are tags or post-it notes attached to routes to help identify them. There can be more than one community attached to a route.Communities are recommended to be set at the ingress point. Communities need be applied only onceadministrative burden and complexity is greatly reduced.When routes egress, filtering can be based on one or more community strings.Sample Communities Regional, by Peer, Customer, Internal, Peer, Transit

  • Communities Set at IngressAS701AS4355transitrouter bgp 4355network route-map make-greennetwork route-map make-red207.69.0.0/16i198.99.146.0/24i4.0.0.0/8i bgp 4355neighbor a.a.a.a remote-as 701neighbor a.a.a.a route-map make-blue in i i

  • Communities Used to Filter on EgressAS701AS3703AS4355transitcustomer207.69.0.0/16i198.99.146.0/24i4.0.0.0/8i 4335 i 4335 i207.69.0.0/164335 irouter bgp 4355neighbor b.b.b.b remote-as 3703neighbor b.b.b.b route-map blue-green out4.0.0.0/8701 i i

  • Community Categories Route TypeCustomer Routes4006:65150Private Peering4006:65140Transit4006:65130Public Peering4006:65120

    Internal Routes (OPN-visible)4006:65110Internal Routes (Global-visible)4006:65100

  • Other Peoples Networks (OPNs)To expand our national coverage, Mindspring utilized third party networks dialup facilities. These networks are what we term as OPNs.

    Prefixes for Core Services which we want restricted to MindSpring customers and not visible to the rest of the world (e.g. news, radius, smtp) are announced to our OPNs alone.This has the added advantage of protecting against abuse of our services by non-customers.

    With communities, we can tag routes for export to OPNs alone.

  • Community Categories Route Ingress LocationField Peering4006:65020Exchange Point Peer4006:65010

    Northeast Region Peering (DC)4006:65030Southeast Region Peering (Atlanta)4006:65040Northcentral Region Peering (Chicago)4006:65050West Peering Region (Palo Alto)4006:65060Southcentral Region Peering (Dallas)4006:65070

  • Community Categories SpecialsNo Export to any external BGP peerNo-ExportDo Not Advertise to any peer (Well Known)No-AdvertiseAlways Prefer (proposed Well Known)Prefer-Me (65535:65519)Always Avoid (proposed Well Known)Avoid-Me (65535:65504)

  • Community Categories Origin ASAlso add a community string for the origin AS

    If the route comes from UUNet, then add 4006:701If the route comes from Sprint, then add 4006:1239

  • router bgp 4355neighbor b.b.b.b remote-as 4006neighbor b.b.b.b route-map setlocpref90 in

    router bgp 4355neighbor c.c.c.c remote-as 701neighbor c.c.c.c route-map setlocpref60 in

    Local PreferenceAS4006AS3703AS4355peeringcustomer165.200.1.0/241003703 i165.200.1.0/241 3703 i165.200.1.0/24irouter bgp 4355neighbor a.a.a.a remote-as 3703neighbor a.a.a.a route-map setlocpref100 in 3703 iAS701165.200.1.0/241239 3703 i transit165.200.1.0/24904006 3703 i

  • Local Preference HierarchyThe higher the Local Preference, the more desirable the route.Customers ALWAYS come first we never want to send their traffic to a peer, regardless of AS-Path paddingPrivate Peering is always more desirable than Public PeeringTransit is less desirable than private peering for economic reasons

  • Local Preference HierarchyAlways Preferred250Customer Routes100Customer Backup Routes90Private Peering80Less Preferred Private Peering (congested)70Paid Transit 60 Less Preferred Paid Transit (congested)50Public Peering (ATM NAPs)40Less Preferred Public Peering (FDDI NAPs)30Never Preferred1

  • Peer TypesLocal sub-AS Peer (within a POP)Confederation Peers (other POPs or sub-ASes)Transit Peers (we buy transit from them)Public/Private Peering Customer Peers

  • Local sub-AS PeersAll peers within a POP are members of this group.The update source for these BGP sessions will be the loopback address of the router.Communities must be recognized.Option to use full-mesh or route-reflectors.

    For Each Local Sub-AS Peerneighbor remote-as neighbor description otherlocalrouternameneighbor update-source loopback0neighbor send-communityneighbor version 4

  • Update-Source Loopback AddressThe routers will use loopback address as the source of the bgp packets. Only one session needs to be created even with multiple paths between routers.Peering between loopback addresses increase the stability of the bgp sessions since loopback addresses dont go down.

  • Confederation PeersAll peers that are POP border routers are members of this group.The update source for these BGP sessions will be the facing interface of the router.Inbound Soft Reconfiguration is not necessary.Outbound soft reconfiguration can be done at the remote endCommunities must be recognized.Filtering is done on egress, MEDs are set on ingress.

  • Soft Reconfigurationclear ip bgp drops the TCP session. Soft reconfiguration is much friendlier.

    clear ip bgp soft out issues withdrawals for all advertised routes, recomputes and then resends the routes (low cpu)

    clear ip bgp soft in reevaluates routes received from its peers stored in memory. (high memory requirements)

  • Confederation Peer ConfigurationPeer-Groupneighbor internal peer-groupneighbor inte