1 routing design in operational networks: a look from the inside david a. maltz, geoffrey xie, jibin...
TRANSCRIPT
1
Routing Design in Operational Networks:A Look from the Inside
David A. Maltz, Geoffrey Xie, Jibin Zhan, Hui ZhangCarnegie Mellon University
Gisli Hjalmtysson, Albert GreenbergATT Labs Research
2
The Problem of Routing Design
HostA
HostB
Ho
stC
Ho
stD
eBGPiBGP
OSPFRIP
EIGRP
ACLspolicy
3
Many Routing Designs Possible
A
B
C
A
B
C
A B C
A
B
C
A
B
C
Drop A<->B
Drop A<->B
Drop A<->B
AS1
AS2
AS3
Multiple OSPF instances
Packet filters
Multiple AS’s & BGP
4
Routing Design
• Selecting routing protocols• Configuring their boundaries• Setting the policies that control their interaction• Adding packet filters, other mechanisms
Routing design fundamentally establishes the network’s properties
• Remains constant as network expands• Details of protocol, path computation are
second-order effects• Topology doesn’t say much about reachability
5
Reachability Example
• Enterprise with two remote offices• Only A&B should be able to talk to server C
Internet
A
B
C
Permit B
->C
Permit A->C
6
Reachability Example
• Network designers add two links for robustness• Configure routing protocols to use new links in
failure
Internet
A
B
C
Permit B
->C
Permit A->C
7
Reachability Example
• Designers apply packet filters to new links
Internet
A
B
C
Permit B
->C
Permit A->C
Permit A->C
Perm
it A->
C
Per
mit
B->
C
8
Reachability Example
Internet
A
B
CPermit A->C
Permit A->C
Perm
it A->
C
Per
mit
B->
C
9
Reachability Example
• Packet from B->C dropped!• Testing under normal conditions won’t find this
error!
Internet
A
B
CPermit A->C
Permit A->C
Perm
it A->
C
Per
mit
B->
C
10
How Are Routing Designs Expressed Today?
interface Ethernet0
ip address 6.2.5.14 255.255.255.128
interface Serial1/0.5 point-to-point
ip address 6.2.2.85 255.255.255.252
ip access-group 143 in
frame-relay interface-dlci 28
router ospf 64
redistribute connected subnets
redistribute bgp 64780 metric 1 subnets
network 66.251.75.128 0.0.0.127 area 0router bgp 64780 redistribute ospf 64 match route-map 8aTzlvBrbaW neighbor 66.253.160.68 remote-as 12762 neighbor 66.253.160.68 distribute-list 4 in
access-list 143 deny 1.1.0.0/16access-list 143 permit anyroute-map 8aTzlvBrbaW deny 10 match ip address 4route-map 8aTzlvBrbaW permit 20 match ip address 7ip route 10.2.2.1/16 10.2.1.7
11
Lots of Configuration Files
Router ID8810
Lines in
config file
2000
1000
0
12
Problems with State of the Art
No good way to visualize or describe routing design
Impossible to establish linkage between configurations and routing design
Only a few “textbook” routing designs are widely known
13
Approaches?Option 1: High-level design compiled down to
configuration commands• Feasible?• What are the constructs?• How to capture design intents?
Our starting point: Bottom-up white-box approach• Start with router configuration files• Reverse-engineer the routing design
14
Contributions
Abstractions for modeling routing design• Routing Process Graph• Routing Instance Graph
Reverse-engineering methodology• Anonymization of configuration files• Tools to convert configurations into model
Study of 31 production networks using model• Verified some bits of common wisdom• Found counter examples for other bits
15
Router Model
OSPF BGP OSPF
Route Selection
Route Table
Router 1
16
Route Redistribution
OSPF BGP OSPF
Route Selection
Route Table
Router 1
Routing policy 1 Routing policy 2
17
Routing Protocol Adjacencies
OSPF BGP OSPF
Route Selection
Route Table
OSPF
RS
Route Table
Router 1Router 2
Routing policy 1 Routing policy 2
18
Reverse-Engineering Overview
Configuration files
Find links
Find adjacent routing processes
Construct Routing Process Graph
Condense adjacent routing processes
Construct Routing Instance Graph
Construct Layer 3 Topology
OSPF #1 OSPF #2BGP AS1
AS2
19
Reconstruct the Layer 3 Topology
interface Serial1/0.5
ip address 1.1.1.1 255.255.255.252
….
Router 1 Config
interface Serial2/1.5
ip address 1.1.1.2 255.255.255.252
….
Router 2 Config
Internet
20
Construct the Routing Process Graph
OSPF BGP OSPF
Route TableRT
OSPF
RT
OSPF
RT
OSPF
RT
OSPF
EB
GPPolicy1 Policy2
Internet
21
Abstract to a Routing Instance Graph
• Pick an unassigned Routing Process• Flood fill along process adjacencies, labeling processes• Repeat until all processes assigned to an Instance
OSPF #1 OSPF #2BGP AS1
EBGPAS2
Policy1 Policy2
OSPF BGP OSPF
Route TableRT
OSPF
RT
OSPF
RT
OSPF
RT
OSPF
22
Abstract to a Routing Instance Graph
Router1 Router2
OSPF BGP OSPF
Route TableRT
OSPF
RT
OSPF
RT
OSPF
RT
OSPF
Router2Router1
OSPF #1 OSPF #2BGP AS1
EBGPAS2
Policy1 Policy2
23
A Study of Operational Production Networks
Obtained anonymized configuration files for 31 active networks (>8,000 configuration files)
Networks include:• 6 Tier-1 and Tier-2 Internet backbone networks• 25 enterprise networks
Sizes between 10 and 1,200 routers• 4 enterprise networks significantly larger than the
backbone networks
Networks created by diverse set of designers and companies
24
Textbook Routing Design for Enterprise Networks
• Border routers speak eBGP to external peers• BGP selects a few key external routes to redistribute
into OSPF• 7 of 25 enterprise networks follow this pattern
OSPFBGP
AS #1
EBGPEBGP
AS2
AS3
25
Reality: A Diversity of Unusual Routing Designs
• Network broken up into compartments, each with only 1 to 4 routers
• Each compartment has its own AS number• Hub and spoke logical topology• Why? Lots of control over how spokes communicate
BGPAS #1
BGPAS #4
BGPAS #2
BGPAS #3
BGPAS #5
EBGPEBGP
EBGP
EBGP
Rest of the
World
Rest of the
World
26
Routing Design for 900 Router Network
27
Reality: A Diversity of Unusual Routing Designs
• Network broken up into many compartments, each running EIGRP, some with 400+ routers
• BGP used to filter routes passed between compartments• Compartments themselves pass information between BGP speakers• Why? Little need for IBGP; few routers speak BGP; Lots of control
over how packets move between compartments
BGPAS #1
EBGP
EBGPRest of
the World
Rest of the
World
EIGRP
BGPAS #2
EIGRPEIGRP
BGPAS #3
BGPAS #4
Rest of the
World
Rest of the
World
EBGP
EBGP
28
Myth: Policy Enforced at Edge of Network
Conventional wisdom:• Place packet filters on the edge to defend
infrastructure• Routing policy applied where networks touch
29
Reality: Policy Exists Throughout Networks
Packet filters commonly used on internal links• Protect routers from attack• Implement reachability matrix
– Prevent some hosts from communicating with others– Localize traffic, particularly multicast
30
Summary
Developed abstractions to model routing design• Routing Instance – abstracts away details• Reverse-engineer routing design from configs We presented our extracted design to designers• They agreed we captured their design intentFocusing on individual protocols is not enough• Understanding composition is equally important
First step towards turning routing design from an art into a science
31
Applications of Routing Design Analysis
Enables static analysis of network properties
Reachability/security analysis– Route leaks? Reachability violations?
Robustness analysis– How sensitive is the network to external
events such as route announcements?
Resource usage analysis– Will a particular configuration cause the
routing table of a router to overflow?
32
The Value of Investigating Routing Design:Next Steps
Found many different designs in use• Do we need so many designs?• Framework to ask and answer questions of
scalability, completeness, optimality
Do we have the right abstractions?• Is this the right way to program routers?• Suggest improvements to protocols and
configuration languages• Can the network be run using abstractions?
33
Questions?
34
Textbook Routing Design for Backbone Network
• Border routers speak eBGP to external peers• All routers speak iBGP with each other• All routers participate in both BGP and OSPF (learning
infrastructure routes from OSPF, external routes from BGP)
OSPFBGP
AS #1
EBGPEBGP
EBGP
EBGP
EBGP
EBGPIGBP MESH
AS2
AS3
ASn
…
35
Real Routing Designs for Backbone Networks
• All 6 backbone networks used basic OSPF/BGP pattern
OSPFBGP
AS #1
AS2
AS3
ASn
36
Real Routing Designs for Backbone Networks
• All 6 backbone networks used basic OSPF/BGP pattern• 3 of 6 include many additional routing instances• Used to exchange routes with customers
OSPFBGP
AS #1
AS2
AS3
ASn
RIP
EIGRP
Customer
EIGRP
37
BGP Used an IGP
38
What do Designers do Today?
Network designers balance many goals• Scalability• Resiliency to failure• Make it easy to expand network
Many “rules of thumb” in use• Instability results from overloaded routers• Too much routing state is bad• Use routing boundaries to control spread of change
Routing Design is currently an art – can we add more science?
39
Approaches?Need deeper understanding than network topology
Need broader study than backbone networks
Interviewing network designers isn’t enough• No language/visualization exists for communicating
about routing design• Documentation is out-of-date or non-existent
Our approach: Bottom-up white-box• Start with router configuration files• Reverse-engineer the routing design
40
Potential ApproachesTop-down design problem• How should networks be designed?
First must understand what happens in real networks
Bottom-up black-box approach• Send probe traffic to explore network properties• Very successful at recovering topology [RocketFuel]
[Skitter] [Mercator]
Measured topology a result of a routing design --- it does not expose the routing design itself
Our approach: Bottom-up white-box• Start with router configuration files• Reverse-engineer the routing design
41
Router Configuration Files
42
Lots of Configuration Files