seamless bgp migration with router grafting eric keller, jennifer rexford princeton university kobus...
TRANSCRIPT
![Page 1: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/1.jpg)
Seamless BGP Migration withRouter Grafting
Eric Keller, Jennifer Rexford
Princeton University
Kobus van der Merwe
AT&T Research
NSDI 2010
![Page 2: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/2.jpg)
Dealing with Change
2
• Networks need to be highly reliable– To avoid service disruptions
• Operators need to deal with change– Install, maintain, upgrade, or decommission equipment– Deploy new services– Manage resource usage (CPU, bandwidth)
• But… change causes disruption– Forcing a tradeoff
![Page 3: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/3.jpg)
Why is Change so Hard?
• Root cause is the monolithic view of a router (Hardware, software, and links as one entity)
3
![Page 4: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/4.jpg)
Why is Change so Hard?
• Root cause is the monolithic view of a router (Hardware, software, and links as one entity)
4
Revisit the design to make dealing with change easier
![Page 5: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/5.jpg)
Our Approach: Grafting
• In nature: take from one, merge into another– Plants, skin, tissue
• Router Grafting– To break the monolithic view– Focus on moving link (and corresponding BGP session)
5
![Page 6: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/6.jpg)
Why Move Links?
6
![Page 7: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/7.jpg)
Planned Maintenance
7
• Shut down router to…– Replace power supply– Upgrade to new model– Contract network
• Add router to…– Expand network
![Page 8: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/8.jpg)
Planned Maintenance
• Could migrate links to other routers– Away from router being shutdown, or– To router being added (or brought back up)
8
![Page 9: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/9.jpg)
Customer Requests a Feature
Network has mixture of routers from different vendors* Rehome customer to router with needed feature
9
![Page 10: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/10.jpg)
Traffic Management
Typical traffic engineering: * adjust routing protocol parameters based on traffic
Congested link
10
![Page 11: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/11.jpg)
Traffic Management
Instead…* Rehome customer to change traffic matrix
11
![Page 12: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/12.jpg)
Understanding the Disruption (today)
12delete neighbor 1.2.3.4
Add neighbor 1.2.3.4
BGP updates
1) Reconfigure old router, remove old link
2) Add new link link, configure new router
3) Establish new BGP session (exchange routes)
updates
![Page 13: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/13.jpg)
Understanding the Disruption (today)
13
1) Reconfigure old router, remove old link
2) Add new link link, configure new router
3) Establish new BGP session (exchange routes)
Downtime (Minutes)
![Page 14: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/14.jpg)
Router Grafting: Breaking up the router
14
Send state
Move link
![Page 15: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/15.jpg)
Router Grafting: Breaking up the router
15
Router Grafting enables this breaking apart a router (splitting/merging).
![Page 16: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/16.jpg)
Not Just State Transfer
16
Migrate session
AS100AS200 AS400
AS300
![Page 17: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/17.jpg)
Not Just State Transfer
17
Migrate session
AS100AS200 AS400
AS300
The topology changes(Need to re-run decision processes)
![Page 18: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/18.jpg)
Goals
• Routing and forwarding should not be disrupted– Data packets are not dropped– Routing protocol adjacencies do not go down– All route announcements are received
• Change should be transparent– Neighboring routers/operators should not be involved– Redesign the routers not the protocols
18
![Page 19: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/19.jpg)
Challenge: Protocol Layers
BGP
TCP
IP
BGP
TCP
IP
MigrateLink
MigrateState
Exchange routes
Deliver reliable stream
Send packets
Physical Link
A B
C
19
![Page 20: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/20.jpg)
Physical Link
BGP
TCP
IP
BGP
TCP
IP
MigrateLink
MigrateState
Exchange routes
Deliver reliable stream
Send packets
Physical Link
A B
C
20
![Page 21: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/21.jpg)
• Unplugging cable would be disruptive
Remote end-point
Migrate-from
Migrate-to
21
Physical Link
![Page 22: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/22.jpg)
mi
• Unplugging cable would be disruptive
• Links are not physical wires– Switchover in nanoseconds
Remote end-point
Migrate-from
Migrate-to
22
Physical Link
![Page 23: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/23.jpg)
IP
BGP
TCP
IP
BGP
TCP
IP
MigrateLink
MigrateState
Exchange routes
Deliver reliable stream
Send packets
Physical Link
A B
C
23
![Page 24: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/24.jpg)
• IP address is an identifier in BGP
• Changing it would require neighbor to reconfigure– Not transparent– Also has impact on TCP (later)
24
Changing IP Address
mi
Remote end-point
Migrate-from
Migrate-to
1.1.1.11.1.1.2
![Page 25: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/25.jpg)
• IP address not used for global reachability– Can move with BGP session– Neighbor doesn’t have to reconfigure
25
Re-assign IP Address
mi
Remote end-point
Migrate-from
Migrate-to
1.1.1.1
1.1.1.2
![Page 26: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/26.jpg)
TCP
BGP
TCP
IP
BGP
TCP
IP
MigrateLink
MigrateState
Exchange routes
Deliver reliable stream
Send packets
Physical Link
A B
C
26
![Page 27: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/27.jpg)
Dealing with TCP
• TCP sessions are long running in BGP– Killing it implicitly signals the router is down
• BGP and TCP extensions as a workaround(not supported on all routers)
27
![Page 28: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/28.jpg)
Migrating TCP Transparently
• Capitalize on IP address not changing– To keep it completely transparent
• Transfer the TCP session state– Sequence numbers– Packet input/output queue (packets not read/ack’d)
28
TCP(data, seq, …)
send()
ack
TCP(data’, seq’)
recv()app
OS
![Page 29: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/29.jpg)
BGP
BGP
TCP
IP
BGP
TCP
IP
MigrateLink
MigrateState
Exchange routes
Deliver reliable stream
Send packets
Physical Link
A B
C
29
![Page 30: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/30.jpg)
BGP: What (not) to Migrate
• Requirements– Want data packets to be delivered– Want routing adjacencies to remain up
• Need– Configuration– Routing information
• Do not need (but can have)– State machine– Statistics– Timers
• Keeps code modifications to a minimum30
![Page 31: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/31.jpg)
Routing Information
mi
• Could involve remote end-point– Similar exchange as with a new BGP session– Migrate-to router sends entire state to remote end-point– Ask remote-end point to re-send all routes it advertised
• Disruptive – Makes remote end-point do significant work
31
Remote end-point
Exchange Routes
Migrate-from
Migrate-to
![Page 32: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/32.jpg)
Routing Information (optimization)
mi
Migrate-from router send the migrate-to router:
• The routes it learned– Instead of making remote end-point re-announce
• The routes it advertised– So able to send just an incremental update
32
Remote end-point
Migrate-from
Migrate-to
IncrementalUpdate
Send routes advertised/learned
![Page 33: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/33.jpg)
Migration in The Background
RemoteEnd-point
Migrate-to
Migrate-from
33
• Migration takes a while– A lot of routing state to transfer– A lot of processing is needed
• Routing changes can happen at any time
• Disruptive if not done in the background
![Page 34: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/34.jpg)
While exporting routing state
In-memory:p1, p2, p3, p4
Dump:p1, p2
RemoteEnd-point
Migrate-to
Migrate-from
34
BGP is incremental, append update
![Page 35: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/35.jpg)
While moving TCP session and link
RemoteEnd-point
Migrate-to
Migrate-from
35
TCP will retransmit
![Page 36: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/36.jpg)
While importing routing state
RemoteEnd-point
Migrate-to
Migrate-from
36
In-memory:p1, p2
Dump:p1, p2, p3, p4
BGP is incremental, ignore dump file
![Page 37: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/37.jpg)
Special Case: Cluster Router
37
SwitchingFabric
Blade
Line card
Line card
Line card
Line card
A
B
C
D
Blade
A B C D
• Don’t need to re-run decision processes
• Links ‘migrated’ internally
![Page 38: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/38.jpg)
Prototype
• Added grafting into Quagga– Import/export routes, new ‘inactive’ state– Routing data and decision process well separated
• Graft daemon to control process
• SockMi for TCP migration
38
ModifiedQuagga
graftdaemon
Linux kernel 2.6.19.7
SockMi.ko
Graftable Router
HandlerComm
Linux kernel 2.6.19.7-click
click.ko
Emulatedlink migration
Quagga
Unmod.Router
Linux kernel 2.6.19.7
![Page 39: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/39.jpg)
Evaluation
• Impact on migrating routers
• Disruption to network operation
• Overhead on rest of the network
39
![Page 40: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/40.jpg)
Evaluation
• Impact on migrating routers
• Disruption to network operation
• Overhead on rest of the network
40
![Page 41: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/41.jpg)
Impact on Migrating Routers
• How long migration takes– Includes export, transmit, import, lookup, decision– CPU Utilization roughly 25%
41
Between Routers0.9s (20k) 6.9s (200k)
Between Blades0.3s (20k) 3.1s (200k)
![Page 42: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/42.jpg)
Disruption to Network Operation
• Data traffic affected by not having a link– nanoseconds
• Routing protocols affected by unresponsiveness– Set old router to “inactive”, migrate link, migrate TCP, set
new router to “active”– milliseconds
42
![Page 43: Seamless BGP Migration with Router Grafting Eric Keller, Jennifer Rexford Princeton University Kobus van der Merwe AT&T Research NSDI 2010](https://reader036.vdocument.in/reader036/viewer/2022062515/56649c905503460f94949aed/html5/thumbnails/43.jpg)
Conclusions and Future Work
• Enables moving a single link/session with…– Minimal code change– No impact on data traffic– No visible impact on routing protocol adjacencies– Minimal overhead on rest of network
• Future work– Explore applications– Generalize grafting
(multiple sessions, different protocols, other resources)
43