Protocol implementation

• Next-hop resolution

• Reliability and graceful restart

What is a next-hop• The destination of the packets I am sending

– Not the same as the interface– An ethernet interface will have many nodes behind it – Directly connected next hop is 1 hop away

• E.g. RSVP sends a PATH message to the next downstream node – Next hop may be directly connected (strict ERO)– Or not (loose ERO)

• OSPF sends an LS update to the other end of a link or a neighbor on an eithernet – Always directly connected

• BGP has an iBGP-next hop for each of its paths – Not directly connected

Next-hop

• If the next hop is not directly connected the way to reach it depends on the IGP– May change when IGP routing changes– Will have to use a different interface to reach it – Need to keep track of these changes

• Next hop resolution

Next hop resolution • Periodic resolution

– may take a bit more time • But next-hops will not be too many• Or will they? Tunnels, VLANs …

– Quagga uses this approach • Through the IPV4_LOOKUP_NEXTHOP command

• Registration/notification – RSVP would tell zebra which nexthops it is interested

in – Zebra will notify RSVP when something changes in the

IGP path to it• Better scaling for RSVP • Difficult to ensure good scaling inside zebra

– Various protocols may register 1000s of next hops

• More complex code in zebra

Network Reliability • Availability: How many nines?

– 99.999% is 5.26 min down time/year– 99.9999% is 31.5 sec down time/year

• Telephone networks are between 5 and 6 nines– Internet will have to get there – Currently at 4 nines? (vendors claim 5) – Very important with the new types of traffic

• Voip, Ipvt

• What can go wrong (% of failures for US telephone network ca. 1992): – Hardware failures (19%)– Software failures (14%) – Human errors (49%) – Vandalism/Terrorism – Acts of nature (11%)– Overload (6% but had the largest impact on customers)

Hardware failures • Link failures

– Protocols can cope with that • Re-route, may be slow • More aggressive repair methods

– we will see them later

• Router failures– Can not do much just add redundancy

• Power supplies, fans, disks, etc

– Line-card failure is similar to a link failure– Control processor failure is more serious

• Always have two of them• Primary and backup

Modern Router architectures

• Dual controllers – For running the control plane

• Multiple line-cards – Can operate without the controllers – Router can forward traffic even when the

control plane crashes– Called non-stop forwarding or head-less

operation

Software failures • When primary fails start using backup

– Switchover

• Must be as fast as possible – Things in the network change in the meanwhile – Need to minimize this window

• What happens with the control software – Need to keep primary and backup instance in

sync– How tight is this synchronization?

Tight synchronization• Both primary and backup are active, keep

them in sync by:• Send them both the same input (I.e.

duplicate control packets)– Fastest possible switchover – Expensive, may need to duplicate packets– Does not work for TCP based protocols

• The primary keeps sending state updates to the backup – May need to send too many messages

• Being totally in-sync is not easy – Needs transactional communication

Loose synchronization

• Backup is idle– But we keep configuration up to date – Each configuration change on the primary is mirrored on the

backup

• Backup instance is started when the primary fails– Switchover will take longer

• Much-much simpler – Configuration changes are much less

• Variation:– Keep only the RIB process in sync in both primary and

backup

Non-stop forwarding

• Key concept– forwarding happens in the line cards

– Even if control processor fails forwarding can continue

– Non stop forwarding, head-less operation

• Old Common sense: when router s/w crashes do not use the router – But with head-less operation it is ok to continue using

routers that their s/w crashed

– Assuming their s/w will be operational again soon

Special Case

• Planned restart – For s/w upgrade

• These are a significant percentage of downtime

– For refresh • Memory is leaking but s/w still operational

• Restart to get a clean start

• I can use graceful restart

Graceful Restart• Other routers in the network will keep using a

neighbor router – Even if is looks like its control plane has failed – Assuming it will come back soon

• Needs coordination– The failed router needs to do some special processing

when it comes back– It has to tell its neighbors first that it supports graceful

restart

• Zero impact on the network – The failed router will have the chance to restart its s/w

and come back– Nobody in the rest of the network will know that

something happened

How does it work• Used for all protocols by now

– OSPF, BGP, RSVP-TE…

• The neighbor will discover that the router is dead or it has restarted – HELLO timeout, different information in the

HELLOs etc… – But will ignore it for a certain time period

• If the failed router comes back within this period – It will re-sync its state (database exchange for

OSPF, resend all the LSPs for RSVP, …)– And all is back to normal

Example RSVP • Use HELLOs

• Special recovery label messages

• Restarting router needs to remember the labels it allocated before the crash – Where?

• Shared memory

• recover them from the forwarding plane

– Why? • Must use the same labels again

• Must make sure it does not use an allocated label for some other LSP

Example OSPF

• Trick is to re-establish the adjacencies after a failure

• Remember the set of neighbors – Shared memory or in the backup controller

• After restart do not originate any LSAs

• Just re-establish adjacencies and re-sync database

Graceful restart catches • All routers in the network should implement

this to work

• Mostly for planned restarts:– S/w upgrades – Refreshes (if a router runs low on memory)– But it is possible to use for crashes too!

• It can not work if something changes in the network while the restart is going on– There may be routing loops

Router self-monitoring • Automatically restart failed or stuck

processes

• A separate monitor process – Keeps an eye on other processes– If there is a failure the failed process is restarted

• Of course it may fail again

– Heart-beats to determine liveness – Failure may not necessarily be a crash

• Could be a software bug that causes an infinite loop or very-very slow processing

Why is it important

• Remember the PoP structure – Need dual routers for reliability – If I had a single router that was extra-reliable I

could save a lot of money

Issues

• Strict Isolation – VMs – Other methods

• Global resource coordination– For example memory