designenterprise network availability - sunet-projekt...

1© 1999, Cisco Systems, Inc. 5050911_04F9_c3

HighHigh--AvailabilityAvailabilityEnterprise NetworkEnterprise Network

DesignDesign

[email protected]@cisco.com

25050911_04F9_c3 © 1999, Cisco Systems, Inc.

Staying On TargetStaying On TargetHA Focus HA Focus vs vs Distractions!Distractions!

“Flat networksare easier”beware!

Five nines is job one!

Inherited complexityhard to purge

The latest cool stuffolder is more stable

“Variety” of vendors, protocols, designs, etc.

“Featurerich”let’s use all the knobs!

Change is hard, sometimes $$$


HA Features of the Catalyst 6500HA Features of the Catalyst 6500Consider for Backbones & Server FarmsConsider for Backbones & Server Farms

✔ Fabric Redundancyswitch fabric modulein CatOS 6.1

✔ Supervisor RedundancyHA feature in CatOS 5.4.1stateful recoveryimage versioning on the fly

✔ MSFC Redundancyconfig-sync featureIOS 12.1.3 CatOS 6.1HSRP pair


Thinking Outside the BoxThinking Outside the Box

For HA/HP design “outside the box”

☛ the logical design is critical☛ network features & protocols☛ geophysical diversity is powerful

Inside: “HA”, RAID, UPS, MTBF, etc.


Dramatis Dramatis PersonaePersonaeOur Cast of SymbolsOur Cast of Symbols

✔ Links GE, DPT, SONET, etc.

✔ L2 switchingL2 forwarding in hardware

✔ L3 switchingL3/L2 forwarding in hardware

✔ Routing L3 forwarding (SW or HW)

✔ Control plane = IOS routing protocols & features

✔ QoS where required✔ Application intelligence

Catalyst 4000

Cisco 7500 Cisco 12000

Catalyst 6500

GigE Channel


Client Blocks

Distribution L3

Access L2

HA Gigabit Campus Architecturesurvivable modules + survivable backbone

Backbone

ServerBlock

Server Farm

Distribution L3

Access L2E or FE PortGE or GEC

Ethernet or ATMLayer 2 or Layer 3

☛ Define the missioncritical parts first!


High Availability DesignHigh Availability DesignWhy aWhy a Modular ABC ApproachModular ABC Approach

✔ Many new products, features, technologies

✔ HA and HP application operation is the goal

✔ Start with modular, structured approach (the “logical” design)

✔ Add multicast, VoIP, DPT, DWDM...


Price per 10/100

Catalyst 2912GCatalyst 2948GCatalyst 2980G

242410/100 Ports10/100 PortsGigabit PortsGigabit Ports

24-500+24-500+ 24-350+24-350+3-38+3-38+ 8-64+8-64+

Catalyst 5XXX

32-9632-966-126-12

Catalyst 4XXX

$100

$200

$250

$300

$350

Switching CapacitySwitching Capacity Up to 72 MppsUp to 72 Mpps20 Mpps20 Mpps Up to 150 MppsUp to 150 MppsBackplaneBackplane 24 Gbps24 Gbps 1.2-3.6 + 10Gbps1.2-3.6 + 10Gbps 250+ Gbps250+ Gbps

NewNew

NewNew

New New ModulesModules

Catalyst 6XXX

Design the SolutionDesign the SolutionThen Pick the ProductsThen Pick the Products

New New ModulesModules


HA Design Reality Check!HA Design Reality Check!Assume Things Fail Assume Things Fail -- Then What?Then What?

✔ Networks are complex✔ Things break, people make mistakes✔ What happens if a failure occurs?✔ Simple, structured, deterministic design

required for fast recovery✔ The “tradeoffs”

your choices are important


Layer 2Layer 2

Layer 2Layer 2AccessAccess

DistributionDistribution

BuildingBuilding

Core L3Core L3

ServerServerDistributionDistribution

Server FarmServer Farm

Layer 3Layer 3

3

2

1

5

6

BranchesBranches

WAN

WANbackup

4

Network RecoveryNetwork RecoveryHow Long? What Happens?How Long? What Happens?


Failure ScenarioFailure

Scenario

1,2 server

3,4 uplink

5,6 core

dual-path L3

EtherChannel

L3 routing

L2 general

DPT

1,2 server

3,4 uplink

5,6 core

dual-path L3

EtherChannel

L3 routing

L2 general

DPT

RecoveryMode

RecoveryMode

RecoveryTime

RecoveryTime

Server NIC

HSRP (& UplinkFast)

HSRP track

alternate path used

channel recovery

EIGRP or OSPF

L2 spanning tree

IPS

Server NIC

HSRP (& UplinkFast)

HSRP track

alternate path used

channel recovery

EIGRP or OSPF

L2 spanning tree

IPS

< 2 seconds

tune to 3 seconds

tune to 3 seconds

< 2 seconds

< 1 second

depends on tuning

tune (up to 50 seconds)

50 milliseconds

< 2 seconds

tune to 3 seconds

tune to 3 seconds

< 2 seconds

< 1 second

depends on tuning

tune (up to 50 seconds)

50 milliseconds

Network Recovery TimesNetwork Recovery TimesIf You Follow the RulesIf You Follow the Rules


Design for High AvailabilityDesign for High AvailabilityHow to Build Boring Networks!How to Build Boring Networks!

✔ The Concepts✔ The Rules✔ Design Building Block✔ Design Backbone✔ Notes on Tuning


HA Network Design ConceptsHA Network Design Conceptsthinking outside the boxthinking outside the box

1) Simplicity & Determinism2) Collapse the Sandwich3) Spanning Tree Failure Domain4) Map L3 to L2 to L15) Scaling and Hierarchy6) ABCs of Module + Backbone

Design7) The Four Corners


1) Simplicity and Determinism1) Simplicity and Determinismreducing the degrees of freedomreducing the degrees of freedom

✔ Every Choice Affects Availability!✔ Determinism or Flexibility?✔ Would you support 27 desktop environments?✔ Would you support 13 network vendors?✔ Would you use 57 varieties of Cisco IOS?

FlexibleComplexVaried

SimpleStructuredDeterministic “HA Continuum”

Boring! Interesting!


TraditionalModel

Fiber

SONET

Big Fat Pipe

• Lower equipment cost• Lower operational cost• Simplified architecture• Scalable capacity

Optical Internetworking

Fiber

IP

FR/ATM

IP

2)2) Collapse the SandwichCollapse the Sandwichroute IP over glassroute IP over glass

Service

Traffic Eng

Fiber Mgmt


33)) Minimize the Failure DomainMinimize the Failure Domainpublic enemy number onepublic enemy number one

Where should root go?

What happens when something breaks?

How long to converge?

Many blocking links

Large failure domain!

Broadcast flooding

Multicast flooding

Loops within loops

ST from heck

Times 100 VLANs?

avoid highly meshed, non-deterministic large scale L2 = VLAN topology

Building 1 Building 2

Building 3 Building 4


4)4) Map L3 to L2 to L1Map L3 to L2 to L1

✔ Easier administration & troubleshooting

Clients in subnet 10.0.55.0

VLAN 55

wiring closet “55” on floor 55

access switch “55”

interface VLAN 55

all match and life is good

go fishing with your kids

10/100 BaseT

GE or GEC


5) Scaling and Hierarchy5) Scaling and Hierarchy

Strong hierarchies like telephone system and Internet segment addressing and therefore scale

U

C

N

U

C

N

U

C

N

C complexityU unmanageableN number of devices

Flat L2 Ethernet is easy but does not scale

ATM LANE is logically flat, scales as N squared


6)6) Building Block &Building Block &Backbone Design ABCsBackbone Design ABCs

WAN

EcommerceSolution

PSTN


CoreCore

LAN AccessLAN Access


Server Farm

Internet

A design bb

B design BB

C connect bb to BB

Divide and conquer

Cookie cutter configuration

Deterministic

L3 demarcation

WAN AccessWAN Access


7) Four Square Network Redundancy7) Four Square Network Redundancyor the Four Corners Problemor the Four Corners Problem

One ChassisOne Chassis Two ChassisTwo Chassis

One One SupervisorSupervisor

Two Two SupervisorsSupervisors

SimplestSimplestNo RedundancyNo Redundancy

Most ComplexMost ComplexBelt and SuspendersBelt and Suspenders

GeoPhysicalGeoPhysicalEffectiveEffective

When space When space is limitedis limited

“HA”“HA”

L3L3


Dos and Don’ts for HA DesignDos and Don’ts for HA Design

1) Eliminate STP Loops2) L3 Dual-Path Design3) EtherChannel Across Cards4) Workgroup Servers5) Use HSRP Track6) Passive Interfaces7) Issues with Single-Path Design8) Oversubscription Guidelines9) HA for single attached servers10) Protocol Tradeoffs11) UDLD Protection


Rule 1) Eliminate STP LoopsRule 1) Eliminate STP Loopsin the backbone and mission critical pointsin the backbone and mission critical points

No blocking links to waste bandwidth

Avoids slow STP convergence

Very deterministic

Routed links not VLAN trunks

L2 Gigabit switch in backbone

subnet X = VLAN X

Too many cooks spoil the brothL3 control is better

X.2 X.3X.1

Root VLAN X


Rule 2) Dual EqualRule 2) Dual Equal--Cost Path L3Cost Path L3

✔ Load balance - don’t waste bandwidthunlike L1 and L2 redundancy

✔ Fast recovery to remaining pathdetect L1 down & purge - about 1s

✔ Works with any routed fat pipes

Path A

Path B

Destination network X

Equal cost routes to XPath APath B


Rule 3) Rule 3) EtherChannel EtherChannel Across CardsAcross Cards

Increased availability✔ Sub second recovery✔ Spans cards on 6500✔ Up to 8 ports in channel

Small complexity increase✔ Single L2 STP link✔ Single L3 subnet✔ less if channel set “on”


Rule 4a) Connect Workgroup ServerRule 4a) Connect Workgroup Server�With no L2 recovery path, what happens if link

breaks ….

Workgroup server X.100attached to distribution layerL2 path to client X.1

Client X.1 VLAN X in purple includes clients and workgroup servers attached at different places.

A B

C

Links to core

Link CB breaks ….


Rule 4b) Connect Workgroup ServerRule 4b) Connect Workgroup Server• Subnet X now discontiguous• Incoming traffic gets dropped


Client X.1

Routers A & B continue to advertise reachability of subnet X ...

A B

C

X.1 not reachable

X.100 not reachable


Rule 4c) Connect Workgroup ServerRule 4c) Connect Workgroup Server• Introduce L2/STP redundancy • Adds a loop (band-aid fix)


Client X.1•VLAN trunk AB forms L2 loop •recovery path for STP•prevents black hole

A B

C


Rule 4d) Connect Workgroup ServerRule 4d) Connect Workgroup Server

Real Lessons:☛ Enterprise Server Farms are better☛ L3 demarcation is better☛ Example of why extended L2 is difficult


Rule 5a) Use HSRP TrackRule 5a) Use HSRP Track• Review - Hot Standby Router Protocol• Fast recovery can be tuned to 3s or less

X is M.100HSRP PrimaryPriority 200

Y ( becomes M.100)HSRP BackupPriority 100

Z

Router X acts as gateway router for subnet M, IP address M.100. If link Z fails router Ywill take over as M.100 gatewaywith same MAC address

10/100 BaseT

GE or GEC

Subnet Mhosts M.1 M.2 M.3


Rule 5b) Use HSRP TrackRule 5b) Use HSRP Track• Track extends HSRP to monitor links to backbone• Ensures shortest path - best outbound gateway

Track interface A - lower priority 75Track interface B - lower priority 75HSRP triggers if both A and B lost

10/100 BaseT

GE or GEC

X is M.100HSRP PrimaryPriority 200

Y ( becomes M.100)HSRP BackupPriority 100

Z

Subnet Mhosts M.1 M.2 M.3

A B


Rule 6a) Use Passive InterfacesRule 6a) Use Passive Interfaces

• L3 switches X & Y in distribution layer• 4 VLANs per wiring closet• 10 wiring closets

X Y

ABCD EFGH IJKL MNOP

… Ten totalWiringclosetswitch

Distributionswitch


Rule 6b) Use Passive InterfacesRule 6b) Use Passive Interfaces

• What X and Y see is 4*10=40 routed links• Increased protocol overhead & CPU

X Y

A

CB

DEFGEtc.

A.1

C.1B.1

D.1E.1F.1G.1Etc.

A.2

C.2B.2

D.2E.2F.2G.2Etc.


Rule 6c) Use Passive InterfacesRule 6c) Use Passive Interfaces

☛ Turns off routing updates & overhead☛ Leave two routed links for redundant paths☛ CDP, VTP, HSRP etc. still function on all links

X Y

A

CB

DEFGEtc.

A.1

C.1 (passive)B.1 (passive)

D.1 (passive)E.1F.1 (passive)G.1 (passive)Etc.

A.2

C.2 (passive)B.2 (passive)

D.2 (passive)E.2F.2 (passive)G.2 (passive)Etc.


Rule 7a) Issues With Single Path Rule 7a) Issues With Single Path DesignsDesigns

✔ L3 engine MSFC on core-X reloads

✔ Lights are on but nobody home - HSRP does not recover

✔ Remove passive interface to wiring closet subnets A, B

✔ Provide longer routed recovery path

Single pathto core

GE

Subnet A Subnet B

X

HSRPprimary

Core L3

Access L2

Y

New, longer outbound routes

Outbound case ...


Rule 7b) Issues with SingleRule 7b) Issues with Single--Path Path DesignDesign

✔ Recovery must take place in both directions

✔ Routing protocol recovers longer route from X to subnets A, B

✔ Therefore dual-path L3 is better & faster than single-path

Single pathto core

GE

Subnet A Subnet B

X

HSRPprimary

Core L3

Access L2

Y

New, longer routes to A, B

Inbound case ...


Rule 8a) Rule 8a) Oversubscription Oversubscription GuidelinesGuidelines

✔ Oversubscription part of all networks - not bad

✔ Non-blocking switches do not mean a non-blocking network

✔ You determine the amount of “blocking”

GE

GE

Non-blockingdesign

GE

GE

Blocking design 2:1

GE


Rule 8b) Rule 8b) Oversubscription Oversubscription GuidelinesGuidelines

✔ Oversubscription rules of thumb work well

✔ 20:1 at wiring closet ✔ Less in distribution and

server farm✔ QoS required IFF

congestion occurs✔ Protect real time flows

at congested points

n:1

20:1

Core L3use non-blocking switches

Dual-linkGEC

200 100BaseT

GE8 uplinks

DistributionL3


Rule 9) Dual SupervisorsRule 9) Dual SupervisorsHA for Single Attached ServersHA for Single Attached Servers

✔ Single point of failure✔ Dual supervisors - fast stateful recovery✔ No increase in complexity

10/100 BaseT

GE or GEC

Single attached servermission critical application

HA dual supervisorsCatalyst 6XXX

Redundant uplinks


Rule 10)Rule 10) Protocol TradeoffsProtocol TradeoffsAutomatic or Manual ConfigurationAutomatic or Manual Configuration

✔ Configuration up front rather than CPU overhead later, for example:

➙ set VTP mode transparent➙ set/clear VLANs for each trunk➙ set trunks on or off➙ set channel on or off

✔ Choose flexibility or determinism


Rule 11) Rule 11) UniDirectional UniDirectional Link Link Detection Detection

✔ UDLD detects mismatch when physical layer checks out OK

✔ Prevents various failure conditions including crossed wiring

Tx Fiber

Rx Fiber

The lights are on, BUT …..


Building Block Means SurvivableBuilding Block Means SurvivableSelfSelf--contained Backbonecontained Backbone

✔ Autonomous Survivability Unit - HSRP

✔ L3 Broadcast Multicast demarcation

✔ Cookie cutter configuration✔ L3 Demarcation of failure

domain✔ Simple, repeatable,

deterministic✔ Redundancy adds 15% cost

at mission critical points like server farm

L2L3

ASUdelimits failure domain


Building Block TemplatesBuilding Block TemplatesUse “As Is” or CombineUse “As Is” or Combine

1) Standard Modelsimple, structured

2) VLAN Modelmore flexible

3) Large Scale Server Farm Model

accommodate dual NIC4) Small Scale Server Farm

Modelaccommodate dual NIC


1) Standard Building Block1) Standard Building Blockno loops no loops -- no STP complexityno STP complexity

HSRP PrimarySubnets/VLANs10, 12, 14, 16

HSRP PrimarySubnets/VLANs11, 13, 15, 17

Access L2root switchVLAN 10/11

Subnet 10Subnet 11

GE/GECVLAN Trunks

10/100 BaseT

GE or GEC Dual Path with Tracking

Subnet 12Subnet 13

Subnet 14Subnet 15

Subnet 16Subnet 17

Highly DeterministicL1 maps L2 maps L3No blocking linksShortest path alwaysNot “flexible”


2) VLAN Building Block2) VLAN Building Blockmake L2 design match L3 designmake L2 design match L3 design

All All VLANs VLANs terminate at L3 boundaryterminate at L3 boundary

STP rootVLANs 10 12 14 16

HSRP primarysubnets 10 12 14 16

STP rootVLANs 11 13 15 17

HSRP primarysubnets 11 13 15 17

L2L3

All VLANsAll Subnets

GE/GECVLAN Trunks

Dual Path with Tracking




L2 Path

10/100 BaseT

GE or GEC

More flexibleFO forwarding oddBE blocking even etc.

FEBO

FOBE

FEBO

FOBE

FEBO

FOBE

FEBO

FOBE

L2L3

Uplink-Fast


3) Large3) Large--Scale Server Farm Scale Server Farm Building BlockBuilding Block

Dual-NIC ServerExample Fault Tolerant Mode (FTM)Same IP Address - seamless recovery

GE/GECVLAN Trunks


L2 Path

Access L2UplinkFast

10/100 BaseT

GE or GEC

based on VLAN building blockaggregates traffic - high BW

L2L3

L2L3

STP rootVLANs EVEN

HSRP primarysubnets EVEN

STP rootVLANs ODD

HSRP primarysubnets ODD


4) Small4) Small--Scale Server Farm Scale Server Farm Building BlockBuilding Block

Dual-NIC ServerExample Fault Tolerant Mode (FTM)Same IP Address - seamless recovery


L2 Path

10/100 BaseT

GE or GEC

Simplified building block with no STP loops

Use if port density permits

Use if no oversubscription (non-blocking) is a requirement

L2L3

L2L3 HSRP primary

subnets EVENHSRP primarysubnets ODD


Redundant Backbone ModelsRedundant Backbone Modelsall good all good -- increasing scaleincreasing scale

1) Collapsed L3 Backbone2) Full Mesh3) Partial Mesh4) Dual-Path L2 Switched5) Dual-Path L3 Switched


Core L3

Access L2

1) Collapsed L3 Backbonelarge building or small campus

Clients

Collapsed Backbone

GE/GECScale depends on physical plant and policy more than performance

Server Farm10/100 BaseT

GE or GEC


Client Blocks

Distribution L3

Access L2

2) Full Mesh Backbonesmall campus - n squared limitation

ServerBlock

Distribution L3

Access L2

Note importance of passive wiringcloset interfaces in meshed designs!

2 blocks - 6 peerings3 blocks - 15 peerings4 blocks - 28 peerings5 blocks - 45 peerings

E or FE PortGE or GEC


Distribution/Core L3

Access L2

Client Blocks

Distribution L3

Access L2

3) Partial Mesh Backbone3) Partial Mesh Backbonemedium campus medium campus -- traffic flow to server farmtraffic flow to server farm

ServerBlock


Predominant traffic pattern


4) Dual4) Dual--Path L2 Switched BackbonePath L2 Switched Backboneno STP loops or VLAN trunks in coreno STP loops or VLAN trunks in core

South

ClientBlocks

Dual L2 Backbone

Distribution L3

Core L2

Access L2

“red” coresubnet=VLAN=ELAN

“blue” coresubnet=VLAN=ELAN

WestNorth



5a) Benefits of a L3 Backbone5a) Benefits of a L3 Backbone

✔ Multicast PIM routing control✔ Load balancing✔ No blocked links✔ Fast convergence EIGRP/OSPF✔ Greater scalability overall✔ Router peering reduced✔ IOS features in the backbone


Distribution L3

Access L2

5b) Dual-Path L3 Backbonelargest scale, intelligent multicast

Core L3

ServerFarmBlock

Distribution L3

Access L2

All routed links,consider subnetcount !

ClientBlock



Restore ConsiderationsRestore Considerations

✔ Restoring can take longer in some cases - more complex -schedule

✔ On power up L1 may come up before L3 builds routing table -temporary black hole for HSRP

✔ Use “preempt delay” for HSRP

✔ Restoring can take longer in some cases - more complex -schedule

✔ On power up L1 may come up before L3 builds routing table -temporary black hole for HSRP

✔ Use “preempt delay” for HSRP


Campus Failover Layer 2 Campus Failover Layer 2 Recovery & TuningRecovery & Tuning

STPTune ‘diameter’ on root switchImproves recovery time maxage

PortFastServer or desktop ports only 1 sMove directly from linkup into forwarding

UplinkFastNo tuning, 2 seconds, wiring closet onlyOnly applies with forwarding & blocking link

BackbonefastConverges 2 sec + 2xFwd_delay for indirect link failuresEliminates maxagetimeout


Campus Failover Layer 3 Campus Failover Layer 3 Recovery & TuningRecovery & Tuning

Caution with aggressive tuning

Good when network is stable, highly summarized

HSRP (fast LAN links)Tune hello timer 1 sec, dead timer 3 sec<4s to converge

OSPF (fast LAN links)Tune hello timer 1 sec, dead timer 3 sec<4s to recognize problem, then converge

EIGRP (fast LAN links)Tune hello timer 1 sec, hold timer 3 sec<4s to recognize problem, then converge


Keeping Networks Available!Keeping Networks Available!

✔ KISS - eliminate complex L2 ✔ ASU - building blocks✔ Redundant backbone ✔ Redundant L3 paths✔ L3 segments failure domain

designenterprise network availability - sunet-projekt...

Documents