small to medium data centre designsd2zmdbbm9feqrf.cloudfront.net/2015/anz/pdf/brkdct-2218.pdf#clmel...

#clmel

Small to Medium Data Centre Designs

BRKDCT-2218

Nic Rouhotas - Data Centre Consulting Engineer

© 2015 Cisco and/or its affiliates. All rights reserved.BRKDCT-2218 Cisco Public

Abstract

• Network design for the data centre has evolved over time, yet typically there

has been the common requirement for networked connectivity to all

applications and their respective resources of physical and virtual compute,

storage and network services, as well as to other required services and

locations. Many of the technical design challenges are the same regardless

the size of the organisation. This session will discuss example architectures for

small to medium data centres, starting from entry-level and then illustrate

transition points to increase scale and capacity whilst providing support for

additional features and functionality. The Nexus switching product range will be

referenced in the examples and guidance provided around optimisation of

features and protocols. Also included is a discussion on connecting to remote

data centres as well as considerations for extending workloads to public clouds

3


Cisco Live Melbourne Related Sessions

4

BRKDCT-2048 Deploying Virtual Port Channel (vPC) in NXOS

BRKDCT-2049 Data Centre Interconnect with Overlay Transport Virtualisation

BRKDCT-2334 Data Centre Deployments and Best Practices with NX-OS

BRKDCT-2404 VXLAN Deployment Models - A Practical Perspective

BRKDCT-2615 How to Achieve True Active-Active Data Centre Infrastructures

BRKDCT-3640 Nexus 9000 Architecture

BRKDCT-3641 Data Centre Fabric Design: Leveraging Network Programmability

and OrchestrationBRKARC-3601 Nexus 7000/7700 Architecture and Design Flexibility for Evolving

Data Centres


Cisco Live Melbourne Related Sessions

5

BRKACI-2000 Application Centric Infrastructure Fundamentals

BRKACI-2001 Integration and Interoperation of Existing Nexus Networks into an

ACI Architecture

BRKACI-2006 Integration of Hypervisors and L4-7 Services into an ACI Fabric

BRKACI-2601 Real World ACI Deployment and Migration

BRKVIR-2044 Multi-Hypervisor Networking - Compare and Contrast

BRKVIR-2602 Comprehensive Data Centre & Cloud Management with UCS

Director

BRKVIR-2603 Automating Cloud Network Services in Hybrid Physical and Virtual

Environments

BRKVIR-2931 End-to-End Application-Centric Data Centre

BRKVIR-3601 Building the Hybrid Cloud with Intercloud Fabric - Design and

Implementation


Start small

6

…then grow

Blade Runner, BrickWorld US

…..then evolve


juggling many pieces…

© 2015 Cisco and/or its affiliates. All rights reserved.BRKDCT-2218 Cisco Public 8

Which pieces to select?

BRKDCT-2218 Cisco Public© 2015 Cisco and/or its affiliates. All rights reserved.

Agenda

• Introduction

• Spine/Leaf Primer

• Initial Design Options

• Scale Up or Out

• Data Centre Interconnect Solutions

• Programmability

• Automation & Orchestration

• Cloud Considerations

9


Typical Requirements

Minimum pair of dedicated DC Switches

Transition from collapsed core

Workloads mostly virtualised, some physical

Connect to network periphery

Scalable

Size for current needs

Reuse components in larger designs

Design Options

Feature choice + priority = tradeoffs

Driving efficiency: SDN, Programmability, Orchestration, Automation

“Cloud with Control”

Designing Small to Medium sized Data Centres

FC

FCoE

iSCSI / NAS

L3-----------

L2

Campus

Client Access

WAN / DCI


Single-Tier, Dual-Tier, Spine/Leaf

Small Spine/Leaf

VXLAN

Dual Tier DC

Single Layer DC

Scalable Spine/Leaf DC Fabric

VXLAN


Connectivity & Usage Needs Drive Design Choices

14

VM VMVM

FCoE

iSCSI

FC

NFS/

CIFS

VM VMVM

Hypervisor Network VirtualisationRequirements

– vSwitch vSS/vDS, OVS, Hyper-V, Nexus 1000v/AVS

Automation/Orchestration

– Abstraction

– APIs/Programmability/Orchestration

– VMM’s ; Fabric

Connectivity Model

– 10 or 1-GigE Server ports

– NIC/HBA Interfaces per-server

– NIC Teaming models

14

Compute Form Factor

– Unified Computing Fabric

– 3rd Party Blade Servers

– Rack Servers (Non-UCS Managed)

Storage Protocols

– Fibre Channel (FC)

– FCoE

– IP (iSCSI, NAS)


Data Centre Fabric Needs

15

EAST – WEST TRAFFIC

NO

RT

H-

SO

UT

HT

RA

FF

IC

FC

FCoE

iSCSI / NAS

Server/Compute

Site BEnterprise

Network

PublicCloud

Internet

DATA CENTREFABRIC

Mobile

Services

Storage

Orchestration/

Monitoring

Offsite DC

API

• “North-South”: end-users

and external entities.

• “East-West”: clustered

applications, workload

mobility.

• High throughput, low latency

• Increasing high availability

requirements.



Planning Physical Data Centre Pod Requirements

17

Compute

Rack

Network/Storage

Rack

(2)N2232

FEX

(32) 1RU

Rack

Servers

• Map physical Data Centre needs to a flexible fabric topology.

• Plan for growth in a modular, pod-based repeatable fashion.

• Your own “pod” definition may be based on compute, network, or storage requirements.

• Access Pod TOR switching becomes the leaf switches of a spine/leaf topology.

• How many current servers/racks, and what is the expected growth?

• Intra Row cabling distances ToR to EoR/MoR

• Impact of Spine Leaf to cabling

• Reuse of MMF cabling with 40G BiDi optics

(2) N5672UP

Storage

Arrays

Term Server,

Management Switch

PATCH

Today’s

Server

Racks

Tomorrow’s

Data Centre

Floor

MDF


Core, Aggregation and

Access

Spine-Leaf

Design Selection: Traditional Multi-Tier vs. Spine-Leaf

19


Agenda

• Introduction



• Scale Up or Out


• Programmability



20


Why Spine-Leaf Design? Pay as You Grow Model

Need more host

ports?

Add a leaf

96 ports

2x48 10G (960 Gbps total)

Need even more

host ports?

Add another leaf

To speed up flow

completion times, add

more backplane,

spread load across

more spines

Lower FCT = FASTER

APPLICATIONS

* FCT = Flow Completion Times

144 ports

3x48 10G (1440 Gbps total)192 ports

4x48 10G (1920 Gbps total)

Pe

r S

pin

e

Utiliz

ation

FC

T

FC

T

FC

T

Pe

r Sp

ine

Utilis

atio

nF

CT

FC

T

FC

T

10G host ports

40G fabric ports


Host

1

Host

3

Host

2

Host

4H

ost

5

Host

7

Host

6

Spine/Leaf DC Fabric ≅ Large Non-Blocking Switch

Host

1

Host

3

Host

4Host

5

Host

7

Host

2Host

6


Spine/Leaf DC Fabric ≅ Large Modular Switch

Host

1

Host

3

Host

2

Host

4H

ost

5

Ho

st

7

Host

6

Lin

e

Ca

rd

Lin

e

Ca

rd

Lin

e

Ca

rd

Lin

e

Ca

rd

Lin

e

Ca

rd

Lin

e

Ca

rd

Lin

e

Ca

rd

Lin

e

Ca

rd

Lin

e

Ca

rdL

ine

Ca

rd

Fabric

Module

Fabric

Module

Fabric

Module


Impact of Link Speed – the Drive Past 10G Links

20×10Gbps

Downlinks

20×10Gbps

Uplinks

20×10Gbps

Downlinks

2×100Gbps

Uplinks

20

0G

Aggre

ga

te

Ba

nd

wid

th

20

0G

Aggre

ga

te

Ba

nd

wid

th

20×10Gbps

Downlinks

5×40Gbps

Uplinks

• 40 & 100Gbps fabric provide very similar performance for fabric links

• 40G provides performance, link redundancy, and low cost with BiDi


1 2 3 4 5

Statistical Probabilities of Efficient Forwarding

1 2

1 2 20

Probability of 100% throughput ≅ 3%



20×10Gbps

Uplinks2×100Gbps

Uplinks

11×10Gbps flows

(55% load)

5×40Gbps

Uplinks


Lower

FCT is

Better

Impact of Link Speed on Flow Completion Times

0

2

4

6

8

10

12

14

16

18

20

30 40 50 60 70 80

FC

T

(no

rma

lize

d t

o o

pti

ma

l)

Load (%)

Avg FCT: Large (10MB,∞) background flows

OQ-Switch

20x10Gbps

5x40Gbps

2x100Gbps

• 40/100Gbps fabric: ~ same FCT as non-blocking switch

• 10Gbps fabric links: FCT up 40% worse than 40/100G

Flow Completion is dependent on queuing and

latency.

40G is not just about faster ports and optics,

it’s about

Faster Flow Completion.


Agenda

• Introduction



• Scale Up or Out


• Programmability



28


DC and Cloud Networking Portfolio – Nexus Family

Nexus 5000/5600

Nexus 7000/7700

Nexus 3548/3100Nexus

2000/2300

Nexus 9000

Nexus 1000V/AVS

OPENAPIs/ Open Source/ Application Policy Model

HIGH PERFORMANCE FABRIC

1/10/40/100 GE

SCALABLE SECURE SEGMENTATION

VXLAN, BGP-EVPN

ACI Ecosystem

Resilient, Scalable Fabric

Workload Mobility Within/ Across DCs

LAN/SAN Convergence

Operational Efficiency—P-V-C

Architectural Flexibility

Nexus 6000

29


FC

Single Layer Data Centre, Nexus 5500 • Dedicated Nexus 5500-based switch pair

FCoE

iSCSI / NAS

1Gig/100M

Servers

Rack Servers

10 or 1-Gig attached

(i.e. UCS C-Series)

Rack Servers

10-GigE

(i.e. UCS C-Series)

L3-----------

L2Nexus 5500

Campus

Client Access

WAN / DCI

Nexus

2000

Positive

Unified Port on all ports – Max Flexibility

Can work as FC/FCOE access transition switch

Non-blocking, Line Rate 10Gpbs L2

~2us Latency

Supports FabricPath, Unified Fabric Automation

160G Layer-3 with L3 daughter card or GEM

Supports 24 FEX, A-FEX, VM-FEX

Referenced in most validated designs (i.e. FlexPod)

Negative

L3 card: 160G max, not cumulative

Unified Fabric Automation “L2 ONLY Leaf”

No VXLAN HW support

No BiDi Optics support

No ACI support

No native DCI support

ISSU not supported w/L3

FEX count lower w/L3 (16)

Q: 5500 or 5600?

Models:

Nexus 5548P (1RU); Nexus 5548UP (1RU); Nexus 5596UP (2RU); Nexus 5596T (2RU)

* = Roadmap

Blade Chassis

(i.e. UCS B-Series)



Positive

Low Price/High Performance

Unified Ports – Good Flexibility (not all ports)

Supports VXLAN, FabricPath, Unified Fabric Automation, BiDi Optics

VXLAN Bridging & Routing

VXLAN Flood & Learn; VXLAN EVPNControl Plane *

Non-blocking, Line Rate L2/L3

Native 40G/10G, breakout

~1us Latency

Supports 24 FEX, A-FEX, VM-FEX

New models with higher 40G density

Negative

No ACI support



VXLAN EVPN Control Plane in future s/w

NX-API support in future

FC

FCoE

iSCSI / NAS

1Gig/100M

Servers

10-GigE

UCS C-Series

L3-----------

L2Nexus 5600

Campus

Client Access

WAN / DCI

Models:

Nexus 5672UP (1RU); Nexus 5624Q (1RU); Nexus 5648Q (2RU); Nexus 56128P (2RU)

Nexus 5696Q (4RU)

* = Roadmap

Blade Chassis

(i.e. UCS B-Series) Rack Servers


(i.e. UCS C-Series)

Nexus

2000


Single Layer Data Centre, Nexus 6000• Positioned for rapid scalability and a 40-GigE Fabric

FC

FCoE

iSCSI / NAS

1Gig/100M

Servers

10-GigE

UCS C-Series

L3-----------

L2Nexus 6004

Campus

Client Access

WAN / DCI

Nexus

2000

Positive

Unified Ports – Good Flexibility with expansion

Non-disruptive scale-up

96*40G or 384*10G

Supports FabricPath, Unified Fabric Automation


Native 100G/40G/10G, BiDi, breakout support

~1us Latency

Supports 48 FEX (L2 only), 24 FEX(with L3), A-FEX, VM-FEX

Same software train across N5k/N6k

Negative

No VXLAN Bridging & Routing support

No ACI support


FEX count Lower w/L3


Higher initial cost for modular LEMversion

Certain models transitioned to popular Nexus 5600 family

Q: 6000 or 5600?

Models:

Nexus 6004 (4RU); Nexus 6004EF (4RU); Nexus 6001P/T (> Nexus 5672UP/56128P)

Nexus 6004X now 5696Q (4RU)

Blade Chassis



(i.e. UCS C-Series)



iSCSI / NAS

1Gig/100M

Servers

10-GigE

UCS C-Series

L3-----------

L2Nexus 9300

Campus

Client Access

WAN / DCI

Nexus

2000

Positive

Low Price/High Performance

VXLAN Support in h/w

VXLAN Bridging & Routing

VXLAN Flood & Learn; VXLAN EVPNControl Plane (NX-OS Standalone mode)

ACI mode Spine/Leaf

NX-OS Standalone mode Spine/Leaf


Native 40G & 10G; BiDi; <1us Latency

Hot/Cold Patching, Graceful Insertion/Removal *

FEX Support – 16

NX-API Programmability

Negative

No FC, Unified Ports (GEM variations in future)

FCoE* in future s/w

No FabricPath support

No Unified Fabric Automation support


Breakout on some 40G ports

ISSU not supported

ACI Spine <> ACI Leaf (different models)

Models:

Nexus 9372TX (1RU); Nexus 9396TX (2RU) ; Nexus 93120TX* (1RU) ; Nexus 93128TX (3RU)

Nexus 9372PX (1RU) ; Nexus 9396PX (2RU) ; Nexus 9332PQ (1RU)

Nexus 9336PQ (ACI Spine only) (2RU)

* = Roadmap

Blade Chassis



(i.e. UCS C-Series)


Single Layer Data Centre, Nexus 7000/7700• Highly Available Virtualised Chassis Access/Aggregation Model

L3-----------

L2

Nexus 7700

WAN / DCI

Campus

Client Access

iSCSI / NAS

1Gig/100M

Servers

10-GigE

UCS C-Series

Nexus

2000

FCoE

Positive

More feature rich platform

Modular, easy scale up

Flexible L2/L3 with ISSU

MPLS, LISP, OTV, FEX, FCoE, FP, VXLAN*

Supports Unified Fabric Automation

VXLAN Bridging & Routing *

VXLAN Flood & Learn *; VXLAN EVPNControl Plane *

Native 100G, 40G & 10G,BiDi, breakout

FEX Support 32(Sup2)/64(Sup2E)

ISSU

VDC, PBR, WCCP, MACSec, ITD, RISE

Chassis variety (2-slot* to 18-slot)

Negative

Higher initial capital cost of modular platform

No Unified Ports

VXLAN support in future s/w

No ACI Support

NX-API support in future

Physical Footprint (for larger chassis) (3RUto 26RU)

Models:

Chassis: Nexus 7004/7009/7010/7018; Nexus 7702*/7706/7710/7718

I/O Modules : M1 (10M/100M/1000G ; 1G ; 10G) , M2 (10G; 40G; 40/100G), F2E (1/10G), F3 (1/10G; 40G; 100G)

* = Roadmap

Blade Chassis

(i.e. UCS B-Series)

Rack Servers


(i.e. UCS C-Series)


Single Layer Data Centre, Nexus 9500• Highly Available Chassis Access/Aggregation Model

iSCSI / NAS

10 or 1-Gig attached UCS C-Series

L3-----------

L2Nexus 9500

WAN / DCI

Campus

Client AccessPositive

Modular, easy scale up

Flexible L2/L3 with ISSU*

Hardware support for FEX*, FCoE*, VXLAN*

VXLAN Bridging & Routing *

VXLAN Flood & Learn *; VXLAN EVPNControl Plane * (NX-OS Standalone mode)

Native 100G, 40G & 10G, BiDi, breakout

Supports 32 FEX*

ACI Spine/Leaf support* ; NX-OS Standalone Spine/Leaf support

Hot/Cold patching; Graceful Insertion/Removal *

NX-API Programmability

Negative

Higher initial capital cost of modular platform

No FC, Unified Ports (future)

No FabricPath Support

No Unified Fabric Automation support

No VDC

No native DCI (complement with N7k/ASR)

FEX, VXLAN, FCoE support in future s/w

ACI Modular Leaf support in future

No ISSU (in future; many use cases covered by Hot/Cold patching and GIR)

Models:

Chassis: Nexus 9504; Nexus 9508; Nexus 9516

I/O Modules: 94xx (NX-OS) ; 95xx (NX-OS, ACI) ; 96xx (NX-OS) ; 97xx (ACI)

* = Roadmap


Agenda

• Introduction



• Scale Up or Out


• Programmability



37


Classic/Traditional: Single to Dual-layer Switching

38

L3-----------

L2Nexus 6004EF/77xx

Single Layer

Nexus 56xx/93xx

Single Layer

• Modular switches suited to core/agg(Availability/Density/Features)

• Storage connectivity with FC/FCoE will require N5k/N6k/MDS, or N7k(FCoEonly)

• Fixed config for ToR access tier

• Network services connected at L3 boundary

• Access Tier can make use of FEX to scale-up

• Traditional hierarchical design limitations

Dual Layer DC

FEX

Switch models are examples only


Traditional Multi-Tier Hierarchical Design

…

core1 core2

agg1 agg2 aggX aggY

• Extremely wide customer-deployment footprint

• Scales well, but scoping of failure domains imposes some restrictions

– L3 Boundary

– VLAN extension / workload mobility options limited

– Default Gateway Placement

• Network Services placement is a challenge

• Discrete device management

• Automation is challenging

L2

L3

L3


Modern DC Design Evolution

• Moving to Spine/Leaf construct

• No Longer Limited to two aggregation boxes

• Created Routed Paths between “access=leaf” and “core=spine”

– Routed based on MAC, IP, or VNI

• Layer 2 can be anywhere even with routing

• Automation/Orchestration; removing human error.

41

Ro

ute

d

Do

ma

in

L2

Domain


STP

Data Centre “Fabric” Journey

MAN/WAN

FabricPath

/BGP

MAN/WAN

VXLAN

/BGP EVPN

MAN/WAN

ACI

VPC

MAN/WAN

FabricPath

(Flood & Learn) VXLAN

(Flood & Learn)


Fabric Server Access Starter Pod

Two Racks, 96x10G ports (960GB)***

24x40G fabric ports needed for non-oversubscribed

72x40G available

10G host ports

40G fabric ports

5600 starter4x5672UP

Full SW Bundle

(including DCNM)

~250K US list

ACI starter2x9336PQ

4x9396PX

3xAPIC & 192 Port Leaf licensing

~250K US list

*** Server/Rack density dependent on required load, available power and cooling (geo-diverse)


Scaling with Spine/Leaf:

Two Racks, 96x10G ports (960GB)


72x40G available

Three Racks, 144x10G ports (1440GB)


72x40G available


72x40G available

Four Racks, 192x10G ports (1920GB)Five Racks, 240x10G ports (2400GB)


72x40G available


72x40G available

Six Racks, 288x10G ports (2880GB)

10G host ports***

40G fabric ports

*** This example is 100% non-blocking, non-oversubscribed. Could build an oversubscribed model with FEX or

fewer fabric links. Server/Rack density dependent on load, power, cooling (geo-diverse)


When do you add/upgrade spines?

Six Racks, 288x10G ports (2880GB)


72x40G available


144x40G now available, smaller failure impact

Eight Racks, 384x10G ports (3840GB)


144x40G available

10G host ports***

40G fabric ports

*** This example is 100% non-blocking, non-oversubscribed. Could build an oversubscribed model with FEX or

fewer fabric links. Server/Rack density dependent on load, power, cooling (geo-diverse)


When do you add/upgrade spines?

Eight Racks, 384x10G ports (3840GB)


140x40G available


2x36 in each modular spine, 280x40G, LC Redundancy, Spine ISSU, etc.

10G host ports***

40G fabric ports


Q: Okay, have my Spine-Leaf topology now what?

Choice for Fabric mode of operation:

L2 vPC

[Traditional]

L2 Routed Fabric

i.e. FabricPath

L3 Routed Fabric

with Overlay(Flood and Learn)

L3 Routed Fabric

with Overlay +

Control Plane

Managers/Controllers (optional)

ACI

APIC Controller


Flexible Data Centre Fabrics

Hosts

VM

OS

VM

OS

Virtual

Physical

Create Virtual Networks on

top of an efficient IP network

• Mobility

• Segmentation + Policy

• Scale

• Automated &

Programmable

• Full Cross Sectional BW

• L2 + L3 Connectivity

• Physical + Virtual

Use VXLAN to Create DC Fabrics

L3

L2/L3

VNI

Gateways


SVI/VNI/VLAN Scoping and Provisioning

All VNIs/SVIs everywhere

• Umbrella catch-all provisioning

• Full ARP state on all Leaf Nodes

• Can be manually provisioned up-front

• Open to L2 Flooding everywhere

Orchestration leads to scale optimisation

VNIs/SVIs scoped as hosts attach

• Provision on host attach/policy

• ARP state only for local subnets

• Requires VXLAN Control Plane

• Requires orchestration (i.e. ACI ,VTS*)

• L2 Flooding is scoped51

L3 Fabric

L3 GWY L3 GWY L3 GWY L3 GWYL3 GWY L3 GWY

L3 Fabric

L3 GWY L3 GWY L3 GWY L3 GWYL3 GWY L3 GWY

Mgmt


NAS

Sample L3 Routed Spine-Leaf Fabricusing VXLAN with BGP EVPN


UCS C-SeriesUCS B-Series

Systems

OTV/DCI

Nexus 7700 DCI &

MPLS


Rack Servers Cisco B22 FEX for Blade

Chassis Access

Enterprise

Core

iSCSI

FC/FCoE

Spine Tier

Leaf TierDistributed

Anycast Gateway

IP Storage

FC/FCoE Attached Storage

N2k/FIFI

N5k/MDS

N9kN9kN9kN9kN9k

SAN

N9kN9k

TEPTEP TEP TEP TEP TEP TEP TEP TEP TEP

RR RR

IP Routed Fabric

(with VXLAN)

VLAN

(eg. 10)

VNI (eg. 10010)

VNI (eg. 10020)

VLAN

(eg. 20)

VNI (eg. 10000)

VRF

(eg. Org-A)

g/w g/w

Built-In Multi-Tenancy

VXLAN with MP-BGPEVPN Control plane

FC/FCoEFC/FCoE


Agenda

• Introduction



• Scale Up or Out


• Programmability



58


Ethernet

MPLS

IP

LAN Extension for DCITechnology Selection

Over dark fiber or protected D-WDM

VSS & vPC

Dual site interconnectionFabricPath

Multiple site interconnection

MPLS Transport

EoMPLS

Transparent point to point

VPLS

Large scale & Multi-tenants, Point to Multipoint

E-VPN

Large scale & Multi-tenants, Point to Multipoint

IP Transport

OTV

Enterprise style Inter-site MAC Routing

LISP

For Subnet extension and Path Optimisation

VXLAN (future for DCI)

Emerging limited A/A site interconnect (requires BGP EVPN & anycast gateway)

Metro style

SP style

IP style


Data Centre Interconnect Options• Options for L2 Interconnect

L3-----------

L2

Campus

Client Access

WAN / DCI

VM VMVMVM VMVM

Virtualised Servers with Nexus

1000v, vPath, CSR 1000v

Virtual DC

Services in

Software

L3-----------

L2

WAN / DCICampus

Client Access

VM VMVMVM VMVM

Virtualised Servers with Nexus

1000v, vPath, CSR 1000v

Virtual DC

Services in

Software

CSR1000v

ASR1000

ASR1000

N7K


Agenda

• Introduction



• Scale Up or Out


• Programmability



61


Programmability Sample Use Cases

Application Monitoring Super Commands Topology Mapper

Off-Box On-Box Off-Box

63


Nexus Programmability

Protocols and

Data Models

XMPP Shipping Shipping Future

LDAP Shipping Shipping Shipping

NetConf/XML Shipping Shipping Shipping

YANG Future Future Future

REST Future Future Shipping

Provisioning &

Orchestration

Puppet/Chef Future Shipping Shipping

PoAP Shipping Shipping Shipping

OpenStack Shipping Shipping Shipping

Programmatic

Interfaces

Native Python Shipping Shipping Shipping

Integrated container Future Future Shipping

Guest Shell Future Future Shipping

OpenFlow Future Shipping Shipping

OpFlex Future Future Future

NXAPI (JSON/XML) Future Future Shipping

Nexus 7K Nexus 5K / 6K Nexus 9K


Programming for Many Boxes – Git Hub Repository

https://github.com/datacenter/

https://github.com/datacenter/nexus9000/blob/master/nx-os/nxapi/getting_started/nxapi_basics.py


Here’s an example that uses the NXAPI for the N9K. It can automate mundane configuration tasks: you launch it remotely (from your Mac/PC) and use it to get an inventory of the switch, configure new interfaces, etc:


Here’s another one that collects the output of several “show commands” and puts them together to create a “super command” which nice NxOS-style formatting:

https://github.com/datacenter/nexus9000/blob/master/nx-os/python/samples/showtrans.py

There are a few others such as a CRC error check here:

https://github.com/datacenter/nexus7000/blob/master/crc_checker_n7k.py

Programming Examples


https://github.com/datacenter/nexus9000/blob/master/nx-os/python/samples/showtrans.py

https://github.com/datacenter/nexus7000/blob/master/crc_checker_n7k.py


Agenda

• Introduction



• Scale Up or Out


• Programmability



67


Open Standards and Choice of Deployment

Scale, security, flexibility

Open Standards

Controller Choice:

3rd Party

Cisco VTS* Controller

Controllerless deployment

Any Hypervisor

Application-centric policy

Physical & Virtual

Telemetry

L4-7 automation

Open Standards

Large adoption

Added functionality for Programmability / Devops

Open APIs

Modernised NXOS

DC PODs

DC Core

EXISTING 2, 3-TIER

DESIGNS

APPLICATION

CENTRIC

INFRASTRUCTURE

Routed Fabric with

Overlays (VXLAN with

BGP EVPN)

APIC


UCS Manages Compute through Abstraction

69

LAN

SAN

Motherboard Firmware

BIOS Configuration

Adapter Firmware

Boot Order

RAID configuration

Maintenance Policy

LAN Connectivity Configuration

SAN Connectivity Configuration

Service Profile


ACI Manages Communications through Abstraction

70

QoS QoS QoS

Network Path

Forwarding AC

L

Exte

rna

l C

on

ne

ctivity

QoS

FW

Co

nfig

ura

tio

n

SL

B C

on

fig

ura

tion

Ho

st C

on

ne

ctivity

AC

L

AC

L

QoS

QoS

SL

B C

on

fig

ura

tion

FW

Co

nfig

ura

tio

n

Ho

st C

on

ne

ctivity

Ho

st C

on

ne

ctivity

Application Network

Profile


Different Modes of Operation with Nexus 9000

NX-OS Working w/ multiple SDN controllers

(inclusive for NfV)

APIC data object / policy model integrated natively with NX-OS

running on Nexus 9000 switches (spines and leaves)

Loosely coupled integration

(custom integration and open programmability)Tightly coupled integration – Out of the box ready system

Deploy for multiple topologies

Leaf/Spine, 2-Tier Aggregation, Full Mesh

Deployed as a well-known CLOS topology.

It’s a system approach.

Interoperable w/ 3rd Party ToR Switches

and WAN gear

Must be Nexus 9000 hardware for leaves and spines as well as ACI

Software (switch code and APIC controller)

1/10/40/100GE

Common Platform

Nexus 9000 Standalone

(with choice of Controllers)Application Centric Infrastructure (with APIC)

VTS

NCS

71


Workload Automation & Open Environment

72

Advantages

Any workload, Anywhere, Anytime

Open Integration: Orchestration

Automated Scalable Provisioning

Workload aware fabric

Network Services Controller

Published Schema

Network & Network Services Policies

Compute & Storage Policies

Open APIs

UCS Director

Fabric Management

VTS


Agenda

• Introduction



• Scale Up or Out


• Programmability



74


DC/Private Cloud

End User and IT Admin Portals

Secure Fabric Extender Network,

Compute, and Storage

vSphere

Hyper-V*

KVM*

Xen*

Intercloud Fabric

for Business

EC2 APIs

Azure APIs

Intercloud

Fabric for

Providers

Provider Clouds

Intercloud Ecosystem

Intercloud

Fabric for

Providers

Cisco Powered Services and Cloud

Providers

Cisco Intercloud Fabric: Solution Overview

* Available in subsequent releases


UCSD

Cisco Intercloud Fabric Architectural Details

Public

VM

InterCloud

Director

InterCloud

Switch

InterCloud Provider

Enablement Platform

VM

Manager

Private

Cisco Global InterCloud / Cisco Powered

Services and Cloud Providers

IT AdminsEnd Users

VM VM

InterCloud

Extender

InterCloud Services

VM

InterCloud Secure Fabric

Administrator installs

InterCloud Director

Installed and configured

through InterCloud

Director

SP Admin deploys

ICPEP

Cisco Global

Intercloud

Services


InterCloud Director

UCSD-based, separate interface

InterCloud Secure Fabric

N1Kv-based, doesn’t require a full N1Kv install VMM Support for vSphere 5.1/5.5 – Ent Plus license *NOT* required

vNIC from intercloud connecter into the vSwitch

Network Services: Intercloud Fabric Firewall (VSG based) and IntercloudFabric Gateway (CSR1000v based), only for AWS (initial release)

InterCloud Provider Enablement Platform

ICF-Provider Edition implemented by Provider

InterCloud Components

Key Takeaways


Cisco has many options for building DC solutions

All solutions can start small and grow

Does not have to be a “rip and replace”

Spine-Leaf does not have to be expensive

L3 Routed Fabrics with integrated Overlays using Protocol Based learning provides single architecture for small to hyper scale

Automated fabrics can provide new tools for simplified operations

Cloud technologies can expose new operational models

Key Takeaways


Recommended Reading

81


Give us your feedback and receive a

Cisco Live 2015 T-Shirt!

Complete your Overall Event Survey and 5 Session

Evaluations.

• Directly from your mobile device on the Cisco Live

Mobile App

• By visiting the Cisco Live Mobile Site

http://showcase.genie-connect.com/clmelbourne2015

• Visit any Cisco Live Internet Station located

throughout the venue

T-Shirts can be collected in the World of Solutions

on Friday 20 March 12:00pm - 2:00pm

Complete Your Online Session Evaluation

Learn online with Cisco Live!

Visit us online after the conference for full

access to session videos and

presentations. www.CiscoLiveAPAC.com

http://showcase.genie-connect.com/clmelbourne2015

http://www.ciscoliveapac.com/

Additional Resources


Follow up information for more details:

ACI home page on CCO: http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/index.html

Promise Theory for Dummies (careful, adult language): https://www.socallinuxexpo.org/scale11x/presentations/promise-theory-dummies

Meta Data in the Software Defined Data Center: https://www.youtube.com/watch?v=e29hQ7kCcNs&list=PLinuRwpnsHaf7ePRWHZ4Jb5gvTSrxkwpw&index=5

Additional Resources

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/index.html

https://www.socallinuxexpo.org/scale11x/presentations/promise-theory-dummies

https://www.youtube.com/watch?v=e29hQ7kCcNs&list=PLinuRwpnsHaf7ePRWHZ4Jb5gvTSrxkwpw&index=5

Access Pod Feature Details


Access Pod Features: Virtual Port Channel (vPC)

88

Virtual Port Channel

L2

SiSi SiSi

Non-vPC vPC

Physical Topology Logical Topology

• vPC provides port-channel link aggregation across a pair of separate physical switches.

• Allows the creation of resilient Layer-2 topologies based on Link Aggregation.

• Spanning Tree Protocol (STP) is no longer the primary means of loop prevention.

• Provides more efficient bandwidth utilisation since all links are actively forwarding.

• vPC maintains independent control and management planes.

• Two peer vPC switches are joined together to form a vPC domain.

Re

f


Access Pod Features: Nexus 2000 Fabric Extension

89

Dual NIC 802.3adServer

Dual NIC Active/Standby

Server

• Using FEX provides Top-of-Rack presence in more racks with fewer points of management, less cabling, and lower cost.

• In a “straight-through” or single-homed FEX configuration, each Nexus 2000 FEX is only connected to one parent switch.

• FEX parent switch may be Nexus 5000, 6000 or 7000 9000 Series.

• Nexus 2000 includes 1/10GigE TOR models with 10 or 40GigE uplinks, plus the B22 models for use in blade server chassis from HP, Dell, Fujitsu, and IBM.

Design note: Verify platform-specific FEX compatibility and scale numbers on cisco.com.

Nexus 2000 FEX

Nexus Parent Switch

End/Middle of Row Switching with FEX

Re

f


Nexus Fabric Features: Enhanced vPC (EvPC)Dual-homed FEX with addition of dual-homed servers

90

Dual NIC 802.3adDual NIC Active/Standby

Single NIC

• In an Enhanced vPC configuration, server NIC teaming configurations or single-homed server connections are supported on any port.

• No vPC ‘orphan ports’ on FEX in the design.

• All components in the network path are fully redundant.

• Supported FEX parent switches are Nexus 6000, 5600 and 5500.

• Provides flexibility to mix all three server NIC configurations (single NIC, Active/Standby and NIC Port Channel).

Design Notes:

Port Channel to active/active server is not configured as a “vPC”.

N7000 planned to support dual-homed FEX (without dual-homed servers) targeted in NX-OS 7.1

Nexus 6000/5600/5500

FEX

Re

f


Nexus Fabric Features: Unified Ports and FCoESeamless transport of both storage and data traffic at the server edge

Unified Ports:

• May be configured to support either native Fibre Channel or Ethernet

• Available on Nexus 5500/5600UP switches, or as an expansion module on Nexus 6004.

Fibre Channel over Ethernet (FCoE):

• FCoE allows encapsulation and transport of Fibre Channel traffic over a shared Ethernet network

• Traffic may be extended over Multi-Hop FCoE, or directed to an FC SAN

• SAN “A” / “B” isolation is maintained across the network

FC

Servers with CNA

Nexus Ethernet/FC Switches

FCoE

Links

SAN-BSAN-A

Fibre

Channel

Traffic

Ethernet

or Fibre

Channel

Traffic

Fibre

Channel

Any Unified Port can be configured as:

Disk Array

Re

f


Planning Physical Data Centre Pod Requirements

92

Compute

Rack

Network/Storage

Rack

(2)N2232

FEX

(32) 1RU

Rack

Servers

• Map physical Data Centre needs to a flexible fabric topology.

• Plan for growth in a modular, pod-based repeatable fashion.

• Your own “pod” definition may be based on compute, network, or storage requirements.

• Access Pod TOR switching becomes the leaf switches of a spine/leaf topology.

• How many current servers/racks, and what is the expected growth?

(2) N5672UP

Storage

Arrays

Term Server,

Management Switch

PATCH

Today’s

Server

Racks

Tomorrow’s

Data Centre

Floor

Re

f


Data Centre Service Integration Approaches

93

VM VMVMVM VMVM

Network

Core

Virtualised Servers with

Nexus 1000v and vPath

Physical DC

Service Appliances

(Firewall, ADC/SLB,

etc.)

Virtual DC

Services in

Software

Data Centre Service Insertion Needs• Firewall, Intrusion Prevention

• Application Delivery, Server Load Balancing

• Network Analysis, WAN Optimisation

Physical Appliances/Switch Modules• Typically introduced at Layer 2/3 Boundary;

spine/aggregation or “services leaf” switches.

• Traffic direction with VLAN provisioning, Policy-Based Routing, WCCP.

Virtualised Services• Deployed in a distributed manner along with

virtual machines.

• Traffic direction with vPath and Nexus 1000v.

• Application Centric Infrastructure (ACI) automated framework for service insertion.

L3-----------

L2

Re

f


Working with 40 Gigabit Ethernet

94

QSFP-40G-SR4 with direct MPO and 4x10

MPO-to-LC duplex splitter fiber cables

QSFP-40G-CR4

direct-attach cablesQSFP+ to 4-SFP+

direct-attach cables

(splitter)

Nexus family switches support QSFP-based 40 Gigabit Ethernet interfaces.*

On most platforms, splitter cables can be used to provision 4x10GigE ports out of 1 QSFP.*

40 Gigabit Ethernet cable types:• Direct-attach copper [QSFP <-> QSFP] and [QSFP <-> 4 x

SFP+]. Passive cables at 1/3/5m, active cables at 7 and 10m.

• SR4 uses bit-spray over 4 fiber pairs within a 12 fiber MPO/MTP connector to reach up to 100/150m on multimode OM3/OM4

• CSR4 is a higher powered SR4 optic with reach up to 300/400m on multimode OM3/OM4

• LR4 uses CWDM to reach up to 10km on a single-mode fiber pair.

* Verify platform-specific support of capabilities and roadmap

Reference


QSFP-BIDI vs. QSFP-40G-SR4Run 40 GigE over existing duplex multimode cable plant

TX/RX

TX/RX

2 x 20G

2 x 20G

Duplex (two strand) multimode fiber with Duplex LC connectors at both ends

Duplex Multimode Fiber

Use of duplex multimode fiber lowers cost of

upgrading from 10G to 40G by leveraging

existing 10G multimode infrastructure

QSFP-BIDI

Duplex Multimode Fiber

QSFP-BIDITX

RX

4 x 10G

4 x 10G

12-Fiber ribbon cable with MPO connectors at both ends

12-Fiber infrastructure

Higher cost to upgrade from 10G to

40G due to 12-Fiber infrastructure

QSFP SR

12-Fiber infrastructure

QSFP SR

95


OM4 Fiber

PlantMMF LC

Patch cord

MMF LC

Patch cordSFP-10G-SR

$995

SFP-10G-SR

$995

OM4 Fiber

Plant

MP

O

MP

O

Used Fiber PairUsed Fiber PairUsed Fiber PairUsed Fiber Pair

Used Fiber Pair

QSFP-40G-SR4

$2995

QSFP-40G-SR-BD

$1095

OM4 Fiber

PlantMMF LC

Patch cord

MMF LC

Patch cordUsed Fiber Pair

Distance <= 125m with

OM4

QSFP-40G-SR4

$2995

QSFP-40G-SR-BD

$1095

40G BiDi Optics Preserve Existing 10G Cabling

Configuration Best Practices:vPC with Layer-2, Layer-3


vPC Options: Auto-Recovery

• By default, both parents must be present for a newly connected vPC to be brought active

• Auto-Recovery Allows vPC’s to be established with only a single parent present.

• Addresses multiple scenarios:

After a power failure with a partial restore where only one parent switch is present.

New vPC-attached devices to be configured or powered on during a hardware issue with one of the parent switches.

Ongoing operations based off of either the configured vPC Primary or Secondary parent when one is down for any reason.

“Missing” vPC Peer

N6004-a(config)# vpc domain 10

N6004-a(config-vpc-domain)# auto-recovery

vPC Domain

98


vPC Options: Orphan Ports Suspend

• An orphan port is a device attached to only one member of a vPC pair.

• Intended for devices that do not support port-channel. Other devices should be dually connected by vPCs.

• If the vPC peer-link were to go down, the vPC secondary peer device shuts all its vPC member ports as well as designated orphan ports.

• Configure switch ports for single attached devices (like Firewall or Load Balancer) as orphan-port.

• Configuration allows consistent behavior of orphan ports with vPC member ports.

• Active/Standby Server NIC teaming also uses Orphan Ports.

S2-SecondaryS1 -Primary

vPC peer-link

Keepalive

Orphan port

Active or Standby

Active or Standby

S1(config)# int eth 1/1

S1(config-if)# vpc orphan-ports suspend

S2(config)# int eth 1/1

S2(config-if)# vpc orphan-ports suspend

Server with Active/Standby NIC

Teaming

99


vPC Options: vPC Peer SwitchUnifies Spanning Tree processing across vPC peers

• For use on vPC pairs acting as root bridge of an STP domain (Not needed on FabricPath edge)

• Allows ongoing STP processing without a root bridge transition in the event of a switch failure.

• STP configuration and priority settings must be identical on both peer switches

• vPC Peer-link operates in forwarding state for all vPC VLANs

vPC Peer-link

S1 S2

vPC Primary vPC Secondary

vPC1 vPC2

S1 S2

vPC Primary vPC Secondary

Peer-switch

RootRoot Root

S,0

,SLogical representationPhysical representation

Root

Peer-switch

100


vPC Options: vPC Peer Gateway

Non-RFC compliant end hosts:

• The vPC peer-gateway functionality allows a vPC switch to act as the active gateway for packets that are addressed to the router physical MAC address of the vPC peer.

• Some non-compliant devices use the MAC address of the sender device (Switch physical MAC instead of virtual MAC)

• Certain NAS devices (i.e. NETAPP Fast-Path or EMC IP-Reflect) have been found to do this.

vPC Peer Gateway Feature

• Allows a vPC peer to respond to both the HSRP virtual and the real MAC address of both itself and it’s peer

Switch B

Layer 2/3

Layer-2Access

• Physical IP A• Physical MAC A• Virtual IP • Virtual MAC• Physical MAC B

• Physical IP B• Physical MAC B• Virtual IP• Virtual MAC• Physical MAC A

Switch A

VLAN 100 VLAN 200

101


vPC Options: ip arp synchronize

• When the peer-link connection is first established, perform an ARP bulk-sync using CFS over Ethernet to the peer switch

• Improves convergence times for layer 3 flows after recovery of a peer relationship

Primary vPC

Secondary vPCS

P

P S

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

ARP TABLE

IP1 MAC1 VLAN 100

IP2 MAC2 VLAN 200

IP1 MAC1 IP2 MAC2

SVIs

S1(config-vpc-domain)#

ip arp synchronize

S2(config-vpc-domain)#

ip arp synchronize

102


Routing Protocol Peering between vPC PeersDeployment specifics for Nexus Switches

• Nexus 5000/6000 series only support using a VLAN over the vPC Peer-Link

• Do not provision a separate physical link for router peering on Nexus 5000/6000

Nexus 7000 Layer 2/3

Nexus Access Layer-2

Layer-3CoreLayer-3

Links

Nexus 5/6000 Layer 2/3

Nexus Access Layer-2

Layer-3CoreLayer-3

Links

L3 SVI’s with VLAN over

Shared Peer Link

SeparateL3 Physical

Port-Channel

• Nexus 7000 series allow use of a separate physical port channel for Layer-3 Peering

• This is optional and can provide greater control of behavior for service integrations

103

Spine/Leaf Design Elements


Designing Switching with Oversubscription: Balancing Cost and Performance

105

3:1

Oversubscription: • Most servers will not be consistently filling a 10 GigE

interface.

• A switch may be a line-rate non-blocking device, but still introduce oversubscription into an overall topology by design.

• Consider Ethernet-based storage traffic when planning ratios, keep plans on the conservative side.

Example device numbers assuming all ports connected: Nexus 5672UP: 48x10Gig + 6x40Gig uplink = 48:24

or 2:1 oversubscription

Nexus 2232PP FEX: 32x10Gig + 8x10Gig uplink = 32:8 or 4:1 oversubscription

Actual oversubscription can be controlled by how many ports and uplinks are physically connected.

4:1

Spine

Leaf

FEX

Servers


Value of FabricPath/vPC+ in Spine/Leaf DesignsAdding FabricPath to a traditional physical DC Topology

106

VM VMVM

FabricPath

vPC+

Spine

Leaf

FEX

UCS Rack Servers

vPC becomes vPC+ when used at the edge of a FabricPath network, the Peer Link also runs FabricPath.

FabricPath Benefits:

• Topology flexibility beyond the vPC limitation of using switches in pairs.

• Ease of configuration.

• Completely eliminates STP from running between Leaf and Spine.

• No Orphan Port isolation on Leaf switch vPC Peer-link loss.

• Improved Multicast and routing support with vPC+.


VXLAN Overlay EncapsulationDynamic network segmentation across traditional Layer-3 boundaries

107

Outer IP Header

Outer MAC Header

Outer UDP Header

FCSVXLANHeader

Original Ethernet Frame

VTEP VTEP

VTEP VTEP

• Overlay encapsulations allow fabric segmentation beyond VLAN limits for greater flexibility and scale.

• Software-only VXLAN implementations can provide Layer-2 workload mobility, but with limited visibility into the physical network.

• Nexus 9000, 7000-F3, 6000X, and 5600 platforms support VXLAN in hardware.

• An optimal control plane will utilise the benefits of VXLAN encapsulation, while integrating directly with the underlying physical network.


Unified Fabric AutomationModular building blocks for migration to an automated fabric

108

Leaf

Nexus 7k, 6k, 5k

Spine

Nexus 7k,6k

WAN / DCI

Fabric

Client Access

Border-Leaf

Nexus 7k, 6k

DCNM

Central Point of

Management

• Integration with cloud orchestration platforms and supports dynamic workload mobility.

• Provides a distributed default gateway in the leaf layer to handle traffic to and from any subnet or VLAN.

• Implements segment-id in frame header to eliminate hard VLAN scale limits and support multi-tenancy.

• Provides central point of fabric management (CPOM) for network, virtual-fabric and host visibility.

• Auto-configuration of new switches to expand the fabric using POAP, also provides cable plan consistency checking.

• Built into DCNM and NX-OS, supported with select Nexus switches (N5600; N6000; N7000/7700)

DCNM


Application Centric Infrastructure (ACI)

109

ACI Leaf

ACI Spine

WAN / DCI

ACI Fabric

• Centralised provisioning and abstraction layer for control of the switching fabric.

• Simplified automation with an application-driven policy model.

• Controller provides policy to switches in the fabric but is not in the forwarding path.

• Normalizes traffic to a VXLAN encapsulation with Layer-3 Gateway and optimisedforwarding.

• Decouples endpoint identity, location, and policy from the underlying topology.

• Provides for service insertion and redirection.

Application Policy

Infrastructure

ControllerClient Access

APIC APICAPIC

Border-Leaf

Nexus 9000

APIC controller-managed fabric based on Nexus 9000 hardware innovations

small to medium data centre designsd2zmdbbm9feqrf.cloudfront.net/2015/anz/pdf/brkdct-2218.pdf#clmel...

Documents