hong kong next generation server access architecture with ... · next generation server access...

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Connect 1 1 © 2012 Cisco and/or its affiliates. All rights reserved.

Hong Kong

March 6-7, 2013

Eddie Tan Principal Engineer

Next Generation Server Access architecture with Cisco Nexus 6000

Agenda

Introducing Nexus 6000 Series

Nexus 6004 and Nexus 6001 Overview

Nexus 6000 Positioning

Nexus 6000 Designs

Key Takeaways

Q& A

IT is Moving Towards a World of Many Clouds

Cost

Agility

Flexibility

As Apps Evolve They Demand Infrastructure

Adapt at an Ever Increasing Rate

Public

Private

Hybrid

Communit

y

Is the

Network

Ready?

Key Attributes

Cloud Infrastructure

High-Performance

Intelligent

Programmable

Flexible

Lowest TCO

Introducing

Nexus 6001 1RU 10G switch

Nexus 2248-PQ 1RU 1G/10G FEX

Q2CY13

Q1CY13

48 SFP+ ports and 4 QSFP+ ports

48 SFP+ ports and 4 QSFP+ Uplinks

Performance and Scale

L2 + L3

….. 1 2 32*

1us

NX-OS

7.68 Tbps in 4 RU

FEX

3 X port density of

nearest competitor

Single Management Point

1500+ 10G Servers

Industry leading 10G/

40G ports per RU

First 40G FCoE switch

* Roadmap

Intelligent

Sampled Netflow*

1

31

2

.. Network C

Network B

Network A

* Software Roadmap

Span: Drop,

High Latency*

Microburst Monitoring*

Active Buffer Monitoring*

Latency Monitoring*

Flexible- Platform

40G

FET**

10G

100G*

40G

FC Port*

* Nexus 6004 Roadmap

** Roadmap

….. 1 2 32*

FEX

Connectivity Flexible Choice of 10G or 40G

Convergence FCoE or FC*

Versatile Design with FEX

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 10

Nexus 6000 Overview

Nexus 6004 Feature Highlights

High Performance High Scalability Feature-Rich Visibility and Analytics

• Line rate L2 and L3

with all ports and all

features and all frame

sizes

• 1-microsecond port-to-

port latency with all

frame sizes

• 40-Gbps flow

• 40-Gbps FCoE

• Cut-through switching

for 40 and 10 GE

• 25-MB buffer per three

QSFP interfaces

• Line-rate SPAN

• Sampled NetFlow*

• Buffer monitoring

• Latency monitoring*

• Conditional SPAN-

SPAN on drop-SPAN

on higher latency*

• Micro-burst monitoring*

• L2 and L3 features

• FEXlink

• vPC FabricPath TRILL

• FabricPath with

segment ID

• Adapter-FEX

• /VM-FEX

• 96x40 GE in 4RU

• 384x10 GE in 4RU

• Up to 256,000 MAC

• Up to 128,000 ARP

• 32,000 LPM

• 16,000 bridge

domains*

• 31 Bidirectional SPAN

sessions * Some features in this slide are not supported by software at FCS


Cisco Nexus 6004 Chassis Port-Side View

• Chassis depth: 30 in.

• Chassis width: 17.5 in.

30 in.

4RU

48 Fixed QSFP

Interfaces

12 QSFP ports

Expansion Module


Cisco Nexus 6004 Chassis Power Supply and Fans

• The chassis has six power supply slots; a minimum of three is required.

They support both 3 + 1 and 3 + 3 redundancy.

• Each power supply is rated 1100W, 90—240 VAC.

• The chassis has four fan trays. A minimum of three is

required. Power Supply 3 + 3

Grid Redundancy

or 3 + 1 Redundancy

Console Mgmt0 USB

Fan Module 3+1

Redundancy


Cisco Nexus 6004 Airflow

• Front-to-back or back-to-front airflow

• Port-side exhaust at FCS

• Port-side intake (reversed airflow) with Harbord maintenance release –Q2CY13

• New power supply and fan modules required for different airflow directions

Airflow direction at

FCS


12xQSFP Line-Card Expansion Modules

Provide 12 QSFP interfaces

Support 40, 10GE, FCoE

Offer same performance, features,

and scalability as fixed ports

Support OIR


Unified Port Expansion Module

• 16 SFP+ unified ports expansion

module

• 2/4/8/FC port

• 10GE and FCoE

• For Nexus 6004 only

• SFP+ port allows support for more

variety of optical transceiver

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Target

2H CY13

Pre-CC


100G Expansion Module

• 4 port 100G unified ports expansion

module

• 100GE support including FCOE

• For Nexus 6004 only

• Not CC’d and EC’d. Form factor and

features subject to change

Target

1H CY14

Pre-CC

1

2

3

4


Nexus 6004 Control Plan Elements

• Single built-in supervisor

• Support L2 ISSU with first software release

CPU Four core 2-GHz Intel Gladden

DRAM 16 GB of DDR3 in four DIMM slots

Program Store 80 GB of iSATA flash for base system storage and partitioned to store image, configuration, log

Boot/BIOS flash memory

8 GB to store upgradable and golden version of (BIOS + bootloader) image

Onboard fault log (OBFL)

64 MB of flash memory to store hardware-related fault and reset reason

NVRAM 6 MB of SRAM to store syslog and licensing information

Management

Interfaces

RS-232 console port

10/100/1000BASE-T management port

USB port


Cisco Nexus 6000 Key Forwarding Tables

• Host table: 256,000 -entry hashing table;

actual capacity is slightly less than 256,000

• Host table: Shared between MAC, ARP, and

ND and /32 host route

• Host table FCS carving: 128,000 MAC, 128,000 IP

host

• LPM table: 32,000 entries. Also known as summary

routes

• Mroute table: 64,000 entries. 8K by softwrae at FCS

IP Host Region (ARP/ND/Host

route/(*,G))

MAC Region Summary Routes (S,G)

Host Table LPM Table(32,000) Mroute Table(64,000)

256,0

00 E

ntr

ies

* Hardware table size. Please check configuration limit for software scaling


Multi-tenant with FabricPath Segment ID

L3 Cloud

• 24-bit segment ID is embedded in FabricPath

frame. Extend “VLAN” space to 16 million

• Provide great scale to support large number of

tenants in the fabric

• Supported by N7k F3, N6k and N3548

Traffic from VM with local VLAN

Ingress leaf switch maps local

VLAN to 24-bit segment ID

Egress leaf switch maps

segment ID back to local VLAN


Flexible Buffer Management

• More Packet Buffer

• 25MB per ASIC(3x40G or 12x10G)

• 800MB for Nexus 6004

Dedicated Buffer

• Predictable performance

• No interference among ports with congestion

• Shared Buffer

• Increase burst absorption capacity

• Efficient use of buffer

• Tunable Buffer Allocation

• Flexible and get benefit of both dedicated and shared buffer scheme

• VOQ Architecture

• Leverage buffer from multiple ASIC

Port 1

Dedicated

Port 2

Dedicated

Port 3

Dedicated

Shared Packet Buffer

SPAN Control

Ingress Buffer 16MB per UPC

Multicast Buffer

Egress Buffer 9MB per UPC

Unicast Buffer


Virtual Output Queuing

• Take advantage of ingress buffer from multiple port or ASIC for unicast burst

absorption.

• Ensure fairness among multiple ingress ports


Long Distance FCoE

300m FCoE at FCS (optical transceiver distance limitation)

• 10-km FCoE for 10 GE and 40 GE ports with global QoS policy (roadmap)

‒ Current software implements global network-qos policy to tune the buffer for long-distance FCoE.

‒ Global network-qos policy increases FCoE buffer for all the ports.

‒ >10-km FCoE for 10 GE is possible from perspective of the hardware.

300m FCoE

FCS

10-km FCoE

Future Software Roadmap


Intelligent SPAN for Better Visibility

Large number of active SPAN sessions

• 31 bidirectional active SPAN sessions supported by

hardware

• 16 bidirectional active SPAN sessions supported at FCS

Line-rate SPAN

• Line-rate SPAN for multiple 40 GE ports

• Built-in extra capacity in fabric to accommodate SPAN

traffic

Intelligent SPAN

• Prioritize data over SPAN in case of fabric link

oversubscription due to SPAN traffic

• Conditional SPAN: SPAN on drop. SPAN on high latency

• Sampled SPAN

• Truncated SPAN

• ERSPAN with 1588 timestamp


Visibility with SPAN-on-Drop

• Copy the dropped packet to capture device.

• Correlate the packet drop with application

• Software support at Q3CY13

Data Flow

Data Flow

SPAN-on-drop

packets Capturing

Device

Nexus 6000

N6004# sh monitor session 1

session 1

---------------

type : span-on-drop

state : up

acl-name : acl-name not specified

source ports : Eth1/8 Eth1/9

destination ports : Eth1/11

1/8

1/9 1/11


High Performance SPAN

• No penalty to production data traffic with SPAN

• Extra bandwidth capacity built in for SPAN traffic

• Line-rate SPAN throughput with extra fabric link bandwidth

• First product in the market to support 16 line-rate SPAN sessions

Unified

Crossbar

Fabric

120-Gbps

Data

448

Gbps

448

Gbps

Scheduler Prioritizes

Production Data

Data

Span

Extra Fabric

Bandwidth

for SPAN

224

Gbps

Separate Buffer

Pool for SPAN

120-Gbps

Data


Cisco Nexus 6004 Features at FCS

In General All Cisco Nexus® 5000 Features up

to 5.2 Release(with the exception of 1G, FC

port, PTP, Trustsec and Flexlink) Are Supported:

• FEX support: 24 FEX for L2 and L3; all Cisco

Nexus 2200 FEX and B22 HP and B22 Fujitsu

are supported.

• vPC

• FabricPath

• IPv4 and IPv6

• PIM-SM, PIM-SSM, BiDir PIM, and MSDP

• FCoE including 40-Gbps FCoE

• Adapter-FEX/VM-FEX

• QoS with flexible buffer carving

• SPAN and ERSPAN source sessions: 16

sessions

• Egress interface drop counters

• PoAP, Python for PoAP

• Increased table scalability

• VRF-Lite

• PACL, VACL, and RACL logging

• VRF Route Leak

• BGP Local-AS

• BGP Allow-AS in

• BGP AS-Path Relax

• BGP disable-peer-as-check

• BGP passive neighbor

• BGP MD5 authentication for passive neighbors


Cisco Nexus 6000 Feature Timeline*

*Not comprehensive…Subject to change

• FEX support

• vPC

• FabricPath

• IPv4 and IPv6

• PIM-SM, PIM-SSM, BiDir

PIM, and MSDP

• FCoE

• Adapter-FEX/VM-FEX

• QoS with flexible buffer

carving

• ACL and ACL logging

• SPAN and ERSPAN source

sessions: 16 sessions

• PoAP

• Increased table scalability

• VRF-Lite

• VRF route leaking

• BGP enhancement

• Reverse airflow for Cisco

Nexus® 6004

• QSFP-LR4

• Nexus 6001

• Dell FEX

• Ingress policing

• QSFP-FET

• FEX storm control

• PBR

• BFD

• EEM

• 32 FEX support in L2

• TCAM carving

• Forwarding table carving

Harbor(EC)—Q1CY13 Harbord Maintenance

(EC)—Q2CY13

Harbord Plus

(CC)—Q3CY13


Cisco Nexus 6004 Key Scalability Numbers at FCS

FEXLink

ACL

SPAN

FabricPath

L3

L2

* Single-dimension scalability. Multidimension scalability will be provided at FCS.

• 24 FEX per Cisco Nexus® 6004 in L2 mode

• 24 FEX per Cisco Nexus 6004 in L3 mode

• 4,000 VLANs

• 115,000 MAC

• 32,000 summary routes

• 8,000 mroutes

• 115,000 ARP/host routes

• 128 switch IDs

• 4,000 VLANs

• 16 bidirectional SPAN

• VACL 1024 ACE

• 64,000 STP logical interfaces

• 8,000 IGMP snooping entries

• 256 OSPF peers

• 256 BGP peers

• 256 EIGRP peers

• 2 topologies

• 4 multidestination trees

• IFACL 1152 ACE

• QoS ACL 448 ACE

• 256 PIM peers

• 500 HSRP groups

• 4,000 VRF instances

• RACL 1024 ACE


Physical Connection Choices

• FCS support: QSFP-SR4 (100m over OM3); copper 40 GE–40 GE cable and copper breakout cable

• FCS stretch: QSFP-CSR4 (300m over OM3); post FCS: QSFP-LR4 (10 KM over SMF)

40G QSFP+

Interfaces

40G QSFP+

Interfaces

40G QSFP+

Interfaces

40G QSFP+

Interfaces

40G QSFP+

Interfaces

40G QSFP+

Interfaces

10G QSFP+

Interfaces

10G QSFP+

Interfaces

QSFP-SR4 QSFP-SR4

1M, 3M, 5M Passive

7M, 10M Active

QSFP-SR4

QSFP-4x10G-7M

QSFP-4x10G-10M


Supported Transceiver

Type Distance

QSFP-40G-LR4 10 km with SMF

QSFP-40G-SR4 100m with OM3

150m with OM4

QSFP-40G-CSR4 300m with OM3

400m with OM4

QSFP-4x10G-AC7M 7m

QSFP-4x10G-AC10M 10m

QSFP-H40G-1M 1m

QSFP-H40G-3M 3m

QSFP-H40G-5M 5m

QSFP-H40G-AC7M 7m

QSFP-H40G-AC10M 10m

QSFP QSFP

10 km

QSFP SFP+

300m

QSFP SFP+

300m

QSFP QSFP

10 km

FEX

Cisco Nexus®

2248PQ-10G

• QSFP-40G-LR4 will be supported with Q2CY13 software release


Low Cost FEX Connectivity FET-40G

Cisco Nexus 6000 Cisco Nexus 6000

Cisco Nexus 2232PP and 2232TM-E 2232TM

Cisco Nexus 2248TP-E 2248TP Cisco Nexus 2248PQ-10G

FET-40G

FET-40G

FET-10G

FET-40G

• Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus® 6004

• Supported on Cisco Nexus 6004 and Nexus 2248PQ-10G

• Interoperable with FET-10G

• Support for 100m distance with OM3


Nexus 6000 Positioning


LAN/SAN LAN SAN

Cisco Unified Fabric Switching Innovations

CISCO NX-OS: From Hypervisor to Core CISCO DCNM: Single Pane of Management

DELIVERING TO YOUR DATA CENTER NEEDS

Resilient, High Performance,

Scalable Fabric

Operational Efficiency/

Consistency—P-V-C

Workload Mobility

Within/Across DCs

Secure Separation/

Multitenancy

LAN+SAN Convergence

Cisco MDS 9200 Series

Cisco ® MDS 9500 Series

Cisco Nexus 1000V

Cisco Nexus ®

1010

Cisco Nexus 4000

Cisco MDS 9100 Series

Cisco Nexus 5000

Cisco Nexus 3000

Cisco Nexus 2000

Cisco Nexus 7000 Cisco

Nexus 6000

• 10GbE Access Scale/Compact 40GbE Aggregation • Low Latency Spine/Leaf • 10GbE/40GbE Converged Fabric


Data Center Switching Nexus 7000 and Nexus 6000: DC Considerations

Customer Requirements: Decision Points

Decision Criteria in the Aggregation/Spine

Virtualization Scalability DCI/Mobility Environmentals

L4-7 Services High Availability Latency Investment Protection

Recommended when:

Scale and Flexibility 10G/40G/100G

Highest Availability (HA)

Investment Protection

Multi-Protocol / Services

FEX, VDC, OTV, MPLS/VPLS, LISP

Recommended when:

High density compact 10/40G

Low footprint & low power

Low latency & jitter

Advanced FEX Features

Nexus

7000

Series

17 Tbps

Nexus

6004

Up to 7.68 Tbps

Lead Platform: Modular, High-End Solution Fixed, Mid-Range Solution


Data Center Switching DC Access Layer Considerations

* Roadmap

Customer Requirements: Decision Points

Decision Criteria in the Access Layer

Connectivity Options Table Sizes 40G Density Environmentals

Line Rate L3 FC Latency FEX Scale

Recommended when:

Medium to High density compact

10G deployments

Integrated and line rate L3

Enhanced scale

(Tables, FEX, throughput)

NAT, PIM-BiDir, Analytics

Recommended when:

Low to medium density 10G deployments

UP/Native FC support

Nexus

6001

Series

Nexus

5500

Series

Nexus 6001 Nexus 5500

Nexus 6004 Hardware Pricing Summary

Starting as low as $90,000 for a 24 port configuration

Flexible software and hardware upgrade options

24 ports

HW bundle

$90,000

SW Lic

$40,000

SW Lic

$40,000 LEM

$40,000

LEM

$40,000 LEM

$40,000

LEM

$40,000


Nexus 6000 Designs

Access Nexus 6001/Nexus 5000/Nexus 2000

• Different connectivity

options 1G/10G/10GBase-

T/FCoE

• Support all HA design

• N6001: 40G uplink, large

MAC table

Aggregation

Nexus 6004 • High density 10G/40G

• Compact form factor

• Power efficient

• Large MAC and ARP

table

• vPC/vPC+

Nexus 6000 Access Agg vPC Design

N6004-1 N6004-2

vPC domain 100

L2

L3

slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8

blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8











N5k/N6k

N7k/N6k

L3 Core

Nexus 6000 Design FabricPath • . Easy to scale out. Up to 8000x10G ports with 4x Nexus 6004 as spine

• 3:1 oversubscription at FEX

• Leaf to spine 1:1 or 2:1 oversubscription ratio

• Physical hosts and VMs mobility across entire fabric

• Leverage big host table size on N6k for gateway function

10GE 10GE

10G FEX: 2248PQ

Nexus 6004

10GE 10GE

24xFEX

Nexus 6004

Nexus 6004

10GE 10GE

10G FEX: 2248PQ

10GE 10GE

24xFEX

Nexus 6004 L3 Core

FabricPath

L2 Spine

L2 leaf Routed

leaf

FabricPath Architecture Evolution Anycast HSRP Gateway(Future)

Default

Gateway

Default

Gateway

Anycast Gateway

Today(vPC+ with HSRP)

• A pair of FP node running vPC+ support

HSRP Active/Active

• Limit to 2 FP nodes as gateway for a given

VLAN

• Can be placed at spine or have dedicated leaf

to provide routing

FP Anycast HSRP Gateway

• More than two FP nodes as default gateway

• N-way ECMP for Inter-VLAN traffic

• Can be placed at spine or routed leaf

• No need for vPC+ related configuration

Vlan 100-200 Vlan 201-300

Vlan 100-300

FabricPath

FabricPath

L3

L2

L3

L2

Nexus 6000 Design vPC+L3 High Density 1G and 10G

10GE 10GE

10G FEX: 2248PQ

Nexus 7K/6004

10GE 10GE

24xFEX

L3

L2

L3

Nexus 6004

10G POD

1G FEX: 2248TM-E 22xFEX

L3

L2 Nexus 6001

1G POD

L3 Core

• N6k+N2k as POD

• L2 L3 boundary at leaf N6k.

• Easy to scale out spine layer and add more POD for future growth

• 32K ARP support at leaf N6k with first software release. Higher ARP per N6k with future software release

10GE 10GE

10G FEX: 2248PQ

10GE 10GE

24xFEX

L3

L2 Nexus 6004

10G POD

Key Takeaways • High performance compact 10G/40G L2 L3 switch for Datacenter

• Address the challenges of scalability in Datacenter

• Flexible for different architecture design

• Innovative features


Thank you.

hong kong next generation server access architecture with ... · next generation server access...

Documents