cisco nx-os software architectured2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/brkarc-3471.pdfcisco...

Cisco NX-OS Software Architecture

BRKARC-3471

Ron Fuller, CCIE#5851 (R&S/Storage)

Technical Marketing Engineer, Nexus 7000

@ccie5851

© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3471 Cisco Public

Agenda

• NX-OS Origins & Overview

• NX-OS Modular Architecture

• High-Availability Infrastructure

• High-Availability Features & Capabilities

• Command Line Interface

• Operational & Management Features

• Licensing & Lifecycle

• Innovation

• Conclusion

3


NX-OS: Designed for the Data Center

NX-OS SAN-OS

IOS

CatOS

MDS 9000

Catalyst 6500

4

Nexus 9x00/7x00/6000/5x00/

4000/3000/1000V


Cisco NX-OS Adoption

5

Shipping for 6+ years

50,000+ customers

600,000 systems

Validated Design Guides and Case Studies

Differentiating features driving the adoption and being deployed

Major Certification and Deployment milestones


NX-OS Uptime

Feltham, England

Kernel uptime is 1313 day(s)

Nexus 7018

Lawrenceville, Georgia


Nexus 5010


MDS9513

Eschbon, Germany


NEXUS 5020


NEXUS 7018

NY, New York


MDS IBM FC Bladecenter

Hong Kong


Nexus 7010

UK


Nexus 7018

Houston


MDS

Malaysia


Nexus 7010

> 3.5Y RTP, North Carolina

Kernel uptime is 1363 day

Nexus 5010

> 3.5Y

> 3.5Y

> 1.5Y

> 2.5Y

Chicago

Kernel uptime is 675 days

MDS

Germany

System uptime: 1619 days

Nexus 7010

Rome, Italy

Kernel uptime is 1813 day(s),

Nexus 7010

Ireland, UKI

System uptime: 2612 days

MDS9509

> 4.5Y > 7.0Y

> 3.5Y

> 3.5Y

> 3.5Y

> 4.0 Y

> 4.0 Y

> 3.0 Y

> 3.5 Y

Italy

1364 days

MDS 9513

> 3.5Y

Sweden

1475 days

Nexus 7018

> 4.0 Y

VI, Virginia Beach


Nexus 7010

6


Nexus Validation Testing (NVT)

• NVT provides a comprehensive, end-to-end systems test of Nexus powered data centers

• Includes Nexus 7000, 6000, 5000, 3000, and UCS in multiple Pods and Data Centers

– IPv4 & IPv6

– FabricPath, vPC, vPC+ and OTV

• Testing on pre-Cisco.com images

7

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nvt/index.html




NX-OS Operating System

Data Center Network Manager (DCNM)

Nexus 2000 Nexus 3000

Nexus 1000V Nexus 7x00 MDS 9x00 Nexus 5x00

Nexus 4000

The Cisco Unified Fabric Family

8

• Complete data center class switching portfolio

• Consistent operating system across all platforms

• Infrastructure scalability, transport flexibility and operational manageability

Nexus 6000 Nexus 9x00


Comprehensive Data Center Feature Set

9

Engineered to meet evolving data center landscape

• IPv4, IPv6 • Multicast • Interface / L2 / L3 Scale • WCCP L2 • Security • DHCP v6 Relay • VRRPv3

• Python • onePK • OpenFlow • Chef and Puppet Agents • Containers

• OTV • LISP • VPLS • OTV + VLAN Translation

• DCB/FCoE • Unified Ports • Multi-Hop FCoE

• Virtual Port Channel (VPC)

• Fabric Extender (FEX)

• eVPC

• FabricPath

• FabricPath Multi-Topology

• Anycast HSRP

• DFA– Enhanced Forwarding

• VRF

• VDC (4 8)

• VM FEX / Adapter FEX

• QinQ

• MPLS L3 VPN

• VXLAN

• DFA - SegmentID

• SNMP / XML

• Netflow (Full and Sampled)

• IPSLA

• SPAN/ERSPAN +

enhancements

• Advanced Network Analytics

• Single Point of Management

• Hitless ISSU

• Non-Stop Forwarding

• Stateful Switchover

• BFD

• BFD additional clients

• BGP PIC Edge

• Software Patching

Roadmap generally applicable to Nexus 7K, 6K, 5K, 3K

R&S Baseline SDN / Programmability Data Center Interconnect Storage Convergence

Fabric Technologies Virtualization / Multi-

Tenancy Monitoring / Management High Availability

• Industry leading Data Center solutions

• Enable Catalyst to Nexus migration for DC use-cases

• Focus on Operational excellence NX-OS Focus


Certifications for NX-OS

10

• IPv6 Ready Logo Phase I Certified https://www.ipv6ready.org/db/index.php/public/logo/01-000556/

• FIPS 140-2 Certified – Completed in April 2011 - Cert# 1533, 1534

– http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140val-all.htm

• EAL4 Common Criteria Certified – Completed in April 2011

– http://www.niap-ccevs.org/st/vid10349

• NX-OS 6.2 is targeted for IPv6 Phase II and FIPS 140-2 and CC certification

https://www.ipv6ready.org/db/index.php/public/logo/01-000556/



http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140val-all.htm





http://www.niap-ccevs.org/st/vid10349





NX-OS Distributed Architecture Distributed Forwarding and Control-plane

11

• OS designed to leverage distributed hardware architecture

• Fabric & forwarding engine removed from supervisor

• Each I/O module has independent control-plane and forwarding hardware

• Control-plane & data-plane separation (same on Nexus 6000 & 5x00)

• Fully distributed system for non-disruptive SSO & ISSU (SSO only available on dual-sup Nexus 7x00 and 9500)

Supervisor

(Control-Plane)

Fabrics

I/O Module

(Forwarding Engine)

EO

BC

* Module 1

Module 2

Module 3

*EOBC: Ethernet Out Of Band Channel


Agenda








• Innovation

• Conclusion

12


NX-OS Modular Architecture

13

Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

Feature API

API

Feature Velocity

Faster Defect Resolution

Consistency

HA

Infrastructure

API


NX-OS Kernel

14

• Linux 2.6 kernel – N7K, N6K, N5K

• Linux 3.4 kernel – N9K

• Brings the benefits of Linux

oResilient Pre-emptive Multitasking Multi-threaded

oScalable Multi-CPU/Core support

oConstant development and enhancement

Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

Feature API

API

HA

Infrastructure

API


NX-OS Platform Specific Portion

15

• Chipset specific code

• Provide Hardware Abstraction Layer (HAL)

• Ported per platform

Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

Feature API

API

HA

Infrastructure

API


NX-OS Netstack

16

• Complete network stack implemented in user space • L2 Packet Management/ARP

• IPv4/IPv6

• ICMPv4/ICMPv6

• TCP/UDP & Socket Library

• Added Functionality • Virtualization (VDCs/VRFs)

• High-Availability (SSO)

• Added system stability Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

Feature API

API

HA

Infrastructure

API


NX-OS Management Infrastructure

17

• Provides CLI and configuration interfaces

• Provides SNMP agent

• Provides NETCONF/XML interface

• Provides Python

• Provides Cisco ONE Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

Feature API

API

HA

Infrastructure

API


Python Software Architecture

18

CLI Interpreter

Python

Interpreter

Other NX-OS component

(BGP, OSPF, etc…)

Operator

Console/Telnet/SSH

Python can run from the switch CLI

Python script can be run once or at specific

intervals

Configuration / show commands can be

executed from Python Interpreter mode

Call a different python script from a script

Ability to automatically run at bootup

Parse show outputs and perform

conditional actions (syslog, email, traps)

Integration with PoAP

Check RIB/FIB tables

Restrict access (security)


NX-OS Feature/Service Granularity

19

• Highly granular implementations

• Each service is an individual memory protected process – Including multiple instances of

particular service

• Effective fault isolation between services

• Individually Monitored & Managed

Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

HA

Infrastructure

UDLD FCF FCoE STP

HSRP OTV vPC LISP

OSPF EIGRP BGP PBR


NX-OS Conditional Features

20

• Services (Protocols/Features) can be explicitly enabled/disabled N7K-3-Core1(config)# feature ?

<snip>

lacp Enable/Disable LACP

ldap Enable/Disable ldap

lisp Enable/Disable Locator/ID Separation Protocol (LISP)

lldp Enable/Disable LLDP

msdp Enable/Disable Multicast Source Discovery Protocol (MSDP)

mvrp Enable/Disable MVRP

netflow Enable/Disable NetFlow

ntp Enable/Disable NTP

ospf Enable/Disable Open Shortest Path First Protocol (OSPF)

<snip>

• Disabling a service releases associated resources, configuration and CLI


1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

Forwarding Engine VOQs VOQs

Fabric ASIC

To Fabric Modules

10G MAC 10G MAC 10G MAC 10G MAC

4:1 Mux 4:1 Mux 4:1 Mux 4:1 Mux 4:1 Mux 4:1 Mux 4:1 Mux 4:1 Mux

10G MAC 10G MAC 10G MAC 10G MAC

Replication

Engine

Replication

Engine

Replication

Engine

Replication

Engine

Front Panel Ports

LC

CPU

EOBC To Central Arbiter

NX-OS Runs on the Linecard

21

• Microcode version of NX-OS powers the linecards

• Runs on linecard control-plane CPU

• Service processes on the linecards are for hardware and functional support

• Reinforces highly distributed architecture

• In Service Upgrade capabilities (ISSU)

Kernel

Netstack

Management

Infrastructure

Hardware Drivers

HA

Infrastructure

Port Manager

ACL/QoS

Manager

FIB Manager


Control-Plane/Data-Plane Separation

22

I/O Module

Static RIP EIGRP IS-IS OSPF BGP

Unicast Routing Information Base (uRIB)

uFDM

uFDM & FIB Manager

FIB Hardware

Supervisor


NX-OS Platform Packaging and Delivery

23

• Modular nature of NX-OS allows delivery of “permutations” based on hardware capabilities

• Kernel, core infrastructure code, and APIs remain consistent

• Minimizes development

• Maximizes code reuse & feature velocity

Kernel

Netstack

Management

Infrastructure

Hardware Drivers

HA

Infrastructure

UDLD FCF FCOE STP

ACL &

QoS OTV vPC LISP

OSPF EIGRP BGP PBR

Kernel

Netstack

Management

Infrastructure

Hardware Drivers

HA

Infrastructure

UDLD FCF FCOE STP

ACL &

QoS OTV vPC LISP

OSPF EIGRP BGP PBR

Kernel

Netstack

Management

Infrastructure

Hardware Drivers

HA

Infrastructure

UDLD vPath FCOE LACP

ACL &

QoS OTV vPC LISP

OSPF EIGRP BGP PBR

Nexus 7x00

Nexus 6000/5x00 Nexus 1000v


Agenda








• Innovation

• Conclusion

24


NX-OS High-Availability Infrastructure

25

• Actually composed of 3 sub-services

–System Manager

–Message & Transaction Service (MTS)

–Persistent Storage Service (PSS)

Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

Feature API

API

HA

Infrastructure

API System

Manager

PSS

MTS


NX-OS System Manager

26

• Center of service management and fault recovery

• Acts like Unix-like ‘init’ process.

• Starts up configured features/services

• Heartbeats received from services

HA

Infrastructure

System

Manager

PSS

MTS


NX-OS Message & Transaction Service

27

• Message relay system for IPC communications

• Provides reliable unicast & multicast delivery

• Used for service-to-service and module-to-module messaging

HA

Infrastructure

System

Manager

PSS

MTS


NX-OS Persistent Storage Service

28

• Lightweight key/value database

• Provides store options for DRAM or NVRAM

• API for services to store data

• Used to maintain runtime data/state

• PSS updated in NX-OS 6.2 to increase overall scale

• PSS in NX-OS 6.2 is multi-core and multi-CPU “aware”

HA

Infrastructure

System

Manager

PSS

MTS


Agenda








• Innovation

• Conclusion

29


NX-OS Stateful Process Restart

30

If a fault occurs in a process…

• HA manager determines best recovery action (restart process, switchover to redundant supervisor)

• Process restarts with no impact on data plane

• State is recovered, operation resumes

• Total recovery time: ~10s ms

Restart

process!

Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

HA Infrastructure

UDLD SSH IGMP STP

HSRP 1 OTV vPC HSRP 2

OSPF 1 EIGRP BGP OSPF 2

Control-Plane

Data-Plane

NX-OS services checkpoint their runtime state to the PSS for recovery in the event of a failure

OSPF


Active and Standby Supervisor Syncing

31

Services start in standby mode

Active SUP

PSS

Service

System manager

MTS

Standby SUP

PSS

System manager

MTS Standby Online (all services gsync)

Service

Determine

Active/Standby

1

Request Initial States (gsync) 2

Snapshot of Initial

States

3

Services Set Initial

States

4

Event-driven

Syncing

5

Initial State for

Services Runtime config

Runtime states/data


Stateful Supervisor Switchover

32

• Active/Standby

• Fast switchover time – State is already in place

• Switchover initiated if:

repeated critical process restart failures

kernel failures

supervisor hardware failure detected by diagnostics (GOLD)

Nexus-Dual-Sup# show system redundancy status Redundancy mode --------------- administrative: HA operational: HA This supervisor (sup-1) ----------------------- Redundancy state: Active Supervisor state: Active Internal state: Active with HA standby Other supervisor (sup-2) ------------------------ Redundancy state: Standby Supervisor state: HA standby Internal state: HA standby


In-Service Software Upgrade

33

N7K# install all kickstart bootdisk:6.2-kickstart system bootdisk:6.2-system N7K#

N7K#

Sup 2 Sup 1

Upgrade standby supervisor 1

Reload standby supervisor 2

Upgrade standby supervisor 4

Perform SSO 3

Upgrade LCs & FEX in series* 6

Release

6.2 Reload standby supervisor 5

Release 6.1 • Parallel upgrade of the I/O modules supported on the

Nexus 7000 from 5.2 (3 at the same time)

• Parallel upgrade of FEX supported on the Nexus 7000 from

6.1 (10 at the same time)

Active

5.2 Active

6.2

Standby

6.2 Standby

5.2

Standby

6.2 Standby

5.2

Release

6.2


Hitless ISSU on the Nexus 5x00/6000

34

• Difference in the detailed operation from Nexus 7K

– Single supervisor/control-plane vs. dual supervisor

– L3 ISSU not supported on the 5K/6K

– ISSU not possible if non-edge STP designated port (only works in the access)

• During ISSU, control plane functions are temporarily suspended.

• Control Plane restored within 80 seconds

• Hitless ISSU of the Nexus 2000s (Nexus 5x00/6000)

• Supported from NX-OS 4.2(1)N1


Agenda








• Innovation

• Conclusion

35


NX-OS CLI Highlights Improved Over IOS Model

36

N7K(config)# int e1/1 N7K(config-if)# ip address 192.168.0.1/23

Support for CIDR ‘slash’ notation for IPv4/IPv6 masks

N7K(config)# show interface e1/1 Ethernet1/1 is up Hardware: 10/100/1000 Ethernet, address: 001b.54c1.5d44 (bia 001b.54c1.5d44) MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 <snip>

Hierarchy Independent CLI allows ‘show’ commands to be executed from exec-mode or config-mode

N7K# show cli history ? <CR> config-mode Display history of config commands only exec-mode Display history of exec commands only this-mode-only Display history from current mode only unformatted Display just the commands N7K# show cli history config-mode 12 05:20:34 int e1/1 13 05:20:42 where detail

Mode-aware CLI history


Review Configuration with Flexibility

37

N7K# show running-config ntp ntp server 171.68.10.80 use-vrf management ntp server 171.68.10.150 use-vrf management ntp source 172.26.244.101 clock format 12-hours clock format show-timezone

Compare between startup- and running-configuration N7K# copy running-config startup-config [########################################] 100% N7K# config terminal Enter configuration commands, one per line. End with CNTL/Z. N7K(config)# feature telnet N7K(config)# sh running-config diff *** Startup-config --- Running-config *** 1,11 **** feature lacp --- 1,11 ---- + feature telnet feature lacp

Identify the line number and

difference between startup-

config and running-config

Display feature-specific configuration


NX-OS Running-config Permutations

38

• “show running-config” provides many enhancements

N7K# show running-config ?

<CR>

> Redirect it to a file

aaa Display aaa configuration

all Current operating configuration with defaults

am Display am information

arp Display arp information

bgp Display bgp information

callhome Display callhome configuration

cdp Display cdp configuration

cmp Display CMP information

copp show running config for copp

dhcp Display dhcp snoop configurations

diagnostic Display diagnostic information

diff Show the difference between running and startup configuration

dot1x Display dot1x configuration

eem Show the event manager running configuration

eigrp Display eigrp information

icmpv6 Display icmpv6 information

igmp Display igmp information

interface Interface configuration

ip Display ip information

ipqos show running config for ipqosmgr


NX-OS CLI Tips

39

Dude, where am I?!

*Who* typed that command?!

N7K-3-Core1(config-router)# where

conf; router ospf 100 admin@N7K-3-Core1%default

N7K-3-Core1# show accounting log

Tue Apr 22 08:04:38 2014:type=update:id=vsh.19970:user=chad:cmd=switchto ; confi

gure terminal ; interface Vlan616 ; ip access-group mgt-outbound out (SUCCESS)

Tue Apr 22 08:04:38 2014:type=update:id=vsh.19970:user=chad:cmd=switchto ; confi

gure terminal ; interface Vlan616 ; no ip redirects (SUCCESS)

I don’t want to read the entire log N7K-3-Core1# show log last 10

2014 May 14 04:59:53 N7K-3-Core1 %BFD-5-SESSION_MOVED: BFD session 0x4200000d: I

nstalled on LC 5

2014 May 14 04:59:54 N7K-3-Core1 %BFD-5-SESSION_CREATED: BFD session to neighbor

10.1.0.9 on interface Eth5/1/1 has been created

Typing is hard N7K-3# show cli alias

CLI alias commands

==================

agg :switchto vdc Agg1

agg2 :switchto vdc Agg2


CLI Parsing Many options to parse output N7K-3-Core1# show int e5/1/1 | ?

cut Print selected parts of lines.

diff Show difference between current and…

temp files: remove them with 'diff…

on commands with big outputs, like…

egrep Egrep - print lines matching a pattern

grep Grep - print lines matching a pattern

head Display first lines

human Output in human format

last Display last lines

less Filter for paging

no-more Turn-off pagination for command output

section Show lines that include the pattern…

that are more indented than matching…

sed Stream Editor

sort Stream Sorter

source Run a script (python, tcl,...) from

bootflash:scripts

top Run the command before the pipe in a loop with

set…

tr Translate, squeeze, and/or delete characters

uniq Discard all but one of successive identical lines

vsh The shell that understands cli command

wc Count words, lines, characters

xml Output in xml format (according to .xsd defini…

xmlin Convert CLI show commands to their XML formats

xmlout Output in xml format (according to the latest …

xpath Run xpath query on xml output (to be used after…

begin Begin with the line that matches

count Count number of lines

end End with the line that matches

exclude Exclude lines that match

include Include lines that match

40


CLI Parsing - continued

41

N7K-3(config)# cli alias name cop sh pol int con | eg

"class|violate(d| rate)" | sed "s/$.*class-map.*$ (match-

any)/\n\1/" | eg -v "violate rate 0 bytes/sec“

N7K-3# cop

class-map copp-system-p-class-critical

violated 0 bytes,

violated 0 bytes,

class-map copp-system-p-class-important

violated 0 bytes,

violated 0 bytes,

<snip>


IOS to NX-OS Conversion Tool

42

• Available today on cisco.com http://tools.cisco.com/nxmt

• Migrate Catalyst 6500/4500 configuration to Nexus 7x00/5x00/6000

http://tools.cisco.com/nxmt


Agenda








• Innovation

• Conclusion

43


Embedded WireShark Analyzer (Ethanalyzer)

44

• Real-time, on-the-device protocol analyzer provide ultimate visibility into various traffic hitting CPU from remote locations

Control

Processor Data

Traffic mgmt0

Inband

Monitor traffic from inband and

mgmt0 interfaces to the Control

Processor

Extensive capture and display

options, including to file (.pcap)

Capture rules/filters


NX-OS Configuration

45

• Facilitate change-management with configuration snapshots

• Checkpoint & Rollback

System and user generated checkpoints

System checkpoint automatically created

when any conditional features are

disabled

User-defined checkpoint can be initiated

from CLI

Rollback to any checkpoint allows easy

recovery

Current Running

Configuration

System

Checkpoint

New Running

Configuration

User-Defined

Checkpoint

Ro

llb

ack


Checkpoint & Configuration Rollback

46

N7K(config)# no feature vpc N7K(config)# sh checkpoint summary System Checkpoint Summary ------------------------------------- 1) system-fm-vpc: Created by admin Created at Fri, 16:51:40 06 Nov 2009 Size is 24,567 bytes Description: None

System-checkpoint created automatically upon feature removal

User-defined checkpoint with description simplifies configuration management N7K# checkpoint 2009-11-06 description SQL DC ACL Update N7K# sh checkpoint summary User Checkpoint Summary ------------------------------------- 1) 2009-11-06: Created by admin Created at Fri, 18:33:41 06 Nov 2009 Size is 25,773 bytes Description: SQL DC ACL Update

Flexible option for configuration rollback N7K# rollback running-config checkpoint 2009-11-11 ? <CR> atomic Stop rollback and revert to original configuration (default) best-effort Skip errors and proceed with rollback stop-at-first-failure Stop rollback at the first error verbose Show the execution log

Default name for system-checkpoint,

‘system-fm-xxx’

Timestamp of checkpoint

help configuration

management


Control Plane Policing (CoPP)

47

• Protects switch’s CPU from network traffic and improves stability of the platform

• Only the traffic sent to the CPU via the inband interface is subject to CoPP

– ARP, ICMP, SNMP, routing protocols, etc.

• Default CoPP policy works in majority of environments

– Can be customized for specific requirements

• Application requirements and/or scale

– Life cycle management of CoPP critical

• Monitor, tune, monitor, evaluate, repeat

• CoPP is updated to include new features & protocols

Linecard

FE

Linecard

FE

Transit

Packets

Transit

Packets

Layer 2 Protocols Layer 3 Protocols

VLAN PVLAN

OSPF BGP

EIGRP

GLBP HSRP IGMP

UDLD CDP

802.1X STP LACP PIM CTS SNMP

… …

Control Plane

Supervisor

Logical Representation of the Fabric Modules


NX-OS Embedded Event Manager (EEM)

• Action can be

• CLI Command

• Python Script

• Multiple actions per Event

• Rich set of Events

• Syslog messages

• Monitoring for certain CLI commands

• Memory thresholds

• Module status changes

• Missing fan tray

• Temperature thresholds event manager applet track_1_18_down

event track 1 state down action 1 syslog msg EEM applet track_1_18_down shutting down port eth1/33 as 1/18 went down action 2 cli conf term action 3 cli interface ethernet 1/33 action 4 cli shut

event manager applet track_1_18_up

event track 1 state up action 1 syslog msg EEM applet track_1_18_up bringing up port eth1/33 as 1/18 came up action 2 cli conf term action 3 cli interface ethernet 1/33 action 4 cli no shut

EEM to track interface down

EEM to track interface up

48


NX-OS Scheduler • Multiple schedules and multiple jobs per schedule

• Frequency

• Run Once, Daily, Weekly, Monthly

• Delta (Begin Job at specified time and repeat at specified intervals)

• Run CLI commands or Python scripts

Schedule 1

Run Weekly

Schedule 2

Run Once

Job 2 – Run

Sanity Check

Python Script

Job 1 – Full

System Check

…

Schedule 3

……..

Job 1 – Backup

Running

Configuration Job …… Job …… Job ……

49


NX-OS Scheduler Setup

# configure terminal # feature scheduler

# configure terminal # scheduler logfile size 32

# configure terminal # scheduler job name bkpConfig # cli var name timestamp $(TIMESTAMP) # copy running-config bootflash:/$(SWITCHNAM)-cfg.$(timestamp)

# configure terminal # scheduler schedule name bkpConfig # job name bkpConfig # time daily 23:00

Enable schedule

Configure log file size (kb)

Create job

Schedule job

50


NX-OS Stateful Process Restart & Patching

51

When a process is patched…

• Install process applies new patch

• HA manager restarts process

• Process restarts with patched code and no impact on data plane

• State is recovered, operation resumes

• Total recovery time: ~10s ms

Restart

process!

Kernel

Netstack

Management

Infrastructure

Hardware

Drivers

HA Infrastructure

UDLD SSH IGMP STP

HSRP 1 OTV vPC HSRP 2

OSPF 1 EIGRP BGP BGP

Control-Plane

Data-Plane

• NX-OS services checkpoint their runtime state to the PSS for recovery in the event of a failure

• Install process can use PSS state to recover state BGP


SMU Repository

Copy to Device

SMU Removed

Software Maintenance Update (SMU) Workflow (CLI)

Show Install Active

Show Install Committed

Show Install Inactive

Show Install Packages

SMU Committed

SMU Committed

Router> Install Add

Router> Install Activate

Router> Install Commit Router> Install Deactivate

Router> Install Commit

Router> Install Remove

SMU

.

.

Memory: Process:

Memory: Process:

SMU Applied Memory: Process:

Memory: Process:

Memory: Process:

SMU

52


Cisco Software Manager (Formerly SMU Manager)

Challenge # 1 – Too many devices to manage

Challenge # 2 - Find the appropriate SMU considering the OS Image, the Process, and the SMU interdependencies.

• Maintain inventory of Devices including Image and SMUs supported

• Maintain an inventory of SMUs available on CCO

• Recommend a SMU upgrade path per supported Device

• Support existing Data Center Orchestration Tools (i.e. Chef)

53


CSM

Patch Server

S/W Download via SCP etc.

Initiate S/W Install

Which SMUs are available per image ?

Software SMU:

• BugID

• Image Version

• Severity

• CCO SMU Profile

• Device Image/SMU Profile

• Create CHEF Databag

Internet

Facilitate SMU downloads

Download Databag and Databag items

Download Cookbook to all devices

Pull Databag Item & SMU




SMU

SMU SMU

www.cisco.com

SMU on CCO SMU

Cisco Software Manager ( Formerly SMU Manager)

54


Nexus Options

Custom

Listeners

Nexus DC Switch

NXOS

Linux Container hosting One

PK Apps

Thrift API

OMI

Server OpenFlow

Agent

CLI

Python SNMP Custom

Apps

OMI

Python

Apps

Traditional

Management

OpenFlow

Controller

55


OnePK Architecture – High Level View

Nexus DC Switch

NXOS

Linux Container hosting One PK Apps

Thrift API

OMI

Server

OpenFlow

Agent Chef Agent

Puppet

Agent Custom….

OpenFlow

Controller Chef Puppet OMI

Custom

App

56


Agenda








• Innovation

• Conclusion

57


NX-OS Licensing

58

Grace Period

• Allows trying functionality without a licence for 120 days

• Periodic syslog, callhome and SNMP traps warning when grace period nears expiry

• Self generated license for 90 days (beyond grace period)

• https://tools.cisco.com/SWIFT/LicensingUI/loadDemoLicensee?FormId=4056

Time-Bound Licenses

• License with expiry date

• Used for demo or as an emergency

• Periodic syslog, callhome and SNMP traps warning when time bound license nears expiry

• After expiry date feature will continue to run if grace period has not been exhausted

License PAK

(product activation key)

www.cisco.com

PAK +

chassis serial #

<xml...

licA ...>

license

file

Licenses are enforced on the switch # show license host-id

License tied chassis serial # stored in dual redundant NVRAM

modules on backplane

Licenses are issued in the form of a digitally signed text file

# install license bootflash:N7K-1234.lic

https://tools.cisco.com/SWIFT/LicensingUI/loadDemoLicensee?FormId=4056



http://www.cisco.com/


NX-OS Platform Universal System Image License-based Feature Management

59

Single NX-OS System Image

A+B

A+B

+C

A+C

A+B

+D

A+C

+D

A+D

Multiple SW Images NX-OS Base (A)

Enterprise

License

(B)

Advanced

License

(C)

Storage

Features

(D)

Future

License

Packages

Finding the right image

can be a challenge


Enterprise LAN

IP routing

OSPFv2

OSPFv3

IS-IS

BGP for IPv4 & IPv6

EIGRP for IPv4 & IPv6

BFD

IP Multicast

PIM: Sparse, Bidir, ASM and SSM for IPv4 & IPv6

Multicast Source Discovery Protocol (MSDP) for IPv4

PBR for IPv4 and IPv6

GRE Tunnels

TrustSec (SGTs and MACSEC)

Advanced

LAN

VDCs

vPC Port Profile WCCP Port Security GOLD EEM TACACS LACP ACL QoS STP

STP Guards UDLD CDP CoPP uRPF IP Source Guard DHCP Snooping CMP ISSU SSO

Dynamic ARP Inspection Smart Call Home SNMP 802.1x SPAN Netflow v5 and v9 IEEE1588 Base

Scalable Services

Enhanced L2

FabricPath

PONG

Intelligent Traffic Director (ITD)

MPLS

MPLS VPN

LDP

MPLS QoS

TE/FRR

mVPN

MPLS OAM

6PE/6VPE

NX-OS Software Packaging – Nexus 7x00 Nexus 7000 Overview

60

FCoE*

Multi-Hop FCoE

FCF

FIP

Storage

Enterprise

Inter VSAN routing

VSAN based access control

* Per Module-based license

Transport

Services

OTV

LISP

+4 VDCs

VDC

Unlocks HW resources on XL Module

Scalability


Base Enterprise

• SVI routed interfaces

• L3 routed ports on non-FEX interfaces

• Static Routing

• RIPv2

• EIGRP for Routed Access (Stub)

• OSPFv2 and OSPFv3

• HSRP

• VRRP

• IGMPv1, v2, v3

• PIM v2 (sparse)

• Routed ACLs

• uRPF

• MSDP

Enterprise LAN

BGPv4

EIGRP

VRF-Lite

v6 Routing (IS-ISv6, BGPv6)

HSRPv6/VRRPv3

ISSU vPC Port Profile LACP ACL QoS STP LLDP XML SNMP 1588 Base

Enhanced L2

FabricPath

FCoE License

FCoE:

• Native Fibre Channel

• FCoE

• NPV

• FC Port Security

• Fabric Binding

NX-OS Software Packaging – Nexus 5x00 Nexus 6000 Overview

61

VM-FEX

VM-FEX


Base Enterprise

• SVI routed interfaces

• L3 routed ports on non-FEX interfaces

• Static Routing

• RIPv2

• EIGRP for Routed Access (Stub)

• OSPFv2 and OSPFv3

• HSRP

• VRRP

• IGMPv1, v2, v3

• PIM v2 (sparse)

• Routed ACLs

• uRPF

Enterprise LAN

BGPv4

EIGRP

VRF-Lite

v6 Routing (IS-ISv6, BGPv6)

HSRPv6/VRRPv3

ISSU vPC Port Profile LACP ACL QoS STP LLDP XML SNMP 1588 Base

Enhanced L2

FabricPath

FCoE License

Storage Services:

• 8-port FC/FCoE License

• Chassis license for 5548 and 5596

• FcoE NPV (available also as standalone license)

NX-OS Software Packaging – Nexus 600x Nexus 5500 Overview

62

FCoE NPV License

FCoE NPV:

• FCoE NPV

VM-FEX

VM-FEX for 5548 and 5596 only


NX-OS Releases

63

• NX-OS on Nexus 7000 Minimum software Recommendation http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/recommended_releases/recommended_nx-os_releases.html

• NX-OS on Nexus 5000 Minimum software Recommendation http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/release/recommended_releases/recommended_nx-os_releases.html

• NX-OS Software Release Strategy Document http://www.cisco.com/en/US/prod/collateral/iosswrel/ps9494/ps9372/guide_c07-658595.html

http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/recommended_releases/recommended_nx-os_releases.html





http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/release/recommended_releases/recommended_nx-os_releases.html




http://www.cisco.com/en/US/prod/collateral/iosswrel/ps9494/ps9372/guide_c07-658595.html





5.2 is a long-lived NX-OS train on N7K and N5K (EOS)

6.2 is a long-lived NX-OS train on N7K and N5K/N6K

NX-OS Software Life Cycle

64

Short Lived Release

FCS EoSM EOL

Maintenance Release

(1 or 2 releases)

(8-12 weeks)

End of

Maintenance

EoS

12

Months 6

Months

End of

Support

54

Months

Long Lived Release

FCS EoSM EOL

Maintenance Release

Introduction Phase

(8-12 weeks)

Maintenance Release

Mature Phase

(6-12 months)

End of

Maintenance

EoS

Up to 36 Months 12 Months

End of

Support

48 Months


Agenda








• Innovation

• Conclusion

65


Virtual Device Contexts (VDCs)

66

• VDCs are not… – The ability to run different OS levels on the same box at

the same time

– based on a hypervisor model; there is a single ‘infrastructure’ layer that handles h/w programming…

• VDC—Virtual Device Context – Flexible separation/distribution of Software

Components

– Flexible separation/distribution of Hardware Resources

– Securely delineated Administrative Contexts

Infrastructure

Layer-2 Protocols Layer-3 Protocols

VLAN mgr

STP

OSPF

BGP

EIGRP

GLBP

HSRP

VRRP

UDLD

CDP

802.1X IGMP sn.

LACP PIM CTS SNMP

RIB RIB

Protocol Stack (IPv4 / IPv6 / L2)

Layer-2 Protocols Layer-3 Protocols

VLAN mgr

STP

OSPF

BGP

EIGRP

GLBP

HSRP

VRRP

UDLD

CDP

802.1X IGMP sn.

LACP PIM CTS SNMP

RIB RIB

Protocol Stack (IPv4 / IPv6 / L2)

Kernel

VDC A

VDC B

VDC A VDC B

VDC n


Virtual Device Contexts (VDCs)

67

• Typical silo/stovepipe design • Production, Development, Test

• Intranet, Internet, DMZ, Extranet

• Application A, Application B, Application C

• Customer A, Customer B, Customer C

• VDCs enable collapsing of physical

infrastructure into logical infrastructure

• Preserves security, administration, and organizational

boundaries, & fault isolation

• FIPS 140-2 and Common Criteria EAL4+ certified

Physical network

islands are virtualized

onto common data

center infrastructure

VDC Extranet

VDC Prod

VDC DMZ


Fabric Extenders

68

• Nexus 7x00/600x/5x00 + FEX is like a “Virtual Chassis” • Nexus 2000 FEX is an “intelligent patch panel” to its “parents” • No Spanning Tree between the FEX and its “parent” • No local switching on the FEX • NX-OS Linecard code runs on the 2148/2248/2232/2248

Fabric Extender (FEX)

Nexus 5x00 Nexus 7x00 Nexus 600x Nexus 9x00


Access Layer with Nexus 2000

69

Physical view (Efficient cabling)

Logical view (Efficient Management)

Combines benefit of ToR and EoR architecture

• Reduces cable runs

• Reduce management points in the network

• Easier to ensure feature consistency across hundreds or thousands of server ports


Virtual Port Channel (vPC) Objectives

70

• Provides a loop-free topology

• Maximises bandwidth / lower over-subscription

• Improved convergence & availability

vPC on Nexus

logical equivalent


Up-to 32 Ports

vPC Topology Example

71

• Two layers of vPC peers can be connected back-to-back e.g. N7k to N5k

• Opportunity for very high bandwidth using an evolutionary development of STP

• Up to 32-way port-channel

*Note* Use unique domain IDs

• Back to Back

vPC member

Routed

Interface

Host Port

Nexus

7x00

Nexus

5x00/600x

Nexus

2000


FabricPath: an Ethernet Fabric Shipping on Nexus 7x00, Nexus 600x and Nexus 5x00

72

N7K(config)# interface ethernet 1/1

N7K(config-if)# switchport mode fabricpath

Eliminates Spanning tree limitations

High resiliency, fast network re-convergence

Any VLAN, Anywhere in the Fabric

Connect a group of switches using an arbitrary topology

With a simple CLI, aggregate them into a Fabric

FabricPath


FabricPath Emulated Switch ID

• Anycast HSRP data plane allows for binding an “Anycast Switch ID” (ASID) with the vMAC of HSRP group ID.

• ASID – uses a similar concept as an “Emulated Switch ID” deployed in vPC+ environments where every Anycast HSRP Gateway router apart from its real Switch-ID, also advertises ASID

• Since ASID is mapped to multiple switches, any InterVLAN or Routed traffic can leverage more than one exit point in the network*.

A B

VLAN 10 VLAN 20

WAN

L2

L3

C

HSRP vMAC

SID:10 SID:20 SID:30 SID:40

SID:50 SID:60 SID:70 SID:80

ASID ASID ASID ASID

*FabricPath IS-IS facilitates building ECMP with all available shortest paths

73


Overload Bit

• Additional Spine switch is brought up and starts sending updates with Overload Bit set

s1 s2 s3 s4

FabricPath

Up

date

Advertise

overload bit

in topology

updates

Too many SPF updates: back off.

Routing table incomplete maintain overload bit, no blackholing

74


Overload Bit - continued

• Spine clears “Overload bit” and now is ready to become a transit path for traffic

s1 s2 s3 s4

FabricPath

Up

date

Overload bit

cleared: S1 is

operational

Routing table complete, clear overload bit and start routing…

75


Dynamic Fabric Automation (DFA)

76

Fabric

Management

Optimized

Network Virtual Fabrics Workload

Automation

Fabric

Management

Workload

Automation

Virtual Fabrics Optimized

Network

Bundled functions are modular and simplified for scale and automation

More details in BRKDCT-2385 – Dynamic Fabric Automation Architecture


Today’s DC Challenges

Many devices to manually configure

Protocol Restrictive Function

Deficient SW overlays

Network elasticity constraints

Disparate workload provisioning

Static resource allocation

Auto-configuration at scale

Protocol independent Function Integration

HW-Based Fabric Optimized Functions

Any workload (physical/virtual) anywhere anytime

Automated Workload Workflow

Dynamic Resource Management

Operational Complexity

Architecture Rigidity

Infrastructure

Inefficiency

Are the result of…

SIMPLIFY

OPTIMIZE

AUTOMATE

Dynamic

Fabric

Automation

Architecture

77

© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3471 Cisco Public 78

Advantages

Any subnet, anywhere, rapidly

Reduced Failure Domains

Extensible Scale & Resiliency

Profile Controlled Configuration

Any/all subnets on any leaf

Any/all Leaf Distributed Default Gateways

Full bisectional bandwidth (N spines)

Network Config profile Network Services Profile

n1000v# show port-profile name WebProfile

port-profile WebServer-PP

description:

status: enabled

system vlans:

port-group: WebServers

config attributes:

switchport mode access

switchport access vlan 110

no shutdown

security-profile Protected-Web-Srv

evaluated config attributes:

switchport mode access

switchport access vlan 110

no shutdown

assigned interfaces:

Veth10

Cisco Dynamic Fabric Automation Scale, Resiliency and Efficiency


Any subnet anywhere => Any leaf can instantiate any subnet

All leafs share gateway IP and MAC for a subnet (No HSRP)

ARPs are terminated on leafs, No Flooding beyond leaf

Facilitates VM Mobility, workload distribution, arbitrary clustering

Seamless L2 or L3 communication between physical hosts and virtual machines

GW IP: 11.11.11.1

GW MAC:

0011:2222:3333

GW IP: 10.10.10.1

GW MAC:

0011:2222:3333

L3

L2

Anycast Gateway

DFA - Distributed Gateway at the Leaf

79


Virtual Fabrics

Introducing Segment-ID Support

Traditionally VLAN space is expressed

over 12 bits (802.1Q tag)

Limits the maximum number of segments in a

data center to 4096 VLANs

DFA leverages a double 802.1Q tag for a

total address space of 24 bits

Support of ~16M L2 segment (10K targeted at

FCS)

Segment-ID is hardware-based innovation

offered by leaf and spine nodes part of the

Integrated Fabric

FabricPath Frame Format

Integrated Fabric Frame Format

Segment-ID =

80

802.1Q 802.1Q

DFA Frame


Virtual Fabrics

802.1Q Tagged Traffic to Segment-ID Mapping

Segment-IDs are utilized for providing

isolation at L2 and L3 across the

Integrated Fabric

802.1Q tagged frames received at the

leaf nodes from edge devices must be

mapped to specific Segments

The VLAN-Segment mapping can be

performed on a leaf device level

VLANs become locally significant on the

leaf node and 1:1 mapped to a Segment-ID

Segment-IDs are globally significant,

VLAN IDs are locally significant

81

WAN

802.1q Trunk 802.1q Trunk

VLANs VLANs

Segment-IDs (Global)

Segment-ID

5000

vlan 10

mode fabricpath

vn-segment 5000

vlan 10

mode fabricpath

vn-segment 5000


Simplifying Fabric Management & Optimizing Fabric Visibility

82

Advantages

Device Auto-Configuration

Cabling Plan Consistency Check

Automated Network Provisioning

Common point of fabric access

Network, vFabric & Host Visibility

TFTP

Services

DHCP

Services

XMPP

Server

LDAP

Message

Broker DCNM (CPoM)


Workload Automation & Open Environment

83

Advantages

Any workload, anywhere, anytime

Open Integration: orchestration

Automated scalable provisioning

Workload aware fabric

Services

Controller

Fabric Mgmt

Provisioning

Open

APIs

Published

Schemas

Network & Network

Services Policies

Cloud Stacks

Compute & Storage

Policies

UCS Director


Overlay Transport Virtualization Enable L2 Extension over IP Networks

84

West

OTV

East

South

MAC Addresses

Advertisements OTV

OTV

Custom built technology to solve specific challenges

• No pseudo-wire state maintenance

• Optimal multicast replication

• Multi-point connectivity

Active control plane protocol brings massive benefits

• Failure boundary preservation

• Built-in loop prevention

• Automated multihoming

• Site independence


Cisco Integrated Traffic Director (ITD)

Benefits: • Order of magnitude OPEX savings : reduction in

configuration, and ease of deployment

• Order of magnitude CAPEX savings : Wiring,

Power, Rackspace and Cost savings

• Scalability : Multi-Terabits/s, large number of

nodes, no CPU overhead.

• High availability : N + M redundancy

ITD Overview: • ASIC based multi-Tbps L4 load-balancing at line-rate

• VIP based server load-balancing

• Capability to create clusters of devices, eg, Firewalls

• Redirect line-rate traffic to any devices, for example

web cache engines, WAE, video-caches, etc..

• No service module or external load-balancer needed.

• IP-stickiness

• ACL along with redirection and load balancing

simultaneously.

• Monitoring the health of servers/appliances.

• Supports both IPv4 and IPv6

ITD Deployment

Scalable solution for L3/L4 load-balancing, redirection and clustering

ITD

Redirect

Clients

ACL to select traffic

Select the traffic

destined to VIP

Load-balance

85 85


Cisco Integrated Traffic Director (ITD) Enabling Scalable and highly available data-centers

Application/Services scaling

Multi-Tbps Scale

VIP based L3/L4 server load-balancing Redirect traffic to web cache,

video-cache, WAE, etc.

Create multi-Tbps firewall

Significant OPEX reduction

Investment protection : Supported on all LCs and Sups on both N7000 and N7700* *F1 Modules do not support ITD

86 86

© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3471 Cisco Public 87

Problems being addressed:

VLAN scale – VXLAN extends the L2 segment ID field to 24-bits, potentially allowing for up to 16 million unique L2 segments over the same network

Layer 2 segment elasticity over Layer 3 boundary – VXLAN encapsulates L2 frame in IP-UDP header

High Level Technology Overview:

MAC-in-UDP encap

Leverages multicast in the transport network to simulate flooding behavior for broadcast, unknown unicast and multicast in the same segment

Leverage ECMP to achieve optimal path usage over the transport network

NX-OS and VXLAN What does VXLAN solve/address?


NX-OS and VXLAN Supported Functionalities

88

VXLAN to VLAN Bridging (L2 Gateway) VXLANORANGE

Ingress VXLAN packet on

Orange segment

Egress interface chosen

(bridge may .1Q tag the packet)

VXLAN L2

Gateway

Egress is a tagged interface.

Packet is routed to the new VLAN VXLAN to VLAN Routing (L3 Gateway)

VXLANORANGE


Orange segment

VXLAN

Router

Destination is in another segment.

Packet is routed to the new segment

VXLANORANGE VXLANBLUE


Orange segment

VXLAN

Router

VXLAN to VXLAN Routing (L3 Gateway)

*Check hardware platform for capability


Agenda








• Innovation

• Conclusion

89


NX-OS Software Architecture Top Things to Remember

90

• NX-OS built around High-Availability as a core principle

• NX-OS highly-granular modularity for improved efficiency and fault isolation

• NX-OS built to compartmentalize, scale (up or down), be portable, and extendable

• Based on proven SAN-OS/IOS & secure/standard features implementation

• Enabling virtual mobility and cloud services


NX-OS Software Architecture Summary

91

N

S

Highly Available and Secure

Modular and Efficient

Full-Featured and Cloud Ready

X

O

cisco nx-os software architectured2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/brkarc-3471.pdfcisco...

Documents