deploying sdn on asr - …d2zmdbbm9feqrf.cloudfront.net/2015/usa/pdf/brkspg-2722.pdfdeploying sdn on...

Deploying SDN on ASR 9000

Joel Roberts, Technical Leader, Advanced Services [email protected]

Rob Piasecki, Solutions Architect, Advanced Services

[email protected]

BRKSPG-2722

mailto:[email protected]


• SDN Drivers, Concepts, & Definitions

• ASR9000 SDN Enabled Platform

• BGP-LS

• Stateful PCEP

• OpenFlow

• NETCONF/YANG

• Summary

Agenda

Drivers, Concepts, & Definitions

There Are Many Reasons to Change

Industry Realignment

Significant Traffic Growth

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

2012 2013 2014 2015 2016 2017

Ex

ab

yte

s p

er

Mo

nth

Web/Data (24.2%, 18.9%)

File Sharing (15.7%, 8.1%)

Managed IP Video (21.8%, 21.0%)

Internet Video (38.3%, 52.0%)

50 Billion Connected Things

by 2020

Connected Things Growing 5X Faster than

Mobile Devices

More than 22% of all

networked events will be

Machine Driven by 2017

Emergence of Internet of Everything

Acquires 45%

of Verizon

Wireless from

+

+

+

+

+

+

Evolution & Intersections of Technology

Video

CloudMobility

M2M

There Are Many Key Challenges

Traffic GrowthComplexity

Competition

Total Cost Ownership Speed of Innovation

Video

M2M

Cloud

MobilityEPN

Evolved Programmable Network (EPN)

NCS NCS

APIs

APIs

EDGECORE

UA

VM VM

Edge

Core

VM

Orchestration

VM / Storage Control

Service CatalogService Orchestration Apps

VM

Application

CDN

AGILITY

OPTIMIZE

MONETIZE ¥£€$

Always “ON”

On-DemandServices Anywhere

Dynamic Scale

ApplicationInteraction

SeamlessExperience

Policy

Real-Time Analytics

Services Orchestration

Fully Virtualized

IntelligentConvergence

Automated

Open and Programmable

Unified Access

SDN: Means Many Things to Many People

Practical Attributes:

• Software platform with (abstracted) view of the network

• Northbound APIs for Application interaction to/from network

• Southbound methods to collect and program information from/to network

Optical

IP/MPLS

NB API

Centralized Platform with

Global View

Apps

Today’s Infrastructure: Many Silos, Complex…

Data

Center

CustomersWide Area Network Cloud

Optical

IP/MPLS

Software Engineering (Applications)

System Engineering

(Compute, Storage, Virtual

Machines)

Network Engineering

(WAN Services, NfV)Enterprise

Virtual

Evolved Programmable Network

Data

Center


Optical

IP/MPLS


System Engineering


Machines)

Network Engineering


Virtual

Physical and Virtual

API’s

Orchestration

API’s

Applications

SDN: Place In Network

Data

Center

Access/CustomersWide Area Network Cloud

Optical

IP/MPLS

DC/Cloud

SDN

Enterprise

SDN

Applications DC-WAN X-Domain OrchEnt-SP WAN X-Domain Orch

SP SDN

(e.g WAN Orchestration)

SDN: Place In Network

Data

Center Access/Customers Wide Area Network Cloud

Optical

IP/MPLS

DC/Cloud

SDN

Enterprise

SDN

Applications DC-WAN X-Domain OrchEnt-SP WAN X-Domain Orch

SP SDN

(e.g WAN Orchestration)

APIC

EMAPIC

DC

Many SDN Software Platform Options

“Buy” “Build/Develop”

Tail-F NCS

Orc

hestra

tion

NETCONFPCEPBGP-LSNETFLOWSNMP CLI

Multiple Methods to/from Physical & Virtual Devices

Scope

Applications

Orchestration

API’s + Protocols

Orchestration – Then and Now

OSS/BSS

WAN CPE

Data Center

CPE

Prov

WAN Prov

DC

Prov

OSS/BSS

WAN CPE

Data Center

Orchestration Platform

YANG Service Models

YANG Device Models

REST, Netconf, …

Customized BSS/OSS and Provisioning Systems Model-driven Network and Service

Orchestration

Services Platform Evolution Modular Framework for SDN and NFV

Customer-Facing

Services (Broker)

Service Provider OSS

Service Provider Existing Portal and Catalog

Portal/Service Catalog RT-OSS (Service Fulfillment, Inventory, Assurance)

Service

Orchestration

with Tail-f

Applications and

Controllers

DCN Compute and

Storage

Orchestration Engine

Netconf/

YANG

Access WAN

WAE

Network Services Orchestration with Tail-f Technology

OSCPCEP

Infrastructure

DCNCPE SP Network Compute and Storage

MERAKIAPIC

EM VTS APIC VNF-M

SDN Strategy for SPs – High Level View

• Model driven, end-to-end service lifecycle and customer experience focus

• Seamless integration with existing and future OSS/BSS environment

• Loosely-coupled and modular architecture leveraging open APIs and standard protocols

• Orchestration across Multi-domain & multi-layer provides centralized policy and services across entire network

Service-Intent API

Orchestration, Service & Policy Implementation

Branch/CPE

ControlEMS/NMS

DC & NFV

ControlMulti-layer WAN

SDN

CLI/

SNMP

SDN / APIs

Openstack /

vCenterOpenflowPCEP

Segment

Routing

Netconf/

YANGBGP

BSS

OSS (Fulfillment & Assurance)

Multi Vendor End to End Management & Orchestration(Physical & Virtual)

Metro and Access WAN Data CentreCPE

WAE

WAN Automation Engine

Network Plan

WAN Automation Software Suite

Design and Network Planning

Network

PlanningOptimization Failure

Analysis

Visualization, Analytics, BI, Inventory

Weather MapBusiness

IntelligenceNetwork

Inventory

Service, Network,

and Analytics

REST APIs

Optimization and Prediction

Deployer Collector

New ModelCurrent Model

CalendaringAnalytics

Collection Drivers

NetFlowCLISNMP BGP-LS ...PCEPOSCNC/YA

NG

OrchestrationRESTful APIs

DeployerCollection

Use-Case: Bandwidth Scheduling (On-Demand)

WAN

R1

R2

R3Data Center #1 Data Center #2

5

1

2 3

4

Congested!!

Challenge/Opportunity:

Customer has “on demand”

need for a DC backup

Network conditions reported

to collector (continuous)1

Customer requests DC1 to

DC2 bandwidth NOW2

Demand admission request:

<R1-R3, B/W, NOW>3

Orchestration returns option

and customer confirms4

R1-R3 LSP tunnel

programmed 5


DeployerCollection

Use-Case: Bandwidth Calendaring

WAN

R1

R2


5

1

2 3

4

Congested!!

Customer uses self-service portal to request and schedule bandwidth between DC’s


to collector (continuous)1


DC2 bandwidth at future date2

Demand admission request:

<R1-R3, B/W, Future Date>3

Orchestration returns

confirmation of date/time4

R1-R3 LSP tunnel

programmed 5

Use Case: Coordinated Maintenance

Select

• Network Element

• ID circuits traversing node

Evaluate

• Impact to global Network

• Network capacity to reroute LSPs

Schedule

• Time changes prior to outage

• Time normalization

ASR 9000 SDN Enabled Platform

ASR 9001 ASR9904 ASR 9006 ASR 9010 ASR9912 ASR 9922

Size 2RU 6RU 10RU 21RU 30RU 44RU

# of I/O Slots 2 MPAs 2 4 8 10 20

Air Flow Side to side Side to side Side to back Front to back Front to back Front to back

Fabric BW/slot N/A 770G/385G 440G/220G 440G/220G 770G/660G 770G/660G

Chassis

scalability/slot120Gbps 2Tbps 880Gbps 880Gbps 2Tbps 2Tbps

Total Chassis BW 120Gbps 8Gbps 3.5 Tbps 7.0 Tbps 20Tbps 40Tbps

Cisco ASR 9000 Portfolio Evolution

ASR 9001-S

Virtual XR DP

IOS XRv 9000

ASR9K SDN Enabled Platform & Strategy

Controller

OpenFlow

OpenFlow Agent

Applications

Policy Servers OSS/BSS User AppAnalytics

Orchestration

Control PlaneManagement

Plane

Controller

Controllers3

Program Policies for Optimized Experience

BGP-LSAPIs and Protocols

OpenFlow Agent ,

BGP-LS, PCEP,

NETCONF

2

SDN Enabled Data

Plane

1

E-PBR Data Plane

Policy Fwd Flows

ASR 9000

SDN Platforms

Orchestration,

Analytics, Services, etc

4

VSM

Harvest

Network

Intelligence

Controller

PCEP

Applications5

Various Tools for Testing & Evaluation

OpenDaylight (ODL)

https://wiki.opendaylight.org/view/Getting_started

http://www.opendaylight.org/software/downloads

REST Client:

Chrome: Postman REST Client

Firefox: REST Client

Wireshark: https://www.wireshark.org/

IOS XR 5.2.2 with physical routers and

Virtual Internet Routing Lab (VIRL): virl.cisco.com

For YourReference

https://wiki.opendaylight.org/view/Getting_started

ASR9000 BGP Link-State (BGP-LS)

Network Discovery & Data Collection

• Traditional

• Methods: SNMP, SNMPCollect (polling), Netflow, CLI Parsing

• Platforms: Network Management Systems (NMS)

• Are and will continue to be used…

• Some Challenges:

• Delay in topology changes to NMS • E.g. delay based on polling intervals

• Unreliable transport• e.g. missed SNMP traps

• Must Augment with (near) real-time information about the state of the network

Overview BGP Link-State (BGP-LS)• Use BGP to advertise LSDB and TED of a

network

• New link-state address family

• Support for OSPF and ISIS LSDB

• Advantages

• Single upstream topology feed (BGP)

• IGP isolated from external entities

• Leverage well-known BGP security,transport and policy knobs

• Enables operator control

• Support introduced in IOS XR 5.1.1

• Implementations: WAN Automation Engine (WAE), OpenDaylight (ODL)

Domain 1 Domain 2

Domain 0

BGP-LS

BGP-LS BGP-LS

RR

PCE

TED

LSP DB

Reference Diagram: BGP-LS

OpenDaylight

192.168.96.0/29

.1

.2

192.168.x.x/3164.4/31

64.0/31

192.4/31 192.6/31

128.2/31

128.0/31

192.0/31

AG04-4

10.99.8.8/32

AG04-2

10.99.2.2/32

AG06-3

10.99.5.5

AG06-1

10.99.3.3

AG06-2

10.99.4.4

AG06-4

10.99.6.6

AG04-1

10.99.1.1

172.18.152.3/24

For YourReference

BGP-LS

REST

router ospf 1distribute bgp-ls instance-id 1router-id 10.99.8.8area 0

router bgp 65000bgp router-id 10.99.8.8address-family link-state link-state!neighbor 192.168.96.2remote-as 65000update-source Loopback0address-family link-state link-state!!

ASR9000 BGP Link State Configuration

Distribute link state

database into BGP-LS

Specify BGP-LS peer

Enable address-family

link-state

BGP Link State Prefixes

• BGP-LS prefix string has the following general format

[NLRI-Type][Area][Protocol-ID][Local node descriptor][Remote node descriptor][Attributes]/prefix-length

• Node descriptors and attributes consists of potentially multiple TLVs

• Node descriptors and attributes are shown as

[X[TLV1][TLV2]…]

• Where X identifies object (e.g. local node, remote node, link, etc.)

• TLVs are shown in the format

[yVALUE]

• Where y identifies field type (e.g. AS number, interface address, etc.)

For YourReference

RP/0/RSP0/CPU0:ASR9K-4#sh bgp link-state link-state Tue May 6 00:09:45.523 UTCBGP router identifier 10.99.8.8, local AS number 100Status codes: s suppressed, d damped, h history, * valid, > best

i - internal, r RIB-failure, S stale, N Nexthop-discardOrigin codes: i - IGP, e - EGP, ? - incompletePrefix codes: E link, V node, T IP reacheable route, u/U unknown

I Identifier, N local node, R remote node, L link, P prefixL1/L2 ISIS level-1/level-2, O OSPF, D direct, S statica area-ID, l link-ID, t topology-ID, s ISO-ID,c confed-ID/ASN, b bgp-identifier, r router-ID,i if-address, n nbr-address, o OSPF Route-type, p IP-prefixd designated router address

Network Next Hop Metric LocPrf Weight Path*> [V][O][I0x1][N[c100][b10.99.8.8][a0.0.0.0][r10.99.1.1]]/376

0.0.0.0 0 i *> [E][O][I0x1][N[c100][b10.99.8.8][a0.0.0.0][r10.99.1.1]][R[c100][b10.99.8.8][a0.0.0.0][r10.99.2.2]][L[i192.168.192.2][n192.168.192.3]]/792

0.0.0.0 0 i

ASR9000 BGP Link State Verification

Prefix codes

Node

Link

BGP-LS *OpenDaylight Sample Configuration

<type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:bgp:rib:impl">prefix:rib-impl</type>

<name>example-bgp-rib</name>

<rib-id>example-bgp-rib</rib-id>

<local-as>65000</local-as>

<bgp-id>192.168.96.2</bgp-id>

<bgp>

<type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:bgp:listener">prefix:listener</type>

<name>example-bgp-peer</name>

<host>10.99.8.8</host>

</bgp>

* Approximately Lines 55 to 67 of <install-path>/etc/opendaylight/karaf/41-bgp-sample.xml

BGP-LS peer (ASR9000)

OpenDaylight Controller

RESTCONF* Sample OpenDaylight BGP LS Topology

{ "topology": [{ "link": [

{ "link-id": "bgpls://Ospf:1/type=link&local-as=100&local-domain=174262280&local-area=0&local-router=174261509&remote-as=100&remote-domain=174262280&remote-area=0&remote-router=174261766&ipv4-iface=192.168.128.0&ipv4-neigh=192.168.128.1",

"l3-unicast-igp-topology:igp-link-attributes": {"ospf-topology:ospf-link-attributes": {

"ospf-topology:ted": {} },"l3-unicast-igp-topology:metric": 10 },

"destination": {"dest-node":

"bgpls://Ospf:1/type=node&as=100&domain=174262280&area=0&router=174261766","dest-tp": "bgpls://Ospf:1/type=tp&ipv4=192.168.128.1”},

"source": {"source-tp": "bgpls://Ospf:1/type=tp&ipv4=192.168.128.0","source-node":

"bgpls://Ospf:1/type=node&as=100&domain=174262280&area=0&router=174261509”}

* http://localhost:8080/restconf/operational/network-topology:network-topology/topology/example-linkstate-topology

BGP-LS Sample* Topology Visualization

*Visit DevNet for Sample BGP-LS Application

ASR9000 Path Computation Element

Protocol

Path Computational Element (PCE) • Path computation in large, multi-domain, and multi-layer networks

• Can reside inside router or on an external server platform

• Setup intra-, inter-area (or domain) and inter-layer paths

• Computed Path might be:

• Explicit route identifying a contiguous set of strict hops between the source and destination

• Combination of strict/loose hops between the source and destination

Basic function is to compute paths (e.g. MPLS TE LSPs)

in MPLS/GMPLS Networks

PCE Definitions

• Traffic Engineering Database (TED)

• Contains topology and resource information

• Inputs from IGP LSDB

• PCE Server (PCE)

• Path Computation Client (PCC)

• Agent on router(s) that interact withPCE Server

• PCE Protocol (PCEP)

• Protocol that runs between PCC on router and PCE server

• Stateless

• PCE has no knowledge of previously established paths

• Stateful

• Synchronization between PCCs and PCEs

• PCC maintains state synchronization with PCE

• PCC may delegate LSP control to PCE

• Either PCE or PCC can initiate LSP setup

• PCC always owns LSP state

Stateless and Stateful PCE

PCEP

Stateful PCE

TED

LSP DB

PCC

PCE-initiated

LSP

Stateful PCE• LSP Database

• Contains info/status on active LSPscommunicated by PCCs in LSP state report messages

• Active Stateful PCE

• References LSP DB for path computations

• Programs LSP state in network

• PCC reports LSP status to PCE

• PCE may reprogram an existing LSP

• LSP Delegation

• PCC delegates LSP control responsibility to PCE LSP=Label Switch Path

ASR9000 PCE-Initiated Tunnels• Treated as dynamically created tunnels

(auto-tunnel)

• Router does NOT verify or compute path that PCE provides

• treated as verbatim path

• PCE responsible for LSP re-optimization

• PCE sends an PCEP Update when a better path exists

• Tunnels may be inter-area

• *PCE Initiated tunnel does not appear in configuration

• “show mpls traffic-eng tunnels”

PCEP

Initiate /

Create

Stateful PCE

TED

LSP DB

PCC

Reference Diagram: PCE Initiated LSP

OpenDaylight

192.168.96.0/29

.1

.2

192.168.x.x/3164.4/31

64.0/31

192.4/31 192.6/31

128.2/31

128.0/31

192.0/31

AG04-4

10.99.8.8/32

AG04-2

10.99.2.2/32

AG06-3

10.99.5.5

AG06-1

10.99.3.3

AG06-2

10.99.4.4

AG06-4

10.99.6.6

AG04-1

10.99.1.1

172.18.152.3/24

For YourReference

PCEP

REST

ASR9000 PCC Configurationhostname ASR9K-AG04-4!ipv4 unnumbered mpls traffic-eng Loopback0!mpls traffic-engpcepeer ipv4 192.168.96.2stateful-clientinstantiationdelegation

!auto-tunnel pcctunnel-id min 7000 max 9999

!

Allow PCE-initiated LSP

User defined

tunnel number

range.

Required for

auto-tunnel

PCE server

Delegate statically

configured tunnels

ASR9000 Verification PCE Peer

RP/0/RSP0/CPU0:ASR9K-4#show mpls traffic-eng pce peer Tue May 6 03:21:17.216 UTC

Address State Learned From--------------- ------------ --------------------

192.168.96.2 Up Static config

PCE Add-LSP (OpenDaylight) Sample

• REST URL: http://localhost:8080/restconf/operations/network-topology-pcep:add-lsp

• Method: POST Content-Type:application/xml

<input>

<node>pcc://10.99.8.8</node>

<name>test-tunnel</name>

<arguments>

<endpoints-obj><ipv4>

<source-ipv4-address>10.99.8.8</source-ipv4-address>

<destination-ipv4-address>10.99.6.6</destination-ipv4-address>

</ipv4></endpoints-obj>

</arguments>

<network-topology-ref xmlns:topo="urn:TBD:params:xml:ns:yang:network-topology">/topo:network-topology/topo:topology[topo:topology-id="pcep-topology"]</network-topology-ref>

</input>

PCC

Source

Destination

http://localhost:8080/restconf/operations/network-topology-pcep:add-lsp

ASR9000 Verification PCE Tunnel

RP/0/RSP0/CPU0:ASR9K-AG04-4#show mpls traffic-eng pce tunnels Mon May 19 17:36:20.237 UTCTunnel : tunnel-te7004

Destination : 10.99.6.6State : down

PCE Update LSP (OpenDaylight) Sample

• URL: http://localhost:8080/restconf/operations/network-topology-pcep:update-lsp

• Method: POST Content-Type:application/xml<input>

<node>pcc://10.99.8.8</node> <name>test-tunnel</name>


<arguments>

<operational xmlns:stateful02="urn:opendaylight:params:xml:ns:yang:pcep:crabbe:stateful:02">true</operational>

<ero>

<subobject><loose>false</loose><ip-prefix><ip-prefix>192.168.64.4/32</ip-prefix></ip-prefix></subobject>




</ero>

</arguments>

</input> Explicit-Route Object (ERO)


RP/0/RSP0/CPU0:ASR9K-AG04-4#show mpls traffic-eng pce tunnels Mon May 19 17:39:38.550 UTCTunnel : tunnel-te7004

Destination : 10.99.6.6State : upCurrent path option : 10

PCE Remove LSP (OpenDaylight) Sample

• URL: http://localhost:8080/restconf/operations/network-topology-pcep:remove-lsp

• Method: POST Content-Type:application/xml

<input>

<node>pcc://10.99.8.8</node>

<name>test-tunnel</name>


</input>

http://localhost:8080/restconf/operations/network-topology-pcep:remove-lsp


RP/0/RSP0/CPU0:ASR9K-AG04-4#sh mpls traffic-eng pce tunnels Mon May 19 17:43:35.610 UTCNo PCE tunnels.

ASR9000 Show Commands

• New commands

• show mpls traffic-eng auto-tunnel pcc […]

• Modified Commands

• show mpls traffic-eng pce peer [ all|ipv4 <addr>|node-id <id>|stateful|stateless]

• Other useful commands

• show mpls traffic-eng pce tunnels <id>

• show mpls traffic-eng pce trace […]

• show mpls traffic-eng tunnels tabular

Segment Routing

• Simple routing extensions to implement source routing

• Packet path determined by prepended segment identifiers (one or more)

• Reduced network state

• Increased network scalability and virtualization

• Data-plane agnostic

• MPLS: label as segment id

• IPv6: address as segment id

• The state is no longer in the network but in the packet

• datatracker.ietf.org/wg/spring/documents/

*BRKRST-2124 Introduction to Segment Routing

Traffic Steering into PCE-Initiated Tunnels

Example approaches:

• Autoroute announce

• Policy-based tunnel selection (forwarding class id)

Policy Based Tunnel Selection

• Local mechanism at head-end

• PBR policy sets forwarding class for incoming traffic

• Traffic switched to tunnel with matching forwarding class

• Seven forwarding classes supported (1-7)

• One forwarding class reserved as default (0)

PE1

PE2

Tunnel-te1000

Forwarding class 1

Tunnel-te2000

Forwarding class 0

(default)

Input policy matches

traffic based on ACL

and sets forwarding

class

Traffic Steering PCE Initiated Tunnel

Tunnel-te1000

Forwarding

class 1

Tunnel-te2000

Forwarding

class 0 (default)

Tunnel-te7000

PCE Initiated LSP

Forwarding class 2

ASR9K-1

CETenGigE0/1/0/1

10.201.7.43

ASR9K-2

ASR9K-3

Set Forward ClassID 2


PCEPProgramming

Use-Case: PCE Initiated LSP

WAN

R1

R2


4

3

Service Request1

Analytic to determine LSP2

PCCreate LSP3

Traffic steering onto LSP6

10.201.7.43

2

6

TE LSP Signaling4

Delegate & LSP State Report5

5

ASR9000: OpenFlow

OpenFlow (OF)

• Open Network Foundation (ONF)

• Layer 2 Communication Protocol:

• Access to forwarding plane

• Switch or Router

• OF Controller: Control Plane

• OF Agent (device): Forwarding Plane

OpenFlow Agent/Plugin

• Implements standard OpenFlow switch model.

• Speaks ‘standard’ OpenFlow protocol

• Native dedicated CLI for troubleshooting

Availability: Universal

User Experience: Consistent

Deployment:

End-to-end

• Available* IOS-XR, IOS and IOS-XE, NX-OS

• Common code, base features and CLI across platforms

• Across SP, Data Center, Campus…

* Please check roadmap for details on supported platforms & timelines

OpenFlow

Table:

• Consists of a set of flows (each row is a flow)

• Each flow contains a set of matches and actions.

• Applied only in ingress direction

Sends instructions…

OpenFlow Controller

Table

Match Action

OpenFlow Table

Match:

• Ingress port

• Priority: Highest priority flow entry that matches packet gets selected

• Lowest priority is zero and highest is 32768

Action:

• Packet forwarding actions

• Packet modification actions

• Actions are immediately applied to the packet

OpenFlow: ASR9000

• Support starting IOS XR 5.1.2

• OpenFlow Switch Version 1.3

• IOS XR Images

• asr9k-mini-px.vm

• asr9k-mpls-px.pie (required for L3VPN, L2VPN)

• asr9k-k9sec-px.pie (required for TLS)

• Default port TCP 6653

• Any physical interface for OF controller

OpenFlow Agent

OpenFlow Controller

Control PlaneManagement

Plane

E-PBR Data Plane

Policy Fwd Flows

ASR9K

ASR9000: OpenFlow Hybrid Mode

• Recommended

• OpenFlow and XR forwarding

• Ability to take advantages of both XR features and OpenFlow flexibility

• OpenFlow as a policy based routing (PBR) type of feature.• Can provide ‘fine-tuning’ on incoming traffic based on OpenFlow rule matching and rule actions

ASR9000 OpenFlow Layer 2 Table Options


OpenFlow Controller

L2 Only Table

Match- L2 Header Action: L2 Only

L2_L3 Table

Match: L2 & L3 Hdr Action: L2 Only

Supported L2 Matches & Actions are detailed in

ASR9000 System Management Configuration Guide, Release 5.3.x

ASR9000 OpenFlow Layer 3 Table Options


OpenFlow Controller

L3_V4 Table

Match: L3 IPv4 Hdr Action: L3 Only

L3_DS Table

Match:L3 v4 & v6 Action: L3 Only

Supported L3 Matches & Actions are detailed in


ASR9000 OpenFlow Tables

Layer Table Type Pipeline Supported

Interfaces

Description

Layer 2

L2 129 Bridge-domain, Gigabit ethernet, Bundle,

Bundle-subinterfaces, PWHE-

subinterfaces

• Supports L2 header matches

• Supports L2 actions.

• Can be applied to ingress L2 interface

L2_L3 130 L2Bridge-domain, Gigabit ethernet,

Bundle, Bundle-subinterfaces, PWHE-

subinterfaces

• Supports L2 & L3 (IPv4/IPv6) header

matches

• Supports L2 actions

• Can be applied ingress L2 interface

Layer 3

L3_V4 131 VRF and global interfaces, BVI (IPv4

only), Bridge-domain, Gigabit ethernet,

Bundle, Bundle-subinterfaces

• Supports L3 (IPv4) header matches

• Supports L3 (IPv4) actions

• Can be applied to ingress L3

interfaces

L3_DS 132 VRF and global interfaces, BVI (IPv4

only), Bridge-domain, Gigabit ethernet,

Bundle, Bundle-subinterfaces

• Supports L2 & L3 (IPv4/IPv6) header

matches

• Supports L3 (IPv4/IPv6) actions

• Can be applied to ingress L3

interfaces

For YourReference

OpenFlow Cisco Extension Actions

• Additional OpenFlow features on ASR9000

Cisco Added Actions

OpenFlow Switch Types supported on

ASR9000

Applied L2 Bridge

Domain

Applied L3 or L3 VRF

Interface

Action Description L2

Only

L2_L3 L3_V4 L3_DS

Set IPv4 Next Hop Set IPv4 Nexthop

Address

No No Yes Yes

Set IPv6 Next Hop Set IPv6 Nexthop

Address

No No No Yes

Set Forward Class ID Set Forward Class ID No No Yes Yes

Netflow Enable/Disable Enable/Disable Netflow No No Yes Yes

Openflow for Traffic Steering

• Use Openflow to program classifiers/actions on WAN Edge

• Flow entries would be something like:

• MATCH/Set_IPv4_Next_Hop

• MATCH/Set_IPv6_Next_Hop

• MATCH/Set_FCID (tunnel identifier)

• Useful for services and applications requiring steering of specific flows into a programmed WAN resource

Head-End

PCEP

Create LSP Tunnel

WANEdge Routers

Flow Entries

<MATCH/”Forward to

Tunnel”>

Tunnel

Traffic Steering

App

SDN WAN Orchestration

OF PCEP

OF

Flows


ProgrammingOpenFlow

Use-Case: OpenFlow for Traffic Steering

WAN

R1

R2


3

3

Service Request1 Analytic to determine LSP

2

Create LSP Tunnel with

FCID 23

Use OpenFlow to program

match/action on WAN edge4

L3_DS Table

Match: SRC 10.201.7.43 Action: Set FCID=2

10.201.7.43

2

4

interface TenGigE0/0/0/3l2transport!l2vpnbridge group SDN-1bridge-domain OF-1interface TenGigE0/0/0/3

!openflowswitch 7 pipeline 129tls trust-point local openfCA remote openfCAbridge-group SDN-1 bridge-domain OF-1controller ipv4 172.16.1.45 port 6653 security tls!!

ASR9000 OpenFlow L2 Switch Configuration

Openflow Switch attached to

bridge-domain

OpenFlow Controller

L2 Only Switch

interface TenGigE0/0/0/3l2transport!l2vpnbridge group SDN-2bridge-domain OF-2interface TenGigE0/0/0/4

!openflowswitch 9 pipeline 130tls trust-point local openfCA remote openfCAbridge-group SDN-2 bridge-domain OF-2controller ipv4 172.16.1.45 port 6653 security tls!!

ASR9000 OpenFlow L2 + L3 Switch Configuration

Openflow Switch attached to

bridge-domain

OpenFlow Controller

L2 + L3 Switch

openflow switch 11 pipeline 131interface Bundle-Ether2.1interface GigabitEthernet0/1/0/7interface GigabitEthernet0/0/0/0/4.1301controller ipv4 172.16.1.45 port 6653 security tls

openflowswitch 10 pipeline 131vrf ONEcontroller ipv4 172.16.1.45 port 6653 security tls

ASR9000 OpenFlow L3_V4 Switch Configuration

All interfaces in VRF become

part of OpenFlow Switch

Specify L3

Interfaces

L3_V4 Switch

L3_V4 Switch

openflow switch 100 netflowflow monitor mmap sample smapinterface Bundle-Ether3interface GigabitEthernet0/0/0/0/4.1201controller ipv4 172.16.1.45 port 6653 security tls

ASR9000 OpenFlow Netflow Configuration

• Cisco extension to OpenFlow

• Used to enable/disable NETFLOW on Layer 3 Interfaces

Sample ASR9000 OpenFlow Show CommandsRP/0/RSP0/CPU0:ASR9K-AG06-4#show openflow switch 102 controllerLogical Switch Id: 102

Controller: 1 Address : 192.168.96.3:6653Protocol : tcpVRF : defaultLocal Trustpoint: : Not availableRemote Trustpoint: : Not availableConnected : YesRole : Masterlast_error : Invalid argumentstate : ACTIVEsec_since_connect : 431579

RP/0/RSP0/CPU0:ASR9K-AG06-4#show tcp briefPCB VRF-ID Recv-Q Send-Q Local Address Foreign Address State

0x501e3690 0x60000000 0 0 10.99.6.6:58633 192.168.96.3:6653 ESTAB0x501c8720 0x60000000 0 0 10.99.6.6:38212 10.99.8.8:179 ESTAB

Sample ASR9000 OpenFlow Show Commands

RP/0/RSP0/CPU0:ASR9K-AG06-4#show openflow switch 102 flows Logical Switch Id: 102Total flows: 1Flow: 1

Match: ipv6,ipv6_dst=192:1::2 Actions: output_nh(ipv6=2001:20::2)Priority: 0Table: 0Cookie: 0x1Duration: 196.353sNumber of packets: 0Number of bytes: 0

RP/0/RSP0/CPU0:ASR9K-AG06-4#show openflow switch 102 portsTue Jan 27 14:07:21.025 UTCLogical Switch Id: 102Port Interface Name Config-State Link-State Features

1 Te0/1/0/1 PORT_UP LINK_UP 0

OpenDaylight Controller

Visit DevNet for OpenDaylight and Cisco Open SDN Controller

OpenFlow Show Commands• show openflow switch <>

• show openflow switch <> controllers | stats

• show openflow switch <> ports

• show openflow switch stats

• show openflow switch flows | brief/summary

• show openflow interface switch <>

• show openflow hardware capabilities pipeline <>

• show table-cap table-type <>

For YourReference

OpenFlow Show CommandsShow policy-map commands:

• show policy-map transient list type pbr

• show policy-map transient type pbr pmap-name <>

• show policy-map transient targets summary

PBR platform show commands

• show pbr-pal ipolicy [<policy_name> | all ] location <loc>

• show pbr-pal ipolicy <policy_name> iclass [<iclass_handle> | all ] vmr-info location <loc>

• show pbr-pal ipolicy <policy_name> iclass [<iclass_handle> | all] stats [clear-on-read] location <loc>

• show prm server tcam entries <table> vmr-id <> 100 np0 loc <>

• show prm server tcam summary <table> PBR all loc <>

For YourReference

OpenFlow Debug

Debug for OpenFlow Agent:

• debug openflow switch ovs module ofproto level debug

• debug openflow switch ovs module ofproto-plif level debug

• debug openflow switch ovs module plif-onep level debug

• debug openflow switch ovs module plif-onep-util level debug

• debug openflow switch ovs module plif-onep-wt level debug

Debug for Policy Manager

• debug policymgr all

• debug policymgr trace

• debug policymgr lib all

• debug policymgr lib trace

• Debug for PBR: debug pbr-pal all loc

For YourReference

ASR9000 OpenFlow References

ASR9000 and OpenFlow Whitepaper:

https://supportforums.cisco.com/document/12402506/asr-9000-sdn-openflow-whitepaper


http://www.cisco.com/c/en/us/support/routers/asr-9000-series-aggregation-services-routers/products-installation-and-configuration-guides-list.html

For YourReference

https://supportforums.cisco.com/document/12402506/asr-9000-sdn-openflow-whitepaper

http://www.cisco.com/c/en/us/support/routers/asr-9000-series-aggregation-services-routers/products-installation-and-configuration-guides-list.html

ASR9000 NETCONF/YANG

Traditional: Command Line Interface (CLI)

• Generally a majority of today’s configuration

• Device Specific

• Human Friendly

• Manual (challenging to automate)

• Developer un-friendly

• Such parsing CLI screen scrape

• No Common Data Model

• No Built-In Error Reporting

[joerober@rtp-odl ~]$ ssh [email protected]@10.99.1.1's password: RP/0/RSP0/CPU0:ASR9K-AG04-1#configure terminal Tue Jan 6 17:18:58.350 ESTRP/0/RSP0/CPU0:ASR9K-AG04-1(config)#

NETCONF

• Network Configuration Protocol (NETCONF)

• XML based interface between network device and NMS

• Mechanism to manage, configure, and monitor network device

• Published RFC 4741 (Dec 2006)

• Updated RFC 6241 (Jun 2011)

• RFC 6242 (Jun 2011) Using the NETCONF Protocol over Secure Shell

• Overcome SNMP Limitations

• 2003 IAB Network Management Workshop (RFC3535)

YANG

• Modeling language defined in RFC 6020

• Used by NETCONF to define objects and data in requests and replies

• Models configuration, operational, and RPC data

• Provides semantics to better define NETCONF data

• Provides common data model:

• In order for NETCONF to be useful as network-wide protocol

• To consume NETCONF data from any network device

• YANG modules are for NETCONF what MIBs are for SNMP

Comparing SNMP and NETCONF

SNMP NETCONF

Data Models Defined in MIBs Defined in YANG modules (or

XML schema documents)

Data Modeling Language Structure of Management

Information (SMI)

YANG (and XML schema)

Management Operations SNMP NETCONF

RPC Encapsulation Basic Encoding Rules (BER) XML

Transport Protocol UDP TCP (reliable transport)

NETCONF Layers

• Content

• Configuration and Operational Data

• Operations

• Defines base operations

• What clients do to servers

• Messages/Remote Procedure Call (RPC)

• Transport-independent mechanism for encoding messages

• RPC, RPC-Reply

• Transport

• Reliable communication between client and server

Content

Operations

Messages

Transport

ASR9000 NETCONF Transport & Messages

• SSH NETCONF (1.1) session:

• IOS XR 5.3.1

• On client utilize ssh with –s to request invocation of subsystem on server (router)• ssh [email protected] -p 830 -s netconf

Transport

Messages

Operations

Content

Client Server

<rpc>

<rpc-reply>


NETCONF Operations Operation Description

<get-config> Retrieve all or part of specified configuration datastore

<edit-config> Loads all or part of a configuration to the specified

configuration datastore

<get-schema> Retrieve YANG-based XML Data

<commit> Copy candidate datastore to running datastore

<get> Retrieve running configuration and device state information

<lock> / <unlock> Lock or unlock the entire configuration datastore system

<close-session> Graceful termination of NETCONF session

<kill-session> Forced termination of NETCONF session

Complete List Protocol Operations: https://tools.ietf.org/html/rfc6241

Transport

Messages

Operations

Content

ASR9000 NETCONF Data Stores

• Target of NETCONF Operations

• Data stores are named containers that may hold an entire copy of the configuration

• IOS XR Supported Datastores:

• <running>

• <candidate>

• *XR two stage commit (no startup datastore)

Running CandidateTransport

RPC

Operations

Content

YANG XR 5.3.1 Data Models

• YANG data modules are part of the software image.

• Models can be retrieved from router using <get-schema> operation.

• YANG Model file for each configuration module

• For example, to configure CDP, the relevant yang model is Cisco-IOS-XR-cdp-cfg

• 5.3.1 Data Models: http://www.cisco.com/c/en/us/td/docs/routers/crs/software/crs_r5-3/security/configuration/guide/b-syssec-cg53x-crs/Implementing_the_Network_Configuration_Protocol.pdf#unique_8

http://www.cisco.com/c/en/us/td/docs/routers/crs/software/crs_r5-3/security/configuration/guide/b-syssec-cg53x-crs/Implementing_the_Network_Configuration_Protocol.pdf#unique_8

ASR9000 NETCONF 1.1 Configuration (XR 5.3.1)

• Pre-requisite:

• k9sec pie must be installed

• Crypto keys must be generated

• Enable NETCONF Agent:

ssh server netconf port 830

netconf-yang agent ssh

• Session verification:

show netconf-yang clients

show netconf-yang statistics

! IOS XR 5.3.1

hostname ASR9K-R1

domain name cisco.com

!

ssh server v2

ssh server vrf default

ssh server netconf port 830

netconf-yang agent

ssh

!

XR 5.3.1 Sample NETCONF/YANG Workflow

ASR9000

Orchestration Platform

NETCONF Client

1 XR Image contains YANG

2 Client (application) retrieves

models with <get-schema>

3 Client installs and processes

YANG models

YANG

Data Model

1

YANG

Data Model

3

4 Client sends operation request

Using YANG-based XML data

5 Router understands YANG-based

XML data & is configured

accordingly

2 <get-schema>

<edit-config>4

Candidate Running5 6 Client-router interaction continues

Until network configured as desired

<edit-config>

<PBR> <ServicePolicy>

<Input>PBTS</Input>

</ServicePolicy>

</PBR>


Programming

Use-Case:NETCONF Detailed Policy/Traffic Steering

WAN

R1

R2


3

3

Service Request1 Analytic to determine LSP

2

Create LSP Tunnel FCID 23

NETCONF to program:

-detailed policy*

-apply policy inbound

4

10.201.7.43

2

4

*example policy in reference slides

Starting SSH NETCONF Session nms$ ssh [email protected] -p 830 –s netconf

[email protected]'s password:

<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

<capabilities>

<capability>urn:ietf:params:netconf:base:1.1</capability>

<capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability>

<capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>

<capability>urn:ietf:params:netconf:capability:rollback-on-error:1.0</capability>

<capability>urn:ietf:params:netconf:capability:validate:1.1</capability>

<capability>urn:ietf:params:netconf:capability:confirmed-commit:1.1</capability>

<!– yang capability output omitted -->

</capabilities>

<session-id>2110493906</session-id>

</hello>

]]>]]>

NETCONF Operation <hello>• Used to exchange capabilities

• Initiated by the NETCONF Server

• Must be acknowledged by client before client can send any other messages

IOS XR (5.3.1): <hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><capabilities><capability>urn:ietf:params:netconf:base:1.1</capability><capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability><capability>urn:ietf:params:netconf:capability:candidate:1.0</capability><capability>urn:ietf:params:netconf:capability:rollback-on-error:1.0</capability><capability>urn:ietf:params:netconf:capability:validate:1.1</capability><capability>urn:ietf:params:netconf:capability:confirmed-commit:1.1</capability>

<!– list of yang capability omitted from output --></capabilities>

<session-id>285212672</session-id></hello>

Transport

RPC

Operations

Content

NETCONF Operations - <edit-config>

• Modify configuration of particular data store

• Can only be used on writable data store

• Support on IOS XR:

• Candidate Configuration Capability

urn:ietf:params:netconf:capability:candidate:1.0

Transport

RPC

Operations

Content

Example <edit-config>

<?xml version="1.0" encoding="UTF-8" ?>

<rpc message-id="106" xmlns="urn:ietf:params:netconf:capability:candidate:1.0">

<edit-config>

<target><candidate/></target>

<config>

<Configuration><InterfaceConfigurationTable><InterfaceConfiguration><Naming>

<Active>act</Active>

<InterfaceName>TenGigE0/3/0/0</InterfaceName>

</Naming>

<Description>NETCONF CONFIG</Description>

<IPV4Network><Addresses><Primary>

<Address>10.223.1.1</Address>

<Netmask>255.255.255.0</Netmask>

</Primary></Addresses></IPV4Network>

</InterfaceConfiguration></InterfaceConfigurationTable>

</Configuration></config>

</edit-config> </rpc>

##

Datastore

Interface

Add interface

description

IPv4 Address

Netmask

interface TenGigE0/3/0/0

description NETCONF CONFIG

ipv4 address 10.223.1.1 255.255.255.0

NETCONF Operations - <get-config>

• Used to retrieve all or portions of configuration

• Subtree filtering support:

• Attribute Match Expression• Can only be specified in Table classes (e.g. <InterfaceName Match=“TenGig.*”/> )

• Containment Nodes: • Filtering is by specifying container classes (e.g. <InterfaceConfigurationTable/> )

• Selection Node: • Filtering by specific selection (e.g. <InterfaceName>Loopback0</InterfaceName>

Transport

RPC

Operations

Content

Example <get-config>(Attribute match)



<get-config>

<source><running/></source>

<filter>

<Configuration>

<InterfaceConfigurationTable>

<InterfaceConfiguration>

<Naming>


<InterfaceName Match=“TenGigE.*”/>

</Naming>

</InterfaceConfiguration>

</InterfaceConfigurationTable>

</Configuration>

</filter>

</get-config> </rpc>

Attribute match: all TenGigE interfaces

Datastore

For YourReference

Example <get-config>(Containment node)



<get-config>


<filter>

<Configuration>

<InterfaceConfigurationTable/>

</Configuration>

</filter>

</get-config>

</rpc>

Containment: all interfaces

Datastore

For YourReference

Example <get-config> (selection node)<?xml version="1.0" encoding="UTF-8" ?>


<get-config>


<filter>

<Configuration>

<InterfaceConfigurationTable>

<InterfaceConfiguration>

<Naming>


<InterfaceName>Loopback0</InterfaceName>

</Naming>

</InterfaceConfiguration>

</InterfaceConfigurationTable>

</Configuration>

</filter>

</get-config>

</rpc>

Datastore

Selection: Loopback0 only…

For YourReference

Example <get-config> ASR9000 <rpc-reply><?x<?xml version="1.0" encoding="UTF-8"?>

<rpc-reply message-id="106" xmlns="urn:ietf:params:netconf:capability:candidate:1.0">

<data><Configuration>

<InterfaceConfigurationTable MajorVersion="5" MinorVersion="3">

<InterfaceConfiguration> <Naming><Active>act</Active>

<InterfaceName>Loopback0</InterfaceName>

</Naming>

<InterfaceVirtual>true</InterfaceVirtual>

<IPV4Network MajorVersion="6" MinorVersion="3">

<Addresses><Primary>

<Address>10.99.8.8</Address>

<Netmask>255.255.255.255</Netmask>

</Primary></Addresses></IPV4Network>

</InterfaceConfiguration></InterfaceConfigurationTable>

</Configuration></data>

</rpc-reply>

Tagged Configuration

For YourReference

Sample: Get YANG Model <get-schema> list#229

<rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

<get>

<filter type="subtree">

<netconf-state xmlns= "urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">

<schemas/>

</netconf-state>

</filter>

</get>

</rpc>

##

For YourReference

Example: <get-schema> sample#286

<?xml version="1.0" encoding="utf-8"?>


<get-schema xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">

<identifier>Cisco-IOS-XR-ifmgr-cfg</identifier>

<version>2015-01-07</version>

</get-schema>

</rpc>

##

For YourReference

Closing SSH NETCONF Session

#140



<close-session>

</close-session>

</rpc>

## Connection to 10.99.8.8 closed by remote host.

odl-1$

RPC close-session

Connection Closed

For YourReference

Summary

Traditional Infrastructure

Data

Center


Optical

IP/MPLS


System Engineering


Machines)

Network Engineering


Virtual

Evolved Programmable Network

Data

Center


Optical

IP/MPLS


System Engineering


Machines)

Network Engineering


Virtual

Physical and Virtual

API’s

Orchestration

API’s

Applications


ProgrammingCollection


WAN

R1

R2


5

1

2

3

4Congested!!





to collector1


DC2 bandwidth NOW2

Analytic and confirmation to

determine LSP3

R1-R3 LSP Tunnel

programmed4

Traffic Steering onto

R1-R3 LSP5


ProgrammingCollection


WAN

R1

R2


5

1

2

3

4Congested!!




Augment Collection: BGP-LS1

Service Request2

Analytic, Orchestration

e.g. WAE3

LSP: PCEP, NETCONF, etc

Controller-ODL, Tail-f, etc4

Traffic Steering:

OpenFlow, NETCONF, etc5

Summary-ASR9000 APIs & Protocols

Key Function Protocol/API Comments

IGP Topology BGP Link-State Wraps up LSDB in BGP transport and pushes to BGP speaker

on SDN WAN Orch Platform

Create, Modify and Delete TE

or SR Tunnels

Stateful Extensions to

PCEP

Introduced as part of Stateful PCE effort

Classification and Action Openflow Extensions Leveraging per-flow MATCH/Action semantics

Read/Write of Persistent

Configuration Data on

Network Devices

Netconf/Yang Finally gaining traction with vendor implementations and now on

OpenDaylight Platform

Call to Action

• Visit the World of Solutions for

• Service Provider Section

• ASR9000

• WAN Automation Engine

• Tail-f

• DevNet Zone

• Related labs and sessions

• Cisco Open SDN Controller

• WAN Automation Engine (WAE)-Hands-On Labs

• DevNet: developer.cisco.com

DevNet: Getting Started with WAE

• DevNet – WAE Tutorials and API Documentation https://developer.cisco.com/site/wae/index.gsp

• Learning Labs – Hands on with the WAE APIs over in the DevNet zone https://learninglabs.cisco.com/#/home

• dCloud – demos of WAEhttp://dcloud.cisco.com/

• WAE home page on cisco.com: http://www.cisco.com/go/wae

WAE Learning labs

For YourReference

https://developer.cisco.com/site/wae/index.gsp

https://learninglabs.cisco.com/#/home

http://dcloud.cisco.com/

http://www.cisco.com/go/wae

World of Solutions: Tail-f NCS Overview

• Logically centralized network services

• Data models for data structures

• Structure representations of:

• Service instances

• Network configuration and state

• Mapping service operations to network configuration changes

• Transactional integrity

• Multiprotocol support

• Multivendor support

Device Models Network Element Drivers

Device Manager

Service Manager

Tail-f NCS Service Models

Networkwide CLI and Web UI REST, NETCONF, Java, etc.

Network Engineer

Management Applications

NETCONF, CLI, SNMP, REST, etc.

• Execution management systems (EMSs)

• Applications • Controllers

For YourReference

DevNet: Open SDN Controller (OSC)

• Cisco Distribution of OpenDaylight platform

• Enhancements

• Simplified install

• Admin/Monitoring tools

• Embedded Apps

• Plug-in Clustering

• Technical Support

• developer.cisco.com/site/openSDN/

Community Support Cisco Supported

“HELIUM”

Open SDN Controller

Common Content Incremental

Cisco

Value

Precluded

OpenDaylight

Content

SDNi

Defense4all

PacketCable

PCMM

VTN Project

OpenContrail

Plugin

SNMP4SDN

AAA

BGP-LS

Controller

DLUX

Group Policy

L2 Switch

LISP Flow

Mapping

Openflow

Plugin

OVSDB

Secure Network

Bootstrap Infra

Service Function

Chaining

Yang Tools

MD-SAL

AD-SAL

Log

Aggregation

Metrics

Aggregation

Monitoring

Basic

Clustering

Central Admin

OVA Distribution

Sample Apps

One Click Install PCEP

Plug-in Clustering

To be contributed back

to the “open community”

For YourReference

Cisco Developer Resources• Devloper Network: http://developer.cisco.com

• OpenDaylight: http://www.opendaylight.org/

• SDLC - https://developer.cisco.com/site/networking/one/sdlc/overview/

• EEM - https://supportforums.cisco.com/community/netpro/network-infrastructure/eem

• ONE Forums -https://developer.cisco.com/site/devnet/forums/index.gsp#L2CiscoONE

• APIC-EM - https://developer.cisco.com/site/networking/one/apic/enterprise-module/

• APIC-DC

• APIs https://developer.cisco.com/site/networking/routers-switches/nexus9000/documents/

• GitHub - https://github.com/datacenter/nexus9000

For YourReference

http://developer.cisco.com

https://supportforums.cisco.com/community/netpro/network-infrastructure/eem

https://developer.cisco.com/site/devnet/forums/index.gsp#L2CiscoONE

https://developer.cisco.com/site/networking/one/apic/enterprise-module/

https://developer.cisco.com/site/networking/routers-switches/nexus9000/documents/

https://github.com/datacenter/nexus9000

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

• Related sessions

Thank you

deploying sdn on asr - …d2zmdbbm9feqrf.cloudfront.net/2015/usa/pdf/brkspg-2722.pdfdeploying sdn on...

Documents