OpenFlow in Service Provider Networks

AT&T Tech TalksOctober 2010

Rob SherwoodSaurav Das

Yiannis Yiakoumis

Talk Overview

• Motivation• What is OpenFlow• Deployments• OpenFlow in the WAN

– Combined Circuit/Packet Switching– Demo

• Future Directions

Million of linesof source code

5400 RFCs Barrier to entry

500M gates10Gbytes RAM

Bloated Power Hungry

We have lost our way

Specialized Packet Forwarding Hardware

OperatingSystem

App App App

Routing, management, mobility management, access control, VPNs, …

SoftwareControl

Router

HardwareDatapath

Auth

entica

tion,

Secu

rity, A

ccess

Contro

l

HELLO

MPLS

NATIPV6

anycastmulticas

tMobile IP

L3 VPN

L2 VPN VLANOSPF-TE

RSVP-TEHELLOHELLO

Firewall

Multi layer m

ulti

region

iBGP,

eBGP

IPSec

Many complex functions baked into the infrastructureOSPF, BGP, multicast, differentiated services,Traffic Engineering, NAT, firewalls, MPLS, redundant layers, …

An industry with a “mainframe-mentality”

DeploymentIdea Standardize

Wait 10 years

Glacial process of innovation made worse by captive standards process

• Driven by vendors• Consumers largely locked out• Glacial innovation

New Generation Providers Already Buy into It

In a nutshellDriven by cost and controlStarted in data centers….

What New Generation Providers have been Doing Within the Datacenters

Buy bare metal switches/routers Write their own control/management applications on a

common platform

6

Specialized Packet Forwarding Hardware

App

App

App

Specialized Packet Forwarding Hardware

App

App

App

Specialized Packet Forwarding Hardware

App

App

App

Specialized Packet Forwarding Hardware

App

App

App

Specialized Packet Forwarding Hardware

OperatingSystem

OperatingSystem

OperatingSystem

OperatingSystem

OperatingSystem

App

App

App

Network Operating System

App App App

Change is happening in non-traditional markets

App

Simple Packet Forwarding Hardware

Simple Packet Forwarding Hardware

Simple Packet Forwarding Hardware

App App

Simple Packet Forwarding Hardware Simple Packet

Forwarding Hardware

Network Operating System

1. Open interface to hardware

3. Well-defined open API2. At least one good operating system

Extensible, possibly open-source

The “Software-defined Network”

Windows(OS)

Windows(OS)

Linux MacOS

x86(Computer)

Windows(OS)

AppApp

LinuxLinuxMacOS

MacOS

Virtualization layer

App

Controller 1

AppApp

Controller2

Virtualization or “Slicing”

App

OpenFlow

Controller 1NOX(Network OS)

Controller2Network OS

Trend

Computer Industry Network Industry

Simple common stable hardware substrate below+ programmability + strong isolation model + competition above = Result : faster innovation

What is OpenFlow?

Short Story: OpenFlow is an API

• Control how packets are forwarded• Implementable on COTS hardware• Make deployed networks programmable

– not just configurable

• Makes innovation easier• Result:

– Increased control: custom forwarding– Reduced cost: API increased competition

Ethernet Switch/RouterEthernet Switch/Router

Data Path (Hardware)Data Path (Hardware)

Control PathControl PathControl Path (Software)Control Path (Software)

Data Path (Hardware)Data Path (Hardware)

Control PathControl Path OpenFlowOpenFlow

OpenFlow ControllerOpenFlow Controller

OpenFlow Protocol (SSL/TCP)

Controller

PC

HardwareLayer

SoftwareLayer

Flow Table

MACsrc

MACdst

IPSrc

IPDst

TCPsport

TCPdport Action

OpenFlow Firmware

**5.6.7.8*** port 1

port 4port 3port 2port 1

1.2.3.45.6.7.8

OpenFlow Flow Table Abstraction

OpenFlow BasicsFlow Table Entries

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport

Rule Action Stats

1. Forward packet to port(s)2. Encapsulate and forward to controller3. Drop packet4. Send to normal processing pipeline5. Modify Fields

+ mask what fields to match

Packet + byte counters

ExamplesSwitching

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Action

* 00:1f:.. * * * * * * * port6

Flow Switching

port3

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport

Action

00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6

Firewall

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Forward

* * * * * * * * 22 drop

ExamplesRouting

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport Action

* * * * * 5.6.7.8 * * * port6

VLAN Switching

*

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport

Action

* * vlan1 * * * * *

port6, port7,port9

00:1f..

OpenFlowSwitch.org

Controller

OpenFlow Switch

PC

OpenFlow UsageDedicated OpenFlow Network

OpenFlow Switch

OpenFlow Switch

OpenFlowProtocol

Aaron’s code

Rule Action Statistics

Rule Action Statistics Rule Action Statistics

Network Design Decisions

Forwarding logic (of course)

Centralized vs. distributed control

Fine vs. coarse grained rules

Reactive vs. Proactive rule creation

Likely more: open research area

Centralized vs Distributed Control

Centralized Control

OpenFlow Switch

OpenFlow Switch

OpenFlow Switch

Controller

Distributed Control

OpenFlow Switch

OpenFlow Switch

OpenFlow Switch

Controller

Controller

Controller

Flow Routing vs. AggregationBoth models are possible with OpenFlow

Flow-Based

Every flow is individually set up by controller

Exact-match flow entriesFlow table contains one entry

per flowGood for fine grain control, e.g.

campus networks

Aggregated

One flow entry covers large groups of flowsWildcard flow entriesFlow table contains one entry per category of flowsGood for large number of flows, e.g. backbone

Reactive vs. Proactive Both models are possible with OpenFlow

Reactive

First packet of flow triggers controller to insert flow entries

Efficient use of flow tableEvery flow incurs small

additional flow setup timeIf control connection lost, switch

has limited utility

Proactive

Controller pre-populates flow table in switchZero additional flow setup timeLoss of control connection does not disrupt trafficEssentially requires aggregated (wildcard) rules

OpenFlow Application: Network Slicing

• Divide the production network into logical sliceso each slice/service controls its own packet forwardingo users pick which slice controls their traffic: opt-ino existing production services run in their own slice

e.g., Spanning tree, OSPF/BGP

• Enforce strong isolation between sliceso actions in one slice do not affect another

        • Allows the (logical) testbed to mirror the production network

o real hardware, performance, topologies, scale, users

o Prototype implementation: FlowVisor

Add a Slicing Layer Between Planes

DataPlane

Rules Excepts

Slice 1Controller

Slice 2Controller

Control/DataProtocol

SlicePolicies

Slice 3Controller

Network Slicing Architecture

• A network slice is a collection of sliced switches/routers

• Data plane is unmodified

– Packets forwarded with no performance penalty

– Slicing with existing ASIC

• Transparent slicing layer

– each slice believes it owns the data path

– enforces isolation between slices

• i.e., rewrites, drops rules to adhere to slice police

– forwards exceptions to correct slice(s)

Slicing Policies

• The policy specifies resource limits for each slice:

– Link bandwidth

– Maximum number of forwarding rules

– Topology

– Fraction of switch/router CPU

– FlowSpace: which packets does the slice control?

FlowSpace: Maps Packets to Slices

Real User Traffic: Opt-In

• Allow users to Opt-In to services in real-timeo Users can delegate control of individual flows to Sliceso Add new FlowSpace to each slice's policy

• Example:o "Slice 1 will handle my HTTP traffic"o "Slice 2 will handle my VoIP traffic"o "Slice 3 will handle everything else"

• Creates incentives for building high-quality services

FlowVisor Implemented on OpenFlow

CustomControlPlane

StubControlPlane

DataPlane

OpenFlowProtocol

Switch/Router

Server

Network

Switch/Router

Servers

OpenFlowFirmware

Data Path

OpenFlowController

Switch/RouterSwitch/Router

OpenFlowFirmware

Data Path

OpenFlowController

OpenFlowController

OpenFlowController

FlowVisorOpenFlow

OpenFlow

FlowVisor Message Handling

OpenFlowFirmware

Data Path

AliceController

BobController

CathyController

FlowVisorOpenFlow

OpenFlow

Packet

Exception

Policy Check:Is this rule allowed?

Policy Check:Who controls this packet?

Full Line RateForwarding

Rule

Packet

OpenFlow Deployments

OpenFlow has been prototyped on….

• Ethernet switches– HP, Cisco, NEC, Quanta, + more underway

• IP routers– Cisco, Juniper, NEC

• Switching chips– Broadcom, Marvell

• Transport switches– Ciena, Fujitsu

• WiFi APs and WiMAX Basestations

Most (all?) hardware switches now based on Open vSwitch…

Most (all?) hardware switches now based on Open vSwitch…

Deployment: Stanford

• Our real, production networko 15 switches, 35 APso 25+ userso 1+ year of useo my personal email and

web-traffic!

• Same physical network hosts Stanford demoso 7 different demos

Demo Infrastructure with Slicing

Deployments: GENI

(Public) Industry Interest

• Google has been a main proponent of new OpenFlow 1.1 WAN features– ECMP, MPLS-label matching

– MPLS LDP-OpenFlow speaking router: NANOG50

• NEC has announced commercial products– Initially for datacenters, talking to providers

• Ericsson– “MPLS Openflow and the Split Router Architecture: A Research

Approach“ at MPLS2010

OpenFlow in the WAN

OPEX: 60-70%

CAPEX: 30-40%

… and yet service providers own & operate 2 such networks : IP and Transport

DDC

DDC

DDC

DDC

IP/MPLSIP/MPLS IP/MPLS

IP/MPLS

IP/MPLSIP/MPLS

IP/MPLSIP/MPLS

CDD

CDD

CDD

DD

DD

DD

DD

C C

DD

DD

GMPLS

Motivation

• managed and operated independently

• resulting in duplication of functions and resources in multiple layers

• and significant capex and opex burdens

… well known

IP & Transport Networks are separate

DDC

DDC

DDC

DDC

IP/MPLSIP/MPLS IP/MPLS

IP/MPLS

IP/MPLSIP/MPLS

IP/MPLSIP/MPLS

CDD

CDD

CDD

DD

DD

DD

DD

C C

DD

DD

GMPLS

Motivation

• IP links are static

• and supported by static circuits or lambdas in the Transport network

IP & Transport Networks do not interact

What does it mean for the IP network?IP backbone network design

– Router connections hardwired by lambdas– 4X to 10X over-provisioned

• Peak traffic• protection

IP

DWDM

Big Problem

- More over-provisioned links

- Bigger Routers

How is this scalable??*April, 02

Bigger Routers?Dependence on large Backbone Routers• Expensive• Power Hungry

Juniper TX8/T640

TX8

Cisco CRS-1

How is this scalable??

Functionality Issues!Dependence on large Backbone Routers• Complex & Unreliable

Network World05/16/2007

Dependence on packet-switching• Traffic-mix tipping heavily towards video

• Questionable if per-hop packet-by-packet processing is a good idea

Dependence on over-provisioned links

• Over-provisioning masks packet switching simply not very good at providing bandwidth, delay, jitter and loss guarantees

How can Optics help?

• Optical Switches– 10X more capacity per unit volume (Gb/s/m3)– 10X less power consumption– 10X less cost per unit capacity (Gb/s)– Five 9’s availability

• Dynamic Circuit Switching– Recover faster from failures– Guaranteed bandwidth & Bandwidth-on-demand– Good for video flows– Guaranteed low latency & jitter-free paths– Help meet SLAs – lower need for over-provisioned IP links

DDC

DDC

DDC

DDC

IP/MPLSIP/MPLS IP/MPLS

IP/MPLS

IP/MPLSIP/MPLS

IP/MPLSIP/MPLS

CDD

CDD

CDD

DD

DD

DD

DD

C C

DD

DD

GMPLS

Motivation

• IP links are static

• and supported by static circuits or lambdas in the Transport network

IP & Transport Networks do not interact

What does it mean for the Transport network?

IP

DWDM

*April, 02

Without interaction with a higher layer• there is really no need to support dynamic services• and thus no need for an automated control plane• and so the Transport network remains manually controlled via NMS/EMS• and circuits to support a service take days to provision

Without visibility into higher layer services• the Transport network reduces to a bandwidth-seller

The Internet can help…• wide variety of services• different requirements that can take advantage of dynamic circuit characteristics

What is needed

… Converged Packet and Circuit Networks

• manage and operate commonly

• benefit from both packet and circuit switches

• benefit from dynamic interaction between packet

switching and dynamic-circuit-switching

… Requires• a common way to control

• a common way to use

But

… Convergence is hard

… mainly because the two networks have very different architecture which makes integrated operation hard

… and previous attempts at convergence have assumed that the networks remain the same

… making what goes across them bloated and complicated and ultimately un-usable

We believe true convergence will come about from architectural change!

FlowNetwork

DDC

DDC

DDC

DDC

IP/MPLSIP/MPLS IP/MPLS

IP/MPLS

IP/MPLSIP/MPLS

IP/MPLSIP/MPLS

CDD

CDD

CDD

DD

DD

DD

DD

C C

DD

DD

GMPLS

UCP

FlowNetwork

… that switch at different granularities: packet, time-slot, lambda & fiber

Simple,network of Flow Switches

Research Goal: Packet and Circuit Flows Commonly Controlled & Managed

pac.c

52

… a common way to control

Exploit the cross-connect table in circuit switches

Packet FlowsSwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport

Action

52

Circuit Flows

Signal Type

VCG52 Signal Type

VCG

The Flow Abstraction presents a unifying abstraction

… blurring distinction between underlying packet and circuit and regarding both as flows in a flow-switched network

OpenFlow Protocol

PacketSwitch

CircuitSwitch

Packet & Circuit Switch

NETWORK OPERATING SYSTEM

Variable Bandwidth

Packet Links

Variable Bandwidth

Packet Links

DynamicOptical Bypass

DynamicOptical Bypass

Unified RecoveryUnified

Recovery

UnifiedControl Plane

Unifying Abstraction

Networking Applications

Packet & Circuit Switch

PacketSwitch

VIRTUALIZATION (SLICING) PLANE

Underlying Data Plane Switching

Traffic Engineering

Traffic Engineering

Application-Aware QoSApplication-Aware QoS

… a common way to useUnified Architecture

Congestion Control

Congestion Control

Example Application

..via Variable Bandwidth Packet Links

OpenFlow Demo at SC09

Video Clients Video Server

192.168.3.12192.168.3.10λ1 1553.3 nm

λ2 1554.1 nm

192.168.3.15

OpenFlowController

OpenFlow Protocol

GE to DWDM SFP convertor

GE

O-E

NF2

GE

E-O

NetFPGA based OpenFlow packet switch NF1

25 km SMF

to OSA

to OSA

AWG

WSS based OpenFlow circuit switch

1X9 Wavelength Selective Switch (WSS)

Lab Demo with Wavelength Switches

Openflow Circuit Switch

25 km SMF

OpenFlow packet switch OpenFlow packet switch

GE-Optical

GE-Optical

Mux/Demux

Lab Demo with Wavelength Switches

OpenFlow Enabled Converged Packet and Circuit Switched Network

Stanford University and Ciena Corporation

Demonstrate a converged network, where OpenFlow is used to control both packet and circuit switches.

Dynamically define flow granularity to aggregate traffic moving towards the network core.

Provide differential treatment to different types of aggregated packet flows in the circuit network:

VoIP : Routed over minimum delay dynamic-circuit path Video: Variable-bandwidth, jitter free path bypassing

intermediate packet switches HTTP: Best-effort over static-circuits

Many more new capabilities become possible in a converged network

SANFRANCISCO

HOUSTON

NEW YORK

Controller

OpenFlow Protocol

OpenFlow Enabled Converged Packet and Circuit Switched Network

Demo Video

Issues with GMPLS

• GMPLS original goal: UCP across packet & circuit (2000)

• Today – the idea is dead

•Packet vendors and ISPs are not interested

• Transport n/w SPs view it as a signaling tool available to the mgmt system for

provisioning private lines (not related to the Internet)

• After 10 yrs of development, next-to-zero significant deployment as UCP

• GMPLS Issues

Issues are when considered as a unified architecture and control plane

• control plane complexity escalates when unifying across packets and circuits because it

• makes basic assumption that the packet network remains same: IP/MPLS network – many years of legacy L2/3 baggage• and that the transport network remain same - multiple layers and multiple vendor domains

• use of fragile distributed routing and signaling protocols with many extensions, increasing switch cost & complexity, while decreasing robustness

• does not take into account the conservative nature of network operation

• can IP networks really handle dynamic links? • Do transport network service providers really want to give up control to an automated control plane?

• does not provide easy path to control plane virtualization

Issues with GMPLS

Conclusions

• Current networks are complicated• OpenFlow is an API

– Interesting apps include network slicing

• Nation-wide academic trials underway• OpenFlow has potential for Service Providers

– Custom control for Traffic Engineering– Combined Packet/Circuit switched networks

• Thank you!

Conclusions

• Current networks are complicated• OpenFlow is an API

– Interesting apps include network slicing

• Nation-wide academic trials underway• OpenFlow has potential for Service Providers

– Custom control for Traffic Engineering– Combined Packet/Circuit switched networks

• Thank you!

Backup

• It is well known that Transport Service Providers dislike giving up manual control of their networks

• to an automated control plane• no matter how intelligent that control plane may be• how to convince them?

• It is also well known that converged operation of packet & circuit networks is a good idea

• for those that own both types of networks – eg AT&T, Verizon• BUT what about those who own only packet networks –eg Google

• they do not wish to buy circuit switches• how to convince them?

• We believe the answer to both lies in virtualization (or slicing)

Practical Considerations

OpenFlow Protocol

C

C C

FLOWVISOR

OpenFlow Protocol

CK

CK

CK

PP

CK

P

CKP

Basic Idea: Unified Virtualization

OpenFlow Protocol

C C C

FLOWVISOR

OpenFlow Protocol

CK

CK

CK

PP

CK

P

CKP

ISP ‘A’ Client Controller

Private Line Client Controller

ISP ‘B’ Client Controller

Under Transport Service Provider (TSP) control

IsolatedClient

Network Slices

SinglePhysical

Infrastructureof Packet &

Circuit Switches

Deployment Scenario: Different SPs

Demo Topology

PKTPKTETH

ETH

ETH

ETH

PKTPKTETH

ETH

ETH

ETH

ISP# 1’s NetOSISP# 1’s NetOS

AppApp AppApp AppApp

PKTPKTETH

ETH

ETH

ETHP

KT

ETH

ETH

SONET

SONET

TDM

PKTPKTETH

ETH

ETH

ETHPKTPKT

ETH

ETH

ETH

ETH

PKT

ETH

ETH

SONET

SONET

TDM

ISP# 2’s NetOSISP# 2’s NetOS

AppApp AppApp AppApp

PKTPKTETH

ETH

ETH

ETH

PKT

ETH ETH

SONET SONET

TDM

Transport Service Provider’s (TSP) virtualized networkInternet Service Provider’s

(ISP# 1) OF enabled networkwith slice of TSP’s network Internet Service Provider’s (ISP# 2)

OF enabled network with another slice of TSP’s networkTSP’s private line customer

Demo Methodology

We will show:1.TSP can virtualize its network with the FlowVisor while maintaining operator control via NMS/EMS.

a) The FlowVisor will manage slices of the TSP’s network for ISP customers, where { slice = bandwidth + control of part of TSP’s switches }

b) NMS/EMS can be used to manually provision circuits for Private Line customers

2.Importantly, every customer (ISP# 1, ISP# 2, Pline) is isolated from other customer’s slices.

1. ISP#1 is free to do whatever it wishes within its slicea) eg. use an automated control plane (like OpenFlow)b) bring up and tear-down links as dynamically as it wants

2. ISP#2 is free to do the same within its slice3. Neither can control anything outside its slice, nor interfere with other slices4. TSP can still use NMS/EMS for the rest of its network

ISP #1’s Business Model

ISP# 1 pays for a slice = { bandwidth + TSP switching resources }

1. Part of the bandwidth is for static links between its edge packet switches (like ISPs do today)

2. and some of it is for redirecting bandwidth between the edge switches (unlike current practice)

3. The sum of both static bandwidth and redirected bandwidth is paid for up-front.

4. The TSP switching resources in the slice are needed by the ISP to enable the redirect capability.

ISP# 1’s network

PKTPKTETH

ETH

ETH

ETHP

KT

ETH

ETH

SONET

SONET

TDM

PKTPKTETH

ETH

ETH

ETH

PKT

ETH

ETH

SONET

SONET

TDM

PKTPKTETH

ETH

ETH

ETH

PKT

ETH ETH

SONET SONET

TDM

PKTPKTETH

ETH

ETH

ETH

PKTPKTETH

ETH

ETH

ETH

PKTPKTETH

ETH

ETH

ETH

Packet (virtual) topology

Actual topology

Notice the spare interfaces

..and spare bandwidth in the slice

ISP# 1’s network

PKTPKTETH

ETH

ETH

ETHP

KT

ETH

ETH

SONET

SONET

TDM

PKTPKTETH

ETH

ETH

ETH

PKT

ETH

ETH

SONET

SONET

TDM

PKTPKTETH

ETH

ETH

ETH

PKT

ETH ETH

SONET SONET

TDM

PKTPKTETH

ETH

ETH

ETH

PKTPKTETH

ETH

ETH

ETH

PKTPKTETH

ETH

ETH

ETH

Packet (virtual) topology

Actual topology

ISP# 1 redirects bw between the spare interfaces to dynamically create new links!!

ISP #1’s Business Model Rationale

Q. Why have spare interfaces on the edge switches? Why not use them all the time?

A. Spare interfaces on the edge switches cost less than bandwidth in the core1. sharing expensive core bandwidth between cheaper edge

ports is more cost-effective for the ISP2. gives the ISP flexibility in using dynamic circuits to create

new packet links where needed, when needed3. The comparison is between (in the simple network shown)

a) 3 static links + 1 dynamic link = 3 ports/edge switch + static & dynamic core bandwidth

b) vs. 6 static links = 4 ports/edge switch + static core bandwidthc) as the number of edge switches increase, the gap increases

ISP #2’s Business Model

ISP# 2 pays for a slice = { bandwidth + TSP switching resources }

1. Only the bandwidth for static links between its edge packet switches is paid for up-front.

2. Extra bandwidth is paid for on a pay-per-use basis

3. TSP switching resources are required to provision/tear-down extra bandwidth

4. Extra bandwidth is not guaranteed

ISP# 2’s network

Packet (virtual) topology

Actual topology

PKTPKTETH

ETH

ETH

ETH

PKTPKTETH

ETH

ETH

ETH

PKT

ETH

ETH

SONET

SONET

TDM

PKTPKTETH

ETH

ETH

ETH

PKT

ETH

ETH

SONET

SONET

TDM

PKT

ETH ETH

SONET SONET

TDM

PKTPKTETH

ETH

ETH

ETH PKTPKT

ETH

ETH

ETH

ETH

PKTPKTETH

ETH

ETH

ETH

ISP# 2 uses variable bandwidth packet links ( our SC09 demo )!!

Only static link bw paid for up-front

ISP #2’s Business Model Rationale

Q. Why use variable bandwidth packet links? In other words why have more bandwidth at the edge (say 10G) and pay for less bandwidth in the core up-front (say 1G)

A. Again it is for cost-efficiency reasons. 1. ISP’s today would pay for the 10G in the core up-front

and then run their links at 10% utilization.2. Instead they could pay for say 2.5G or 5G in the core,

and ramp up when they need to or scale back when they don’t – pay per use.