short tutorial @ ecoop 2015

From dumb to smarter switches in software defined networks: towards a stateful data plane

Antonio CaponePolitecnico di Milano – ANTLab

A. Capone: ECOOP -‐ NetPL 2015

ECOOP 2015, NetPL, Prague, July 6

BebaBEhavioural BAsed forwarding

joint work with: G. Bianchi, M. Bonola, S. Pontarelli– Univ. Rome Tor VergataC. Cascone, L. Pollini, D. Sanvito– Politecnico di Milano

supported by EU project

www.beba-‐project.eu

Outline1) Switches cannot remain dumb: ➔ Data plane evolution. ¤ How far should we go?

2) Not too much not too little: ➔ OpenState and stateful data planes

3) Applied smartness: stateful applications¤ How to incorporate statefulness in languages?

A. Capone: ECOOP -‐ NetPL 2015 2

Setting the scene: OpenFlow and dataplane abstraction

The future has already arrived. It's just not evenly distributed yet. [William Gibson]


Forwarding HW

OS

Classic network paradigmDistributed network functions

Forwarding HW

OS

Forwarding HW

OS

State distribution mechanism(standard protocols)

Router/switch/applianceA. Capone: ECOOP -‐ NetPL 2015 4

Closed platform!

SDN paradigm

Simple forwarding HW




App App App

Data plane abstraction

Programming interface


(protocol for enforcement and notifications)

N

SW ENetwork OS

Several attempts …Active Networks, IETF ForCES, …


… but one winnerOpenFlow

OpenFlow Protocol

OpenFlow controller

OpenFlow• OpenFlow has been proposed as a clean slate approach for new generation network


• … bit it is actually a pragmatic approach to SDN based on a simple HW abstraction that can be implemented with current HW commercial platforms

HW vs SW switches• Soft switching based on general CPU• Switching based on network processors• Switching based on dedicated chips


10x

10x

8

Data plane abstraction: Flow tableMatch Actions Counters

1. Forward (one or more ports)2. Drop3. Encapsulate and send to controller4. Header rewrite5. Push/pop MPLS label / VLAN tag6. Queues + bitrate limiter (bit/s)7. Etc..

Bytes + packets

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

L4sport

L4dport

VLANpcp

IPToS

Slide courtesy: Rob Sherwood


Dataplane Abstraction


OF1.0: Single Match Table (SMT)

TCAMTernary Content Addressable Memory

10

Flow table

Redirect to controller

Packet

Apply actions, forward

Drop

OpenFlow Switch Specification Version 1.1.0 Implemented

Table0

Table1

Tablen

Packet Execute Action

Set

PacketIn

ActionSetAction

Set = {}

OpenFlow Switch

PacketOut...

Ingressport

Packet +ingress port +

metadata

ActionSet

(a) Packets are matched against multiple tables in the pipeline

Match fields:Ingress port +metadata +

pkt hdrs

Action set

Flow Table

➀ Find highest-priority matching flow entry

➁ Apply instructions: i. Modify packet & update match fields (apply actions instruction) ii. Update action set (clear actions and/or write actions instructions) iii. Update metadata

➂ Send match data and action set to next table

➀

➁

➂Action set

Match fields:Ingress port +metadata +

pkt hdrs

(b) Per-table packet processing

Figure 2: Packet flow through the processing pipeline

The flow tables of an OpenFlow switch are sequentially numbered, starting at 0. Pipeline processingalways starts at the first flow table: the packet is first matched against entries of flow table 0. Other flowtables may be used depending on the outcome of the match in the first table.

If the packet matches a flow entry in a flow table, the corresponding instruction set is executed (see4.4). The instructions in the flow entry may explicitly direct the packet to another flow table (using theGoto Instruction, see 4.6), where the same process is repeated again. A flow entry can only direct a packetto a flow table number which is greater than its own flow table number, in other words pipeline processingcan only go forward and not backward. Obviously, the flow entries of the last table of the pipeline cannot include the Goto instruction. If the matching flow entry does not direct packets to another flow table,pipeline processing stops at this table. When pipeline processing stops, the packet is processed with itsassociated action set and usually forwarded (see 4.7).

If the packet does not match a flow entry in a flow table, this is a table miss. The behavior on ta-ble miss depends on the table configuration; the default is to send packets to the controller over the controlchannel via a packet-in message (see 5.1.2), another options is to drop the packet. A table can also specifythat on a table miss the packet processing should continue; in this case the packet is processed by the nextsequentially numbered table.

6

OF1.1: Multiple Match Table (MMT)

Switches cannot remain dumb: Data plane evolution

One man alone can be pretty dumb sometimes, but for real bona fide stupidity, there ain't nothin' can beat teamwork. [Edward Abbey]


Models can be perfect and clean, reality is dirty!

• The match/action model can ideally be used to program any network behavior and to get rid of protocol limitations at any level

• But unfortunately, with OF:– Matches can be done only on a set of predefined header fields (Ethernet, IPv4, MPLS, VLAN tag, etc.)

– Actions are limited to a rather small set– Header manipulation (like adding label/tags, rewriting of fields, etc.) is limited to standard schemes

• As a result, OF is not really protocol independent and standards (including OF standards) are still necessary


Breaking the limitation of specialized pipelines


[BOS13] P. Bosshart, G. Gibb, H.-‐S. Kim, G. Varghese, N. McKeown, M. Izzard, F. Mujica, and M. Horowitz, “Forwarding

metamorphosis: Fast programmable match-‐action processing in hardware for SDN”, in ACM SIGCOMM 2013.

Input Channels

Logical Stage 1

...Switch State(metadata)

Select

... ... VLIWAction

Match Tables

Statistics State

Prog. Parser

HeaderPayload

...

Packets

1

K

...

Logical Stage N

Recombine

Output Channels

...

1

K

Configurable Output Queues

Packets

New Header

(a) RMT model as a sequence of logical Match-Action stages.

Physical Stage 1

PhysicalStage 2

Logical Stage 1

Logical Stage 2

Physical Stage M

...

: Ingress logical match tables

: Egress logical match tables

Logical Stage N

(b) Flexible match table configuration.

Packet Header Vector

Action Input Selector (Crossbar)

Action Memory OP code

VLIW Instruction Memory

Ctrl

Action Unit


...

Src 1Src 2Src 3

Src 1Src 2Src 3

OP code(from inst

mem)

Match Results

...

Match Tables

Action Unit

Very Wide

Header Bus

(c) VLIW action architecture.

Figure 1: RMT model architecture.

2. Flexible Resource Allocation Minimizing Resource Waste:A physical pipeline stage has some resources (e.g., CPU,memory). The resources needed for a logical stage can varyconsiderably. For example, a firewall may require all ACLs,a core router may require only prefix matches, and an edgerouter may require some of each. By flexibly allocating phys-ical stages to logical stages, one can reconfigure the pipelineto metamorphose from a firewall to a core router in the field.The number of physical stages N should be large enoughso that a logical stage that uses few resource will waste atmost 1/N -th of the resources. Of course, increasing N willincrease overhead (wiring, power): in our chip design wechose N = 32 as a compromise between reducing resourcewastage and hardware overhead.

3. Layout Optimality: As shown in Figure 1b, a logicalstage can be assigned more memory by assigning the logicalstage to multiple contiguous physical stages. An alternatedesign is to assign each logical stage to a decoupled set ofmemories via a crossbar [4]. While this design is more flexi-ble (any memory bank can be allocated to any stage), worstcase wire delays between a processing stage and memorieswill grow at least as

p

M , which in router chips that requirea large amount of memory M can be large. While these

delays can be ameliorated by pipelining, the ultimate chal-lenge in such a design is wiring: unless the current matchand action widths (1280 bits) are reduced, running so manywires between every stage and every memory may well beimpossible.In sum, the advantage of Figure 1b is that it uses a tiled

architecture with short wires whose resources can be recon-figured with minimal waste. We acknowledge two disadvan-tages. First, having a larger number of physical stages seemsto inflate power requirements. Second, this implementationarchitecture conflates processing and memory allocation. Alogical stage that wants more processing must be allocatedtwo physical stages, but then it gets twice as much memoryeven though it may not need it. In practice, neither issueis significant. Our chip design shows that the power usedby the stage processors is at most 10% of the overall powerusage. Second, in networking most use cases are dominatedby memory use, not processing.

2.2 Restrictions for RealizabilityThe physical pipeline stage architecture needs restrictions

to allow terabit-speed realization:

Programmable parser

Global states

Header re-‐write with very long instruction word (VLIW) Configurable

queues

Input Channels

Logical Stage 1

...

Switch State(metadata)

Select

... ... VLIWAction

Match Tables

Statistics State

Prog. Parser

HeaderPayload

...

Packets

1

K

...

Logical Stage N

Recombine

Output Channels

...

1

K

Configurable Output Queues

Packets

New Header

(a) RMT model as a sequence of logical Match-Action stages.

Physical Stage 1

PhysicalStage 2

Logical Stage 1

Logical Stage 2

Physical Stage M

...

: Ingress logical match tables

: Egress logical match tables

Logical Stage N

(b) Flexible match table configuration.


Action Input Selector (Crossbar)

Action Memory OP code

VLIW Instruction Memory

Ctrl

Action Unit


...

Src 1Src 2Src 3

Src 1Src 2Src 3

OP code(from inst

mem)

Match Results

...

Match Tables

Action Unit

Very Wide

Header Bus

(c) VLIW action architecture.

Figure 1: RMT model architecture.

2. Flexible Resource Allocation Minimizing Resource Waste:A physical pipeline stage has some resources (e.g., CPU,memory). The resources needed for a logical stage can varyconsiderably. For example, a firewall may require all ACLs,a core router may require only prefix matches, and an edgerouter may require some of each. By flexibly allocating phys-ical stages to logical stages, one can reconfigure the pipelineto metamorphose from a firewall to a core router in the field.The number of physical stages N should be large enoughso that a logical stage that uses few resource will waste atmost 1/N -th of the resources. Of course, increasing N willincrease overhead (wiring, power): in our chip design wechose N = 32 as a compromise between reducing resourcewastage and hardware overhead.

3. Layout Optimality: As shown in Figure 1b, a logicalstage can be assigned more memory by assigning the logicalstage to multiple contiguous physical stages. An alternatedesign is to assign each logical stage to a decoupled set ofmemories via a crossbar [4]. While this design is more flexi-ble (any memory bank can be allocated to any stage), worstcase wire delays between a processing stage and memorieswill grow at least as

p

M , which in router chips that requirea large amount of memory M can be large. While these

delays can be ameliorated by pipelining, the ultimate chal-lenge in such a design is wiring: unless the current matchand action widths (1280 bits) are reduced, running so manywires between every stage and every memory may well beimpossible.In sum, the advantage of Figure 1b is that it uses a tiled

architecture with short wires whose resources can be recon-figured with minimal waste. We acknowledge two disadvan-tages. First, having a larger number of physical stages seemsto inflate power requirements. Second, this implementationarchitecture conflates processing and memory allocation. Alogical stage that wants more processing must be allocatedtwo physical stages, but then it gets twice as much memoryeven though it may not need it. In practice, neither issueis significant. Our chip design shows that the power usedby the stage processors is at most 10% of the overall powerusage. Second, in networking most use cases are dominatedby memory use, not processing.

2.2 Restrictions for RealizabilityThe physical pipeline stage architecture needs restrictions

to allow terabit-speed realization:

13

RMT -‐ Reconfigurable Match Table

other examples: FlexPipe™ (Intel), …

Target programmability: P4


[BOS14] P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A.Vahdat, G. Varghese, and D. Walker “P4: Programming protocol-‐independent packet processors,” SIGCOMM CCR, July 2014

14

• OpenFlow 2.0 proposal by McKeown, Rexford et al. [BOS14]• 3 goals: Protocol independence, Target independence,

Reconfigurability

general interface between the controller and the switches.

That is, we believe that future generations of OpenFlow

should allow the controller to tell the switch how to operate,

rather than be constrained by a fixed switch design. The key

challenge is to find a “sweet spot” that balances the need

for expressiveness with the ease of implementation across a

wide range of hardware and software switches. In designing

P4, we have three main goals:

• Reconfigurability. The controller should be able to re-

define the packet parsing and processing in the field.

• Protocol independence. The switch should not be tied

to specific packet formats. Instead, the controller should

be able to specify (i) a packet parser for extracting header

fields with particular names and types and (ii) a collection

of typed match+action tables that process these headers.

• Target independence. Just as a C programmer does

not need to know the specifics of the underlying CPU, the

controller programmer should not need to know the de-

tails of the underlying switch. Instead, a compiler should

take the switch’s capabilities into account when turning

a target-independent description (written in P4) into a

target-dependent program (used to configure the switch).

The outline of the paper is as follows. We begin by in-

troducing an abstract switch forwarding model. Next, we

explain the need for a new language to describe protocol-

independent packet processing. We then present a simple

motivating example where a network operator wants to sup-

port a new packet-header field and process packets in mul-

tiple stages. We use this to explore how the P4 program

specifies headers, the packet parser, the multiple match+

action tables, and the control flow through these tables. Fi-

nally, we discuss how a compiler can map P4 programs to

target switches.

Related work. In 2011, Yadav et al. [4] proposed an ab-

stract forwarding model for OpenFlow, but with less empha-

sis on a compiler. Kangaroo [1] introduced the notion of pro-

grammable parsing. Recently, Song [5] proposed protocol-

oblivious forwarding which shares our goal of protocol in-

dependence, but is targeted more towards network proces-

sors. The ONF introduced table typing patterns to express

the matching capabilities of switches [6]. Recent work on

NOSIX [7] shares our goal of flexible specification of match+

action tables, but does not consider protocol-independence

or propose a language for specifying the parser, tables, and

control flow. Other recent work proposes a programmatic in-

terface to the data plane for monitoring, congestion control,

and queue management [8, 9]. The Click modular router [10]

supports flexible packet processing in software, but does not

map programs to a variety of target hardware switches.

2. ABSTRACT FORWARDING MODELIn our abstract model (Fig. 2), switches forward packets

via a programmable parser followed by multiple stages of

match+action, arranged in series, parallel, or a combination

of both. Derived from OpenFlow, our model makes three

generalizations. First, OpenFlow assumes a fixed parser,

whereas our model supports a programmable parser to allow

new headers to be defined. Second, OpenFlow assumes the

match+action stages are in series, whereas in our model they

can be in parallel or in series. Third, our model assumes that

actions are composed from protocol-independent primitives

supported by the switch.

Our abstract model generalizes how packets are processed

in di↵erent forwarding devices (e.g., Ethernet switches, load-

balancers, routers) and by di↵erent technologies (e.g., fixed-

function switch ASICs, NPUs, reconfigurable switches, soft-

ware switches, FPGAs). This allows us to devise a com-

mon language (P4) to represent how packets are processed

in terms of our common abstract model. Hence, program-

mers can create target-independent programs that a com-

piler can map to a variety of di↵erent forwarding devices,

ranging from relatively slow software switches to the fastest

ASIC-based switches.

Figure 2: The abstract forwarding model.

The forwarding model is controlled by two types of oper-

ations: Configure and Populate. Configure operations pro-

gram the parser, set the order of match+action stages, and

specify the header fields processed by each stage. Config-

uration determines which protocols are supported and how

the switch may process packets. Populate operations add

(and remove) entries to the match+action tables that were

specified during configuration. Population determines the

policy applied to packets at any given time.

For the purposes of this paper, we assume that configura-

tion and population are two distinct phases. In particular,

the switch need not process packets during configuration.

However, we expect implementations will allow packet pro-

cessing during partial or full reconfiguration enabling up-

grades with no downtime. Our model deliberately allows

for, and encourages, reconfiguration that does not interrupt

forwarding.

Clearly, the configuration phase has little meaning in fixed-

function ASIC switches; for this type of switch, the com-

ACM SIGCOMM Computer Communication Review 89 Volume 44, Number 3, July 2014

other example: POF (Huawei)

Not too much not too little: OpenState and stateful data planes

Too clever is dumb.[Ogden Nash]


Looking for the “right” abstraction• Programmability and real world viability– High levels of (deep) programmability in the data and control planes since ages• Active Networks• IETF ForCES

• Keywords for success:– Pragmatism– Compromise– “Right” mix of programmability: right level of abstraction• Many wonderful programmable platforms buried in the “lab world”



Remember: OF meant to be a compromise[original quotes: from OF 2008 paper]

• Best approach: “persuade commercial name-‐brand equipment vendors to provide an open, programmable, virtualized platform on their switches and routers”– Plainly speaking: open the box!! No way…

• Viable approach: “compromise on generality and seek a degree of switch flexibility that is– High performance and low cost– Capable of supporting a broad range of research– Consistent with vendors’ need for closed platforms.

17

OF forces separation of data and control

Logically-centralized control

DUMB!

SMART!

Events from switchesTopology changes,Traffic statistics,Arriving packets

Commands to switches(Un)install rules,Query statistics,Send packets


Centralized Control: pros and cons• PROS:– Central view of the network as a whole

• Network states, etc– One-‐click network config/deploy

• Platform agnostic switch API is key -‐ e.g. OpenFlow forwarding abstraction

• CONS:– Control latency!!!

• O(second)1s = 300M packets lost @ 100 gbps

– Signalling overhead

Great idea for network-‐wide statesand «big picture»

decisions

Poor idea for localstates/decision, (way!) betterhandled locally

(less delay, less load)



SMART!

SMART!

Events from switches & central rule updatesRestricted to those of interest for GLOBAL decisions

Inject “switch control programs”Change forwarding behavior as specified by “this program” IF(local) events occur

Decision on how network should operateremains at the controller (SDN vision)But “execution” of forwarding plane updates can be locally delegated

Local processing: Ultra low Latency: o(nanosec) versus o(sec)

Local states: lower signalling

Our vision

20

Stateless vs. Stateful data plane


SwitchStateless

ControllerGlobal + local states

SwitchLocal states

ControllerGlobal states

Controlenforcing

Controldelegation

Stateless model(e.g. OpenFlow)

Stateful model

SMART!

DUMB! SMART!

SMART!

Auto-‐adaption

Eventnotifications

21

Eventnotifications

Easier said than done• We need a switch architecture and API which is…

– High performance: state management tasks executed at wire-‐speed (packet-‐based events)

– Platform-‐independent: consistent with vendors’ needs for closed platforms

– Low cost and immediately viable: based on commodity HW

Apparently, far beyond OpenFlow switches…

Our (perhaps surprising?) finding: much closer to OF abstraction than expected!!A. Capone: ECOOP -‐ NetPL 2015 22

Our approach: OpenState


[CCR14] G. Bianchi, M. Bonola, A. Capone, C. Cascone, “OpenState: programming platform-‐independent stateful OpenFlow applications inside the switch”, ACM Computer Communication Review, vol. 44, no. 2, April 2014.[HPSR15] S. Pontarelli, M. Bonola, G. Bianchi, A. Capone, C. Cascone, “Stateful Openflow: Hardware Proof of Concept”, IEEE HPSR 2015, Budapest, July 1-‐4, 2015.[DRCN15] A. Capone, C. Cascone, A.Q.T. Nguyen, B. Sansò, "Detour Planning for Fast and Reliable Failure Recovery in SDN with OpenState", IEEE DRCN 2015, Kansas City, USA, March 24-‐27, 2015 – BEST PAPER RUNNER UP.[ARX14] G. Bianchi, M. Bonola, A. Capone, C. Cascone, S. Pontarelli, “Towards Wire-‐speed Platform-‐agnostic Control of OpenFlow Switches”, available on ArXiv, 2014.

23

Example: Port Knocking firewallknock «code»: 5123, 6234, 7345, 8456

IPsrc=1.2.3.4 Port=5123 Drop(); 1.2.3.4 à 1° knock OK



IPsrc=1.2.3.4 Port=8456 Drop(); 1.2.3.4 à OPEN port SSH

IPsrc=1.2.3.4 Port=22 Forward()


Port Knocking @ controllerIPsrc=any Port=any

DROP

controller

Encapsulate & forward ALL packets of ALL flows

IPsrc=any Port=any

Maintain Knock state per flow

When knock sequence finalized, add entry <Ipsrc, port=22; forward>

Lots of overhead!!


Can we move the state machine in the switch?

Can transform in a flow table? Yes: MATCH: <state, port> à ACTION: <drop/forward, state_transition>

Plus a state lookup/update

state event

DEFAULT

STAGE-‐1

Port=5123

Port=6234

STAGE-‐2

STAGE-‐3

Port=7345

Port=8456

OPEN Port=22

OPEN

*

Port=*

Port=*

Match fields Actions

action Next-‐state

drop

drop

STAGE-‐1

STAGE-‐2

drop

drop

STAGE-‐3

OPEN

forward OPEN

drop

drop

OPEN

DEFAULT

IPsrc PortMetadata:State-‐label

State DB

State DB

IpsrcàOPEN

Ipsrc: ??


Putting all together

Flow key stateIPsrc= … …Ipsrc= … …

… … …… … …

IPsrc=1.2.3.4IPsrc=5.6.7.8

STAGE-3OPEN

IPsrc= no match DEFAULTIPsrc= … …

State Table

… … …

IPsrc=1.2.3.4 Port=8456

1) State lookup

state eventDEFAULTSTAGE-1

Port=5123Port=6234

STAGE-2STAGE-3

Port=7345Port=8456

OPEN Port=22OPEN

*Port=*Port=*

XFSM TableMatch fields Actions

action Next-statedropdrop

STAGE-1STAGE-2

dropdrop

STAGE-3OPEN

forward OPENdropdrop

OPENDEFAULT

IPsrc=1.2.3.4 Port=8456STAGE-32) XFSM state transition

IPsrc=1.2.3.4 Port=8456

OPEN

3) State update

write

Write: OPEN

1 «program» XFSM table for all flows(same knocking sequence)

N states, one per (active) flowA. Capone: ECOOP -‐ NetPL 2015 27

Cross-‐flow state handling

MACdst MACsrcFlow key state

48 bit MAC addr Port #lookup

State Table

MACdst MACsrcFlow key state

48 bit MAC addr Port #update

State Table

state eventPort# *

action Next-stateforward In-port

XFSM Table

DIFFERENT lookup/update scope

Field 1 Field 2 Field N

Flowkey selector Read/write signal

• Yes but what about MAC learning, multi-‐port protocols (e.g., FTP), bidirectional flow handling, etc?


Wrapping upProof of concept• SW implementation:

– Public domain: openstate-‐sdn.org

• HW implementation:–


S. Pontarelli, M. Bonola, G. Bianchi, A. Capone, C. Cascone, “Stateful Openflow: Hardware Proof of Concept”, IEEE HPSR 2015, Budapest, July 1-‐4, 2015.

Switch: ofsoftswitch13; Controller: Ryu

State Machines• Events: matches, times, all-‐flow states, meters, …

• Machine instantiations: One table/machine per set of flows, parallel machines for different set of flows, multiple stateful stages

Applied smartness: How to incorporate statefulness in languages?

There are science and the applications of science, bound together as the fruit of the tree which bears it. [Louis Pasteur]


Stateful applicationsDONE:• Port Knocking• MAC learning• Label/address advertisement

learning• Reverse Path Forwarding• Flow-‐consistent Load Balancing• Failure recovery• DDoS multi-‐stage flow marking• …

Custom implementation in the controller using the OpenState extension of OF1.3

CHALLENGE:• IDS/DPI• TCP flow processing• Monitoring • …

Need new «actions»Need extra logic (full XFSM)


All (!) otherwise not possible without explicit controller’s involvement

31

SDN languages for applications• Example: Frenetic/Pyretic


NetworkPolicy

Compiler

Policy enforcement

OF commands

PolicyDynamicAdaptation

QueriesNetworkStatus

Notifications

PolicyReconf.

Open question• How can we use a stateful dataplane in programming languages for SDN?


• Easy solutions: create a library of switch state machines to be used as functions in the language

• Can we automatically discover local state machines in a network policy and instantiate them (compiler)?

• Is it a parallelization problem of a global state machine with local and global states and events?

• How much of this parallelization problem should be exposed to the programmer?

Thanks

OpenState: openstate-‐sdn.org

EU project BEBA – http://www.beba-‐project.eu

Extended slide-‐set is available on OpenState web site!


Antonio Capone Email: [email protected]

34

short tutorial @ ecoop 2015

Documents