hyperflow: a distributed control plane for openflow

8/10/2019 HyperFlow: A Distributed Control Plane for OpenFlow

1/6

HyperFlow: A Distributed Control Plane for OpenFlow

Amin TootoonchianUniversity of Toronto

[email protected]

Yashar GanjaliUniversity of Toronto

[email protected]

Abstract

OpenFlow assumes a logically centralized controller, whichideally can be physically distributed. However, currentdeployments rely on a single controller which has ma-

jor drawbacks including lack of scalability. We presentHyperFlow, a distributed event-based control plane forOpenFlow. HyperFlow is logically centralized but phys-

ically distributed: it provides scalability while keepingthe benefits of network control centralization. By pas-sively synchronizing network-wide views of OpenFlowcontrollers, HyperFlow localizes decision making to in-dividual controllers, thus minimizing the control planeresponse time to data plane requests. HyperFlow is re-silient to network partitioning and component failures.It also enables interconnecting independently managedOpenFlow networks, an essential feature missing in cur-rent OpenFlow deployments. We have implementedHyperFlow as an application for NOX. Our implemen-tation requires minimal changes to NOX, and allowsreuse of existing NOX applications with minor modi-

fications. Our preliminary evaluation shows that, as-suming sufficient control bandwidth, to bound the win-dow of inconsistency among controllers by a factor ofthe delay between the farthest controllers, the networkchanges must occur at a rate lower than 1000 events persecond across the network.

1. INTRODUCTION

The minimalism and simplicity in the Internets de-sign has led to an enormous growth and innovationatop, yet the network itself remains quite hard to changeand surprisingly fragile and hard to manage. The rootcause of these problems is the overly complicated con-

trol plane running on top of all switches and routersthroughout the network [2]. To alleviate this problem,previous works propose to decouple the control (deci-sion making) and data (packet forwarding) planes, anddelegate the control functionality to a logically central-ized controller [2, 5, 11]. This separation significantlysimplifies modifications to the network control logic (asit is centralized), enables the data and control planes toevolve and scale independently, and notably decreases

the cost of the data plane elements [6]. In particular,OpenFlow [5] has succeeded in attracting commercialvendors [1].

The initial design and implementation of OpenFlowassumed a single controller for the sake of simplicity.However, as the number and size of production net-works deploying OpenFlow increases, relying on a singlecontroller for the entire network might not be feasible

for several reasons. First, the amount of control trafficdestined towards the centralized controller grows withthe number of switches. Second, if the network has alarge diameter, no matter where the controller is placed,some switches will encounter long flow setup latencies.Finally, since the system is bounded by the process-ing power of the controller, flow setup times can growsignificantly as demand grows with the size of the net-work. Figure 1(a) illustrates these issues in a sampleOpenFlow-based network.

In this paper, we present the design and implemen-tation of HyperFlow, a distributed event-based con-trol plane for OpenFlow, which allows network oper-ators deploy any number of controllers in their net-works. HyperFlow provides scalability while keepingnetwork control logically centralized: all the controllersshare the same consistent network-wide view and locallyserve requests without actively contacting any remotenode, thus minimizing the flow setup times. Addition-ally, HyperFlow does not require any changes to theOpenFlow standard [7] and only needs minor modifica-tions to existing control applications. HyperFlow guar-antees loop-free forwarding, and is resilient to networkpartitioning as well as component failures. Besides, itenables addition of administrative areas to OpenFlowto interconnect independently managed OpenFlow ar-eas. Figure 1(b) shows how HyperFlow addresses theproblems associated with a centralized controller in anOpenFlow network.

To the best of our knowledge, HyperFlow is the firstdistributed control plane for OpenFlow. The only simi-lar design we are aware of is FlowVisor [8] which attacksa slightly different problem. FlowVisor enables multiplecontrollers in an OpenFlow network by slicing network


2/6

(a) An OpenFlow network deploying a single controller (b) The same network deploying HyperFlow

Figure 1: A multi-site OpenFlow network with single and multiple controllers. Switch and controller association is depictedusing colors and shadow pattern. (a) Deploying a single controller increases the flow setup time for flows initiated in site 2 and site 3 by50ms. Also, an increase in flow initiation rates in the remote sites may congest the cross-site links. (b) In HyperFlow, all the requestsare served by local controllers, and the cross-site control traffic is minimal: controllers mostly get updated by their neighbors.

resources and delegating the control of each slice to asingle controller.

An alternative design is to keep the controller state ina distributed data store (e.g., a DHT) and enable localcaching on individual controllers. Even though a deci-sion (e.g., flow path setup) can be made for many flowsby just consulting the local cache, inevitably some flowsrequire state retrieval from remote controllers, resultingin a spike in the control plane service time. Addition-ally, this design requires modifications to applicationsto store state in the distributed data store. In contrast,HyperFlow proactively pushes state to other controllers,

thereby enabling individual controllers to locally serveall flows. Also, HyperFlows operation is transparentto the control applications.

We implemented HyperFlow as an application forNOX [3]. The HyperFlow application is in charge ofsynchronizing controllers network-wide views (by prop-agating selected locally generated controller events), redi-recting OpenFlow commands targeted to a non-directly-controlled switch to its respective controller, and redi-recting replies from switches to the request-originatorcontrollers. To facilitate cross-controller communica-tions, we use publish/subscribe messaging paradigm.The HyperFlows design and implementation are dis-

cussed in the next section. Section 3 discusses when andwhy a controller application must be modified to be-come HyperFlow-compatible. To evaluate HyperFlow,Section 4 estimates the maximum level of dynamicity aHyperFlow-based network can have while keeping thewindow of inconsistency among controllers bounded bya factor of delay between farthest controllers in the net-work.

2. DESIGN AND IMPLEMENTATION

A HyperFlow-based network is composed of OpenFlowswitches as forwarding elements, NOX controllers as de-cision elements each running an instance of theHyperFlowcontroller application, and an event propagation sys-tem for cross-controller communication. All the con-trollers have a consistent network-wide view and runas if they are controlling the whole network. They allrun the exact same controller software and set of ap-plications. Each switch is connected to the best con-troller in its proximity. Upon controller failure, affectedswitches must be reconfigured to connect to an active

nearby controller.1 Each controller directly managesthe switches connected to it and indirectly programsor queries the rest (through communication with othercontrollers). Figure 2 illustrates the high-level view ofthe system.

To achieve a consistent network-wide view amongcontrollers, the HyperFlow controller application instancein each controller selectively publishes the events thatchange the state of the system through a publish/subscribesystem. Other controllers replay all the published eventsto reconstruct the state. This design choice is basedon the following observations: (a) Any change to thenetwork-wide view of controllers stems from the occur-

rence of a network event. A single event may affect thestate of several applications, so the control traffic re-quired for direct state synchronization grows with thenumber of applications, but it is bounded to a smallnumber of events in our solution. (b) Only a very small1Currently, in our test environment, we used proprietaryhardware vendor configuration interface to reconfigure thecontroller address.

2


3/6

Figure 2: High-level overview of HyperFlow. Each controller runs NOX with the HyperFlow application atop, subscribes to thecontrol, data, and its own channel in the publish/subscribe system (depicted with a cloud). Events are published to the data channel andperiodic controller advertisements are sent to the control channel. Controllers directly publish the commands targeted to a controller toits channel. Replies to the commands are published in the source controller.

fraction of network events cause changes to the network-wide view (on the order of tens of events per second

for networks of thousands of hosts [3]). The majorityof network events (i.e., packet in events) only requestservice (e.g., routing). (c) The temporal ordering ofevents, except those targeting the same switch, does notaffect the network-wide view. (d)The applications onlyneed to be minimally modified to dynamically identifythe events which affect their state (unlike direct statesynchronization which requires each application to di-rectly implement state synchronization and conflict res-olution).

2.1 Event Propagation

To propagate controller events to others, HyperFlow

uses publish/subscribe messaging paradigm. The pub-lish/subscribe system that HyperFlow uses must pro-vide persistent storage of published events (to provideguaranteed event delivery), keep the ordering of eventspublished by the same controller, and be resilient againstnetwork partitioning (i.e., each partition must continueits operation independently and upon reconnection, par-titions must synchronize). The publish/subscribe sys-tem should also minimize the cross-site2 traffic requiredto propagate events, i.e., controllers in a site should getmost of the updates of other sites from nearby con-trollers to avoid congesting the cross-region links. Fi-nally, the system should enforce access control to ensure

authorized access.We implemented a distributed publish/subscribe sys-

tem satisfying the above requirements using WheelFS [9].WheelFS is a distributed file system designed to offerflexible wide-area storage for distributed applications.2A site is a highly-connected component of the network witha large bisection bandwidth. However, the bandwidth andconnectivity between regions is limited. Typically networkdevices in a single site are geographically co-located.

It gives the applications control over consistency, dura-bility, and data placement according to their require-

ments via semantic cues. These cues can be directlyembedded in the pathnames to change the behaviorof the file system. In WheelFS, we represent chan-nels with directories and messages with files. To imple-ment notification upon message arrival (i.e., new files inthe watched directories) HyperFlow controller applica-tion periodically polls the watched directories to detectchanges.

Each controller subscribes to three channels in thenetwork: the data channel, the control channel, andits own channel. All the controllers in a network aregranted permissions to publish to all channels and sub-scribe to the three channels mentioned. The HyperFlow

application publishes selected local network and appli-cation events which are of general interest to the datachannel. Events and OpenFlow commands targeted toa specific controller are published in the respective con-trollers channel. Additionally, each controller must pe-riodically advertise itself in the control channel to fa-cilitate controller discovery and failure detection. Ac-cess control for these channels are enforced by the pub-lish/subscribe system.

HyperFlow is resilient to network partitioning be-cause WheelFS is. Once a network is partitioned, WheelFSon each partition continues to operate independently.Controllers on each partition no longer receive the ad-

vertisements for the controllers on the other partitionsand assume they have failed. Upon reconnection of par-titions, the WheelFS nodes in both partitions resyn-chronize. Consequently, the controllers get notified ofall the events occurred in the other partition while they

3


4/6

were disconnected, and the network-wide view of all thecontrollers converges.3

Finally, we note that WheelFS can be replaced byany publish/subscribe system satisfying the above men-tioned requirements. We chose WheelFS, primarily be-cause not only it satisfies HyperFlows requirements,but also enables us to rapidly build a prototype. How-ever, as we show in Section 4, there is room for signif-

icant improvements to the existing publish/subscribesystem.

2.2 HyperFlow Controller Application

HyperFlow application is a C++ NOX application wedeveloped to ensure all the controllers have a consistentnetwork-wide view. Each controller runs an instanceof the HyperFlow application. Our implementation re-quires minor changes to the core controller code, mainly,to provide appropriate hooks to intercepts commandsand serialize events. Below, we describe the functionsof the HyperFlow controller application.Initialization:Upon NOX startup, the HyperFlow ap-plication starts the WheelFS client and storage services,subscribes to the networks data and control channels,and starts to periodically advertise itself in the con-trol channel. The advertisement interval must be largerthan the highest round-trip time among controllers ina network. The advertisement message contains infor-mation about the controller including the identifiers ofthe switches it directly controls.Publishing events: The HyperFlow application cap-tures all the NOX built-in events (OpenFlow messageevents) as well as the events that applications registerwith HyperFlow. Then, it selectively serializes (usingthe Boost serialization library) and publishes the oneswhich are locally generated and affect the controllerstate. For that, applications must be instrumented totag the events which affect their state. Furthermore, ap-plications should identify the parent event of any non-built-in event they fire. This way, HyperFlow can traceeach high-level event back to the underlying lower-levelevent and propagate it instead. Using this method weensure that the number of events propagated is boundedby the number of the OpenFlow message events gener-ated by the local controller.

The name of the published messages contains thesource controller identifier and an event identifier lo-

cal to the publisher (see Table 1. This scheme effec-tively partitions the message namespace among con-trollers and avoids the possibility of any write conflicts.Moreover, a cached copy of a message (file) in our sys-tem never becomes stale. Therefore, using semanticcues, we instruct WheelFS to relax consistency require-3We note that this requires the network operator define areplication policy appropriate for the network setup.

Message Type Message Name PatternEvent e : ctrl id : event idCommand c : ctrl id : switch id : event idAdvertisement ctrl id

Table 1: HyperFlows message naming convetion. Allmessage types contain the publisher controller id (ctrlid). Eventsand commands also contain an event identifier (eventid) locallygenerated by the publisher. Commands also contain the identifier

of the switch to which the command is targeted.

ments and fetch cached copies of files from neighboringcontrollers as fast as possible.Replaying events: The HyperFlow application re-plays all the published events, because source controllers with the aid of applications selectively filter out andonly publish the events necessary to reconstruct the ap-plication state on other controllers. Upon receiving anew message on the network data channel or the con-trollers own channel, the HyperFlow application dese-rializes and fires it.Redirecting commands targeted to a non-local

switch: A controller can only program the switchesconnected directly to it. To program a switch not underdirect control of the controller, the HyperFlow applica-tion intercepts when an OpenFlow message is about tobe sent to such switches and publishes the command tothe network control channel. The name of the publishedmessage shows that it is a command and also containsthe source controller identifier, the target switch iden-tifier, and the local command identifier (similar to theevent message identifier).Proxying OpenFlow messages and replies: TheHyperFlow application picks up command messages tar-geted to a switch under its control (identified in the

message name) and sends them to the target switch.To route the replies back to the source controller, theHyperFlow application keeps a mapping between themessage transaction identifiers (xid) and the source con-troller identifiers. The HyperFlow application examinesthe xid OpenFlow message events locally generated bythe controller. If the xid of an event is found in thexid-controller map, the event is stopped from being furtherprocessed and is published to the network data chan-nel. The name of the message contains both controlleridentifiers. The original source controller picks up andreplays the event upon receipt.Health checking: The HyperFlow application listens

for the controller advertisements in the network con-trol channel. If a controller does not re-advertise itselffor three advertisement intervals, it is assumed to havefailed. The HyperFlow application fires a switch leaveevent for every switch that was connected to the failedcontroller. Upon controller failure, HyperFlow config-ures the switches associated with the failed controllerto connect to another controller. Alternatively, either

4


5/6

nearby controllers can serve as a hot standby for eachother to take over the IP address.

3. REQUIREMENTS ON CONTROLLER AP-

PLICATIONS

For most controller applications, HyperFlow only re-quires minor modifications: they must dynamically tag

events which affect their state. However, some of themmust be further modified to ensure correct operationunder temporal event reordering and transiently con-flicting controller views, and guarantee scalability. Be-sides, we discuss how the controller applications mustbe modified to enable interconnection of independently-managed OpenFlow networks.Event reordering: In HyperFlow, correct operationof control applications must not depend on temporalordering of events except those targeting the same en-tity (e.g., the same switch or link), because differentcontrollers perceive events in different orders. Besides,resilience to network partitioning requires control ap-

plications to tolerate out-of-order event delivery (evenlagging several hours) without sacrificing correctness,because each partition is notified of the state of theother partitions upon reconnection.Correctness:Transient inconsistencies among controllersmay lead to conflicting decisions. To ensure correct op-eration in all cases, control applications must forwardrequests to the authoritative controller. The authorita-tive controller for a given flow is the one managing theflows source switch. Consider the switching/routingapplications as an example: To ensure loop-free for-warding, flow paths must be set up by the controllermanaging the flows source switch. Other controllers

must redirect the request to the authoritative controllerin case they receive a flow initiation event. As anotherexample, consider a network with a policy which re-quires both the forward and reverse paths of all flowsto match. To guarantee this, the source controller mustsimultaneously set up both paths upon flow initiation.This modification ensures that the policy is always cor-rectly enforced from the source controllers perspective.Bounded number of possibly effective events: Thenumber of events which possibly affect the state of aHyperFlow-compliant application must be bounded byO(h+ l + s), where h is the number of hosts, l is thenumber of links, and s is the number of switches in

the network. In other words, applications whose statemay be affected by O(f(n)) events, where f(n) is anyfunction of the number of flows in the network, incur aprohibitively large overhead and must be modified.Measurement applications: Applications which ac-tively query the switches perform poorly under HyperFlow,because the number of queries grows linearly with thenumber of controllers. Such applications must be mod-ified to partition queries among controllers in a dis-

tributed fashion and exchange the results (encapsulatedin self-defined events) using HyperFlow. Consider thediscovery application as an example. The discovery ap-plication must only sends link layer discovery protocol(LLDP) probes out of the switches under its direct con-trol. For each link between a directly and a non-directlycontrolled switch pair, the discovery application receivesan LLDP packet generated by another controller signal-

ing the connectivity. Finally, the discovery applicationmust propagate its link events using HyperFlow.Interconnecting HyperFlow-based OpenFlow net-

works: To interconnect two independently managedHyperFlow-based OpenFlow networks (areas), controllerapplications need to be modified to be made area-aware.They must listen for area discovery events from HyperFlow,enforce the area policies declared using a policy lan-guage (e.g., Flow-based Management Language [4]), andexchange updates with the neighboring area througha secure channel providing publish/subscribe service.Applications should encapsulate updates in self-definedevents, and have HyperFlow propagate them to theneighboring areas. HyperFlow removes the need for in-dividual control applications to discover their neighborsand communicate directly; instead, control applications

just fire events locally and HyperFlow delivers them toneighbors. We note that the implementation of thispart is not completed yet.

4. EVALUATION

We performed a preliminary evaluation to estimatethe maximum level of network dynamicity it can sup-port while guaranteeing a bounded inconsistency win-dow among controllers. Throughout our experiments

we used ten servers each equipped with a gigabit NICand running as a WheelFS client and storage node. Inthe near future, we plan to deploy HyperFlow on a largetestbed and characterize its performance, robustness,and scalability with a realistic network topology andtraffic.

Each NOX instance can handle about 30k flow in-stalls per second [10]. However, it can typically processa far larger number of events which do not trigger aninteraction with a switch. Events published throughHyperFlow only affect controller state and should nottrigger any interaction with controllers. Therefore, us-ing HyperFlow, network operators can easily add more

controllers to handle more flow initiation events whilekeeping the flow setup latency minimal. We note that,since in HyperFlow controllers operations do not de-pend on other controllers, they continue to operate evenunder heavy synchronization load. However, as theload increases, the window of inconsistency among con-trollers grows (i.e., the time it takes to have the viewsconverge).

5


6/6

To find the number of events that HyperFlow canhandle while providing a bounded inconsistency windowamong controllers, we benchmarked WheelFS indepen-dently to find the number of 3-KB sized files (sampleserializeddatapath joinevent using the XML archive4)we can write (publish) and read. For that, we instru-mented the HyperFlow application code to measure thetime needed to read and deserialize (with eventual con-

sistency), as well as serialize and write (write locally anddont wait for synchronization with replicas) 1000 suchfiles. We ran each test 10 times and averaged the results.HyperFlow can read and deserialize 987, and serializeand write 233 such events in each second. The limit-ing factor in this case is the number of reads, becausemultiple controllers can publish (write) concurrently.

Based on the above analysis, assuming adequate con-trol bandwidth, HyperFlow can guarantee a boundedwindow of inconsistency among controllers, if the net-work changes trigger less than around 1000 events persecond (i.e., total of 1000 switch and host joins andleaves, and link state changes). We may be able to im-prove HyperFlows performance by modifying WheelFS(whose implementation is not mature yet) or design-ing an alternative publish/subscribe system. Finally,we note that HyperFlow can gracefully handle spikes innetwork synchronization load without losing any events,however in that period the controller views convergewith an added delay.

5. CONCLUSION

This paper presents the design and implementationof HyperFlow which enables OpenFlow deployment inmission-critical networks, including datacenter and en-

terprise networks. HyperFlow enables network opera-tors deploy any number of controllers to tune the per-formance of the control plane based on their needs.Besides, it keeps the network control logic centralizedandlocalizesall decisions to each controller to minimizecontrol plane response time. The HyperFlow applica-tion, implemented atop NOX, synchronizes controllersnetwork-wide views by propagating events affecting thecontroller state. We choose to build up state by re-playing events to minimize the control traffic requiredto synchronize controller state, avoid the possibility ofconflicts in applications state, and minimize the burdenof application modifications. HyperFlow is resilient to

network partitions and component failures, minimizesthe cross-region control traffic, and enables intercon-nection of independently-managed OpenFlow networks.We plan to complete our work on cross-area commu-nication and controller bootstrapping, and perform athorough evaluation of HyperFlow on a large testbed4This is significantly larger than most serialized events.Also, in real deployments we should use binary archiveswhich significantly reduces message sizes.

to further characterize its scalability, performance, androbustness.

6. REFERENCES[1] OpenFlow Consortium.

http://openflowswitch.org/.[2] Greenberg, A., Hjalmtysson, G., Maltz, D. A.,

Myers, A., Rexford, J., Xie, G., Yan, H., Zhan,J., and Zhang, H. A clean slate 4D approach to

network control and management. SIGCOMMComputer Communication Review 35, 5 (2005), 54.

[3] Gude, N., Koponen, T., Pettit, J., Pfaff, B.,Casado, M., McKeown, N., and Shenker, S.NOX: towards an operating system for networks.SIGCOMM Computer Communication Review 38, 3(2008), 105110.

[4] Hinrichs, T. L., Gude, N. S., Casado, M.,Mitchell, J. C., and Shenker, S. Practicaldeclarative network management. In WREN 09:Proceedings of the 1st ACM Workshop on Research onEnterprise Networking(New York, NY, USA, 2009),ACM, pp. 110.

[5] McKeown, N., Anderson, T., Balakrishnan, H.,Parulkar, G., Peterson, L., Rexford, J.,

Shenker, S., and Turner, J. OpenFlow: enablinginnovation in campus networks. SIGCOMM ComputerCommunication Review 38, 2 (2008), 6974.

[6] Naous, J., Erickson, D., Covington, G. A.,Appenzeller, G., and McKeown, N. Implementingan OpenFlow switch on the NetFPGA platform. InANCS (2008), M. A. Franklin, D. K. Panda, andD. Stiliadis, Eds., ACM, pp. 19.

[7] OpenFlow Consortium. OpenFlow switchspecification.http://openflowswitch.org/documents.php.

[8] Sherwood, R., Gibb, G., Yap, K.-K.,Appenzeller, G., Casado, M., McKeown, N., andParulkar, G. FlowVisor: A Network VirtualizationLayer. Tech. Rep. OPENFLOW-TR-2009-01,

OpenFlow Consortium, October 2009.[9] Stribling, J., Sovran, Y., Zhang, I., Pretzer,X., Li, J., Kaashoek, M. F., and Morris, R.Flexible, wide-area storage for distributed systemswith wheelfs. In Proceedings of the 6th USENIXSymposium on Networked Systems Design andImplementation (NSDI 09)(Boston, MA, April 2009).

[10] Tavakoli, A., Casado, M., Koponen, T., andShenker, S. Applying nox to the datacenter. InProceedings of workshop on Hot Topics in Networks(HotNets-VIII)(2009).

[11] Yan, H., Maltz, D. A., Ng, T. S. E., Gogineni,H., Zhang, H., and Cai, Z. Tesseract: A 4d networkcontrol plane. In Proceedings of the 4th USENIXSymposium on Networked Systems Design andImplementation (NSDI 07)(2007).

6

hyperflow: a distributed control plane for openflow

Documents