Inter-controllerTrafficinONOSClustersforSDNNetworks

AbubakarSiddiqueMuqaddas,AndreaBianco,PaoloGiacconeGuidoMaier

13thItalianNetworkingWorkshop:SanCandido,ItalyJanuary13-15,2016 1

SoHwareDefinedNetworking

• highflexibility(vendoragnosPc)• programmability

• centralizedviewofthenetworkstate• simplifydevelopmentofnetworkapplicaPons

Wellknownadvantages

• scalabilityforlargenetworks

Oneofthemanyconcerns

2

Distributedcontrollers

• fault-toleranceandresilience• loadbalancing,thushigherscalabilityforlargenetworks

MoPvaPon

• howthenetworkstateisdistributedacrossthecontrollerstoallowacentralizedlogicalview

CriPcalissue

3

Control-planeindistributedcontrollers

•  mustbein-bandinlargenetworks(e.g.WAN)•  switch-controllertraffic– standardprotocols(e.g.OpenFlow)

4

Inter-controller Traffic in ONOS Clustersfor SDN Networks

Abubakar Siddique Muqaddas†, Andrea Bianco†, Paolo Giaccone†, Guido Maier‡† Dip. di Elettronica e Telecomunicazioni, Politecnico di Torino, Italy

‡Dip. di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano, ItalyE-mails:{abubakar.muqaddas,andrea.bianco,paolo.giaccone}@polito.it, guido.maier@polimi.it

Abstract—In distributed SDN architectures, the network iscontrolled by a cluster of multiple controllers. This distributed ap-proach permits to meet the scalability and reliability requirementsof large operational networks. Despite that, a logical centralizedview of the network state should be guaranteed, enabling thesimple development of network applications. Achieving a consis-tent network state requires a consensus protocol, which generatescontrol traffic among the controllers whose timely delivery iscrucial for network performance.

We focus on the state-of-art ONOS controller, designed toscale to large networks, based on a cluster of self-coordinatingcontrollers, and concentrate on the inter-controller control traffic.Based on real traffic measurements, we develop a model to quan-tify the traffic exchanged among the controllers, which dependson the topology of the controlled network. This model is usefulto design and dimension the control network interconnecting thecontrollers.

I. INTRODUCTION

The standard centralized approach for SDN is based on asingle controller managing all network devices. Even if thissimplifies the network management and the development ofnetwork applications, it poses severe limitations to networkscalability and reliability. Indeed, a single centralized con-troller is a single point of failure. Moreover, a single controllermay not be able to handle a large number of devices, becausethe communication load and the processing overhead for thecontroller increases with device number. Finally, in very largenetworks (as in WANs), devices can be physically very farfrom the controller and, due to the propagation delays, flowmodifications in network devices can experience large latency.

Distributed SDN controllers face with all the above impair-ments. Multiple instances of the controller manage the wholenetwork, which is divided into different domains, each of themunder the control of one controller instance. Distribution of thecontroller functions over multiple physical servers improvesthe robustness of the control plane, by providing backupcontrol resources for each network node. Furthermore, largenetworks can be handled, because the device control is dis-tributed among the controllers and the processing load can bebalanced. Finally, being the control servers also geographicallydistributed across the network area, they can reduce the device-to-controller delay, thus improving the controller reactivity asperceived by the network node.

However, a logical centralized view of the network statehas to be guaranteed also with controller distribution, toease the development of advanced network applications. This

transparent behavior for the network operator comes at thecost of keeping all the shared data structures synchronizedamong the controllers through some consensus protocol. Forexample, the same network topology must be known at eachcontroller to take correct routing decisions. However, sinceeach controller is responsible of a subset of devices, it is ofparamount importance to distribute any topology update in atimely fashion to avoid routing misbehaviors (e.g. loops), ashighlighted in [1].

In large networks, the control plane distributed among thecontrollers is implemented in-band, without the possibility ofrelaying on an out-of-band high-performance network as in thedata center scenario [2]. This poses technical challenges in de-signing the control network, that not only interconnects devicesto controllers, but also supports the communication betweencontrollers. Due to the complexity of the adopted consensusprotocols, the reactivity of the controllers as perceived by thedevices depends also on the bandwidth and delays experiencedin inter-controller communications.

In our work we focus on the control traffic exchangedamong the controllers, which is often neglected in the lit-erature. We consider the state-of-art ONOS controller [3],which is an open-source project developed by ON.Lab [4]and supported by a large community of network operatorsand vendors. Differently from the well-known OpenDaylightproject [5], ONOS has been designed specifically to copewith reliability and scalability issues arising in large ISP/WANnetworks. It natively supports a distributed version of thecontroller, running on a cluster of control servers.

A. Our contributions

We run an experimental testbed mastered by a cluster ofONOS controllers and evaluate the amount of traffic exchangedamong the controllers. We vary the topology (in terms of nodesand links) of the network under control to develop a model,

SDN Controller BSDN Controller A

Eastbound/Westbound APIs

Controller A Domain

Controller B Domain

Fig. 1: Distributed SDN architecture with two controllers

Switch-controller

traffi

c

Inter-controllertraffic

Inter-controllertraffic

•  neededtocoordinatedistributedcontrollers•  ad-hocprotocolsontheeast-westinterfaces•  supportforconsistencyprotocols– shareddatastructures– differentmodelsforconsistency

5

Consistencymodels

•  assumeasharedtable=(key,value)•  strongconsistency–  anyread(key)returnsalwaysthesamevalue

•  eventualconsistency–  anyread(key)returnseventuallythesamevalue

•  aHersometransientPme,thesamevalue•  adoptedmodelaffectsheavily–  themechanismstodistributeandupdatethedata–  thereacPvityoftheSDNcontrollersperceivedbythenetworkdevices

–  thecorrectbehaviorofthenetwork

6

OurcontribuPon

•  deviseanexperimentalmodeltoevaluatetherequiredbandwidthforinter-controllertraffic– usefultodesignanddimensionthetransportnetworkforthecontrolplane

– weneglectswitch-controllertraffic•  complementarytointer-controllertraffic•  canbeindependentlyevaluatedbasedonthenetworkapplicaPon,flowdynamics,andadoptedprotocol(e.g.OpenFlow)

7

DistributedSDNcontrollers

•  focusedmainlyonserviceprovidersandWANs•  faulttoleranceandstatedistribuPonacrosscontrollers•  developedbyOn.Labandsupportedbynetworkoperators(e.g.,AT&T)andbyvendors(Ciena,NEC,Huawei)

ONOS

•  primarilyfocusedondatacentersbutsuitablealsoforWANs•  onecontrollerdesignedtorulealltheothers•  internalabstracPonsstructuredtobecompaPbletoanyfuncPonality

•  supportedbytheLinuxFoundaPonandbymanyITindustries(ADVA,Cisco,Ciena,Corian,etc.)

OpenDaylight

8

Consistencyincontrollers

• eventualconsistency• networktopology• flowrules• flowstaPsPcs

• strongconsistency• switch-controllermapping

• distributedlocks

ONOS

•  strongconsistency•  allshareddatastructures

OpenDaylight

9

ONOSconsistencyalgorithms

•  eventualconsistency•  updatesarelocalinprimarycontrollerandpropagatesinthebackgroundwithgossipalgorithms•  every5sec,eachcontrollerpicksatrandomanothercontroller,comparereplicasandreconciledifferencesbasedonPmestamps

AnP-entropy

•  strongconsistency•  updatesarecentralizedtotheleader•  anupdateiscommidedwheneveramajorityofcontrollersackstotheleader

RAFT

10

Methodology

•  mininettoemulateanytestnetworktopology–  isolated–  linear– star

•  dedicatedLXCcontainerforeachONOSinstance– ONOS1.2Cardinal(rel.May2015)

•  trafficsniffertoevaluatetheinter-controllertraffic

11

ONOSController A

ONOSController B

MininetSniffer

ONOSController A

ONOSController B

Mininet

ONOSController C

Sniffer

Modelforinter-controllertraffic

12

• switchnetworktopologyanddomain• numberofswitches• numberofintra-domainlinks• numberofinter-domainlinks

Input

• analyPcalformulasfortheinter-controllertraffic

Output

ONOSController A

ONOSController B

ONOSinter-controllertraffic•  Memoryofpastnetworkstates– addandremovingswitcheschangesthebaselinebw– dueto``tombstone’’datakeptforfasterrecoveryincaseoffailures

•  Eachexperimentrequiredtorebootthecontainerstoavoidtombstonetraffic

13 40 50 60 70 80 90

100 110 120 130

0 50 100 150 200 250 300 350

Zero Bandwidth 1

Transient 1 Steady State

Zero Bandwidth 2

Transient 2

Ban

dwid

th [k

bps]

Time [s]

Scenariowith2controllers

•  lineartopologyconnectedtocontrollerA

14

50

100

150

200

250

300

350

0 20 40 60 80 100

Ban

dw

idth

[kbps]

Number of switches (S)

A→B

Curve FittedLower ConfMeanUpper Conf

50

100

150

200

250

300

350

0 20 40 60 80 100

Ban

dw

idth

[kbps]

Number of switches (S)

B→A

Curve FittedLower ConfMeanUpper Conf

Scenariowith3controllers

•  lineartopologyconnectedtocontrollerA

15

50

100

150

200

250

300

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

A→B,C

Curve FittedLower ConfMeanUpper Conf

50

100

150

200

250

300

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

B,C→A

Curve FittedLower ConfMeanUpper Conf

AnalyPcalmodelfor3controllers

16

ONOSController C

BA→BC BB→AC

ONOSController A

ONOSController B

this consistency protocol with respect to RAFT. Globally, thebandwidth exchanged B in each direction is still proportionalto the size of the topology store. Considering that the topologyis only added to controller A, then the following system ofequations can be written with the notation in Table III:

8>><

>>:

BLA!BC = S ⇥ bsA!BC + (S � 1)⇥ blA!BC + b0

BLBC!A = S ⇥ bsBC!A + (S � 1)⇥ blBC!A + b0

BIA!BC = S ⇥ bsA!BC + b0

BIBC!A = S ⇥ bsBC!A + b0

which can be solved by considering the linear interpolatingcurve fitting the experimental data. The zero bandwidth be-tween any two controllers here is b0 = 43 kbps (obtained with5.5% accuracy at 95% confidence level). Thus, we computed

blA!BC = 0.93 kbps blBC!A = 0.53 kbps (4)bsA!BC = 1.12 kbps bsBC!A = 0.35 kbps (5)

To extend model to any topology, arbitrarily partitionedamong the two controller domains, the star topology in Fig. 8was considered albeit with 3 controllers with no switch addedto controller C, in which we varied the number of switchesand thus the inter-domain links. In this scenario, the bandwidthoriginating from each controller to the other two controllerswill be different, since different number of switches are addedto each controller. Hence, the three bandwidths in considera-tion are: A ! BC, B ! AC and C ! AB. Furthermore,the average unidirectional bandwidth per inter-domain link inthis case will be bd for controllers A and B but will be be

for controller C, since the links are external to it but of inter-domain type. Now the observed bandwidth in one direction

40

60

80

100

120

140

160

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

A→B,C

Curve FittedLower ConfMeanUpper Conf

40

60

80

100

120

140

160

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

B,C→A

Curve FittedLower ConfMeanUpper Conf

Fig. 9: Isolated topology added to controller A in the scenariowith 3 controllers

50

100

150

200

250

300

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

A→B,C

Curve FittedLower ConfMeanUpper Conf

50

100

150

200

250

300

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

B,C→A

Curve FittedLower ConfMeanUpper Conf

Fig. 10: Linear topology added to controller A in the scenariowith 3 controllers

TABLE III: Notation for the bandwidth in the scenario with 3controllers

Symbol MeaningB, (BI , BL) generic unidirectional bandwidth (in isolated or linear topology)

b0 generic zero bandwidthbs average unidirectional bandwidth per switchbl average unidirectional bandwidth per intra-domain linkbd average unidirectional bandwidth per inter-domain link

shared also by target controllerbe average unidirectional bandwidth per inter-domain link

external to target controllerBI

A!BC bandwidth A ! B and A ! C in isolated topologyBI

BC!A bandwidth B ! A, C ! A, B ! C and C ! Bin isolated topology

BLA!BC bandwidth A ! B and A ! C in linear topology

BLBC!A bandwidth B ! A, C ! A, B ! C and C ! B

in linear topologybsA!BC average bandwidth A ! B and A ! C per switchbsBC!A average bandwidth B ! A, C ! A, B ! C and

C ! B per switchblA!BC average bandwidth A ! B and A ! C per linkblBC!A average bandwidth B ! A, C ! A, B ! C and

C ! B per link

is obtained by summing the following contributions: BI for1 switch to model the switch in A domain; BI for S � 1switches to model the S� 1 switch in B domain; S� 1 timesthe average bandwidth per inter-domain link bd and S�1 timesthe average bandwidth per external inter-domain link be.

Using the same methodology before, and exploiting theestimated values obtained so far, we were able to estimate thatthe average bandwidth per inter-domain link as:

bd = 0.87 kbps be = 0.7 kbps (6)

By combining the results so far and the estimated band-widths in (4), (4) and (6), we can claim the following, byreferring to Fig. 11:

Property 2: In a scenario with 3 controllers A, B and C,managing a generic network topology arbitrarily partitionedamong the three controllers, the exchanged bandwidth can beevaluated as follows:

BA!BC = 43+1.12⇥SA+0.93⇥LA+0.35⇥ (SB+SC)

+0.53⇥(LB+LC)+0.87⇥(LAB+LAC)+0.7⇥LBC [kbps]

BB!AC = 43+1.12⇥SB+0.93⇥LB+0.35⇥ (SA+SC)

+0.53⇥(LA+LC)+0.87⇥(LAB+LBC)+0.7⇥LAC [kbps]

BC!AB = 43+1.12⇥SC +0.93⇥LC +0.35⇥ (SA+SB)

+0.53⇥(LA+LB)+0.87⇥(LAC+LBC)+0.7⇥LAB [kbps]

where we used the notation in Table IV

V. RELATED WORK

The need of a distributed SDN architecture has beenthoroughly advocated in literature, since it provides resiliencyand scalability as compared to a centralized single controllerimplementation. Onix [10] is a type of distributed SDNcontroller which uses partitioning to introduce scalability indistributed SDN. In Onix, although the network topologyinformation known as Network Information Base (NIB) is

this consistency protocol with respect to RAFT. Globally, thebandwidth exchanged B in each direction is still proportionalto the size of the topology store. Considering that the topologyis only added to controller A, then the following system ofequations can be written with the notation in Table III:

8>><

>>:

BLA!BC = S ⇥ bsA!BC + (S � 1)⇥ blA!BC + b0

BLBC!A = S ⇥ bsBC!A + (S � 1)⇥ blBC!A + b0

BIA!BC = S ⇥ bsA!BC + b0

BIBC!A = S ⇥ bsBC!A + b0

which can be solved by considering the linear interpolatingcurve fitting the experimental data. The zero bandwidth be-tween any two controllers here is b0 = 43 kbps (obtained with5.5% accuracy at 95% confidence level). Thus, we computed

blA!BC = 0.93 kbps blBC!A = 0.53 kbps (4)bsA!BC = 1.12 kbps bsBC!A = 0.35 kbps (5)

To extend model to any topology, arbitrarily partitionedamong the two controller domains, the star topology in Fig. 8was considered albeit with 3 controllers with no switch addedto controller C, in which we varied the number of switchesand thus the inter-domain links. In this scenario, the bandwidthoriginating from each controller to the other two controllerswill be different, since different number of switches are addedto each controller. Hence, the three bandwidths in considera-tion are: A ! BC, B ! AC and C ! AB. Furthermore,the average unidirectional bandwidth per inter-domain link inthis case will be bd for controllers A and B but will be be

for controller C, since the links are external to it but of inter-domain type. Now the observed bandwidth in one direction

40

60

80

100

120

140

160

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

A→B,C

Curve FittedLower ConfMeanUpper Conf

40

60

80

100

120

140

160

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

B,C→A

Curve FittedLower ConfMeanUpper Conf

Fig. 9: Isolated topology added to controller A in the scenariowith 3 controllers

50

100

150

200

250

300

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

A→B,C

Curve FittedLower ConfMeanUpper Conf

50

100

150

200

250

300

0 20 40 60 80 100

Ban

dw

idth

[k

bp

s]

Number of switches (S)

B,C→A

Curve FittedLower ConfMeanUpper Conf

Fig. 10: Linear topology added to controller A in the scenariowith 3 controllers

TABLE III: Notation for the bandwidth in the scenario with 3controllers

Symbol MeaningB, (BI , BL) generic unidirectional bandwidth (in isolated or linear topology)

b0 generic zero bandwidthbs average unidirectional bandwidth per switchbl average unidirectional bandwidth per intra-domain linkbd average unidirectional bandwidth per inter-domain link

shared also by target controllerbe average unidirectional bandwidth per inter-domain link

external to target controllerBI

A!BC bandwidth A ! B and A ! C in isolated topologyBI

BC!A bandwidth B ! A, C ! A, B ! C and C ! Bin isolated topology

BLA!BC bandwidth A ! B and A ! C in linear topology

BLBC!A bandwidth B ! A, C ! A, B ! C and C ! B

in linear topologybsA!BC average bandwidth A ! B and A ! C per switchbsBC!A average bandwidth B ! A, C ! A, B ! C and

C ! B per switchblA!BC average bandwidth A ! B and A ! C per linkblBC!A average bandwidth B ! A, C ! A, B ! C and

C ! B per link

is obtained by summing the following contributions: BI for1 switch to model the switch in A domain; BI for S � 1switches to model the S� 1 switch in B domain; S� 1 timesthe average bandwidth per inter-domain link bd and S�1 timesthe average bandwidth per external inter-domain link be.

Using the same methodology before, and exploiting theestimated values obtained so far, we were able to estimate thatthe average bandwidth per inter-domain link as:

bd = 0.87 kbps be = 0.7 kbps (6)

By combining the results so far and the estimated band-widths in (4), (4) and (6), we can claim the following, byreferring to Fig. 11:

Property 2: In a scenario with 3 controllers A, B and C,managing a generic network topology arbitrarily partitionedamong the three controllers, the exchanged bandwidth can beevaluated as follows:

BA!BC = 43+1.12⇥SA+0.93⇥LA+0.35⇥ (SB+SC)

+0.53⇥(LB+LC)+0.87⇥(LAB+LAC)+0.7⇥LBC [kbps]

BB!AC = 43+1.12⇥SB+0.93⇥LB+0.35⇥ (SA+SC)

+0.53⇥(LA+LC)+0.87⇥(LAB+LBC)+0.7⇥LAC [kbps]

BC!AB = 43+1.12⇥SC +0.93⇥LC +0.35⇥ (SA+SB)

+0.53⇥(LA+LB)+0.87⇥(LAC+LBC)+0.7⇥LAB [kbps]

where we used the notation in Table IV

V. RELATED WORK

The need of a distributed SDN architecture has beenthoroughly advocated in literature, since it provides resiliencyand scalability as compared to a centralized single controllerimplementation. Onix [10] is a type of distributed SDNcontroller which uses partitioning to introduce scalability indistributed SDN. In Onix, although the network topologyinformation known as Network Information Base (NIB) is

Conclusions

•  experimentalevaluaPonoftheinter-controllertrafficinONOSSDNclusters•  around1kbpsforeachnetworkelement(switch/port)•  eveniflowbandwidth,thiscontrolinformaPoniscriPcalforthecorrectnetworkbehavior

•  highlightsscalabilitylaws•  globalinter-controllertrafficgrows•  quadraPcallywiththenumberofcontrollers•  linearlywiththenumberofnetworkelements

•  controlplanemustbecarefullydesignedanddimensioned

ContribuPons

17