Dynamic Scaling of Virtualized, Distributed Service Chains:A Case Study of IMS

Jingpu Duan1, Chuan Wu1, Franck Le2, Alex Liu3, Yanghua Peng1

1The University of Hong Kong, Hong Kong 2IBM T.J. Watson Research Center, U.S.A.3Michigan State University, U.S.A.

AbstractThe emerging paradigm of network function virtualiza-tion advocates deploying virtualized network functions(VNF) on standard virtualization platforms for significantcost reduction and management flexibility. There havebeen system designs for managing dynamic deploymentand scaling of VNF service chains within one cloud datacenter. Many real-world network services involve geo-distributed service chains, with prominent examples ofmobile core networks and IMSs (IP Multimedia Subsys-tems). Virtualizing these service chains requires efficientcoordination of VNF deployment across different geo-distributed data centers over time, calling for new man-agement system design. This paper designs a dynamicscaling system for geo-distributed VNF service chains,using the case of an IMS. IMSs are widely used sub-systems for delivering multimedia services among mobileusers in a 3G/4G network, whose virtualization has beenbroadly advocated in the industry for reducing cost, im-proving network usage efficiency and enabling dynamicnetwork topology reconfiguration for performance opti-mization. Our scaling system design caters to key control-plane and data-plane service chains in an IMS, combin-ing proactive and reactive approaches for timely, cost-effective scaling of the service chains. We evaluate oursystem design using real-world experiments on both em-ulated platforms and geo-distributed clouds.

1 IntroductionTraditional hardware-based network functions are noto-riously hard and costly to deploy, maintain and upgrade.The recent trend towards Network Function Virtualization(NFV) promotes deploying software network functions invirualized environments over off-the-shelf servers, to sig-nificantly simplify deployment, maintenance and scalingat much lowered costs [6].

Despite the good news brought by NFV, many prob-lems remain when applying it to practical network ser-vices, especially when scaling their service chains – an or-

dered collection of virtual network functions (VNFs) thataltogether compose a network service – over a large geo-graphical span. Existing work either focus on designingarchitectural improvement for NFV software, or creatingmanagement systems that scale VNF service chains on asingle server cluster. These management systems are ad-equate for service chains that protect client-server basedsystems. For example, the entry service chain consistingof a firewall and an intrusion detection system (IDS) toaccess database and web services is typically deployed inthe on-premise cluster/data center of the service provider.However, they cannot be directly applied to other servicechains that provide inter-connection services for users re-siding widely apart, such as those in IP Multimedia Sub-systems (IMS) and mobile core networks [1] [13]. Puttingsuch a service chain in a single datacenter would be un-favourable as compared to distributing its VNFs acrossseveral datacenters, to enable low-delay access of theusers to the network functions.

Designing a management system for scaling geo-distributed service chains imposes new challenges overthe existing approaches. How to provision VNF instanceson different datacenters, how to adjust the provisioningover time with variation of traffic, and what VNF in-stances a service chain should go through at different dat-acenters are all important problems that must be well ad-dressed.

This paper presents ScalIMS, a management systemthat enables dynamical deployment and scaling of VNFservice chains across multiple datacenters, using the caseof representative control-plane and data-plane servicechains in an IMS. ScalIMS is designed to provide goodscaling performance (minimal VNF instance deploymentand bounded flow end-to-end delays), using both runtimestatistics of VNF instances and global traffic information.The IMS is chosen as the target platform to showcaseScalIMS because of its important role in telecom core net-works as well as the accessibility of open source softwareimplementation of IMS network functions. ScalIMS hastwo important characteristics that distinguish itself fromthe existing scaling systems:

1

arX

iv:1

702.

0285

3v1

[cs

.NI]

9 F

eb 2

017

Figure 1: IMS: an architectural overview

. Multi-datacenter scaling: ScalIMS deploys multi-ple instances of the same network function onto differentdata centers dynamically according to the traffic demandand distribution of users. In this way, the network pathstaken by a service chain is optimized for small end-to-enddelays and QoS guarantee of user traffic.. Hybrid Scaling Strategy: Most existing scaling sys-

tems [29] [31] [20] rely on reactive scaling strategies,reacting to the change of runtime status of the networkfunctions. ScalIMS combines reactive scaling with proac-tive scaling, using predicted traffic volume based on his-tory. This hybrid scaling strategy exploits all opportuni-ties for timely scaling of network functions and improvesthe overall system performance.

We evaluate ScalIMS on both our own computing clus-ter and the IBM SoftLayer cloud. The experiment re-sults show that ScalIMS significantly improves QoS ofuser traffic compared with scaling systems that use onlythe reactive or proactive scaling approach. ScalIMSachieves this improvement by either launching VNF in-stances timely or re-routing traffic to datacenters with re-dundant VNF instances, using almost 50% less VNF in-stances. Even though we make some special design de-cisions for the IMS, the design principles of ScalIMS canbe easily applied to other NFV systems that benefit fromservice chain deployment in multiple datacenters.

2 Background

2.1 Overview of IMS

An IP Multimedia System (IMS) is a core part in 3G/4Gtelecom networks (e.g., 3GPP, 3GPP2) [14] [5], responsi-ble for delivering multimedia services (e.g., voice, video,messaging) over IP networks. IMS is a fully standard-ized solution for multimedia service delivery across thetelecom industry [1], as compared to its proprietary alter-natives (e.g., Skype, Facetime). It is a complex systemconsisting of multiple service chains. We investigate twomost important service chains as follows. An illustration

is given in Fig. 1.Control Plane Service Chain includes three main net-work functions, Proxy-CSCF (P-CSCF), Interrogating-CSCF (I-CSCF, which is optional), and Serving-CSCF(S-CSCF), which user call requests traverse for user reg-istration and authentication, call setup and charging. Thenetwork functions rely on the IETF Session Initiation Pro-tocol (SIP) [9] to interoperate with external terminals (i.e.,mobile users) in the Internet. The three types of networkfunctions act as different roles of SIP proxies and servers.In addition, S-CSCF relies on an external server calledHome Subscriber Server (HSS), which is a user databasethat contains the subscription-related information for S-CSCF to query.Data Plane Service Chain contains a sequence of net-work functions that the actual multimedia traffic betweenusers traverses, for security (e.g., firewall, deep packetinspection, intrusion detection), connectivity (e.g., NAT,IPv4-to-IPv6 conversion), quality of service (e.g., trafficshaping, rate limiting, ToS/DSCP bit setting), and mediaprocessing (e.g., transcoding). While 3GPP has standard-ized the IMS control plane for interoperability reasons,the exact set of deployed network functions for the dataplane varies per operator.

The two service chains collectively handle two impor-tant procedures of an IMS system, which are user regis-tration and session origination. When a user would liketo make a call, he first announces his IP address to theIMS by initiating a SIP REGISTRATION transaction overthe control plane. When the registration is done, S-CSCFtemporarily stores this (user, IP address) binding for ses-sion origination. When a registered caller initiates a SIPINVITE transaction to a callee over the control plane,i.e., session origination starts, S-CSCF uses the bindingsaved during user registration procedure to retrieve the IPaddress of the callee and routes the SIP message to thecallee. After the callee responds, a call is successfully setup. Subsequent media flows between the caller and thecallee are routed through data plane service chain. Whenthe call is finished, a SIP BYE transaction between thecaller and the callee is carried out to close the call overthe control plane.

2.2 Related Work

Performance and scalability of using VNFs in the place ofhardware middleboxes have been the focuses of investiga-tion in the existing literature on NFV.Performance. Running VNF software (e.g., data planepacket processing software [22]) on VMs incurs signifi-cant context switching cost [27], limiting the maximumthroughput of a VNF. To solve this problem, systems suchas ClickOS [23] and NetVM [21] map packets directlyfrom NIC receive queues to the memory and fetch packets

2

directly from user space [3] [26], which greatly improvethe throughput of VNF.Scalability. The existing studies have investigated scal-ing of VNFs in a single server, in a computing clusterand in a datacenter. RouterBricks [19], xOMB [16] andCoMB [28] focus on scaling VNFs in a single serverfor better performance. Their designs typically rely oncustomized VNF software design and may lack general-ity and flexibility. E2 [25] scales VNF instances withina computing cluster connected through SDN enabledswitches. Both Stratos [20] and Slick [17] study howto scale VNF in a SDN-enabled datacenter. Stratos [20]scales VNF instances by jointly considering VNF place-ment and flow distribution within a datacenter, using on-demand VNF provisioning and VM migration to mitigatehotspots. Slick [17] focuses on providing a program-ing interface for managing VNF instances in a datacen-ter. To our knowledge, there do not exist managementframeworks that scale VNF service chains across geo-distributed datacenters. And existing works can’t be di-rectly extended to multi-datacenter scaling. The primaryreason is that SDN [24] is extensively used by existingapproach to control routing, scaling and loadbalancingwithin a single datacenter, but it is extremely hard for aSDN controller to set up flows on other datacenters, whichincurs too much delay and hurts flow performance [30].One possible way is to deploy one scaling system on eachdatacenter, but there’s no existing works to coordinate thebehavior of each independent scaling system. However,scaling systems on different datacenters need to agree oncomplicated task such as coordinated VNF instance pro-vision and collective flow routing. And all these problemscall for an efficient design of a multi-datacenter scalingsystem.

ScalIMS uses similar methodologies as in[20] [17] [31] when scaling NFV service chainswithin a datacenter, but adopts a novel distributed flowrouting framework and proactive scaling to scale NFVservice chains across multiple datacenters.

3 Motivation and Design Highlights

Motivation. The key motivation of this study is the factthat there are no NFV scaling systems designed to operateunder a multi-datacenter environment, given the follow-ing new challenges compared with scaling service chainswithin a single datacenter.

The first challenge is how to decide the service chainpaths, i.e., the datacenters where instances of VNFs in aservice chain are to be deployed. The deployment has animportant impact on service quality of user traffic alongthe chain. In Fig. 2(a), traffic of IMS system user can gothrough either of the two service chain paths (containing

Figure 2: (a) Service chain path in IMS system: an exam-ple; (b) ScalIMS: an overview

two VNFs each) to reach another IMS user. Choosing thelower path incurs a 50ms end-to-end delay and choosingthe upper path incurs a 100ms delay. A multi-datacenterNFV scaling system should constantly update the servicechain paths so that a good service quality can be guaran-teed for user traffic all the time.

The second challenge lies in that decisions on scalingof network functions (i.e., adding/removing instances ofone VNF upon traffic increase/decrease) and deploymentof service chain paths are coupled in the multi-datacenterscenario. If a service chain path is suffering from over-loading, instead of launching new network function in-stances directly on the same datacenters, the scaling sys-tem may search for available network function instanceson other datacenters and move the service chain path tothose datacenters.

The third challenge is the quest for distributed trafficrouting. Using SDN to control flow routing in a data cen-ter is a common approach adopted by the existing scalingsystems. However, if a service chain path goes throughmultiple datacenters, it becomes impossible for a singleSDN controller to control the end-to-end route. MultipleSDN controllers on a service chain path should work incoordination, upon constant updates of the path, to cor-rectly route user traffic towards the destination.

Highlights of ScalIMS. The overall architecture of Scal-IMS is shown in Fig. 2(b), and we have made the follow-ing design decisions.

First, we adopt a hybrid scaling strategy combiningproactive scaling and reactive scaling for both controlplane and data plane service chains. We divide the systemtime into scaling intervals. At the end of each scalinginterval, proactive scaling is carried out taking as inputglobal information including predicted workload, inter-datacenter delays and the current VNF deployment (thenumbers of instances of each VNF on each datacenter),generating decisions on VNF instance scaling and servicechain path deployment simultaneously for the next scal-ing interval. Reactive scaling produces scaling decisoinsof each VNF based on runtime statistics of each instance

3

locally. It compensates for the inaccuracy of workloadprediction in proactive scaling, improving system perfor-mance under unpredictable traffic volume surges.

Second, ScalIMS enables a synergy of global and localcontrollers (Fig. 2b), to best execute the hybrid scalingstrategy. For proactive scaling, the global controller co-ordinates with all local controllers: it collects input datafrom each local controller, executes the proactive scalingalgorithm, generates scaling/deployment decisions, andbroadcasts the decisions to local controllers. Each localcontroller executes the received decisions by launchingnew VNF instances and adjusting service chain paths. Forreactive scaling, a local controller collects runtime statis-tics from each VNF instance running on the same data-center and produces reactive scaling decision. For flowrouting, a local controller uses flow tags and service chainpaths received from the global controller to determine theVNF instances that a flow should traverse.

Third, we bound the end-to-end delay on service chainpaths to guarantee a good end-to-end performance offlows. We also make some design decisions specially tai-lored for an IMS system, e.g., SIP message content manip-ulation and network address translation. Similar designphilosophies can nevertheless be applied in other servicechain systems as well.

4 Scaling of CP Service Chain

In this section, we present the detailed design of ScalIMSin deployment and scaling of the control plane (CP) ser-vice chain.

4.1 Preliminary

Entry Datacenter Binding: Each user of the system isbound to an entry datacenter. If user a sends a flow touser b, then the flow will enter the service chain from usera’s entry datacenter and exit from user b’s entry datacen-ter, before being sent to user b. User b’s entry datacenteris also referred to as user a’s exit datacenter. We use asimple location service to assign a user to his entry data-center: we map a user’s IP address to a geographical loca-tion by querying an IP-location database (e.g., IP locationfinder [4]), and then assign a datacenter that is closest touser’s current location as his entry datacenter.Fixed CP Service Chain Placement: The CP servicechain of an IMS consists of P-CSCF and S-CSCF. Weuse a fixed placement strategy by deploying P-CSCF in-stances on every entry datacenter and S-CSCF instanceson a fixed datacenter. The delay between the datacenterwhere we place S-CSCF instances and other datacentersshould fall within an acceptable range.

The reason why we adopt such a fixed placement strat-egy is that S-CSCF instances relies on external appli-ances to work. First, S-CSCF constantly queries the HSSdatabase for user information. In addition, S-CSCF in-stances are usually built as a stateless network functionby storing user location information (a temporary map-ping between user name and user registration IP address)in a memcached cluster. The stateless design improvesload balancing among S-CSCF instances, but also impliesthat even if we spread S-CSCF instances across differentdatacenters, each S-CSCF instance still needs to access acentral HSS server and a memcached cluster to processmost of the SIP transactions. This fact urges us to placeS-CSCF instances together with the HSS server and mem-cached cluster in the same datacenter. P-CSCF instancesact as relay points to access S-CSCF instances. By placingthem on each entry datacenter, a user’s flow can alwaysaccess a P-CSCF instance on the closest datacenter.

We use a mixed proactive and reactive strategy for de-ciding the number of instances of P-CSCF and S-CSCF todeploy in the respective data centers.

4.2 Proactive Scaling

Figure 3: Proactive scaling workflow.

Proactive scaling is executed in each scaling inter-val according to the workflow in Fig. 3. The globalcontroller and each local controller work in a request-response manner. In view of possible message loss onthe way, the global controller sets a re-transmision timerthat re-transmits a request after 500ms, if no response isreceived.1. Workload Prediction: Workload along a CP servicechain is described by the number of SIP transactions car-ried out on each entry-exit datacenter pair every second.When a SIP transaction finishes, the S-CSCF instance in-volved uses the location service to determine which entry-exit pair this transaction belongs to (according to the IPaddressed of the caller and the callee). Each S-CSCF in-stance keeps a record of the number of SIP transactionson each entry-exit datacenter pair and reports this numberto the local controller in its datacenter every second . Thelocal controller accumulates CP workload for 5 secondsbefore relaying it to global controller.

At the end of the current scaling interval t, the global

4

controller predicts the workload ut+1 in the next scalinginterval using historic data in several past intervals (10 asin our experiments) based on auto regression [31] [18]:

ut+1 = µ+ φ(ut − µ) (1)

Here µ is the mean of the historic values for several pastscaling intervals, ut is the average value of the collectedCP workload in the current scaling interval and φ is a pa-rameter (it captures the variation of the data series formedby received workload values [31] [18]). The workloadbetween each entry-exit datacenter pair is being predictedthis way.2. Aquiring Current Provision: The global controllerthen broadcasts a message to local controllers, askingthem to send information of the current VNF instanceprovisioning (i.e., the number of instances of each VNFprovisioned in the respective datacenter). Upon receivingthis message, a local controller stops its reactive scalingprocess (to be discussed next) so that it does not interferewith proactive scaling, and then responds to the globalcontroller by sending its current provisioning information.3. Compute Proactive Scaling Results: After receiv-ing information of current provisioning from all local con-trollers, the global controller computes the number of P-CSCF and S-CSCF instances to be deployed in the re-spective datacenters in the next scaling interval. Sinceall S-CSCF instances are placed in the same datacenter,the number can be decided by dividing the total predictedtraffic volume between all pairs of entry-exit datacentersby the processing capacity of one instance. The numberof P-CSCF instances to be deployed in a datacenter canbe computed by dividing the overall predicted volume oftraffic flows, that use the datacenter as either the entrypoint or the exit point, by the processing capacity of oneinstance.4. Broadcast Proactive Scaling Result: The globalcontroller then broadcasts the computed new deploymentnumbers to local controllers. A local controller sends acompletion message to the global controller, after apply-ing the new provisioning result by creating new VNF in-stances (scale-out) or enqueueing unused VNF instancesto tail of buffer queue (scale-in).5. Enter New Scaling Interval: After the global con-troller receives completion messages from all local con-trollers, it broadcasts an “enter new scaling interval mes-sage” to all local controllers. After getting this message,a local controller increments its scaling interval index by1, and shuts down some VNF instances from the head ofbuffer queue. Then the local controller sends a final ac-knowledgement to the global controller. After receivingall final acknowledgements from all local controllers, theglobal controller increases its scaling interval by 1.

In each datacenter, a double-ended buffer queue ismaintained to temporarily hold unused VNF instances,

with one queue for one type of VNF. When a VNF in-stance is to be removed, instead of directly shutting itdown, local controller tags it with the current scaling in-terval index and enqueues it to the tail of the buffer queue.Once a VNF instance is enqueued into the buffer queue,no more flows will be routed to it. Whenever more VNFinstances are to be established, if there are available in-stances in the buffer queue, buffered instances from thetail of the respective queue will be poped out and trans-formed back to working instances as much as possible.The purpose is to avoid frequent creation/deletion of VNFinstances under workload fluctuation. Unused bufferedVNF instances are destroyed after τ scaling intervals (τ isset to 10 scaling intervals in our implementation): whenthe local controller in the datacenter receives an “enternew scaling interval message”, it checks from the headof each buffer queue whether an instance has been en-queued τ intervals ago; if so, the VNF instance at the headwill be dequeued. Dequeuing repeats until all unused in-stances for τ intervals are cleaned. These dequeued VNFinstances gets destroyed by local controller when there’sno active traffic on them.

4.3 Reactive ScalingA statistics reporting agent is running on each VNF in-stance and reports to the local controller runtime statisticsfor CPU usage, memory usage and input packet numberevery second. The local controller maintains time seriesfor each type of metrics of each VNF instance. It decideswhether CPU/memory/network is overloaded during thepast several seconds (5 seconds as in our experiments) bycomparing the respective runtime statistics with a thresh-old. If overload is persistently identified for at least twotypes of metrics (e.g., CPU usage and network, in order toeliminate false alarms) for several consecutive seconds (5seconds as in our experiments), then that VNF instance isreported as overloaded. Its state will be transferred from“normal” to “overload”. The local controller will avoidrouting new traffic flows (i.e., new calls) to overloadedinstances if there are available “normal” instances. In adatacenter, if the states of a majority of VNF instanceswith the same type are “overload”, scale-out is triggeredby adding one new instance of that type. Note that noscale-in decisions (i.e., removing instances) are made re-actively, which is solely handled by proactive scaling.

4.4 Message Routing on Control PlaneWhen a user connects to an IMS, he first issues a queryto a global DNS server, which determines the entry dat-acenter of this user using the location service and ob-tains the IP address of an available P-CSCF instance (non-overloaded) by querying a local DNS server located in the

5

Caller Entry DCController

1. (caller IP, caller send port)→(callee IP)2. (callee IP, callee send port)→(caller en-try IP, caller IP, caller receive port)

Callee Entry DCController

3. (callee IP, callee send port)→(caller IP)4. (caller IP, caller send port)→(callee en-try IP, callee IP, callee receive port)

Table 1: Mappings Saved On Local Controller of the En-try Datacenter

entry datacenter. A P-CSCF instance learns the IP ad-dresses of several available S-CSCF instances (non over-loaded) through DNS query as well. The DNS record ofeach CP VNF instance is updated by the local controllerin the datacenter hosting the VNF instance. P-CSCFinstance maintains several connections to up-stream S-CSCF instances and evenly balance up-stream requestsamong these connections. It regularly (every 30s in Scal-IMS) updates these connections by issuing new DNSquery to obtain an updated view of S-CSCF instances.

The establishment of a call involves that the callersends out a SIP INVITE message and the callee respondswith a SIP OK message. The content of these two mes-sages are modified by a P-CSCF instance in order for thecaller/callee to learn a destination IP address on his en-try datacenter. A SIP INVITE message carries the caller’ssource IP address, receive port and send port. When a SIPINVITE message passes through callee’s entry datacenter,the source IP field is modified to an IP address located onthe callee’s entry datacenter, so that the callee learns abouta destination IP address located on callee’s entry datacen-ter. A similar process is applied to SIP OK message aswell. Besides SIP message modification, a P-CSCF in-stance sends two mappings (shown in Table 1) to its localcontroller after it has received the SIP OK message. Thelocal controller saves these two mappings for use whenit processes the data plane traffic flows. Details will bediscusses in Sec. 5.

5 Scaling of DP Service Chain

We next present the detailed design of ScalIMS in deploy-ment and scaling of the data plane (DP) service chain. DPshares with CP the same reactive scaling mechanism dis-cussed in Sec. 4.3. For proactive scaling, DP follows thesame proactive scaling workflow in Sec. 4.2, with someminor differences.

First, DP proactive scaling scheme, running in Step 3of Fig. 3, not only decides how VNF instances are pro-visioned in each datacenter, but also updates the servicechain path between each entry-exit datacenter pair, byrunning the DP proactive scaling algorithm (see Sec. 5.1).We refer to VNFs in a DP service chain as stages of the

chain and index them following the order of the VNFs inthe chain. For example, DP service chain used in ScalIMScontains firewall (stage 1), IDS (stage 2) and transcoder(stage 3). Since ScalIMS manages multiple datacenters,we define a service chain path between each pair of entry-exit datacenters to be a path of datacenters. For a servicechain with m stages, a service chain path contains m+ 2datacenters. The 0th and m + 1th datacenter are entrydatacenter and exit datacenter of the entry-exit datacenterpair associated with the service chain path. The ith dat-acenter (i = 1, ...,m) hosts instances of stage i VNF inthe service chain. Stage 1 and stage m could be hostedon other datacenters other than entry and exit datacenter.For example, a service chain path of (0, 0, 1, 1, 2) showsthat datacenter 0 is the entry datacenter, stage 1 to stage 3of the service chain are hosted on datacenters 0, 1, and 1,respectively, and the exit datacenter is datacenter 2. Notethat even if instances of a VNF may be deployed on differ-ent datacenters, we maintain only one service chain pathfor each service chain between each entry-exit datacenterpair in each single scaling interval, for path computationefficiency.

Second, Besides provisioning VNF resources, localcontroller also acts as an SDN controller that manages DPflow routing within local controller’s datacenter. Whenthe local controller receives DP proactive scaling resultsin step 4 of Fig. 3, the local controller applies the DP VNFprovision decisions (i.e., add/remove instances) immedi-ately, but saves the new DP service chain paths and usesthem for routing only after receiving an “enter new scalinginterval” message in step 5 of Fig. 3. This 2-stage updateis used to guarantee consistency for distributed flow rout-ing (see Sec. 5.4).

Third, the proactive scaling algorithm takes as inputpredicted workload, predicted inter-datacenter networkdelay, and the current DP VNF deployment. Workloadalong a DP service chain is described as the number ofpackets transmitted over each entry-exit datacenter pairevery second. The local controller acquires input DPworkload using workload measuring OpenFlow rules in-stalled on the entry switch. The local controller reportsDP workload measurements to the global controller ev-ery second. Inter-datacenter delays are constantly mea-sured by local controllers through a ping test among eachother. Each local controller reports the ping delay with theother local controllers to the global controller every sec-ond. At the end of the current scaling interval, the globalcontroller perdicts workload and delays in the next inter-val using the same approach as discussed in Sec. 4.2.

5.1 DP Proactive Scaling Algorithm

The DP proactive scaling algorithm is presented in Alg. 1.It greedily serves the predicted workload of each entry-

6

Algorithm 1: DP Proactive Scaling AlgorithmInput: Predicted delay between each datacenter pair, predicted

workload of each entry-exit datacenter pair, current VNFinstance provision, current service chain paths

Output: New VNF instance provision, new service chain paths1 Compute total available processing capacity of instances of each

VNF in each datacenter;2 foreach entry-exit datacenter pair p, p.entry 6= p.exit do3 if there is enough VNF capacity on p’s current service chain

path and the end-to-end delay threshold is not violated alongp’s current path then

4 use p’s current service chain path and decrease predictedworkload from available processing capacities of VNFson p’s new path;

5 foreach p, p.entry 6= p.exit && p’s new service chain path hasnot been determined do

6 compute a new path for p using Alg. 2;7 if there is not enough capacity on p’s new path then8 scale out by creating new VNF instances sufficient to

serve p’s workload;

9 decrease predicted workload from available processingcapacities of VNFs on p’s new path;

10 foreach p, p.entry = p.exit do11 use p’s current service chain path;12 if there is not enough capacity on p’s new path then13 scale out by creating new VNF instances sufficient to

serve p’s workload;

14 decrease predicted workload from available processingcapacities of VNFs on p’s new path;

15 scale in by enqueuing un-used VNF instances to the respectivebuffer queues;

exit datacenter pair p using the current VNF instance pro-visioning, as long as the capacity is still sufficient andthe end-to-end delay threshold is still guaranteed (lines 2-4). Otherwise, a new service chain path for the entry-exitdatacenter pair is computed using Alg. 2. If there is notenough VNF capacity to serve the predicted workload onp’s new service chain, d(Q−Q′)/Ce new VNF instancesare created for the respective VNF in the respective dat-acenter (whose capacity is in shortage), where Q is p’spredicted workload, Q′ is the total capacity of the VNF inthe respective datacenter and C is the processing capacityof each instance of that VNF (lines 5-9). For an entry-exitdatacenter pair p with the same entry and exit datacenter,the entire service chain path of p is always deployed inthis datacenter. The algorithm processes these entry-exitdatacenter pairs at last so that the VNF capacity can be uti-lized more efficiently (lines 10-14).When workload of allentry-exit pairs is packed, scale-in is carried out to removeun-used VNF instances in the corresponding datacenter.

5.2 Path Computation Algorithm

Besides conforming to the end-to-end delay requirement,a DP service chain path should meet two restrictions: (1)each datacenter in a service chain path should host at least

Algorithm 2: Path ComputationInput: Predicted inter-datacenter delays, p = (entry, exit), p’s

predicted workload, p’s current service chain path, currentavailable processing capacities of VNF instances, theservice chain of m stages, n datacenters

Output: p’s new service chain path1 minProvPath = p’s current path;2 for v = 0, ..., exit− 1, exit+ 1, ..., n− 1 do3 record[0] = entry, record[1] = v,

record[m+ 1] = exit;4 for x = 2, ...,m do5 find out datacenter v1 (v1 6= exit) that has the largest

available capacity for stage-x VNF;6 record[x] = v1;7 if there is path loop on record then8 eliminate loop by adjusting record;

9 path[0, ..., x] = record[0, ..., x];10 path[x+ 1, ...,m+ 1] = exit;11 if path leads to a smaller number of new VNF instances

to create for hanlding predicted workload thanminProvPath and satisfies end-to-end delay requirementthen

12 minProvPath = path

13 path[0] = entry, path[1] = v, path[2, ...,m+ 1] = exit;14 check whether path should be assigned to minProvPath

as in lines 11-12;

15 path1[0] = entry, path1[1, ...,m+ 1] = exit;16 path2[0, ...,m] = entry, path2[m+ 1] = exit;17 check whether path1 or path2 should be assigned to

minProvPath as in lines 11-12;18 if minProvPath violates end-to-end delay requirement then19 find out the shortest-delay datacenter path between entry and

exit;20 run stage placement algorithm and assign the found service

chain path to minProvPath;

21 return minProvPath;

one VNF in the respective service chain; (2) a servicechain path should be loopless i.e., for two VNF stagesx < y, if instances of both stage x and stage y are placedon datacenter i, then instances of stage z, x < z < y,must also be placed in datacenter i, as otherwise a routingloop is created.

Alg. 2 presents the algorithm to compute a good ser-vice chain path for a given service chain between a givenentry-exit datacenter pair, that aims to minimize the num-ber of new VNF instances to be created for handling thepredicted workflow between the entry-exit datacenter pair,while satisfying the end-to-end delay requirement (lines10 ∼ 11). If the DP service chain contains m stages andthere are n datacenters, exhaustive search to identify sucha service chain path requires O(mn) running time. In-stead, Alg. 2 seeks to optimistically find a good path inO(mn) time.

In Alg. 2, array record retains the path under investiga-tion (line 3). The search starts by looping through all dat-acenters except the exit to decide one for hosting VNFsof stage 1 (lines 2 ∼ 3). For each subsequent stage of

7

the service chain, a datacenter (except the exit) with thelargest available capacity of VNF instances of that stageis selected (lines 4 ∼ 6). We might have created a loopin the datacenter path. If so, we eliminate the loop by themethod to be discussed in Loop Elimination below (lines7∼ 8). Once the datacenter to host the stage-xVNF is de-termined, a candidate path is produced (lines 9 ∼ 13): wecompute the number of new VNF instances to be createdalong the path, beyond the existing available capacities,for serving the predicted workload between the entry-exitpair, and end-to-end delay of the path; if the path incursless addition of new VNF instances and satisfies the de-lay requirement than the current best candidate path, weretain it in minProvPath (lines 12-13). The algorithmalso checks some candidate paths that are not generatedby the search loop (lines 13 ∼ 17). Finally, it is possi-ble that all candidate paths found so far fail to satisfy thedelay requirement. If so, we compute the datacenter pathwith the shortest end-to-end delay (e.g., using a shortestpath algorithm) and run the Stage Placement Algorithmbelow to produce the service chain path (lines 18-20).

Loop Elimination: A routing loop may be introducedin lines 5 ∼ 6. For example, suppose the record contains(1, 2, 4, ...), meaning entry datacenter is datacenter 1, in-stances of stage-1 VNF are placed on datacenter 2, andinstances of stage-2 VNF are placed on datacenter 4. Ifdatacenter 2 is selected to host the stage-3 VNF, then therecord becomes (1, 2, 4, 2, ...) and a loop is created. Wecompare two approaches for eliminating the loop. Thefirst is that we deploy VNF instances of stages involved inthe loop in the same datacenter at the start and the end ofthe loop, e.g., deploy stage 2 VNF on datacenter 2 insteadof datacenter 4 to make the path (1, 2, 2, 2, ...). The sec-ond option is that we always choose the datacenter withthe largest available capacity for hosting a stage, withoutadding a routing loop to the existing path. Suppose data-center 3 has the second largest capacity of stage-3 VNF.Then we choose datacenter 3 instead of datacenter 2 tohost stage-3 VNF for this entry-exit pair, creating a path(1, 2, 4, 3, ...). We compare paths created using the twoapproaches and choose the better one requring less newVNF instances to be created to serve predicted workload.

Stage Placement Algorithm: Given the shortestdatacenter-to-datacenter path computed in line 19, we de-cide the optimal placement for deploying VNFs of the ser-vice chain onto the datacenters in the path, which mini-mizes the number of new VNF instances that need to becreated. The minimum overall number of new instancesto be created is decided using a dynamic programmingapproach. Let N(i, j) denote the total number of new in-stances of stage 1 to stage j VNFs, that should be createdto serve the predicted workloadQp for entry-exit pair p, ifstage j is placed on the ith datacenter in the datacenter-to-datacenter path (suppose the path contains k datacenters

and service chain path contains m stages). We have

N(1, 1) = num(1, 1), N(2, 1) = num(2, 1),

N(i, 1) = +∞,∀i = 3, . . . , k,

N(1, j) = N(1, j − 1) + num(1, j),∀j = 2, . . . ,m,

N(i, j) = min{N(i− 1, j − 1) + num(i, j), N(i, j − 1) + num(i, j)},

∀i = 2, . . . , k, j = 2, . . . ,m (2)

where num(i, j) is the number of new stage-j VNF in-stances that need to be created when stage j is placed onthe ith datacenter, i.e.,

num(i, j) =

0, if Q′

i,j ≥ Qp and m− j + 1 ≥ k − id(Qp −Q′

i,j)/Cje, if Q′i,j < Qp and m− j + 1 ≥ k − i

+∞, if m− j + 1 < k − i(3)

Here Q′i,j is the total available capacity of stage-j VNF

instances in the ith datacenter and Cj is the processingcapacity of one stage-j VNF instance.

The rationale behind Eqn. (2) and Eqn. (3) is as fol-lows. If stage j is placed on the ith datacenter, then stagej− 1 can only be placed on the i− 1th datacenter (requir-ing N(i− 1, j − 1) + num(i, j) new VNF instances), oron the ith datacenter (requiring N(i, j − 1) + num(i, j)new VNF instances). Otherwise, the service chain pathviolates the restrictions given at the beginning of Sec. 5.2.When m − j + 1 < k − i, i.e., the remaining numberof stages in the service chain is smaller than the remain-ing number of datacenters in the shortest path (it is notcompulsory to place stage m VNF on exit datacenter k,that’s why we have a plus 1 on left side of inequation), thestage placement is not feasible (num(i, j) is set to +∞)since restriction (2) is violated; otherwise, the number ofnew stage-j VNF instances in datacenter i is computedaccording to how much additional capacity is needed toserve the predicted workload.

After the algorithm computes up to N(k,m), we canfind the optimal service chain path by tracing back fromthe smaller ofN(k,m) andN(k−1,m), to eitherN(1, 1)and N(2, 1).

5.3 Flow Routing On Data PlaneWe next illustrate how DP flow routing is done. We use aDP traffic flow sent by the caller as an example. The samerouting process can be applied to DP traffic flow sent bythe callee.Enter the Service Chain Path. The caller learns an IPaddress located on his entry datacenter when the SIP IN-VITE transaction ends (see Sec. 4.4). The caller uses thisIP address to send DP traffic flow to his entry datacenter.Follow the Service Chain Path. When a flow enters adatacenter, an OpenFlow Packet IN message is sent to the

8

local controller. In the entry datacenter, the local con-troller maps the caller’s source IP and source port to thecallee’s source IP using the first mapping in Table 1. Us-ing the location service, the local controller also gets toknow the exit datacenter of this flow and can then iden-tify the service chain path of the entry-exit datacenter pairfor the current scaling interval. In order for local con-trollers in subsequent datacenters in the service chain pathto learn about the service chain path, the local controllerin the entry datacenter adds a tag to the header of pack-ets in the flow (encoded using the destination port field inour implementation). The tag contains the index of theflow’s entry datacenter, the index of the flow’s exit dat-acenter and the current scaling interval number modulo4. Since a local controller only needs to check whetherthe encoded scaling interval is the previous interval, thecurrent interval or the next interval, we encode the cur-rent scaling interval number modulo 4 (only two bits areneeded to represent the scaling interval). Local controllersin subsequent datacenters learn the service chain paths foreach entry-exit pair as in step 4 of Sec. 4.2.Choose VNF instances. After the local controller in adatacenter learns the service chain path that a flow shouldfollow, it selects VNF instances for each stage of theflow’s service chain, that is deployed in the datacenter, forthe flow to traverse, according to a smallest-workload-firstcriteria. It enables the flow to go through selected VNFinstances by encoding routing information in the flow’spacket header too (using the destination IP address fieldin our implementation).

In particular, a unique index is assigned to each VNFinstance in a datacenter. The smallest index value is thenumber of datacenters that ScalIMS controls (the indexvalues from 0 to datacenter number minus 1 is reservedto be used by inter-datacenter VxLAN tunnel). ScalIMSconfigures each VNF instance with 2 NICs, an entry NICfor accepting ingress traffic and an exit NIC for sendingegress traffic. Each NIC is connected to an independentvirtual switch. A static flow rule is installed on the switchthat each VNF instance’s entry NIC connects to. In Scal-IMS, flow rules installed for stage-i (i = 1, ..., 3) VNFinstances match selected bits in the destination IP addresswith a mask of 255 << 8 ∗ (4 − i). If the matched bitsequal the index of a VNF instance, then the flow is sent tothe corresponding VNF instance.

In ScalIMS, if the flow has passed through stage i(i = 1, ..., 3) VNF and is going to another datacenter forfurther processing, we fill the matched bits with a mask of255 << 8 ∗ [4 − (i + 1)] by the index of the next data-center. Note that the mask 255 << 8 ∗ [4− (i+1)] is themask for stage i+ 1 ( If i = 3, we treat stage 4 as an vir-tual exit stage that is always deployed on exit datacenter,with details in first difference at start of Sec. 5). When theflow comes out of the exit NIC of the last processing stage

in the current datacenter, a pre-installed static rule auto-matically routes the flow to the next datacenter through aVxLAN tunnel [15], if the matched bits indicate the indexof the next datacenter.Exit from the DP Service Chain: When the flow hasbeen processed by VNFs of all stages, it exits from the DPservice chain path. To send the flow to the callee, the localcontroller in the exit datacenter maps the caller’s sourceIP and source port to the callee’s entry IP, callee IP, calleereceive port according to the fourth mapping in Table 1.Then an address translation is carried out to substitute thecaller’s source IP by the callee’s IP, the caller’s destinationIP by the callee’s IP, and the caller’s destination port bythe callee’s receive port.

5.4 Handling Scaling Interval InconsistencyIn Step 5 of Fig. 3, the global controller sends an ‘en-ter new scaling interval’ message to all local controllers.Given that not all local controllers may receive the mes-sage at the same time, there can be temporary inconsis-tency of the scaling intervals each local controller is cur-rently in. Hence when a local controller processes a flow,it may find that the encoded scaling interval in the packetheader indicates the previous scaling interval, the currentscaling interval, or the next scaling interval. For example,suppose local controller 1 has received ‘enter new scalinginterval’ message and local controller 2 has not receivedit. Local controller 1 becomes 1 scaling interval ahead oflocal controller 2. Suppose that a flow enters the servicechain from local controller 1’s datacenter and is imme-diately sent to local controller 2’s datacenter during thisshort time window. Then local controller 2 will find thatthe encoded scaling interval in the flow’s packes is thenext scaling interval. But if flow 1 goes from a reverse di-rection during this time interval (enter service chain fromlocal controller2’s datacenter and be sent to to local con-troller 1’s datacenter), then local controller 1 will find thatflow 1’s encoded scaling interval is the previous scalinginterval.

To counter this problem, three sets of service chainpaths are always retained in each local controller duringthe execution of proactive scaling, for the previous, cur-rent and next scaling intervals, respectively. When a localcontroller receives a new flow, it uses one of the servicechain paths depending on the scaling interval encoded inthe header of the flow packets.

6 Implementation and Evaluation

6.1 ImplementationWe implement a prototype of ScalIMS in Java. The localcontroller is implemented as a module in the FloodLight

9

Table 2: VNF Capacity, Overload Threshold and Inter-datacenter Delays

VNF CapacityCPUthreshold

Memorythreshold

Input pkt/sthreshold

Inter-GroupDelay g0 g1 g2 g3

P-CSCF 500 tran/s 70% 50% 1000 pkt/s g0 0ms 10ms 15ms 50msS-CSCF 200 tran/s 70% 50% 400 pkt/s g1 10ms 0ms 20ms 17msFirewall 35000 pkt/s 90% 50% 35000 pkt/s g2 15ms 20ms 0ms 15msIDS 20000 pkt/s 90% 50% 2000 pkt/s g3 50ms 17ms 15ms 0msTranscoder 15000 pkt/s 90% 50% 15000 pkt/s

SDN controller [2]. The global controller is implementedas a multi-threaded java server program, which commu-nicates with local controllers over regular sockets. Wealso implement a traffic generator based on PJSIP [7]. Wedeploy one traffic generator in each datacenter, produc-ing user arrivals bond to the datacenter at a configurablerate. A global traffic generator coordinator receives noti-fications of generated users and pairs up users in a first-come-first-match manner. A call process is launched be-tween every pair of paired users, which includes all therequired SIP transactions to establish a call, followed by aone-minute voice call at the bit rate of 80kbit/s, as well asnecessary SIP transactions to shutdown the call.

We use P-CSCF, S-CSCF and HSS components fromthe Clearwater Project [8] as our control plane VNFs. Thedata plane service chain contains a firewall (implementedusing user space Click [23]), an intrusion detector (SnortIDS [10]) and a transcoder (implemented using user spaceClick). Each network function runs on a QEMU/KVMvirtual machine (VM). A CP VM is configured with 1core and 2GB RAM and a DP VM is configured with 2cores and 2GB RAM. The capacity and overload thresh-old (to decide scaling out) of each instance of each VNFare shown in Table 2, which are obtained by stress testingVNF instances to overloaded states.

6.2 Evaluation in IBM SoftLayer Cloud

We evaluate the performance of ScalIMS on IBM Soft-Layer Cloud [11] by deploying 4 bare-metal servers in 4Softlayer datacenters, located in Tokyo, Hong Kong, Lon-don and Houston, respectively. Each server is equippedwith two 6-core 2.4GHz Intel CPU, 64GB RAM, 1TBSATA disk. In the SoftLayer cloud, servers in differentdatacenters are connected through a global private net-work [12], which provides a Gigabyte throughput. Scal-IMS creates a VxLAN tunnel mesh in the private networkto route DP traffic. CP VNFs are connected directly overthe private network.

We use two groups of metrics to evaluate the perfor-mance of ScalIMS. (1) The total number of new VNF in-stances created over time: the smaller the number is, themore cost/resource effective ScalIMS is; (2) User trafficquality metrics: the SIP transaction completion time forthe control plane traffic, and the RTT and loss rate fordata plane traffic.

Figure 4: Overall number of new VNF instances created.(a) Simultaneous start. (b) Asynchronous start.

Each traffic generator is configured to start generatingusers at a small rate, which gradually increases to a max-imal rate, and then gradually decreases. The user gen-eration rate is changed in every change interval. Eachscaling interval may contain multiple change intervals.The scaling interval is set to be 50 seconds long and eachbuffer queue retains an unused VNF instance for at most10 scaling intervals. The maximum allowed end-to-endflow delay is set to 250ms.

In the following experiments, we compare the perfor-mance achieved by using proactive scaling only (the localcontroller does not react to overload of instances), reactivescaling only (the global controller initializes a good setof service chain paths, but no subsequent proactive scal-ing decisions are broadcast to local controllers), and bothproactive and reactive scaling enabled (i.e., the combinedscaling stragegy of ScalIMS).

6.2.1 Simultaneous Start

In this set of experiments, each traffic generator startsgenerating users simultaneously at the rate of 1 users/s,which gradually increases to 15users/s and then de-creases to 6users/s. In this way, the peak workload ar-rives almost concurrently in each datacenter. The durationof a change interval is set to 20s, 30s, ... 80s respectively.

Fig. 4(a) and Fig. 5 show that the total number ofVNF instances provisioned and the average SIP transac-tion completion time under the three schemes do not differmuch. The total number of VNF instances provisioned issimilar because the maximum workload on each dataceneris similar. On the other hand, Fig. 6 shows that the com-bined scaling strategy of ScalIMS out-performs the othertwo strategies in terms of data plane traffic quality. Thereason can be explained as follows. Pure reactive scalingadds new instances only when an overload signal arises.During the boot-up time of new instances, traffic contin-ues to arrive at the overloaded VNF instances, resultingin a high packet loss rate and then high RTT. Pure proac-tive scaling adjusts VNF instances once every scaling in-

10

Figure 5: CP traffic quality with simultaneous start.

0 5 10 15 20 25 30 35 40

(a) average flow loss rate (%)

0.5

0.6

0.7

0.8

0.9

1.0

1.1

CD

F

change interval: 30s

proactive

reactive

combined

0 5 10 15

(b) average flow loss rate (%)

0.92

0.94

0.96

0.98

1.00

CD

F

change interval: 70s

proactive

reactive

combined

500 1500 2500 3500

(c) average RTT (ms)

0.0

0.2

0.4

0.6

0.8

1.0

CD

F

change interval: 30s

proactive

reactive

combined

0 500 1000 1500 2000

(d) average RTT (ms)

0.0

0.2

0.4

0.6

0.8

1.0

CD

F

change interval: 70s

proactive

reactive

combined

Figure 6: DP traffic quality with simultaneous start.

terval. During each scaling interval, increased workloadmay have overloaded the system. The best performance isachieved combining both proactive and reactive scaling.The reason for similar CP performance is that scaling inthe CP service chains is not triggered as often as that ofthe DP service chains, since each instance of a CP VNF isable to handle a large number of SIP transactions.

6.2.2 Asynchronous Start

In this set of experiments, traffic generators are not startedat the same time. Traffic generator in the Tokyo datacenteris started first, followed by traffic generator in the HongKong datacenter, in the Houston datacenter and then in theLondon datacenter. The start time of the traffic generatorin a subsequent datacenter is delayed for a configurableperiod of time after that of the previously started trafficgenerator. In this way, the peak workloads in differentdatacenter do not occur at the same time. Each traffic gen-erator increases its user generation rate from 5 users/s to15 users/s and then dereases it to 5users/s, with a 30schange interval.

In Fig. 4(b), the start delay between traffic generatorsin Tokyo ∼ and in Hong Kong is set according to val-ues in the x axis, while start delays between Hong Kong

Figure 7: DP traffic quality with asynchronous start.

and Houston and between Houston and London are set to120s. The figure shows that the number of VNF instancescreated by the combined scaling approach is always thesmallest. Fig. 7 shows that the traffic quality is much bet-ter with the combined approach as well.

Why can the combined scaling perform well even whenit creates a smaller number of VNF instances? The Tokyodatacenter sees the peak workload first, leading to theprovisioning of many VNF instances in the datacenter,which later become redundant and will be buffered for 10scaling intervals. When peak workload arrives at otherdatacenters, the global controller can re-use the existingVNF instances by creating service chain paths travers-ing Tokyo datacenter, increasing resource utilization ef-ficiency. Fig. 8(a) shows the number of VNF instancescreated in each datacenter verses the time after the ex-periment starts. We can see that due to the early arrivalof peak workload, many VNF instances are provisionedin Tokyo datacenter by 341s. After that time, the num-ber of service chain paths that go through Tokyo datacen-ter increases from 7 to 11 (Fig. 8(b)) because the globalcontroller schedules more DP traffic flows to re-use theseexisting VNF instances. Because these existing VNF in-stances are buffered and reused, DP traffic flows do notgo through overloaded instances, improving the qualityexperienced by the DP flows.

6.3 Evaluation in Emulation Cluster6.3.1 Links with Large Delays

Next we evaluate whether ScalIMS is able to effectivelydetect links with high delays and shift the traffic awayfrom these links on our own emulation cluster. We use8 servers in the cluster and emulate inter-datacenter de-lay with Linux TC. The 8 datacenters are separated into 4groups (i.e. g0 to g3, where gi contains datacenter i andi + 1). The delay between datacenters in the same groupis set to 5ms, the delay between datacenters in differentgroup is set according to Table 2.

In this set of experiments, traffic generators increaseuser production rate from 1 users/s to 12 users/s and

11

20 341 663 984

(b) time after experiment starts (s)

0123456789

# of NF instances provisioneddc Tokyo

dc HK

dc Houston

dc London

20 341 663 984

(a) time after experiment starts (s)

0

2

4

6

8

10

12# of service chain paths that go through respective datacenters

dc Tokyo

dc HK

dc Houston

dc London

Figure 8: (a) Number of provisioned VNF instances ineach datacenter verses time. (b) The number of servicechain paths that go through different datacenters.

then decrease the rate to 6 users/s, with a 50s updateinterval. At 350s after the experiments start, we deliber-ately increase the delay between groups 1 and 3 to 65ms.Since the maximum end-to-end delay threshold is 50ms inour DP proactive scaling algorithm, the global controllerwould avoid using links between groups 1 and 3.

Fig. 9 (a) shows that with ScalIMS, the number of flowswhose measured RTT is smaller than 100ms is 15% morethan that with reactive scaling. Fig. 9(b) illustrates that af-ter the global controller detects the links with high delay,it re-computes new service chain paths that avoid usingthose links. At about 400s, the number of service chainpaths that use the high delay links drops from 8 to 0,boosting the quality of the DP media flow.

Figure 9: (a) CDF of flow RTT. (b) Number of servicechain paths that use the links between group 1 and group3.

6.3.2 Scaling Interval Consistency

We also evaluate the impact of our design on maintain-ing scaling interval consistency as discussed in Sec. 5.4,

by comparing it with one where flows are not tagged withscaling intervals and local controllers always route flowsusing servcice chain paths in the current scaling interval.In this set of experiments, we generate 15 calls per sec-ond between datacenter 0 and 1 for 5 minutes. EnoughVNF instances are pre-provisioned on 3 datacenters (dat-acenters 0 − 2) to serve the traffic. The global controlleralternates service chain paths from datacenter 0 to data-center 1 and from datacenter 1 to datacenter 0 from (0, 0,0, 1, 1) and (1, 1, 1, 0, 0) to (0, 2, 2, 2, 1) and (1, 2, 2, 2,0) in each scaling interval. The delays between datacen-ters 0 and 2 and between datacenters 1 and 2 are both setto 400ms. If a local controller finds out that it is not ona flow’s service chain path, it applies a rule that drops allthe packets of this flow.

We find that without scaling interval tagging, 18 out of8722 flows (0.2%) are completely not received by their re-ceivers. On the contrary, all 8724 flows are received with0% loss rate when the scaling interval tagging is enabled.This is because when service chain paths from datacenter0 to datacenter 1 and from datacenter 1 to datacenter 0 areswitched from (0, 2, 2, 2, 1) and (1, 2, 2, 2, 0) to (0, 0, 0,1, 1) and (1, 1, 1, 0, 0), datacenter 2 enters the next scal-ing interval earlier. There is a small time window whendatacenter 2 starts using service chain path (0, 0, 0, 1, 1)and (1, 1, 1, 0, 0) while datacenter 0 and datacenter 1 arestill in the previous scaling interval, using service chainpath (0, 2, 2, 2, 1) and (1, 2, 2, 2, 0). Datacenters 0 and1 continue sending flows to datacenter 2 during that timewindow, but datacenter 2 finds that it is not on the servicechain path of this flow, thus dropping all the packets. Weomit the related figures due to space constraint.

7 Conclusion

Scaling VNF service chains has become an appealingtopic in this era. In this paper, we propose ScalIMS, ascaling system designed to operate in geo-distributed dat-acenter for scaling IMS system. We evaluate our proto-type implementation on both IBM SoftLayer cloud andam emulated cluster. The evaluation result shows that:First, ScalIMS is able to improve user traffic quality bya large margin when compared with pure reactive scal-ing approach. Second, when peak workload arrives asyn-chronously, ScalIMS is able to decrease total number ofVNF instances while providing excellent user traffic qual-ity. Third, ScalIMS is very efficient in avoiding inter-datacenter network link with large delays. Finally, weshow that distributed routing framework of ScalIMS ac-curately route flows across geo-distributed datacenters.

12

References[1] 3GPP specification: 23.228. http://www.

3gpp.org/ftp/Specs/html-info/23228.htm.

[2] Floodlight OpenFlow Controller. http://www.projectfloodlight.org/floodlight/.

[3] Intel Data Plane Development Tool Kit. http://dpdk.org/.

[4] IP Location Finder. https://www.iplocation.net/.

[5] LTE. http://www.3gpp.org/technologies/keywords-acronyms/98-lte.

[6] Network functions virtualisation. http://www.3gpp.org/DynaReport/23228.htm.

[7] PJSIP - Open Source SIP, Media, and NAT TraversalLibrary. http://www.pjsip.org/.

[8] Project Clearwater. http://www.projectclearwater.org/.

[9] SIP: Session Initiation Protocol. https://www.ietf.org/rfc/rfc3261.txt.

[10] Snort intrusion detection system. https://www.snort.org/.

[11] SoftLayer. www.softlayer.com.

[12] SoftLayer Global Private Network. http://www.softlayer.com/network.

[13] The Evolved Packet Core.http://www.3gpp.org/technologies/keywords-acronyms/100-the-evolved-packet-core.

[14] Universal Mobile Telecommunications System.http://www.3gpp.org/technologies/keywords-acronyms/103-umts.

[15] Virtual eXtensible Local Area Network (VXLAN):A Framework for Overlaying Virtualized Layer 2Networks over Layer 3 Networks. https://tools.ietf.org/html/rfc7348.

[16] J. W. Anderson, R. Braud, R. Kapoor, G. Porter,and A. Vahdat. xomb: extensible open middle-boxes with commodity servers. In Proceedings ofthe eighth ACM/IEEE symposium on Architecturesfor networking and communications systems, pages49–60. ACM, 2012.

[17] B. Anwer, T. Benson, N. Feamster, and D. Levin.Programming slick network functions. In Proceed-ings of the 1st ACM SIGCOMM Symposium on Soft-ware Defined Networking Research, page 14. ACM,2015.

[18] G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M.Ljung. Time series analysis: forecasting and con-trol. John Wiley & Sons, 2015.

[19] M. Dobrescu, N. Egi, K. Argyraki, B.-G. Chun,K. Fall, G. Iannaccone, A. Knies, M. Manesh, andS. Ratnasamy. Routebricks: exploiting parallelismto scale software routers. In Proceedings of theACM SIGOPS 22nd symposium on Operating sys-tems principles, pages 15–28. ACM, 2009.

[20] A. Gember, R. Grandl, A. Anand, T. Benson, andA. Akella. Stratos: Virtual middleboxes as first-classentities. UW-Madison TR1771, 2012.

[21] J. Hwang, K. Ramakrishnan, and T. Wood. Netvm:high performance and flexible networking usingvirtualization on commodity platforms. Networkand Service Management, IEEE Transactions on,12(1):34–47, 2015.

[22] E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F.Kaashoek. The click modular router. ACM Transac-tions on Computer Systems (TOCS), 18(3):263–297,2000.

[23] J. Martins, M. Ahmed, C. Raiciu, V. Olteanu,M. Honda, R. Bifulco, and F. Huici. Clickos andthe art of network function virtualization. In Pro-ceedings of the 11th USENIX Conference on Net-worked Systems Design and Implementation, pages459–473. USENIX Association, 2014.

[24] N. McKeown, T. Anderson, H. Balakrishnan,G. Parulkar, L. Peterson, J. Rexford, S. Shenker, andJ. Turner. Openflow: enabling innovation in campusnetworks. ACM SIGCOMM Computer Communica-tion Review, 38(2):69–74, 2008.

[25] S. Palkar, C. Lan, S. Han, K. Jang, A. Panda, S. Rat-nasamy, L. Rizzo, and S. Shenker. E2: a frame-work for nfv applications. In Proceedings of the 25thSymposium on Operating Systems Principles, pages121–136. ACM, 2015.

[26] L. Rizzo. Netmap: a novel framework for fast packeti/o. In 21st USENIX Security Symposium (USENIXSecurity 12), pages 101–112, 2012.

[27] L. Rizzo, G. Lettieri, and V. Maffione. Speedingup packet i/o in virtual machines. In Proceedingsof the ninth ACM/IEEE symposium on Architectures

13

for networking and communications systems, pages47–58. IEEE Press, 2013.

[28] V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, andG. Shi. Design and implementation of a consoli-dated middlebox architecture. In Presented as partof the 9th USENIX Symposium on Networked Sys-tems Design and Implementation (NSDI 12), pages323–336, 2012.

[29] J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy,S. Ratnasamy, and V. Sekar. Making middleboxessomeone else’s problem: network processing as acloud service. ACM SIGCOMM Computer Commu-nication Review, 42(4):13–24, 2012.

[30] A. Tootoonchian, S. Gorbunov, Y. Ganjali,M. Casado, and R. Sherwood. On controllerperformance in software-defined networks. Hot-ICE, 12:1–6, 2012.

[31] T. Wood, P. J. Shenoy, A. Venkataramani, and M. S.Yousif. Black-box and gray-box strategies for virtualmachine migration. In USENIX NSDI, 2007.

14