Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

it 4/2013

Special Issue ���

A Federated CloudNet Architecture:The PIP and the VNP RoleEine föderierte CloudNet-Architektur: Die PIP und die VNP Rolle

Johannes Grassler, Gregor Schaffrath, Stefan Schmid∗, TU Berlin & T-Labs

∗ Correspondence author: stefan@net.t-labs.tu-berlin.de

Summary We present a generic and flexible architectureto realize CloudNets: virtual networks connecting geograph-ically distributed cloud resources (such as storage or CPU)with resource guarantees. Our architecture is federated andsupports different (and maybe even competing) economicalroles, by providing explicit negotiation and provisioning inter-faces. Contract-based interactions and a resource descriptionlanguage that allows for aggregation and abstraction, pre-serve the different roles’ autonomy without sacrificing flexi-bility. Moreover, since our CloudNet architecture is pluginbased, essentially all cloud operating systems (e. g., Open-Stack) or link technologies (e. g., VLANs, OpenFlow, VPLS)can be used within the framework. This paper describes tworoles in more detail: The Physical Infrastructure Providers (PIP)which own the substrate network and resources, and the Vir-tual Network Providers (VNP) which can act as resource andCloudNet brokers and resellers. Both roles are fully imple-mented in our wide-area prototype that spans remote sites andresources. ��� Zusammenfassung Wir beschreibeneine generische und flexible Architektur, um Cloud Netzwerke

(kurz: CloudNets) zu realisieren. CloudNets sind virtuelle Netz-werke, die weltweit verteilte Cloud-Ressourcen verbinden (mitQualitätsgarantien). Unsere Architektur ist föderiert und unter-stützt mehrere (eventuell sogar konkurrierende) ökonomischeSpieler, indem sie explizite Verhandlungs- und Administra-tionsschnittstellen anbietet. Durch vertragsbasierte Interaktio-nen und eine Ressourcenbeschreibungssprache, welche Aggre-gationen und Abstraktionen zulässt, ist die Autonomie derunterschiedlichen Rollen ohne Flexibilitätsverluste gewährleis-tet. Durch einen Plugin-basierten Aufbau kann die CloudNets-Architektur mit beliebigen Cloud-Betriebssystemen (z. B. Open-Stack) und Linktechnologien (z. B. VLANs, OpenFlow, VPLS)umgesetzt werden. Dieser Artikel befasst sich insbesondere mitden folgenden zwei Rollen: Dem Provider der physikalischenInfrastruktur (PIP), welcher das Substrat und dessen Ressourcenbesitzt und unterhält, und dem Virtuellen Netzwerk Provider(VNP), welcher als Ressourcen- oder CloudNet-Broker auftretenkann. Die VNP-Rolle kann dabei sogar rekursiv sein. Sowohl diePIP- als auch die VNP-Rolle sind in unserem wide-area Prototypvollständig implementiert.

Keywords ACM CSS → Networks → Network architectures; Computer Networks, Virtualization, Cloud Computing ���Schlagwörter Computernetzwerke, Virtualisierung, Cloud Computing

1 IntroductionThe virtualization paradigm is arguably the main in-novation motor in today’s Internet. Especially servervirtualization (a. k. a. node virtualization) revolutionizedthe server business over the last years, and in today’sclouds, resources are fully virtualized. However, node

virtualization alone is meaningless if no access to thecloud resources is provided: to ensure performance andisolation guarantees, virtualization needs to be extendedto the communication network. Software-defined networks(SDN) are one way to implement network and link vir-tualization [3].

it – Information Technology 55 (2013) 4 / DOI 10.1524/itit.2013.1007 © Oldenbourg Wissenschaftsverlag 155

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

Special Issue

The concept of CloudNets (short for cloud networks)takes the virtualization paradigm one step further, envi-sioning a unified approach which combines node andlink virtualization and offers Quality-of-Service (QoS)guarantees, both on the nodes and the links. Basically,a CloudNet describes a virtual network topology wherethe virtual nodes represent cloud resources (e. g., storageor computation) which are connected by virtual links.

While CloudNet concepts are already emerging in thecontext of data centers (e. g., [1; 2]), we expect that inthe future, Internet Service Providers (ISP) will also of-fer flexibly specifiable and on-demand virtual networks,connecting (heterogeneous) cloud resources with con-nectivity guarantees. The CPU and storage resources cannot only be distributed over large datacenters, but mayalso be located, e. g., together with the ISP’s DSLAMs inthe central offices or Points-of-Presence (PoP), or even inthe street cabinets (in “nano-datacenters”). Hosting (andmigrating) services close to the users is known to increaseproductivity and revenues: e. g., a better web performancedirectly translates into higher productivity at Google, andhigher revenue at Microsoft Bing or Amazon [4].

Besides statically connecting distributed resources, anelastic wide-area CloudNet connectivity can be attractive,e. g., for inter-site data transfers or state synchroniza-tion of a distributed application (such as online gaming).Other use cases are the spill-over (or out-sourcing) to thepublic cloud in times of resource shortage in the privatedata center, or the distribution of content to caches.

Note that a CloudNet may not even specify the loca-tions of its constituting resources, and the mapping of theCloudNet to the physical network can hence be subject tooptimization. The resources of the CloudNets may evenbe migrated to different locations over time. For example,latency-critical CloudNets (e. g., realizing a game, an SAP,or a social networking service) can be dynamically mi-grated together with the demand of the (mobile) users(“move with the sun”), while delay-tolerant CloudNets(e. g., for large-scale computations or bulk data storage)are run on physical servers remote from the users (“movewith the moon”). Concretely, the migration of (parts of)a CloudNet can involve the movement of one or multiplevirtual nodes (“virtual machines”); however, sometimesit may be sufficient to migrate virtual links only (e. g.,finding alternative routes in the physical network).

Moreover, resources allocated to a CloudNet can bescaled up or down depending on the demand at thedifferent sites. Finally, the decoupling of the CloudNetfrom the underlying physical infrastructure can also im-prove reliability, as networks can seamlessly switch toalternative cloud and link resources after a failure, or formaintenance purposes.

We expect that several different economic players willbe involved in the provisioning of CloudNets. Besides theservice and infrastructure provider, new business modelsmay emerge, for instance, resource brokers that resellCloudNet resources.

ContributionThe scientific contribution of this paper is the presen-tation of an architecture which supports the CloudNetnegotiation between different economic roles while re-specting their autonomy. In particular, we have alsodescribed interfaces and communication protocols be-tween the physical infrastructure provider and the virtualnetwork provider roles. As a contribution to the researchcommunity, we also present an open-source prototypeimplementation of these concepts.

In particular, we present the anatomy and prototypeimplementation of a flexible and federated CloudNet ar-chitecture. We will describe the Physical InfrastructureProvider (PIP) role and the Virtual Network Provider(VNP) role in more detail. The two roles communicaterequirements and allocations via clear negotiation inter-faces and using a generic resource description language.Since interfaces between players are generic and sincewe do not distinguish between physical and virtual re-sources, a recursive role concept is supported (e. g., a VNPsub-structured into other VNPs). Moreover, a high gen-erality is achieved by a plugin architecture which allowsfor replacement of underlying technologies and operatingsystems.

In contrast to the distributed CloudNet architecturesby Baldine et al. [5] (whose control framework was partlydeveloped in NSF GENI project) or Ericsson’s NetworkEmbedded Clouds [6], our approach seeks to keep (and ex-ploit) the embedding and resource allocation flexibilitiesin the service specification without violating the auton-omy of the different stakeholders.

2 ArchitectureWe first give some background on the CloudNet visionand on network virtualization [7] in general.

The Road to CloudNetsInter-provider QoS connectivity has been discussed formany years now but it is still not widely supported. Sowhy should wide-area and multi-provider CloudNets be-come a reality now? We believe that the answer lies in theeconomical incentives, and the recent virtualization andSoftware-Defined Networking (SDN) trends (e. g., [3]).Content and service providers (e. g., Netflix, Amazon,etc.) as well as content distribution network providers(e. g., Akamai) have become powerful players in the Inter-net and have stringent resource requirements. CloudNetscan help an ISP to monetarize its infrastructure, not onlyby sharing resources but also by offering a flexible servicedeployment. For example, in the architecture describedin this paper, physical infrastructure providers participatein the service negotiation, which allows them to becomea service partner at eye-level (rather than being a purebitpipe provider).

We expect that CloudNets will first be deployed in-side a single ISP. Such an innovative ISP may benefitfrom a more efficient resource management, and may

156

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

A Federated CloudNet Architecture: The PIP and the VNP Role ���

have a first-mover advantage by offering new flexible andelastic services. Over time, it can make sense for so farindependent CloudNet providers to collaborate. For ex-ample, two providers with a local footprint can offer moreglobal CloudNets.

Economical RolesThe advent of CloudNets may create a tussle amongservice and infrastructure providers over who shouldoperate and who should manage the corresponding net-works. In [8], we identified two additional roles (i. e.,potential market players) besides Physical InfrastructureProviders (PIPs) and Service Providers (SPs): Virtual Net-work Providers (VNPs) (essentially “resource brokers”)for assembling virtual resources from one or multiplePIPs into a virtual network, and Virtual Network Op-erators (VNOs) for the installation and operation of theCloudNet provided by the VNP according to the needs ofthe SP. Our CloudNet architecture defines standardizedinterfaces between the players to automate the setup ofvirtual networks (by using a common control plane).

2.1 Architecture & Prototype

Specification and SpecificityOur architecture is based on a Resource and CloudNetDescription Language (RDL) [9] (a. k. a. FleRD) which re-volves around basic NetworkElement (NE) objects (forboth nodes and links!) that are interconnected via Net-workInterfaces (NI) objects. Keeping these objects generichas the side effect that descriptions of resource aggrega-tions, or non-standard entities (e. g., clusters of providers)is trivially supported: they may be modeled as NEs of anappropriate type and included as topological elements.Thus, we can for example also describe mappings in thecontext of a reseller. Concretely, NE properties are rep-resented as a set of attribute-value pair objects labeled asResource and Features. The meaning of resources here iscanonic and resources may be shared amongst NEs. Fea-tures represent any type of property that is not a resource(e. g., CPU architecture).

One challenge in the communication of specificationsand RDL elements across different roles is to ensure con-sistency, especially for non-topological requirements. Forinstance, imagine a service provider specification onlyrequires that two nodes are binary compatible (e. g., 32or 64 bit). How can a VNP in charge of distributingsuch a CloudNet across multiple PIPs fulfill the specifica-tion? A simple solution would be that the VNP choosesone of the different options (e. g., both 64 bit) beforeforwarding the request to the different PIPs. However,this solution comes with an unnecessary specificity andhence a loss in flexibility: as the VNP has a limited viewon the infrastructure of the PIPs only, the VNP’s choicemay lead to inefficient allocations (e. g., the two nodesare mapped unnecessarily far away). Alternatively, theVNP may assign the corresponding NEs globally unique

IDs which can be used by the PIPs to agree on a choicethemselves; for instance, a unique identifier may con-sist of a unique VNP ID followed by a VNP-chosen NEidentifier. The required communication to achieve thismay go via the VNP again, or occur directly between thePIPs.

PluginsOur architecture is plugin-based to facilitate extensionsand adaptations to new networking and virtualizationtechnologies. Plugin-based operation hinges on a fea-ture of the resource description language, namely typedNetworkElements: The NetworkElement objects all havea hierarchical attribute field (this also applies to the Net-workInterface, Resource and Feature objects). This fieldis a string of hierarchy levels, like a path in a Unix filesystem (e. g., /node/host/generic).

Much like a CloudNet, any player’s substrate canbe described in terms of the RDL. Hence all substrateNEs and all CloudNet NEs will have an attribute field.The higher order hierarchy levels of a CloudNet’s NEstype will be used in the course of the mapping processto determine which substrate NEs they can be allo-cated to (i. e., both /node/host/mainframe and/node/host/server might be suitable allocationsfor a /node/host/generic CloudNet NE).

Once a CloudNet’s resources have been mapped toa player’s substrate resources, the lower levels of the sub-strate NE’s type will be used to determine which pluginto use to provision the CloudNet NEs in question. De-pending on the substrate NetworkElement’s hardware orsoftware (e. g., Layer 2 VPNs, Ethernet, OpenFlow, MPLS,mainframes, VM hosts), the appropriate embedding plu-gin can be chosen to embed all the CloudNet’s resourcesmapped to the substrate element in question.

In conclusion, attributes are used for two purposes:1) The attribute determines which substrate NE typesa request’s NEs can be mapped to. This is done bymatching up the upper hierarchy layers in the NE’s type,(e. g., /node/host in the example) against a list ofavailable substrate network element types. And 2), givena mapping from a request graph’s NE to a substrateNE, the substrate NE’s type determines which embed-ding plugin is used to implement the requested virtualNE.

PrototypeThe (open-source) CloudNets prototype is a proof-of-concept implementation of the architecture describedabove. Currently, the prototype implements the PIP busi-ness role (the infrastructure provider) as well as the VNPbusiness role (the resource broker). As a link virtual-ization plugin we use tagged VLANs: each virtual linkis realized as a VLAN. The VLANs provide isolation andenable the demultiplexing of frames to the correspondingvirtual machines (to which the connection looks like anEthernet link).

157

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

Special Issue

In the following, we will first give an overview of thetestbed and then discuss the prototype. In the context ofthe prototype, we will refer to virtual nodes as VNodes andto virtual links as VLinks. Moreover, we will sometimesdistinguish between the mapping of a CloudNet (i. e.,where to allocate the CloudNet in the substrate) and theembedding of a CloudNet (the actual implementation ofsaid allocation).

TestbedOur network virtualization prototype is run on two sep-arate testbed environments, one at TU Berlin and oneat NTT DoCoMo Eurolabs, Munich. Both testbeds havea Cisco 4500 series switch carrying both virtual net-work data plane VLANs and testbed management VLANs.Moreover, both testbeds consist of Sun X4150 machineshosting both substrate resources and the virtual machinesrunning the various roles’ management software. Virtualnodes can either be fully virtualized KVM virtual ma-chines where physical substrate nodes are available, orparavirtualized Xen virtual machines.

The substrate resource allocation for the CloudNetembeddings is computed with an optimized Mixed In-teger Program (MIP). Once the resource allocation isdetermined, the physical infrastructure provider createsvirtual machines, establishes the links between them andhands control over to the customer (by providing consoleaccess to the virtual nodes for instance). At the heart ofthe prototype lie MySQL databases where each providerkeeps all information about its substrate topology andembedded virtual networks.

The Embedding PluginIn order to embed CloudNets on the given substrate top-ology, we use a two-stage approach: first, a fast heuristicis used to initially place the CloudNet; subsequently,a Mixed Integer Program plugin (short MIP) is usedto optimize long-lived and “heavy-hitter” CloudNets inthe background, migrating them to a new location [10].In the following, we will focus on the second stageonly; for an overview of fast embedding heuristics, werefer the reader to [11–13], and especially the sur-veys [14; 15]).

The mathematical programming approach is attractiveas it is very general and allows us to specify many differenttypes of embeddings and embedding constraints. Espe-cially embeddings on arbitrary Internet infrastructureswith heterogeneous resources and links requires a highflexibility. Our MIP supports a simple replacement of theoptimization function, without the need to redesign theembedding algorithm. For instance, one possible objec-tive function may arise in scenarios where a CloudNetshould be embedded in such a manner that the maximalload or congestion is minimized. In other scenarios, theCloudNets should be embedded in a compact manner(subject to collocation constraints) in order to be able toshut down the other parts of the physical network, e. g.,

to save energy. Moreover, our MIP plugin also supportsthe migration of (parts of) a CloudNet: As the Cloud-Net requests arriving over time may be hard to predict,certain embeddings computed online in the past maybecome suboptimal, and re-embeddings and migrationsare necessary.

By computing the embedding that would result aftera migration together with the migration cost, the MIPallows us to determine, e. g., the cost-benefit tradeoff ofmigration. For example, our algorithm allows to an-swer questions such as: Can we migrate CloudNets toa more compact form such that 20% of the currently usedresources are freed up, and what would be the corres-ponding migration cost? It is then left to the (potentiallyautomated) administrator to decide whether the changesare worthwhile.

3 PIP and VNP RolesIn the following, we will focus on the PIP (Physical In-frastructure Provider) role and the VNP (Virtual NetworkProvider) role, and will describe their implementation.The VNO (Virtual Network Operator) and SP (ServiceProvider) are not implemented in our prototype yet.

Graph SerializationTo permit interactions between PIP and VNP roles, a firstand important task to be solved is graph serialization.Since the management nodes for the VNP and the PIPsin its substrate graph all run on dedicated machines withdedicated databases, graphs representing the CloudNettopologies have to be serialized into a database indepen-dent representation to be sent across the network. To thisend we developed a YAML representation of the resourcedescription language.

3.1 The PIPThe PIP role supports arbitrary topology specificationand topology modification including live migration ofrunning nodes (within a PIP’s substrate). It mainly con-sists of a Mapping Component that handles the MIPformulation, the result interpretation and Mapping Layergraph generation, and an extensible (plugin based) Em-bedding Component that embeds a (partial) CloudNettopology on the PIP’s substrate given its overlay andmapping layer graphs. Figure 1 (left) gives an overview.

SubstrateThe PIP’s substrate is largely virtualized in our testbedin Berlin: most substrate nodes are KVM virtual ma-chines running on the testbed’s Sun X4150 machines. TheX4150s are interconnected by an IEEE 802.1q-capableswitch. The virtualized substrate nodes are assignedunique Ethernet addresses and connected via physicalEthernet interfaces. The CloudNet VNodes are then cre-ated on aforementioned KVM machines. Each virtual linkis assigned a unique VLAN tag (picked by the PIP fromits assigned range of VLAN tags). These VLAN tags are

158

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

A Federated CloudNet Architecture: The PIP and the VNP Role ���

Figure 1 Left: The CloudNets architecture as implemented by the PIP role. Right: Stages of a CloudNet request: from VNP to PIP.

set on all data plane interfaces (or switch ports) of thesubstrate nodes on the virtual link.

In addition, some of the X4150 machines are config-ured to serve as substrate nodes themselves, on top ofhosting virtualized substrate nodes. They can host fullyvirtualized KVM VNodes (as opposed to the virtualizedsubstrate nodes which can only host paravirtualized XenSNodes).

All substrate nodes store their VM images on anNFS share exported by their controlling PIP. PIPs areresponsible for creating VM images and hypervisorconfigurations, while the substrate nodes’ provisioningscripts take care of the rest.

Interconnections between links that do not involveVNodes are realized by creating bridge devices on ded-icated Tunnel Bridge (TBR) nodes. Unlike regularsubstrate nodes, these do not run OpenVSwitch but theregular Linux bridge. They can host virtual switches andterminate OpenVPN tunnels.

In particular these bridge devices are used to establishVLinks spanning across PIP boundaries (transit links):They interconnect the outside segment of these links(within the specifying VNP’s management scope) withthe internal segment of these links (within the PIP’s man-agement scope). The mechanism behind this is based onOpenVPN TAP style layer two tunnels: These are VPNtunnels terminating in virtual Ethernet devices on bothtunnel bridges involved in the transit link. Said virtualinterfaces trunk all VLinks embedded on the transit linkin question and are bridged to the tunnel bridges’ dataplane interface.

3.1.1 Cloud Operating SystemThe prototype provides similar services as modern CloudOperating Systems:

Automated VM ProvisioningThe Embedding Component automatically creates VMimages for VNodes (typically by making copies of a tem-plate VM image suitable for the hosting substrate node’shypervisor) and hypervisor configuration files. With ev-erything in place these VMs can be started through anXMLRPC interface to the hosting substrate node’s pro-visioning scripts. This interface offers a range of basicVM provisioning functionalities such as start, stop, andpowercycle.

VM Image CachingSince the I/O performance of the X4150 machines israther poor and there is plenty of storage space availablewe developed a caching scheme to speed up CloudNetembedding. There is a daemon, clipd, that maintainsa cache of VM images on every PIP’s exported NFSshare (on disk). The Embedding component can requestVM images from this daemon’s cache, which are thenbe moved to their final location on the NFS share, thusgreatly speeding up the whole embedding process. Theclipd cache is replenished by a cron job while the Map-ping Layer/Embedding is inactive.

Customer Console Access to VNodesSince placement of VNodes is rather opaque to a PIP’scustomers – and not relevant as long as their require-

159

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

Special Issue

ments are met – there is a mechanism to automaticallylook up any given VNode’s hosting substrate node, andoffer a proxied serial console session on that very VNodeto a customer requesting access.

Automated Link ProvisioningVirtual links are provisioned and brought up automati-cally. In the Mapping Layer graph they are split up intosegments, one for each substrate network element theyare mapped to (for instance the two substrate nodeson either end and the switch port the substrate nodes’data plane interfaces are connected to). For each of thesecomponents appropriate Embedding Plugin is called. Todate there are Embedding Plugins handling OpenVSwitchbased hosts, traditional Linux Bridge based hosts andCisco 4500 series switches (the latter are configured viaSNMP).

3.1.2 Negotiation and Provisioning InterfaceA PIP controls its substrate nodes and (physical) switchthrough an XMLRPC configuration interface. The PIP inturn offers two interfaces to the VNP role: The Negotia-tion Interface and the Provisioning Interface. Both consistof a range of XMLRPC methods.

Negotiation follows a two-stage protocol: The Negoti-ation Interface allows for sending preliminary embeddingrequests (and potentially prices) for either topology cre-ation or modification of previously embedded topologies.Once a VNP is sure it wants to embed a topology perma-nently (i. e., after receiving the ‘okays’ from all involvedPIPs), it will send a confirmation request for the prelim-inary topology.

The VNodes of an embedded topology can then bestarted and stopped through the Provisioning Interface.

3.2 The VNPThe VNP’s high-level architecture is similar to the PIP’s,also comprising a Mapping Component and an Embed-ding Component. The Mapping Component is for themost part identical to the PIP’s. The major differences arein the embedding component: (1) since the VNP’s sub-strate consists of entire PIPs rather than physical nodesand links, the VNP has its own set of embedding plug-ins to interact with PIPs’ Negotiation Interfaces; (2) theEmbedding Component contains additional code for thecreation of partial graphs which can be mapped to a singlePIP.

In order to generate the partial PIP networks, the VNPfirst constructs a “helper graph” consisting of all PIPs ithas contracts with. For this helper graph, the PIP net-works are “guessed” by the VNP given the information ithas about them. (In our prototype, we simply aggregatethe PIP’s memory capacities.) The VNP then embedsthe CloudNet onto the helper graph to determine whichNEs should be hosted on which PIP. The NEs mappedto a single PIP then form the partial graph of this PIP,

which is handed down to the infrastructure provider forthe final embedding.

SubstrateThe VNP’s substrate consists of PIP nodes interconnectedby transit links1. Hence the VNP is aware of the topologybetween PIPs (or a subset of said topology) but does notknow anything about the PIPs’ internal topology.

Negotiation and Provisioning InterfaceA library on the VNP level implements the client sidefunctionality of the PIP’s Negotiation and ProvisioningInterface. It is used to send partial topology graphs toPIPs and embed them.

4 Life of a CloudNet RequestFigure 1 (right) gives a graphical outline of the life ofa CloudNet.

Incoming CloudNet RequestThe CloudNet request in the form of an ‘overlay graph’(henceforth referred to as OL0 graph) is submitted to theVNP (as a serialized topology graph, through an XML-RPC interface similar to the PIP’s Negotiation Interface).Subsequently, the VNP’s mapping process is started.

Substrate SynchronizationThe first step in mapping a CloudNet request is substratesynchronization: The VNP uses the Negotiation Inter-face to update the available resources (and their costs)on the PIPs constituting its substrate. In order to bothprotect PIP’s business secrets (namely the details of itssubstrate topology), and give the VNP a rough approx-imation of a given PIP’s available resources, we simplysum up all the resources of a given type in the PIP’ssubstrate. These aggregate resources will then be assignedto the /node/host/pip NE representing this PIP inthe VNP’s substrate graph.

Solution of VNP-level MIPBefore proceeding with the mapping process, the VNPcompletes the overlay OL0 graph, yielding an OL1 graph.This is to account for any vagueness on the upstreamentity’s part: e. g., values are chosen for those requiredresources and features that were left unspecified. For in-stance the virtualization technology might not have beenspecified, allowing the VNP to make a choice of its ownor pass the the vague graph on to downstream players,who will in turn have the freedom of choice. Next itformulates the MIP for mapping the incoming CloudNetrequest to its substrate of PIPs. The MIP is based uponthe following variables: (1) the incoming OL0 graph’sresource and feature requirements and constraints, (2)the available resources and features on the PIPs consti-

1 In our prototype, these are represented by OpenVPN (seehttp://openvpn.net/) tap style tunnels between the PIPs’ tunnel bridgenodes.

160

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

A Federated CloudNet Architecture: The PIP and the VNP Role ���

tuting the VNP’s substrate (as far as they are known tothe VNP), and (3) the available resources and features onthe transit links between PIPs.

Once the MIP has been solved the result is translatedback into a mapping layer graph, mapping all of the OL1graph’s Network Elements (NEs) to PIPs and transit links.

Partial Graph GenerationNow the VNP iterates through the OL1 graph and createsa set of partial graphs, one for each PIP. Each of thesepartial graphs consists of all the NEs mapped to the PIPin question.

In the course of partial graph generation, stubs fortransit links are created. These consist of two spe-cial network element types: /node/host/pip and/link/transit.

The /link/transit NE specifies the VLAN to usefor this transit link (we opted for letting the VNP picktransit VLANs to avoid implementing an error prone PIP-to-PIP negotiation mechanism for this purpose). Thisclosely matches the most likely real-world scenario, too:A VNP might have an embedding plugin for a transitprovider (for instance someone running an MPLS net-work) with black boxes located at both PIPs’ networkedges. This embedding plugin requests a transit link be-tween these two PIPs and receives some kind of linkidentifier (such as a VLAN tag). The VNP then com-municates this link identifier to the PIPs on either end,which allows both PIPs to connect the black box to theirlocal segment of the virtual link.

Serialization and TransmissionOnce the partial graphs have been assembled, each ofthem is serialized and sent to its hosting PIP throughthe Negotiation Interface. This currently takes place ina serial manner in our prototype, i. e., partial graphs aresent to PIPs one after the other. If one of the PIPs in thechain turns out to be unable to embed its partial graphthe VNP rolls back the already embedded partial graphs(deletes them) and reports a failure status to the upstreamentity that sent the original CloudNet request.

Formulation and SolutionUpon receipt of a partial graph a PIP will – much likethe VNP – complete the OL0 graphs to yield OL1 graphs,and formulate the MIP for mapping the incoming partialCloudNet request to its substrate of physical hardware.The program is based on the following variables: (1) theincoming partial graph’s resource and feature require-ments and constraints, (2) the available resources andfeatures on this PIP, and (3) the available resources andfeatures on this PIP’s outgoing transit links leading todestination.

Once the MIP has been solved the result is trans-lated back into a mapping layer graph, mapping all ofthe OL1 graph’s Network Elements to substrate links andnodes. Control passes to the Embedding Component now

●●●●●

●

●

●

●

●

AP1(link)

AP2(link)

AP3(link)

AP1(node)

AP2(node)

AP3(node)

010

2030

40

Data setsD

ownt

ime

dura

tion

(s)

Figure 2 Link vs. node migration.

which will configure the topology’s VLANs on the switchas specified by the Mapping Component and create theVNodes’ images and hypervisor configuration. With ev-erything in place the PIP reports success to the VNPwhich sent the (partial) CloudNet creation request.

Bootup and HandoffOnce success has been reported back to the VNP by allPIPs involved in the CloudNet creation request, the VNPconfirms all these requests through the PIPs’ NegotiationInterface, uses the PIPs’ Provisioning Interface to start thenewly created VNodes (the links will already have beenbrought up in the course of negotiation). Finally, theVNP hands control over the CloudNet to the requestingupstream entity (typically in the form of console inter-faces).

PerformanceFigure 2 shows the performance (downtime) of migrat-ing a live streaming server (size 1 GB) in our prototype(between 3 access points): (1) when just virtual links aremigrated, (2) when the server migrates between physi-cal machines, possibly displacing other, latency-uncriticalCloudNets’ nodes. Link migration turned out to performworse in terms of service outage than node migration.Node migration, on the other hand performed muchworse than link migration in terms of total migrationduration (node migration took a median of 225 s, versusonly a median 23 s for link migration).

References

[1] G. Wang and E. Ng. The impact of virtualization on network per-formance of Amazon EC2 data center. In Proc. IEEE INFOCOM,2010.

[2] C. Wilson, H. Ballani, T. Karagiannis, and A. Rowtron. Betternever than late: meeting deadlines in datacenter networks. In Proc.ACM SIGCOMM, 2011.

161

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

Special Issue

[3] D. Drutskoy, E. Keller, and J. Rexford. Scalable network virtual-ization in software-defined networks. In Proc. Internet Computing,2012.

[4] Bing News. http://velocityconf.com/velocity2009/public/schedule/detail/8523. In Website, 2012.

[5] I. Baldine, Y. Xin, A. Mandal, C. Renci, U. Chase, V. Marupadi,A. Yumerefendi, and D. Irwin. Networked cloud orchestration:A geni perspective. In GLOBECOM Workshops, 2010.

[6] James Kempf et al. The network embedded cloud. In TechnicalReport Ericsson Research, 2012.

[7] M. K. Chowdhury and R. Boutaba. A survey of network virtual-ization. Computer Networks, 2010.

[8] G. Schaffrath, C. Werle, P. Papadimitriou, A. Feldmann, R. Bless,A. Greenhalgh, A. Wundsam, M. Kind, O. Maennel, and L. Mathy.Network virtualization architecture: Proposal and initial proto-type. In Proc. ACM VISA, 2009.

[9] G. Schaffrath, S. Schmid, I. Vaishnavi, A. Khan, and A. Feldmann.A resource description language with vagueness support for multi-provider cloud networks. In Proc. ICCCN, 2012.

[10] G. Schaffrath, S. Schmid, and A. Feldmann. Optimizing long-livedcloudnets with migrations. In Proc. 5th IEEE/ACM UCC, 2012.

[11] K. Chowdhury, M. R. Rahman, and R. Boutaba. Virtual networkembedding with coordinated node and link mapping. In Proc.IEEE INFOCOM, 2009.

[12] J. Fan and M. H. Ammar. Dynamic topology configuration inservice overlay networks: A study of reconfiguration policies. InProc. IEEE INFOCOM, 2006.

[13] S. Zhang, Z. Qian, J. Wu, and S. Lu. An opportunistic resourcesharing and topology-aware mapping framework for virtual net-works. In Proc. IEEE INFOCOM, 2012.

[14] A. Belbekkouche, M. Hasan, and A. Karmouch. Resource discov-ery and allocation in network virtualization. IEEE CommunicationsSurveys Tutorials, (99):1–15, 2012.

[15] A. Haider, R. Potter, and A. Nakao. Challenges in resource alloca-tion in network virtualization. In Proc. ITC Specialist Seminar onNetwork Virtualization, 2009.

Received: March 8, 2013

Preview on issue 5/2013

The topic of the next issue is “Open Source Software” (Guest Editor: D. Riehle) and will containthe following articles:

• A. Wassermann: Community and Commercial Strategies in Open Source Software

• W. Mauerer: Open Source Engineering Processes

• N. Kimmelmann: An Open Source Career? Relevant Competencies for Successful OpenSource Developers

• St. Koch: Open Source Community Processes: Implications on Micro and Macro Level

• J. M. Gonzalez-Barahona and Gr. Robles: Trends in Free, Libre, Open Source SoftwareCommunities: From Volunteers to Companies

Johannes Grassler studies applied computer sci-ence at Beuth-Hochschule für Technik Berlin. Heis a student worker with FG INET since February2011. Before that, he worked for several years atGoogle and Kabel Deutschland. His professionalinterests include Unix systems programming andnetwork virtualization.

Address: TU Berlin & T-Labs, Ernst ReuterPlatz 7, 10587 Berlin, Germany,e-mail: jgrassler@net.t-labs.tu-berlin.de

Dr. Gregor Schaffrath studied computer sci-ence at University of Saarland (minor: Physics).He worked for several years at IFI Zurich, be-fore becoming a PhD student at FG INET (T-Labs). His research interests include network vir-tualization and network-based intrusion detec-tion.

Address: TU Berlin & T-Labs, Ernst ReuterPlatz 7, 10587 Berlin, Germany,e-mail: grsch@net.t-labs.tu-berlin.de

Dr. Stefan Schmid studied computer science atETH Zurich (minor: micro/macro economics,internship: CERN) and received his PhD de-gree from the Distributed Computing Group(Prof. Roger Wattenhofer), also at ETH Zurich.Subsequently, he worked with Prof. ChristianScheideler at the Chair for Efficient Algorithms atthe Technical University of Munich (TUM) andat the Chair for Theory of Distributed Systems atUni Paderborn. He is now a senior research sci-entist at Telekom Innovation Laboratories Berlin.Stefan Schmid is interested in distributed systems,and especially in the design of robust and dy-namic networks.

Address: TU Berlin & T-Labs, Ernst ReuterPlatz 7, 10587 Berlin, Germany,e-mail: stefan@net.t-labs.tu-berlin.de

162

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

Ein Unternehmen von De Gruyter

www.degruyter.com/oldenbourg

www.degruyter.com/oldenbourg

Ein Unternehmen von De Gruyter

Th

is a

rticle

is p

rote

cte

d b

y G

erm

an

co

pyrig

ht la

w. Y

ou

may c

op

y a

nd

dis

tribu

te th

is a

rticle

for y

ou

r pers

on

al u

se o

nly

. Oth

er u

se is

on

ly a

llow

ed

with

writte

n p

erm

issio

n b

y th

e c

op

yrig

ht h

old

er.

Ein Unternehmen von De Gruyter

www.degruyter.com/oldenbourg