DIMACS Cloud Workshop, Dec. 8-9, 2011
PACE: Policy-Aware Application Cloud Embedding
Y. Richard Yang
Joint work w/ Erran Li, G. Wilfong, M. F. Nowlan, V. Liaghat, M. Hajiaghayi, H. Zhao, D. Li, C. Guo, and M. Zhang
DIMACS Cloud Workshop, Dec. 8-9, 2011
Motivation• A key aspect of cloud computing is on-demand allocation
of infrastructure resources to construct applications
2
Cloud
On-demand
App
VM
VM
LUN
Data
Center 1
Data
Center 2
VM
VM
VMLUNLUN
LUN
DIMACS Cloud Workshop, Dec. 8-9, 2011
tier-1
Example Real App
3
InternetCE
Tier-1
F1 (Firewall)
S1
LB1 (Load balancer)
IPS1(Intrusion prevention)
F2
S2
LB2
IPS2
S3S4
VLAN 300 VLAN 400
Tier-2
VLAN 100
R1
S5
IPS3
S6
VLAN 200
Tier-3
Logger
DIMACS Cloud Workshop, Dec. 8-9, 2011
App Structure Properties• A real-world application is not a set of isolated
resources (e.g., VMs/LUNs)
• Real world-applications are constructed with external entities (e.g., policy boxes) to enhance functionality,– 95% web applications have severe vulnerabilities
if deployed alone [WASC]
4
DIMACS Cloud Workshop, Dec. 8-9, 2011
Policy-Aware Application Cloud Embedding
• What is it?– Construct structured applications, according to
application policies, in cloud settings
5
DIMACS Cloud Workshop, Dec. 8-9, 2011
Why Worry about PACE?
6
Example 1: Relocating tier-1 servers in VLAN 300 and 400 to cloud
• Issues• IPS1 is used to monitor
cross VLAN traffic. After relocation, cross traffic will not be monitored by IPS1
• Tier-1 servers cannot reach tier-2 servers tier-1
InternetCE
Tier-1
F1 (Firewall)
S1
LB1 (Load balancer)
IPS1(Intrusion prevention)
F2
S2
LB2
IPS2
S3S4
VLAN 300 VLAN 400
Tier-2
DIMACS Cloud Workshop, Dec. 8-9, 2011
Why Worry about PACE?
7
Example 2: Extending VLAN 200 and 100
VLAN 100
R1
S5IPS3
S6
Data Center 1
VLAN 200
u3 v3
VLAN 100VLAN 200
u’3 v’3
S7
L2 Extension Link
Path1
Path2
Data Center 2
DIMACS Cloud Workshop, Dec. 8-9, 2011
PACE Key Components
8
Infrastructure
Orchestrator
App Policy API
App
App
App/Cloud
StatProvisioning System
DIMACS Cloud Workshop, Dec. 8-9, 2011
Policy Specification: Flow Policy Graph
• For each application, introduce Flow Policy Graph (FPG) as a compact, abstract representation of its embedding policy
• A FPG consists of– Nodes: equivalent application-level entities– Edges: QoS/reliability demand and distributed
information flow control (DIFC) policy for traffic between nodes• Focus on DIFC requirements
9
DIMACS Cloud Workshop, Dec. 8-9, 2011
tier-1
Example: Nodes
10
InternetCE
Tier-1
F1 (Firewall)
S1
LB1 (Load balancer)
IPS1 (Intrusion prevention)
F2
S2
LB2
IPS2
S3S4
VLAN 300 VLAN 400
Tier-2
VLAN 100
R1
S5
IPS3
S6
VLAN 200
Tier-3
LoggerVLAN 500
DIMACS Cloud Workshop, Dec. 8-9, 2011
Example: FPG Nodes
11
LB1 Ex
T1-V300 T1-V400T3-V200 T3-V100 T2-V500
LB2
DIMACS Cloud Workshop, Dec. 8-9, 2011
Distributed Information Flow Control
• Graph based information flow control (IFC) representation
– Default “off” policy: no inter-node communications
– Edge DIFC annotation indicates required declassifier(s) to allow inter-node communication• Declassifiers may be either hardware or software
appliances
12
DIMACS Cloud Workshop, Dec. 8-9, 2011
FPG: IFC Declassifier
• A declassifier is identified by its function class, instance configuration, and optional a name, e.g.,– F: a firewall pipe declassifier– IPS: an IPS pipe declassifier– LOG: a replication declassifier
• A declassifier can have internal state– Two declassifiers with the same name is a single
instance
13
DIMACS Cloud Workshop, Dec. 8-9, 2011
Edge Direction and Multiple DIFC Declassifiers
• Edges are directional: the two directions of traffic between two nodes can have different DIFC policies– We call each direction a link– By default, assume that two links of the same edge
have the same DIFC policy
• Multiple DIFC declassifiers may be specified on the same link– The semantics is ordered execution
14
DIMACS Cloud Workshop, Dec. 8-9, 2011
Example: Annotating FPG with DIFC
15
LB1 Ex
T1-V300 T1-V400T2-V500T3-V200 T3-V100
F/acl1;f1
LB2
F/acl1;f1
DIMACS Cloud Workshop, Dec. 8-9, 2011
Example: Annotating FPG with DIFC
16
LB1 Ex
T1-V300 T1-V400
LB2
T2-V500T3-V200 T3-V100
F1
ips1, log1 ips1,
log1
ips1, log1
ips1,
log1
F2
ips1,
log1 F2
Same instance
required for
correlation
DIMACS Cloud Workshop, Dec. 8-9, 2011
Example: Annotating FPG with DIFC
17
LB1 Ex
T1-V300 T1-V400
LB2
T2-V500T3-V200 T3-V100
F1
ips1, log1 ips1,
log1
ips1, log1
ips1,
log1
F2
ips1,
log1 F2
ips1,log1
≠
log1, ips1
DIMACS Cloud Workshop, Dec. 8-9, 2011
Example: Annotating FPG with DIFC
18
LB1 Ex
T1-V300 T1-V400
LB2
T2-V500T3-V200 T3-V100
F1
ips1, log1 ips1,
log1
ips1, log1
ips2
ips2,
F2
ips3
ips3ips2,
F2
ips1,
log1
F2
ips1,
log1 F2
Same ACL used on
4 edges. Does the
app really want it?
DIMACS Cloud Workshop, Dec. 8-9, 2011
From FPG to Cloud Embedding
• Given – A network N;– Applications and their policies represented by
FPGs
• Determine – Forwarding in the network and/or – Allocation of resources to satisfy FPGs.
19
DIMACS Cloud Workshop, Dec. 8-9, 2011
From FPG to Cloud Embedding
• Embedding depends on multiple factors– New app vs migration/extension
– Declassifier deployment locations• In-network devices (e.g., correlating/learning IPS)• End devices (hypervisors)
– Network forwarding mechanisms• Traditional spanning tree, flow table, flow table w/
encapsulation, L2/L3 tunnels, PSwitch
20
DIMACS Cloud Workshop, Dec. 8-9, 2011
Why is Embedding (Theoretically) Hard?
• A Special Embedding Problem: the k-paths-tree problem: – An application with K+1 nodes
• a root R and • a set of source nodes (VMs) S = {s1, s2, . . . , sK}
already deployed– A set of distinct policy function instances M = {x1,
x2, . . . , xK}, whose locations in the network are already deployed
– Policy: Each edge i, 1 ≤ i ≤ K, si -> R needs delcassifier xi
21
DIMACS Cloud Workshop, Dec. 8-9, 2011
NP-Reduction
22
• If tree-based forwarding, it is NP-hard to determine feasibility of
embedding.
• We reduce the NP-complete problem k-node-disjoint-path problem
to our problem
• Given an instance of k-node-disjoint path problem, we compute an
instance of our policy embedding problem
DIMACS Cloud Workshop, Dec. 8-9, 2011
NP-Reduction
23
t1 t2 tK
s1 s2 sK
G
DIMACS Cloud Workshop, Dec. 8-9, 2011
NP-Reduction
24
t1 t2 tK
s1 s2 sK
G
R
x1 x2 xK
NP-hardness still holds even if DC
is fat-tree.
DIMACS Cloud Workshop, Dec. 8-9, 2011
Flow-Table Forwarding Embedding
Assume app entities and declassifiers’ locations at the network are given, what is the condition to make flow-table forwarding only enforcement possible?
25
LB1 Ex
T1-V400
LB2
T2-V500T3-V200 T3-V100
F1
ips1,
log1ips1,
log1
ips1,
log1
ips2
ips2,
F2
ips3
ips3ips2,
F2
ips1,
log1
F2
ips1,
log1 F2
T1-V300
DIMACS Cloud Workshop, Dec. 8-9, 2011
Embedding w/ Flow Table Forwarding
26
• Consider each app edge As → Ad with x1,x2, ..., xK
For notation, let As == x0, Ad == xK+1
As Ad
x1, x2, …, xK
There exist paths p0, ... pK, where path pi is from xi to xi+1, such that
• No policy conflict: Each pi is conflict free, i.e., it does not traverse a declassifer that conflicts with the
given x1 to xK.
• No forwarding conflict: If a switch x belongs (not tail) to multiple paths in {p0, … pK}, then the
incoming ports are distinct among these multiple paths.
DIMACS Cloud Workshop, Dec. 8-9, 2011
Example: Feasible
27
LB1 T1
ips, log
log, ips
LB1 T1
logips
s1
s2
1 2
3
1
2 3
Src Dst inPort OPort
LB1 T1 1 3
LB1 T1 3 2
…
Src Dst inPort OPort
LB1 T1 1 2
LB1 T1 2 3
LB1 T1 3 1
…
DIMACS Cloud Workshop, Dec. 8-9, 2011
Example: Infeasible
28
LB1 T1
ips, log
LB1 log
T1
ips
s1
s2
1 2
3
1
2
3
1
s3
path(LB1→ ips) and path(log → T1)
forwarding conflict on switch 2/port 1
DIMACS Cloud Workshop, Dec. 8-9, 2011
Handling Forwarding Infeasibility
29
• If infeasibility is due to forwarding conflict, and L2 encapsulation is allowed, add next-hop MAC to resolve forwarding conflict.
DIMACS Cloud Workshop, Dec. 8-9, 2011 30
• Previous Proposition assumes that placement of app and declassifiers is given. What if we have placement flexibility?
Placement of App and Declassifiers
Placement
• App entities (VMs) and
• declassifiers
Orchestrator
Forwarding
to enforce PFG
✔
DIMACS Cloud Workshop, Dec. 8-9, 2011
Placement: Cloud SettingA physical cloud network G(V, E)
Each physical node v in the cloud has capacity: Function C : V → N: C(v) is # of VMs that it can host Function Fx : V → N: Fx(v) is # of contexts of
declassifier type x that it can host. Each physical link has bandwidth:
Function B : E → N: B(e) is amount of available bw
31
DIMACS Cloud Workshop, Dec. 8-9, 2011
Placement: Batch EmbeddingFor each cloud location pair a → b, compute M
paths: R1(a → b), …, RM(a → b)
For each cloud location pair a → b and sequence of declassifiers x1,x2, ..., xK, compute set of feasible flow-table forwarding paths Pa → b
Hence each app with a FPG can be served by picking one feasible flow-table forwarding path for each of its FPG links Need to refactor FPG to extract shared declassifiers
32
DIMACS Cloud Workshop, Dec. 8-9, 2011
Placement: Batch EmbeddingA batch set R of app requests to be handled
Let p be a feasible embedding of app request r
yrp: indicator {0, 1} variable of using p to satisfy r
33
DIMACS Cloud Workshop, Dec. 8-9, 2011
Primal to Dual
34
DIMACS Cloud Workshop, Dec. 8-9, 2011
Online Embedding based on Dual
• Comment– Gives a B-competitive fractional embedding
• use rounding to convert to actual embedding• A negative result: no o(|R|)-comp. online embedding
35
DIMACS Cloud Workshop, Dec. 8-9, 2011
Simplification: Extension/Migration Embedding
• Policy-aware embedding can be simpler if – policy has been enforced
in an original network and
– cloud (public/private data centers) is used for extension/migration
36
Ex
T1-V400
LB2
T2-V500T3-V200 T3-V100
F1
ips1,
log1ips1,
log1
ips1,
log1
ips2
ips2,
F2
ips3
ips3ips2,
F2
ips1,
log1
F2
ips1,
log1 F2
T1-V300
LB1
DIMACS Cloud Workshop, Dec. 8-9, 2011
Setting
37
• Assume each app entity is enforced as a VLAN in original network
Original Network Cloud Private/Public Center
VLAN x VLAN y
Migrate or copy
as template
DIMACS Cloud Workshop, Dec. 8-9, 2011
Solution: PACE Proxy
38
• Introduce Sproxy and Vtarget
Original Network Cloud Private/Public Center
VLAN x VLAN y
Migrate or copy
as template
vtarget
VLAN x VLAN y
Sproxy L2 tunnel
DIMACS Cloud Workshop, Dec. 8-9, 2011
Solution: PACE Proxy
39
• Link each serving switch of each migrated/extension template to Sproxy in original w/ VLAN
label
Original Network Cloud Private/Public Center
VLAN x VLAN y
Migrate or copy
as template
vtarget
VLAN x VLAN y
Sproxy
VLAN yVLAN x
DIMACS Cloud Workshop, Dec. 8-9, 2011
Solution: PACE Proxy
40
• Configure forwarding tables of Sproxy and Vtarget to enforce policies back at original
network
Original Network Cloud Private/Public Center
VLAN x VLAN y
Migrate or copy
as template
vtarget
VLAN x VLAN y
Sproxy
VLAN yVLAN x
Encapsulation using QinQ or MACinMAC
DIMACS Cloud Workshop, Dec. 8-9, 2011
Summary
41
• Cloud deployment faces policy-aware embedding issues• Introduce Flow Policy Graph as a DIFC based approach to
capture policies– Future work: integrated label based DIFC
• Cloud embedding efficiency depends on – Network forwarding mechanisms (flow table is promising)– Placement algorithms (competitive fractional alg. But worst
case can be bad)• Extension/migration can provide simpler solutions for
embedding– Introduced the proxy approach for enforcing at original– May also compute mirrors to enforce at both
DIMACS Cloud Workshop, Dec. 8-9, 2011
Thank You
DIMACS Cloud Workshop, Dec. 8-9, 2011
Preliminary Evaluations• A campus network
– 50 routers and 1000 switches• Policies inferred based on topology properties and route
distribution graph• Scope of a policy
– If both endpoints are in the same VLAN, the scope is all nodes in the broadcast domain of the VLAN
– If endpoints are in different VLAN, the scope is all nodes reachable (based on route distribution graph and ACLs) within the security zone
– Scope is extended to include relevant nodes in remote data center
43
DIMACS Cloud Workshop, Dec. 8-9, 2011
Evaluation (Cont’d)• Policy violation without
PACE– BGP redistributions routes to remote data center into base networks–three types of communication: • UU sessions (among
relocated servers)• UL2domain (between
relocated and those remaining at the same L2 domain)
• UExternal (between relocated and those outside the campus network).
44
DIMACS Cloud Workshop, Dec. 8-9, 2011
Evaluation (Cont’d)• Overhead of policy enforcement
45
DIMACS Cloud Workshop, Dec. 8-9, 2011
Setup for algorithm evaluation
Three typical data center structures Bcube, Dcell, and fat-tree
• Bcube: 2187 leaf nodes and 4 levels of 3-port mini-switches
• Each switch has 150 middlebox instances
• Each leaf node has 300 VMs
• Each link has a bandwidth of 1Gbps
• Dcell(32,1): 1056 leaf nodes
• Each leaf node has 500 VMs
• each switch has 1000 middlebox instances
• each link has bandwidth 1Gbps
• Fat-tree with k = 24: 3456 leaf nodes; three middlebox placements: (I) aggregation switch, (II) half aggregation, half access (III) access
switch
Application request: average number of VM and middleboxes is 50 and 60, average bandwidth is 350Mbps
Evaluation (Cont’d)
46
DIMACS Cloud Workshop, Dec. 8-9, 2011
Comparison of online fractional algorithm with integral one
Evaluation (Cont’d)
47
DIMACS Cloud Workshop, Dec. 8-9, 2011
S1
IPS1
S2
Remote Data Center
u1 v1
Path1
S3
IPS2
S4
S5
IPS3
u’1 v’1
Path2
Problem when end hosts communicate directly in remote:
• a simple example when topology mismatch
Why Worry about PACE?
DIMACS Cloud Workshop, Dec. 8-9, 2011
• Existing policy preservation techniques:– Virtualization layer support: Vmware introduces Virtual Service Domain (VSD) which allows a group of VMs to be protected by virtual appliances– Switch support: data center switch supports port policy migration with attached VM, e.g. Voltaire switch
• However, for performance and availability of functionality, some policies must be enforced in network
Existing Work
DIMACS Cloud Workshop, Dec. 8-9, 2011
Feasibility and Optimality Depends on routing and policy enforcement mechanisms available in
the base network and remote network Feasibility: Key factors in the design space affecting feasibility policies
– Where to enforce• At base• At remote• Both base and remote
– Connection/remote capabilities• Remote is spanning-tree: determine feasibility is NP-hard• Pswitch, and Openflow• L2 or L3 tunnels: feasible by constructing two primitives---Mosaic
Proxy, Mosaic Mirror Optimality: objectives and constraints depends on deployment
scenarios– Objectives:
1. Minimizes total path length2. Minimizes cost: cost of network extension includes VM rental cost,
Internet cost, middlebox cost– Constraints: application performance constraints including bandwidth,
delay, jitter
DIMACS Cloud Workshop, Dec. 8-9, 201151
MobilityData Routing
Energy efficient forwarding
DIMACS Cloud Workshop, Dec. 8-9, 2011
Feasibility: L2 Tunnel??
Key idea: implement a logical Sproxy to enforce policies in the base network•
Can be implemented using QinQ or MACinMAC
52
DIMACS Cloud Workshop, Dec. 8-9, 2011
Feasibility: L2 Tunnel??
R1
ue
L2domain(U)
u’3
u2
SproxyS7
S5
vtarget
v’3
VLAN 100
VLAN 200
IPS3
S6
u3
v3
VLAN 100VLAN 200
VLAN 100
VLAN 200
VLAN 100
VLAN 100VLAN 200
VLAN 200
VLAN 200
VLAN 100
VLAN 200
Remote Data Center
L2 extension
Link
Multihop Path
53
DIMACS Cloud Workshop, Dec. 8-9, 2011
Key idea: compute a minimal topology in remote data center to enforce policy constraints in
remote.
• Compute the layer 2 domain L2domain(U) of migrating end host set U
• Compute the layer 3 cutset L3Cutset(U) of U
• Replicate L2domain(U) and L3Cutset(U)
• Minimize nodes needed in remote data center
• Tunnel traffic from L3Cutset routers to its remote replica
Mirror
54
DIMACS Cloud Workshop, Dec. 8-9, 2011
R1
ue
L3Cutset(U)
L2domain(U)
u2
S7
S5
IPS3
S6
u3
v3
VLAN 100
VLAN 100VLAN 200
VLAN 200
VLAN 200
VLAN 100
VLAN 200
Remote Data Center
uk
Rk R’1
S’7
S’5
IPS’1
S’6
U’3
V’3
VLAN 100
VLAN 100VLAN 200
VLAN 200
VLAN 200
VLAN 100
VLAN 200
U’k
R’k
L3 Secure Tunnel
Multihop Path
vtarget
Mirror
55
DIMACS Cloud Workshop, Dec. 8-9, 2011
Our solution framework
Topology
Annotated with
Device Types &
Capabilities
Remote Data
Center
Capabilities
Policy
Specification
Enterprise network
configure files
Enterprise network
reconfiguration
Remote Data Center
Provisioning
Application cloud embedding steps:
1. Infer application policy from enterprise router,
switch, middlebox config files, and neighbor
info
2. Compute annotated topology
3. Specify remote data center capabilities
4. Automatically reconfigure enterprise network
5. Provision network in remote (application
embedding)