enabling efficient hypervisor-as-a-service clouds with ... · quicksort kernbench netperf. 30...

Dan Williams†, Yaohui Hu‡, Umesh Deshpande∗, Piush K Sinha‡, Nilton Bila†, Kartik Gopalan‡, Hani Jamjoom†

†IBM T.J. Watson Research Center ‡Binghamton University ∗IBM Almaden Research Center

Funded in part by the NSF

Enabling Efficient Hypervisor-as-a-Service Clouds with Ephemeral Virtualization

©2016 IBM Corporation2 29 June 2016

§ Thin and secure layer in the cloud

Hypervisors…

Guest 1

Hypervisor

VM

-- or --

©2016 IBM Corporation3 29 June 2016

Hypervisors can do a lot!

Guest 1

Hypervisor

VM

§ Thin and secure layer in the cloud

§ Feature-filled cloud differentiators– Rootkit detection– Live patching– Intrusion detection– High availability service– Other VMI…

-- or --

©2016 IBM Corporation4 29 June 2016

§ Cloud provider runs hyperplexor– Multiplex hardware

§ Each guest selects a featurevisor– Implements rich services

§ “Hypervisor-as-a-service”

§ Can implement with nested virtualization– Pay nesting overhead all of the time

Splitting the hypervisor

Guest 1

Hyperplexor

Featurevisor

VM

©2016 IBM Corporation5 29 June 2016

Ephemeral virtualization

Guest 1

Hyperplexor

VMFeaturevisor

§ VM runs on hyperplexor when not needing featurevisor services

§ VM runs on featurevisor when needing its services

§ Implemented “Dichotomy”– 80ms switching times– Up to 12% better performance

than nested with 2.5s switching interval

©2016 IBM Corporation6 29 June 2016

§ VM runs on hyperplexor when not needing featurevisor services

§ VM runs on featurevisor when needing its services

§ Implemented “Dichotomy”– 80ms switching times– Up to 12% better performance

than nested with 2.5s switching interval

Ephemeral virtualization

Guest 1

Hyperplexor

Featurevisor

VM

©2016 IBM Corporation7 29 June 2016

§ VM runs on hyperplexor when not needing featurevisor services

§ VM runs on featurevisor when needing its services

§ Implemented “Dichotomy”– 80ms switching times– Up to 12% better performance

than nested with 2.5s switching interval

Ephemeral virtualization

Guest 1

Hyperplexor

VMFeaturevisor

VM

©2016 IBM Corporation8 29 June 2016

§ When does this make sense?

§ Design of Dichotomy

§ Future directions

§ Evaluation

§ Related work

§ Conclusions

Roadmap

©2016 IBM Corporation9 29 June 2016

§ One-time– Snapshot, guest patching, VM mgmt …

§ Sample-based– Rootkit detection, logging, VMI …

§ Continuous– HA snapshot (Remus), memory dedup …

Hypervisor-level applications

time

H

F

©2016 IBM Corporation10 29 June 2016

§ One-time– Snapshot, guest patching, VM mgmt …

§ Sample-based– Rootkit detection, logging, VMI …

§ Continuous– HA snapshot (Remus), memory dedup …

Hypervisor-level applications

time

H

F

H

F

©2016 IBM Corporation11 29 June 2016

§ One-time– Snapshot, guest patching, VM mgmt …

§ Sample-based– Rootkit detection, logging, VMI …

§ Continuous– HA snapshot (Remus), memory dedup …

Hypervisor-level applications

time

H

F

H

F

H

F

©2016 IBM Corporation12 29 June 2016

§ One-time– Snapshot, guest patching, VM mgmt …

§ Sample-based– Rootkit detection, logging, VMI …

§ Continuous– HA snapshot (Remus), memory dedup …

Hypervisor-level applications

time

H

F

H

F

H

F

©2016 IBM Corporation13 29 June 2016

Progress during one cycleFeaturevisor is active

t

Rate

of P

rogr

ess

X

©2016 IBM Corporation14 29 June 2016

Progress during one cycleFeaturevisor is active

t

Rate

of P

rogr

ess

X

β XNested

α X

©2016 IBM Corporation15 29 June 2016

Progress during one cycle

t

Rate

of P

rogr

ess

α X

X

β X

Featurevisor is active

Ephemeral

Nested

©2016 IBM Corporation16 29 June 2016

Progress during one cycle

t

Rate

of P

rogr

ess

α X

X

β X

Ephemeral

Nested

Featurevisor is active

©2016 IBM Corporation17 29 June 2016

§ Overhead of nesting

§ Amount of time on hyperplexor vs featurevisor

§ Frequency of switching

§ Overhead of switching– Design goal: minimal switching overhead

Factors affecting performance

©2016 IBM Corporation18 29 June 2016

§ Overhead of nesting

§ Amount of time on hyperplexor vs featurevisor

§ Frequency of switching

§ Overhead of switching– Design goal: minimal switching overhead

Factors affecting performance

Dependent on featurevisor

and workload

©2016 IBM Corporation19 29 June 2016

§ Memory management

§ Switching

Dichotomy

Hyperplexor

Guest

Virtual Guest EPT

Featurevisor

Migration Engine

Performaction

Migration EngineTrigger

Gueststate

Memory Management

©2016 IBM Corporation20 29 June 2016

§ Non-nested

§ Nested

§ Dichotomy

Memory management

©2016 IBM Corporation21 29 June 2016

§ Hypervisor manages mapping between guest-physical and machine-physical pages in the Guest EPT

§ EPT faults may cause hypervisor to update mapping

Memory management: non-nested

L0

Guest

Guest EPTEPT Fault Handler

EPT fault

©2016 IBM Corporation22 29 June 2016

§ Guest pages mapped into L1 hypervisor

§ L1 manages mappings via a virtual EPT for the guest

§ L0 backs virtual EPT with shadow EPT

Memory management: nested

L0

Guest (L2) Virtual Guest EPT

L1

Virtual EPT

Virtual EPT Trap HandlerShadow EPT

EPT Fault Handler

Trampoline

bounce EPT fault

EPT fault

L1 EPT

Virtual EPT modification

©2016 IBM Corporation23 29 June 2016

Memory management: Dichotomy

HyperplexorEPT Fault Handler

Guest Featurevisor

Virtual EPT

Virtual EPT Trap Handler

EPT Fault Handler

Trampoline

EPT fault

F-EPTChange Tracker

Change Receiver

FH

ChangeReceiver

bounce EPT fault

Forward changes

Virtual EPT modification

Dual Guest/ Shadow EPT

©2016 IBM Corporation24 29 June 2016

§ Similar to VM migration– Pause VM– Transfer VCPU, I/O, unsynchronized page table mappings– No copying of memory

Switching

forever:execute guest whilewaiting for trigger

on_trigger:relinquish_guestwait_for_guest

forever:wait_for_guestdo actionrelinquish guestfinish action

Hyperplexor Featurevisor

©2016 IBM Corporation25 29 June 2016

§ Time-based

§ On new network flow?

§ On particular memory access?

§ On program counter value?

§ What’s the right set of triggers?

Triggers

©2016 IBM Corporation26 29 June 2016

§ Multiple featurevisors for one guest– How do featurevisors interact?

§ Multiple guests on one featurevisor– E.g., temporary high-speed memory channel

§ Common hyperplexor services– Dirty page tracking service?

§ What is the right featurevisor interface?

Future work

©2016 IBM Corporation27 29 June 2016

§ Implementation based on KVM/QEMU– Time-based triggers

§ When is Dichotomy better than nested?

§ How small are switching times?

Evaluation

©2016 IBM Corporation28 29 June 2016

§ Featurevisor applications– No-op – Volatility (VMI rootkit detection)– Netmon (1 second tcpdump)

§ Guest workloads– Quicksort 800 MB– Kernbench– Netperf

Evaluation: setup

©2016 IBM Corporation29 29 June 2016

0 4 8 12 16Period (Seconds)

0.0

0.2

0.5

0.8

1.0

Rela

tive

Perfo

rman

ce to

Bas

eDichotomyNestedPre-copy

Workload performance:no-op featurevisor

0 4 8 12 16Period (Seconds)

0.0

0.2

0.5

0.8

1.0

Rela

tive

Perfo

rman

ce to

Bas

e

DichotomyNestedPre-copy

0 4 8 12 16Period (Seconds)

0.0

0.2

0.5

0.8

1.0

Rela

tive

Perfo

rman

ce to

Bas

e

DichotomyNestedPre-copy

§ Dichotomy outperforms nesting if period is sufficiently long (2.5—6 seconds)

§ Pre-copy suffers because of slow switching times

quicksort kernbench netperf

©2016 IBM Corporation30 29 June 2016

Workload performance:volatility featurevisor

§ Service time of volatility is sufficiently long (4s)– Dichotomy always outperforms nesting

quicksort kernbench netperf

0 4 8 12 16Period (Seconds)

0.0

0.2

0.5

0.8

1.0

Rela

tive

Perfo

rman

ce to

Bas

eDichotomyNestedPre-copy

0 4 8 12 16Period (Seconds)

0.0

0.2

0.5

0.8

1.0

Rela

tive

Perfo

rman

ce to

Bas

e

DichotomyNestedPre-copy

0 4 8 12 16Period (Seconds)

0.0

0.2

0.5

0.8

1.0

Rela

tive

Perfo

rman

ce to

Bas

e

DichotomyNestedPre-copy

©2016 IBM Corporation31 29 June 2016

§ 80ms independent of memory size

§ Compared to ~300ms downtime with live migration

How fast is switching time?

0 2 4 6 8

10 12

0 2 4 6 8 10

Switc

hing

tim

e (s

)

Guest memory size (GB)

Pre-copyDichotomy

©2016 IBM Corporation32 29 June 2016

§ On-demand switching to emulation– Taint-based protection (Ho et al., Eurosys 2006)

§ Switching between bare metal and virtualization– On-demand virtualization (Kooburat and Swift, HotOS 2011)

– Mercury (Chen et al., ICPP 2007)

§ Reducing hypervisor to its essentials– Microvisor (Lowell et al., ASPLOS 2004)

– CloudVisor (Zhang et al., SOSP 2011)

§ Disaggregating administrative domain– Breaking up is hard to do (Colp et al., SOSP 2011)

Related Work

©2016 IBM Corporation33 29 June 2016

§ Cloud providers shouldn’t need to chose between small and feature-filled hypervisors

§ Dichotomy: split the role of the hypervisor into hyperplexor andfeaturevisor

§ Switch between with ephemeral virtualization– 80ms switching times– Up to 12% better performance than

nested with 2.5s switching interval

Conclusion

Guest

Hyperplexor

VM VMFeature-visor

VMGaincontrol

Relinquish control

©2015 IBM Corporation