Virtual machines

Jinyang Li

OS sits between h/w and app

OS

hardware

firefox iTunes emacs

syscall

h/w interface (intel manuals)

OS abstracts the h/w interface

VMM virtualizes hardware interface

guest OS

hardware

firefox iTunes emacs

syscall

h/w interfaceguest OS

firefox iTunes emacs

syscall

h/w interface

h/w interface

Virtual machine monitor

VMM hosted architecture

Host OS

hardware

app

syscall

h/w interface (intel manuals)

Guest OS

app app app

syscall

h/w interface

Virtual machine monitor

History of virtualization

• Old idea dating from 1960s– IBM VM/370: a VMM for IBM mainframe– Multiplex multiple OS on expensive h/w– Desirable when few machines around

• Interest died out in the 80s and 90s– PC h/w is cheap

Why VM today?

• Machine consolidation– N virtual machines 1 physical machine– E.g. Amazon’s EC2 cloud

• VM simplifies software management– Bundle OS/libraries/configurations together

• Other cool uses– Security, fault tolerance, debugging …

Similarities of OS and VMM

• OS provides a virtual execution environment for processes

• VMM provides a virtual execution env (virtual hardware) for OSes

Differences btw. virtualization for processes and OSes

• How does the process and OS use hardware resources?

process OS

CPU Non-privileged registers and instructions

+Privileged registers and instructions

memory Virtual memory +Physical memory

exceptions Signals, errors +Traps, interrupts

I/O File system Programmed I/O, DMA, interrupts

Complete machine simulation#define REG_EAX 1;int32_t eip;int32_t regs[8];int32_t segregs[4];...for (;;) { read_instruction(); switch (decode_instruction_opcode()) { case OPCODE_ADD: int src = decode_src_reg(); int dst = decode_dst_reg(); regs[dst] = regs[dst] + regs[src]; break; case .. }

eip += instruction_length;}

Pros/Cons of simulation

• Pros– Controlled execution– Great for debugging

• Cons: too slow– 100x slow down of CPU– The software decode+execution takes

100~1000s cycles to execute one instruction

Virtualization’s goals

• Fidelity– Software on VMM executes identically to its

execution on h/w

• Performance– Majority of guest instructions are directly executed

by hardware

• Safety– VMM manages h/w resources, provides isolation

etc.

Virtualization challenges

• Insight: execute most instructions as they are– ADD $1, %eax

• Challenges: – How to execute privileged instructions?

• lgdt, cli, halt

– How to virtualize the MMU?– How to prevent guest from overwriting host or

other guests?• mv $123, %cr3

– How to virtualize I/O?

Basic CPU virtualization techniques

1. Trap-and-emulate• KVM, QEMU

2. Paravirtualization• Xen

3. Dynamic binary translation• VMWare

Technique #1: trap-n-emulate

• “trap-n-emulate” (classical virtualization)– Run guest OS at “lesser” privilege– Privileged instructions cause “traps”– VMM run simulator on trapped instructions– (Most) non-privileged instructions do not need

traps– Need h/w support

Technique #1: x86 challenges

• Traditional x86 is not amicable to #1

• Problems:– Many privilege instructions do not trap!

• popf does not trap if it cannot modify system flag

– Hardware-managed TLB• On TLB miss, h/w automatically loads from page

table (VMM cannot intercept this event)

Technique #1: h/w support• AMD’s SVM and Intel’s VT extension to x86

– Starting in late 2005 – AMD Athlon 64, Intel P4, Intel Core …

• Many VMMs now utilize this h/w support– VMWare, QEMU, KVM, VirtualBox, …

• More than just simple fixes– I.e. make sure privileged instructions trap

• H/w support’s goal: minimize traps and emulation in VMM

Technique #1: h/w support

OS

app app CPL=3

CPL=0

VMM

Guest OS

app app

Guest OS

Vmx non-root

Vmxroot

vmrun vmexit

Technique #1: h/w support

• VMM sets up an in-memory VM control data structure (VMCS) per VM

• VMCS virtualizes– System registers:

• %CR0, %CR3, %EIP, %eflags, %CS, %SS, …

• VMCS allows VMM to specify exit controls:– E.g. whether to trap upon “HLT”, “LGDT”

instructions

• Effects: fewer traps

Technique #2: paravirtualization

• Fancy word for “we have to modify and recompile OS”

• Popular back when x86 is not easily virtualizable

• VMM runs at privileged mode, VMs run unprivileged mode

• Modified OS to call into VMM for memory, I/O, interrupts setup, etc..– ~3000 LoC modifications for Linux, ~5000 LoC for

XP

Technique #3: dynamic binary translation

• We have seen BT before. Where?– Eraser intercepts all memory reads/writes to check

for lock protection

• How BT enables software virtualization: – find all privileged instructions in OS and replace

them with call-ins to VMM for emulation

• Why not static binary translation?• Popularized by VMWare

– QEMU also supports BT

Technique #3: binary translation

void clearbal() { while (balance>0) balance--;}

… 804836d: a1 8c 95 04 08 mov 0x804958c,%eax8048372: 83 e8 01 sub $0x1,%eax8048375: a3 8c 95 04 08 mov %eax,0x804958c804837a: a1 8c 95 04 08 mov 0x804958c,%eax804837f: 85 c0 test %eax,%eax8048381: 7f ea jg 804836d8048383: c3 ret…

translationengine

code cache

90d:mov…sub…mov…mov…test…jg call<TE_jmp>(804836d)call<TE_ret>

Original Cache804836d 90d… …

jg 90d

Technique #3: binary translation

• Is BT applied on user-level programs?• BT performance

Most instructions can be executed identically Incur translation overhead only for the first time code

is executed Intercepting and emulating privileged instructions is

expensive• e.g. syscalls

BT slows down call/ret control flow

Memory virtualization

PA=0 4G

OS1

PA=0 1G

OS2

PA=0 1G

MA=0

%cr3 pa Can h/w use this page table?

%cr3 ma

pa

pa

ma

ma

VMM gives the correspondingshadow page table to h/w

VA VA

VA

VA

VA

Maintain shadow page tables

• Correctness requires: – A shadow pg table must be consistent with its actual pg table

• Strawman 1: – On switching address space (“mov %cr3 …”), construct a fresh

shadow pg table Incurs expensive addr space switch overhead

• Strawman 2:– On switching address space, use an empty shadow pg table– Upon incurring page faults, modify shadow PTE according to

actual PTE Incurs many hidden pg faults

Maintain shadow page tables

• Can VMM cache shadows?– Challenge: what if OS modifies one of the pg tables

w/o knowledge of VMM?

• Insight: write protect actual pg tables.– Referred to as “memory traces”

• VMM may choose not to populate all shadow PTEs at once– saves addr space switch time– Less hidden pg faults than strawman #2 because

shadow PTEs are cached

More h/w support

• Intel/AMD added h/w support for memory virtualization– e.g. Intel Core i7 (Q4 2008)– Add new table from PA to MA– h/w traverses two pg tables VAPA,

PAMA to fill TLB

Virtualize I/O• OS communicates with I/O devices via

– Special instruction in/out– Memory mapping I/O (PIO)– Interrupts– DMA

• Virtualization– In/out and PIO must trap into VMM– Run simulation of I/O device

• Simulation:– Interrupt: Generate interrupt in CPU simulator– DMA: copy data to/fromt physical memory of VM

Managing memory in VMM

• Configure VMs to use more “physical” memory than actually available

• What happens when running out of memory?• Strawman: use LRU paging at VMM

– OS already uses LRU doubling paging– OS will recycle whatever “physical page” VMM just

paged out– Better to do random eviction

ESX: Reclaiming pages

• Idea: trick OS to return memory to VMM

• OS is better at deciding what to swap– Normally OS uses all available memory– E.g. buffer cache contains old pages, OS

won’t discard if it doesn’t need memory

• ESX trick: baloon driver

baloon driver

VMM

OS1 OS2

Baloon is a special pseudo-device loaded into OS

VMM instructs baloon to inflate or deflate depending

on memory pressure

Baloon inflates by requesting lots of “pinned” memory pages

To accommodate inflated baloon,OS releases/swaps out

some of its memory pages

Baloon tells VMM to recycle its “private” pinned pages

ESX: sharing pages across VMs

• Many VMs run same OS and programs– Many Linux boxes with Apache server

• Idea: use 1 machine page for identical physical pages• Periodically scan to find identical machine pages

– Do copy-on-write to eliminate redundancy

• Optimization: use a hash table keyed by hash(content) – Allows quick lookup based on page content

Idle memory tax• Proportional share memory allocation

– Important VM gets more memory– Reclaim memory from VM with smallest “shares-to-

pages” (S/P) ratio– If SA = 2SB, A can have 2X memory as B

• Problem: – high-share VMs hoard more memory than needed

• Solution: idle memory tax– Instead of S/P, reclaim from VM w/ smallest

S/P(f+k(1-f))– Statistically sample to determine f

f: frac of non-idle pagesk≥1: a configurable

idle page “cost” parameter

Summary: VMM attributes

• Software compatibility– Runs all software

• Low overhead– Near “raw” machine performance

• Complete isolation– Total data isolation between virtual machines

• Encapsulation– VMs are not tied to physical machines– Checkpoint/migration

Example: VMM-based IDS

• Tradeoffs of intrusion detection systems (IDS):– Host-based IDS:

• Good visibility to detect intruder• Weak isolation from intruder disabling IDS

– Network-based IDS:• Good isolation from attacker• Weak visibility of what’s actually going on

• Can we have both visibility and isolation?

Example: VMM-based IDS

• Strong isolation– VMM isolate software in VM from VMM– Compromised OS cannot disable IDS in VMM

• Introspection: peek inside at VM– Examine physical memory, registers, I/O

devices for patterns of break-ins

• Interposition: modify h/w abstraction to enhance security

Compute Utility

• Virtual appliance abstraction– Target specialized environment (e.g. program

development)– Store targeted VMs in centralized repository– Cached on running machines

• Benefits:– Simplified system admin– Mobility: computing environment follows user

around

Transparent replication

• Replicate VMs across multiple physical machines– If one fails, another can take over immediately

• No software modification necessary

• Preserves all active network connections