Evolution of Distributed Operating Systems and

Virtualization

21 November 2009

Jun Nakajima

Principal Engineer

2

Agenda

• History

• Microkernel based Distributed Operating Systems

• Current Generation and Virtualization

• Next Generation (My View)

3

17 ½ Years Ago in Seattle, Washington, US

4

Some of them are still there

5

Distributed UNIX* SVR4 on Chorus

Workshop Proceedings “Micro-kernels and Other Kernel Architectures”,

April 27 – 28, 1992, Seattle, Washington

Process Manger

Object Manger (File)

Stream Manger

IPC Key Manger

6

Microkernel-Based Kernels

• Failed as foundation of distributed operating systems

− Required significant “restructuring” of the OS

• Maintenance /compatibility issues with original OS

− Did not match with the abstractions implemented by OS

• Design/logic changes

• Too simple or generic

− More complexity

• Distributed systems are more complex, hard to debug, APIs keep changing…

• But provided innovations toward modular, optimized, well-

structured OS

− Hybrid kernel

− Nanokernel

• Used by various embedded/real-time OS todayDistributing single OS in components was not a good idea

7

BTW: More Deterministic Latency on Micro-kernel Based OS

Workshop Proceedings “Micro-kernels and Other Kernel Architectures”,

April 27 – 28, 1992, Seattle, Washington

Hypervisor-based Real-time systems can be good

choice

8

Agenda

• History

• Microkernel based Distributed Operating Systems

• Current Generation and Virtualization

• Next Generation (My View)

Virtual Machine Monitor (VMM)

MemoryProcessors I/O Devices

Storage

Network

PhysicalPlatformResources

VM0…

VirtualMachines(VMs)

OS

Apps

VM1

OS

Apps

VM2

OS

Apps

VMn

OS

Apps

VMM is similar to operating system; it monitors

VMs instead of processes

10

Virtualization-Based Distributed OS

• Virtualization on VMM (Virtual Machine Monitor) maintains

existing raw and machine interfaces

− No need to change or restructure the OS

− Well-established interface/behavior based on H/W

− Para-virtualization changes the interface, but the changes are typically

mechanical.

• No design/logic changes

• Virtualization-based approach is ideal for implementing

distributed OS

− Scalability (next slide)

− Reliability (in “Extended topics on virtualization” Class)

− H/W virtualization features

Hardware is fully ready for Virtualization-Based

Distributed Operating System

11

Achieving Scalability Using VMM

• Scalability using single VM

− Scalability within VM

• Increase VCPUs (virtual CPU hot-plug), amount of memory (virtual memory hot-

plug), I/O capabilities (hot-plug, direct I/O, SR-IOV)

− Scalability using VMs

• Add more VMs

• Scalability using multiple VMs

− Use live migration

Physical Host #1

VM Monitor (VMM)

Physical Devices

VM1

Guest OS 1

App #2

Virtual

Devices

VM1

App #2

Virtual

Devices

VM2

Guest OS 3

App #3

Virtual

Devices

VM2

Guest OS2

App #3

Virtual

Devices

VM0

Guest OS 0

App #1

Virtual

Devices

VM0

Guest OS0

App #1

Virtual

Devices

Guest OS1

Physical Host #2

VM Monitor (VMM)

Physical Devices

VM2

App #3

Virtual

Devices

Guest OS2

VM1

App #2

Virtual

Devices

Guest OS1

Physical Host #m

VM Monitor (VMM)

Physical Devices

VM1

App #2

Virtual

Devices

VM2

App #3

Virtual

Devices

Guest OS2 Guest OS1

Physical Host #x

VM Monitor (VMM)

Physical Devices

VM1

App #2

Virtual

Devices

VM2

App #3

Virtual

Devices

Guest OS2 Guest OS1

VM1

App #2

Virtual

Devices

Guest OS1

VM1

App #2

Virtual

Devices

Guest OS1

12

Current Generation and Status

• VMM was not intended to be a distributed OS, but some

features are emerging

− Resource balancing using live migration

− Live migration is not available across VMMs from different venders

• Probably it will not happen in the short term

• Management tools manage resources and VMMs react

− VM configuration

− VM creation/termination

− VM live migration

• Management tools are semi-automatic

− Better than “by hand”

− Not scalable (with e.g. >100)

Current Generation is Incomplete as Distributed

Operating System

13

Agenda

• History

• Microkernel based Distributed Operating Systems

• Current Generation and Virtualization

• Next Generation (My View)

14

Issues with Current Generation

Physical Host #1

VM Monitor (VMM)

Physical Devices

VM1

Guest OS 1

App #2

Virtual

Devices

VM1

App #2

Virtual

Devices

VM2

Guest OS 3

App #3

Virtual

Devices

VM2

Guest OS2

App #3

Virtual

Devices

VM0

Guest OS 0

App #1

Virtual

Devices

VM0

Guest OS0

App #1

Virtual

Devices

Guest OS1

Physical Host #2

VM Monitor (VMM)

Physical Devices

VM1

App #2

Virtual

Devices

VM2

App #3

Virtual

Devices

Guest OS2 Guest OS1

Physical Host #m

VM Monitor (VMM)

Physical Devices

VM1

App #2

Virtual

Devices

VM2

App #3

Virtual

Devices

Guest OS2 Guest OS1

Physical Host #x

VM Monitor (VMM)

Physical Devices

VM1

App #2

Virtual

Devices

VM2

App #3

Virtual

Devices

Guest OS2 Guest OS1

H/W Abstraction Layer

• Minimal H/W abstraction layer to VMs

− Advanced H/W virtualization features are not effectively advertized

• Direct I/O (e.g. VT-d), SR-IOV, HAP (Hardware Assisted Paging, e.g. EPT) or not

− Least common features advertized for live migration

• New H/W features are not utilized in VMs effectively

• VM configuration/placement is ad hoc (e.g. hand)

• Number of VMs and machines would increase in the future

− Multi-cores, Cloud Computing

− More frequent resource adjustment/management

• >10,000 VMs

We will need “real” Virtualization-Based Distributed

Operating System

15

Next Generation: Cloud Operating Systems

• More scalability using systematic and automatic resource management

− View the whole system

− Monitor each VMM and VMs

• Host level: CPU features, # of sockets, # of cores, CPU utilization, # of interrupts, amount memory available, power consumption, I/O throughput, devices available, H/W virtualization features,

• Per VM: VCPU features, # of sockets, # of cores, # of VM exits, # of interrupts, power consumption, I/O throughput and flows, devices available,

− VM creation/placement based on profile/requirements and policy/rules

• Priority

• Type of VM, e.g. large and I/O intensive, HPC, desktop, NUMA,

• Power management – E.g. suspend if idle for a specific time

• Failover, etc.

− Prediction

16

Next Generation: Cloud Operating Systems (cont.)

• Utilize new/advanced H/W features in VMs as much as possible

for cloud computing

− Based on VM profile/requirements

• Performance, power, workload characteristics, security, etc.

− Minimize virtualization overheads

• # of VMs can be very large

• More advance topics

− Grid computing, HPC cloud

• Parallel computing, memory intensive workload, high throughput

17

Conclusions

• Virtualization technologies allow to implement “real” Distributed

Operating System for next generation

− Scalable/reliable cloud operating systems

− High-performance/Grid computing

• Hardware virtualization features are ready for solving

challenging problems

• Time for you to work in virtualization!