Mirage: an OCaml ExokernelAnil Madhavapeddy

University of Cambridge

Computer Laboratory, 15 JJ Thomson Avenue, Cambridge, UK

with Dr. Thomas Gazagnaire (OcamlPro), Dr. Richard Mortier (Nottingham), Dr. Steven Hand (Cambridge), and Prof. Jon Crowcroft (Cambridge)

Motivation: Layers

HardwareHardware

ProcessesProcesses

OS KernelOS Kernel

ThreadsThreads

ApplicationApplication

Motivation: Layers

HardwareHardware

ProcessesProcesses

OS KernelOS Kernel

ThreadsThreads

ApplicationApplication

Language RuntimeLanguage Runtime

Motivation: Layers

HardwareHardware

ProcessesProcesses

OS KernelOS Kernel

ThreadsThreads

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime

Motivation: In Search of Simplicity

HardwareHardware

ProcessesProcesses

OS KernelOS Kernel

ThreadsThreads

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime

Linux KernelMar 1994: 176,250 LoCMay 2010: 13,320,934 LoC

Architecture: Exokernel

HardwareHardware

ProcessesProcesses

OS KernelOS Kernel

ThreadsThreads

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime

HardwareHardware

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime

Architecture: Workflow

HardwareHardware

ProcessesProcesses

OS KernelOS Kernel

ThreadsThreads

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime

HardwareHardware

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime

DevelopDevelop

DeployDeploy

Layer 1: Separation Kernel

Assume { Xen, KVM, L4 } exists

• Abstract Hardware I/O interfaces

• Resource Isolation for memory

• CPU Concurrency and Timers

HardwareHardware

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime

Layer 1: Minimal OS “signature”

module Console : sig type t val create : unit -> t val write : t -> string -> unitend

HardwareHardware

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime let rec fib n = if n < 2 then 1 else fib(n-1) + fib(n-2)

let _ = fib 40

Layer 1: A simple “hello world” kernel

• Xen runs para-virtualized kernels that cooperate with the hypervisor.

• Most code runs unmodified

• Privileged instructions go via Xen hypercalls

HardwareHardware

ApplicationApplication

HypervisorHypervisor

Language RuntimeLanguage Runtime• Linked to a small C library to make a kernel

• Boots in 64-bit mode directly, with starting memory all mapped.

• Is approximately 50-100KB in size.

OS Text and DataOS Text and Data

Network BuffersNetwork Buffers

ReservedReserved

OCaml minor heap

OCaml minor heap

OCaml major heap

OCaml major heap

Mirage: 64-bit Xen Memory Layout

• Single 64-bit address space

• Specialize regions of memory

• No support for:• Dynamic shared libraries• Address Space Randomization• Multiple runtimes (for now)

Mirage: Network Buffers

OS Text and DataOS Text and Data

Network BuffersNetwork Buffers

ReservedReserved

OCaml minor heap

OCaml minor heap

OCaml major heap

OCaml major heap

IP Header

TCP Header

Transmit packet data

IP Header

TCP Header

Receive packet data

Mirage: x86 superpages for OCaml heap

OS Text and DataOS Text and Data

Network BuffersNetwork Buffers

ReservedReserved

OCaml minor heap

OCaml minor heap

OCaml major heap

OCaml major heap

• Reduces TLB pressure significantly.

• Is_in_heap check is much simpler

• Q: Improve GC/cache interaction using PAT registers?

• Q: co-operative GC?

MirageOS: memory performance vs PV Linux

Layer 2: Concurrency and Parallelism

CoreCore CoreCore CoreCore

KernelKernel KernelKernel

CoreCore CoreCore CoreCore CoreCore CoreCore CoreCore

HypervisorHypervisor

ProcessProcess ProcessProcess ProcessProcess ProcessProcess ProcessProcess

ThreadThread ThreadThread ThreadThreadThreadThread ThreadThread ThreadThread

ThreadThread ThreadThread ThreadThreadThreadThread ThreadThread ThreadThread

ThreadThread

ThreadThread

Layer 2: Concurrency

• Xen provides an low-level event interface.

• No need for interrupts: a perfect fit for co-operative threading!

• We always know our next timeout (priority queue)

• So adapted the LWT threading library

Block 5s

Layer 2: OS Signature with Timing

…and parallelism?

• Xen divides up cores into vCPUs, LWT multiplexes on a single core

• Mirage “process” is a separate OS, communicating via event channels

• Open Question: parallelism model (JoCaml, OPIS, CIEL futures)

vCPU 1vCPU 1 vCPU 2vCPU 2Mem 1Mem 1 Mem 2Mem 2SHMSHM

Layer 3: Abstract I/O

module type FLOW = sig

type t type mgr

type src type dst

val read : t -> view option Lwt.t val write : t -> view -> unit Lwt.t val close : t -> unit Lwt.t

module type DATAGRAM = sig

type mgr

type src type dst

type msg

Layer 3: Abstract I/O

module type FLOW = sig

type t type mgr

type src type dst

val read : t -> view option Lwt.t val write : t -> view -> unit Lwt.t val close : t -> unit Lwt.t

val listen : mgr -> src -> (dst -> t -> unit Lwt.t) -> unit Lwt.t

val connect : mgr -> src -> dst -> (t -> unit Lwt.t) -> unit Lwt.t

end

module type DATAGRAM = sig

type mgr

type src type dst

type msg

val recv : mgr -> src -> (dst -> msg -> unit Lwt.t) -> unit Lwt.t

val send : mgr -> dst -> msg -> unit Lwt.t

end

Layer 3: Concrete I/O Modules

module TCPv4 : sig

type t type mgr = Manager.t

type src = (ipv4_addr option * int) type dst = (ipv4_addr * int)

val read : t -> view option Lwt.t val write : t -> view -> unit Lwt.t val close : t -> unit Lwt.t

val listen : mgr -> src -> (dst -> t -> unit Lwt.t) -> unit Lwt.t

val connect : mgr -> src -> dst -> (t -> unit Lwt.t) -> unit Lwt.t

end

module Shmem : sig

type t type mgr = Manager.t

type src = domid type dst = domid

val read : t -> view option Lwt.t val write : t -> view -> unit Lwt.t val close : t -> unit Lwt.t

val listen : mgr -> src -> (dst -> t -> unit Lwt.t) -> unit Lwt.t

val connect : mgr -> src -> dst -> (t -> unit Lwt.t) -> unit Lwt.t

end

Layer 3: Multiple OS modules

OS (Unix)OS (Unix)

OS (Xen)OS (Xen)

StdlibStdlib

IstringTimeClock

ConsoleEthifMain

IstringTimeClock

ConsoleEthifMain

IstringTimeClock

ConsoleEthifMain

IstringTimeClock

ConsoleEthifMain

Layer 3: Multiple OS modules

OS (Unix)OS (Unix)

OS (Xen)OS (Xen)

StdlibStdlib

IstringTimeClock

ConsoleEthifMain

IstringTimeClock

ConsoleEthifMain

IstringTimeClock

ConsoleEthifMain

IstringTimeClock

ConsoleEthifMain

GnttabEvtchnRing

XenbusXenstore

GnttabEvtchnRing

XenbusXenstore

Kernel bindingsKernel

bindings

Xenbindings

Xenbindings

Layer 3: Standard Library Combinations

OS (Unix)OS (Unix)

OS (Xen)OS (Xen)

StdlibStdlib Net (direct)Net (direct)

Net (socket)Net (socket) Unix/socket(ELF binary)Unix/socket(ELF binary)

Unix/direct(ELF binary)Unix/direct(ELF binary)

Xen/direct(microkernel)Xen/direct

(microkernel)

ApplicationApplication

Layer 3: Ocamlbuild Compilation

ocamlopt -output-objocamlopt -output-obj

asmrun.aasmrun.a

minios.aminios.a

ApplicationApplication

cmxcmx aa cmicmi

mlml camlp4camlp4mlimli

cmxcmx

StdlibStdlib

aa cmicmi

mlml camlp4camlp4mlimli

xen.ldsxen.ldsMirage kernelMirage kernel

Layer 3: Ethernet I/O

• I/O arrives via shared-memory Ethernet frames, and parsed via a DSL

• We have Ethernet, ARP, ICMP, IPv4, DHCP, TCPv4, HTTP, DNS, SSH in pure OCaml.

• Performance in user-space is excellent (EuroSys 2007), now benchmarking under Xen.

• Zero-copy, bounds optimisation is vital to performance.

EthernetIP

TCPData

Meta Packet Language (MPL)

packet tcp { source_port: uint16; dest_port: uint16; sequence: uint32; ack_number: uint32; offset: bit[4] value(offset(header_end) / 4); reserved: bit[4] const(0); cwr: bit[1] default(0); ece: bit[1] default(0); urg: bit[1] default(0); ack: bit[1] default(0); psh: bit[1] default(0); rst: bit[1] default(0); syn: bit[1] default(0); fin: bit[1] default(0); window: uint16; checksum: uint16; urgent: uint16 default(0); header_end: label; options: byte[(offset * 4) - offset(header_end)] align(32); data: byte[remaining()];}

OCaml output can both construct and parse packets from this DSL.

Melange: Towards a ‘Functional’ InternetEuroSys 2007, Madhavapeddy et al.

Research Directions

• A more general solution that can handle ABNF, XML, JSON, etc.

• Yakker (AT&T Research)http://github.com/attresearch/yakker

• Dependently typed DSLs (Idris)http://github.com/edwinb/Idris

• LinearML (quasi-linear, reference-counted ML)http://github.com/pika/LinearML

• Goals:

• 10GB/s type-safe network I/O.

• Specify file-systems in this way also.

Research Directions

• Platforms• Bytecode: Simple interpreted runtime• ELF binary: Native code binary running in user-space• Kernel module: Native code binary running in kernel mode• Javascript: Web browser via ocamljs or js_of_ocaml• JVM: virtual machine via ocamljava• 8-bit PIC: via ocamlpic• Microkernel: Xen / KVM / VMWare

• Optimisation

• Whole OS compilation

• LLVM – needed badly for interoperability, not performance

• Profiling

Mirage: roadmap

This work is supported by Horizon Digital Economy Research, RCUK grant EP/G065802/1This work is supported by Horizon Digital Economy Research, RCUK grant EP/G065802/1

Backup Slides

Mirage: concurrency using LWT

• Advantages:

• Core library is pure OCaml with no magic

• Excellent camlp4 extension to hide the bind monad.

• Function type now clearly indicates that it blocks.

• Open Issues:

• Creates a lot of runtime closures (lambda lifting, whole program opt?)

• Threat model: malicious code can now hang whole OS

Moving on from the Socket API (ii)

type packet = | Stream | Datagram

type direction = | Uni | Bi

type consumption = | Blaster | Congestion

val target : packet -> direction -> consumption -> ip_addr -> sockaddr

module Flow : sig type t val read: t -> string -> int -> int -> int Lwt.t val write: t -> string -> int -> int -> int Lwt.t val connect: sockaddr -> (t -> unit Lwt.t) -> unit Lwt.t val listen: sockaddr -> (sockaddr -> t -> unit Lwt.t) -> unit Lwt.tend

OS Text and DataOS Text and Data

Network BuffersNetwork Buffers

ReservedReserved

OCaml minor heap

OCaml minor heap

OCaml major heap

OCaml major heap

Mirage: Typed Memory Allocators

Buddy Allocatordyn_init(type) dyn_malloc(type, size)dyn_realloc(size) dyn_free(type)

Heap Allocatorheap_init(type, pages)heap_extend(type, pages)heap_shrink(type, pages)

Page Grant Allocatorgrant_alloc_page(type)grant_free_page(type)

DNS: Performance of BIND (C) vs Deens (ML)

DNS: with functional memoisation

SQL performance vs PV Linux