Netbus: A Transparent Mechanism for Remote Netbus: A Transparent Mechanism for Remote Device Access in Virtualized SystemsDevice Access in Virtualized Systems

Sanjay KumarSanjay KumarPhD StudentPhD Student

Advisor:Advisor:

Prof. Karsten SchwanProf. Karsten Schwan

Remote Device AccessRemote Device Access

Transparent access to remote devices is becoming Transparent access to remote devices is becoming essential in various computing environmentsessential in various computing environments– datacenters, clusters and even in personal computingdatacenters, clusters and even in personal computing– Remote disks, backup drives, movie on a thin client etc.Remote disks, backup drives, movie on a thin client etc.

Essential in virtualized environment for VM Essential in virtualized environment for VM migrationmigration– Easy for networked devicesEasy for networked devices

iSCSI, NAS, SANiSCSI, NAS, SANBut what about non-networked devices?But what about non-networked devices?

– NDB, DRBD etc. for block devicesNDB, DRBD etc. for block devicesNo support during VM migrationNo support during VM migration

Netbus: Provides transparent and generic access Netbus: Provides transparent and generic access to remote devices to remote devices – Virtual device migration and device hot-swappingVirtual device migration and device hot-swapping

Network

Network

Backup DVD-drive

IPod on networked dockingstation

Blade-Server Blade-Server

Netbus Software ArchitectureNetbus Software Architecture

Targets high-bandwidth, low-latency and reliable Targets high-bandwidth, low-latency and reliable networks with single administrative domainnetworks with single administrative domain– Datacenters, blade-servers, clusters, home and office LANDatacenters, blade-servers, clusters, home and office LAN

Similar to channels in pub/sub systemsSimilar to channels in pub/sub systems– Server exports the device (channel), Client connects (subscribe) Server exports the device (channel), Client connects (subscribe)

to itto it

Logical extension of frontend-backend approach to Logical extension of frontend-backend approach to device virtualizationdevice virtualization– Frontend can communicate with a remote backendFrontend can communicate with a remote backend– Application layer client establishes the connection with serverApplication layer client establishes the connection with server

The fast path is inside kernelThe fast path is inside kernel– Abstraction of a Abstraction of a network busnetwork bus

Generic mechanism with device specific callback Generic mechanism with device specific callback functionsfunctions– Common Netbus header followed by device specific dataCommon Netbus header followed by device specific data– Data passed to device specific callbacks for further processingData passed to device specific callbacks for further processing

Netbus: Software Architecture (contd.)Netbus: Software Architecture (contd.)

Hypervisor

Service VM ServiceVM

Hypervisor

GuestVM

L-BEFE

vdevice device

R-BE

device driver

Local Machine Remote Machine

Netbus

Network

Client

Add remote Dev. to VM

Server

Hypervisor

Service VMGuestVM

BEFE

vdevice

device driver

Local Machine

Add Deviceto VM

device

User

Kernel

Netbus: Software Architecture (contd.)Netbus: Software Architecture (contd.)

Virtual Device MigrationVirtual Device Migration– Migrate virtual devices along with VMsMigrate virtual devices along with VMs– Provides continuous access to VM’s devices after VM Provides continuous access to VM’s devices after VM

migration through Netbusmigration through Netbus– How to deal with pending I/O operations?How to deal with pending I/O operations?

Bring the device into a quiescent state before Bring the device into a quiescent state before migrationmigration– No pending I/O operationsNo pending I/O operations– I/O operations get queued into the FE and complete I/O operations get queued into the FE and complete

after VM migrationafter VM migration

Hypervisor

Service VM ServiceVM

Hypervisor

GuestVM

L-BEFE

vdevice device

R-BE

device driver

Local Machine Remote Machine

Netbus

Network

Client

Add remote Dev. to VM

Server User

Kernel

GuestVM

FE

device driver

vdevice

VM Migration

Virtual Device Migration and Device Hot-swappingVirtual Device Migration and Device Hot-swapping

device

Netbus: Software Architecture (contd.)Netbus: Software Architecture (contd.)

Device Hot-swappingDevice Hot-swapping– Replace the remote device with an ‘equivalent’ local Replace the remote device with an ‘equivalent’ local

device while in operationdevice while in operation– Increases throughput, removes network dependence, Increases throughput, removes network dependence,

hardware maintenancehardware maintenance– Useful for rarely migrated VMsUseful for rarely migrated VMs– How to deal with pending IO operationsHow to deal with pending IO operations

Same as with virtual device migrationSame as with virtual device migration– Can be combined with virtual device migrationCan be combined with virtual device migration

To complete both operations in one shotTo complete both operations in one shot

Netbus’ prototype implemented in Xen and Netbus’ prototype implemented in Xen and works for block and USB devicesworks for block and USB devices

Netbus EvaluationNetbus Evaluation

Testbed descriptionTestbed description– Two Dell Two Dell

Poweredge 2650s Poweredge 2650s connected through connected through a gigabit switcha gigabit switch

– Each machine has Each machine has 2 2.8 GHz, 2-wat 2 2.8 GHz, 2-wat HT Xeon CPUs and HT Xeon CPUs and 2 GB RAM2 GB RAM

– Iozone file I/O Iozone file I/O benchmarks usedbenchmarks used

Write throughput of block devices

Write throughput of block devices without buffer caching

Netbus LatencyNetbus Latency

Latency in accessing block devices

02468

10121416

Stages of access path

Late

ncy

(ms) Local VBD

RVBD (Netbus)

NBD

Latency incurred by various components in accessing IO devices

Latency in accessing USB devices

0

2

4

6

8

10

Stages of access pathLa

tenc

y (m

s)

Local USB

RUSB (Netbus)

Virtual Device MigrationVirtual Device Migration

MySQL server migration in RuBIS online auction benchmark

RuBIS Database server VM migrationRuBIS Database server VM migration

Effect on RUBiS Throughput due to MySQL Server Migration

Effect on Iozone throughput due to VM migration and disk hot-swapping

Questions??Questions??

Remote Device SharingRemote Device Sharing

Host1

Host3 Host4

Host2device1

device3

device2

device4

vdevice4

vdevice2vdevice4