fine-grained fault tolerance using device...

Fine-Grained Fault Tolerance using Device Checkpoints

Asim Kadavwith Matthew Renzelmann and Michael M. Swift

University of Wisconsin-Madison

1

The (old) elephant in the room

2

device drivers

(majority of kernel code)

3rd party developers

+

OSkernel

2

The (old) elephant in the room

2

device drivers

(majority of kernel code)

3rd party developers

+

OSkernel

Recipe for

disaster

2

Improvement System Validation Validation Validation Improvement SystemDrivers Bus Classes

Isolation Nooks [SOSP 03] 6 1 2

XFI [OSDI 06] 2 1 1

CuriOS [OSDI 08] 2 1 2

Type Safety SafeDrive [OSDI 06] 6 2 3

Singularity [Eurosys 06] 1 1 1

Specification Nexus [OSDI 08] 2 1 2

Termite [SOSP 09] 2 1 2

Recovery Shadow Drivers [OSDI 04] 13 1 3

Static analysis tools Windows SDV [Eurosys 06] All All All

Coverity [CACM 10] All All All

Cocinelle [Eurosys 08] All All All

3

Extensive past work on reliability research

3



XFI [OSDI 06] 2 1 1










3

Observation 1: Solutions that limit changes to kernel and apply to lots of drivers have real impact


3



XFI [OSDI 06] 2 1 1










3


Observation 2: Most systems focus on improving isolation and detection and not on recovery

3

Driver failure recovery limited to driver restart

★ Restart driver upon failure★ Safedrive and MINIX approach★ Can break applications

Device Driver

Device

Driver-Kernel Interface

4

Applications

Kernel

Shadow drivers

4

Driver failure recovery limited to driver restart

★ Restart driver upon failure★ Safedrive and MINIX approach★ Can break applications

Device Driver

Device

Shadow Driver

Driver-Kernel Interface

4

Applications

Kernel

★ Restart and replay upon failure★ Shadow driver approach ★ Always record state of driver★ Perform restart and log replay

upon failure★ Transparent to applications

Shadow drivers

4

Problem 1: Restart based driver recovery is slow

5

0ms

500ms

1,000ms

1,500ms

2,000ms

8139too e1000 ens1371 psmouse

Restart times

net net sound input

5

Problem 1: Restart based driver recovery is slow

5

Shadow drivers restart the driver upon failure which can be slow

0ms

500ms

1,000ms

1,500ms

2,000ms

8139too e1000 ens1371 psmouse

Restart times

net net sound input

5

Driver re-initialization probes hardware again

6

Allocate device structures

Set chipset specific ops

Map BAR and I/O ports

Register device operations

Detect chipset capabilities

Cold boot device Verify EEPROM checksum

Device self test

Configure device

Device ready

6

Driver re-initialization probes hardware again

6

★ What does slow device re-initialization hurt?★ Fault tolerance: Driver recovery★ Virtualization: Live migration ★ OS functions: Fast reboot

Allocate device structures

Set chipset specific ops

Map BAR and I/O ports

Register device operations

Detect chipset capabilities

Cold boot device Verify EEPROM checksum

Device self test

Configure device

Device ready

6

Problem 2: Shadow drivers assume drivers follow class behavior

7

★ Class definition includes:★ Callbacks registered with the bus,

device and kernel subsystem

networkdriver

bus

net devicesubsystem

kernel

probe

xmit

confignetwork

card

shadow drivers

7

Problem 2: Shadow drivers assume drivers follow class behavior

7

How many drivers follow class behavior and how much code does this add and

★ Class definition includes:★ Callbacks registered with the bus,

device and kernel subsystem

networkdriver

bus

net devicesubsystem

kernel

probe

xmit

confignetwork

card

shadow drivers

7

Problem 2(a): Drivers do behave outside class definitions

★ Non-class behavior that affects recovery:- procfs/sysfs interactions and unique ioctls

8

$ echo 1 > /sys/class/sound/mixer/device/enable

Windows WLAN card config via private ioctls

Linux sound card config via sysfs

8

Problem 2(a): Drivers do behave outside class definitions

★ Non-class behavior that affects recovery:- procfs/sysfs interactions and unique ioctls

8

At least 16% of drivers have non-class behavior and may not recover correctly using shadow drivers

$ echo 1 > /sys/class/sound/mixer/device/enable

Windows WLAN card config via private ioctls

Linux sound card config via sysfs

8

Problem 2(b): Too many classes

9★ “Understanding Modern Device Drivers” ASPLOS 2012

ata (1%)

cdrom

ide

md (RAID)

mmc

network RAID

mtd (1.5%)scsi (9.6%)floppy

tape

acpiblue tooth

crypto

fire wire

gpu (3.9%)

inputjoy stick

key board

mouse

touch screentablet game port

serio

leds

media (10.5%)

isdn (3.4%)

sound (10%)

pcm

midi

mixer

thermal

tty

char (52%)

block (16%)net (27%)

other (5%)

atm

ethernet

infiniband

wireless

wimax

token ring

Linux

Device Drivers

gpio

tpmserial

display

lcd

back light

video (5.2%)

pata

disk

sata

disk

fiber channel

iscsi

usb-storageosd

raid

drm

vga

bus drivers

xen/lguest

dma/pci libs

video

radio

digital video broadcasting

wan

uwb

driver libraries

9

Problem 2(b): Too many classes

9★ “Understanding Modern Device Drivers” ASPLOS 2012

ata (1%)

cdrom

ide

md (RAID)

mmc

network RAID

mtd (1.5%)scsi (9.6%)floppy

tape

acpiblue tooth

crypto

fire wire

gpu (3.9%)

inputjoy stick

key board

mouse

touch screentablet game port

serio

leds

media (10.5%)

isdn (3.4%)

sound (10%)

pcm

midi

mixer

thermal

tty

char (52%)

block (16%)net (27%)

other (5%)

atm

ethernet

infiniband

wireless

wimax

token ring

Linux

Device Drivers

gpio

tpmserial

display

lcd

back light

video (5.2%)

pata

disk

sata

disk

fiber channel

iscsi

usb-storageosd

raid

drm

vga

bus drivers

xen/lguest

dma/pci libs

video

radio

digital video broadcasting

wan

uwb

driver libraries

Class-specific driver recovery leads to a large kernel recovery subsystem

9

Fine-Grained Fault Tolerance (FGFT)

10

10


10

Fine-grained Isolation

★ Runs driver entry points like transactions

★ Relies on code generation to limit new code in kernel

10


10




Checkpoint-based recovery

★ Provides fast and correct recovery semantics

10


10




★ Requires incremental overhead/changes to drivers

★ Shifts burden of fault tolerance to faulty code


★ Provides fast and correct recovery semantics

10

Outline

11

Introduction

Evaluation and Conclusions

Fine-grained isolation


11

Unit of fault tolerance: Driver entry point

12

networkdriver

network card

probe

xmit

config

12


12

networkdriver

network card

probe

xmit

config

whole driver isolation

12


12

networkdriver

network card

probe

xmit

config

12


12

networkdriver

network card

probe

xmit

config

FGFT isolation

12


12

★ Provide fault tolerance to specific driver entry points

networkdriver

network card

probe

xmit

config

FGFT isolation

12


12

★ Provide fault tolerance to specific driver entry points

networkdriver

network card

probe

xmit

config

★ Can be applied to untested code or code marked suspicious by static or runtime tools

FGFT isolation

12

netdev

Transactional support through code generation

13

networkdriver

get ringparam

netdev

13

netdev


13

networkdriver

get ringparam

netdev

SFInetwork

driver

stubs

stubs

13

netdev


13

networkdriver

get ringparam SFInetwork

driver

stubs

stubs

netdev

13

netdev


13

Range Table

Address Access rights

0xffffa000 Read

0xffffa008 Write

0xffffa00a Readnetworkdriver


driver

stubs

stubs

netdev

13

netdev


13

Range Table


0xffffa000 Read

0xffffa008 Write

0xffffa00a Read

★ Detects and recovers from: ★ Memory errors like invalid pointer accesses★ Structural errors like malformed structures★ Processor exceptions like divide by zero, stack corruption

networkdriver


driver

stubs

stubs

netdev

13

result

netdev


13

Range Table


0xffffa000 Read

0xffffa008 Write

0xffffa00a Read

★ Detects and recovers from: ★ Memory errors like invalid pointer accesses★ Structural errors like malformed structures★ Processor exceptions like divide by zero, stack corruption

networkdriver


driver

stubs

stubs

netdev

netdev

13

Outline

14

Introduction

Conclusion



14

Checkpointing drivers is hard★Easy to capture memory state

15

networkdriver

network card

15


15

networkdriver

network card

checkpoint

15


15

networkdriver

network card

checkpoint

★ Device state is not captured★ Device configuration space

15


15

networkdriver

network card

checkpoint

★ Device state is not captured★ Device configuration space★ Internal device registers and counters

15


15

networkdriver

network card

checkpoint

★ Device state is not captured★ Device configuration space★ Internal device registers and counters★ Memory buffer addresses used for DMA

15


15

networkdriver

network card

checkpoint


★ Unique for every device

15


15

networkdriver

network card

checkpoint


★ Unique for every device

Intuition: Operating systems already capture device state during power management

15

Intuition with power management

16

★ Refactor power management code for device checkpoints★ Correct: Developer captures unique device semantics ★ Fast: Avoids probe and latency critical for applications

★ Ask developers to export checkpoint/restore in their drivers

16

Device checkpoint/restore from PM code

17

Save config state

Save register state

Disable device

Save DMA state

Suspend device

Restore config state

Restore register state

Restore or reset DMA state

Re-attach/Enable device

Device Ready

Suspend Resume

17


17

Save config state

Save register state

Save DMA state

Suspend device





Device Ready

Suspend Resume

17


17

Save config state

Save register state

Save DMA state





Device Ready

Suspend Resume

17


17

Save config state

Save register state

Save DMA state





Device Ready

Resume Checkpoint

17


17

Save config state

Save register state

Save DMA state





Resume Checkpoint

17


17

Save config state

Save register state

Save DMA state




Resume Checkpoint

17


17

Save config state

Save register state

Save DMA state




RestoreCheckpoint

17


17

Save config state

Save register state

Save DMA state




Suspend/resume code provides device checkpoint functionality

RestoreCheckpoint

17

Synergy of isolation and fast checkpoints

18

netdev

networkdriver

netdev

18


18

xmit

netdev

networkdriver

netdev

18


18

netdev

networkdriver

netdev

get ringparam

18


18

netdev

networkdriver

netdev

C

get ringparam

18


18

netdev

networkdriver

netdev

SFInetwork

driver

stubs

stubs

C

get ringparam

18


18

netdev netdevnetdev

networkdriver

SFInetwork

driver

stubs

stubs

C

get ringparam

18


18

netdev netdevnetdev Range Table


0xffffa000 Read

0xffffa008 Write


SFInetwork

driver

stubs

stubs

C

get ringparam

18


18

err R



0xffffa000 Read

0xffffa008 Write


SFInetwork

driver

stubs

stubs

C

get ringparam

18


18

err R

FGFT provides transactional execution of driver entry points



0xffffa000 Read

0xffffa008 Write


SFInetwork

driver

stubs

stubs

C

get ringparam

18

How does this give us transactional execution?

19

19


19

★ Atomicity: All or nothing execution★ Driver state: Run code in SFI module★ Device state: Explicitly checkpoint/restore state

19


19


★ Isolation: Serialization to hide incomplete transactions★ Re-use existing device locks to lock driver★ Two phase locking

19


19


★ Isolation: Serialization to hide incomplete transactions★ Re-use existing device locks to lock driver★ Two phase locking

★ Consistency: Only valid (kernel, driver and device) states★ Higher level mechanisms to rollback external actions★ At most once device action guarantee to applications

19

Outline

20

Introduction

Evaluation & Conclusions



20

Evaluation platform

21

★ Criterion :★ Latency of recovery: How fast is it?★ Correctness of recovery: How well does it work?★ Incremental effort: How much work is it?★ Performance: How much does it cost?

21

Evaluation platform

21

★ Platform : ★ Implemented in Linux 2.6.29★ 2.5 GHz Intel Core 2 Quad

core w/ 4 GB DDR2 DRAM ★ Six drivers across three classes

★ Criterion :★ Latency of recovery: How fast is it?★ Correctness of recovery: How well does it work?★ Incremental effort: How much work is it?★ Performance: How much does it cost?

Driver Class Bus

8139too net PCIe1000 net PCI

r8169 net PCI

pegasus net USB

psmouse sound PCI

ens1371 input serio

21

Recovery speedup

22

8139too e1000 pegasus r8169 ens1371 psmouse0ms

500ms

1,000ms

1,500ms

2,000msRestart recoveryFGFT recovery

Recovery times

22

Recovery speedup

22


500ms

1,000ms

1,500ms

2,000ms

680.00

1030.00

120.00150.00

1800.00

310.00

Restart recoveryFGFT recovery

Recovery times

22

Recovery speedup

22


500ms

1,000ms

1,500ms

2,000ms

680.00

1030.00

120.00150.00

1800.00

310.00410.00

115.000.045.00

295.00

0.07


Recovery times

22

Recovery speedup

22

FGFT provides significant speedup in driver recovery and improves system availability


500ms

1,000ms

1,500ms

2,000ms

680.00

1030.00

120.00150.00

1800.00

310.00410.00

115.000.045.00

295.00

0.07


Recovery times

22

Static and dynamic fault injection

Driver Injected Faults

Native Crashes

8139too 43 43e1000 47 47

r8169 36 36pegasus 34 33ens1371 22 21

psmouse 46 46TOTAL 258 256

23

23



Native Crashes

FGFT Crashes

8139too 43 43 NONEe1000 47 47 NONE

r8169 36 36 NONEpegasus 34 33 NONEens1371 22 21 NONE

psmouse 46 46 NONETOTAL 258 256 NONE

23

23



Native Crashes

FGFT Crashes

8139too 43 43 NONEe1000 47 47 NONE

r8169 36 36 NONEpegasus 34 33 NONEens1371 22 21 NONE

psmouse 46 46 NONETOTAL 258 256 NONE

23

FGFT recovers from multiple failures : 1) restores non-class state and 2) does not affect other threads

23

Programming effort

Driver LOC Isolation annotationsIsolation annotations Recovery additionsRecovery additions

Driverannotations

Kernelannotations

LOC Moved LOC Added

8139too 1, 904 15 20 26 4

e1000 13, 973 32 32 10r8169 2, 993 10 17 5pegasus 1, 541 26 12 22 5ens1371 2, 110 23 66 16 6psmouse 2, 448 11 19 19 6

24

24

Programming effort

Driver LOC Isolation annotationsIsolation annotations Recovery additionsRecovery additions

Driverannotations

Kernelannotations

LOC Moved LOC Added

8139too 1, 904 15 20 26 4

e1000 13, 973 32 32 10r8169 2, 993 10 17 5pegasus 1, 541 26 12 22 5ens1371 2, 110 23 66 16 6psmouse 2, 448 11 19 19 6

24

FGFT requires a loadable kernel module (1200 LOC) and 38 lines of kernel changes to trap processor exceptions

24

Throughput with isolation and recovery

NativeFGFT-‐I/O-‐allFGFT-‐off-‐I/OFGFT-‐I/O-‐1/2

netperf on Intel quad-core machines25

25


0

25

50

75

100

Thr

ough

put

%ag

e (B

asel

ine

844

Mbp

s)

e1000 Network Card



25


0

25

50

75

100100

Thr

ough

put

%ag

e (B

asel

ine

844

Mbp

s)

e1000 Network Card



CPU: 2.4%

25


0

25

50

75

100100

93

Thr

ough

put

%ag

e (B

asel

ine

844

Mbp

s)

e1000 Network Card



CPU: 2.4% 2.4%

25


0

25

50

75

100100

93100

Thr

ough

put

%ag

e (B

asel

ine

844

Mbp

s)

e1000 Network Card



CPU: 2.4% 2.4% 3.4%

25


0

25

50

75

100100

93100

96

Thr

ough

put

%ag

e (B

asel

ine

844

Mbp

s)

e1000 Network Card



CPU: 2.4% 2.4% 2.9%3.4%

25


0

25

50

75

100100

93100

96

Thr

ough

put

%ag

e (B

asel

ine

844

Mbp

s)

e1000 Network Card



CPU: 2.4% 2.4% 2.9%3.4%

FGFT can isolate and recover high bandwidth devices at low overhead without adding kernel subsystems

25

Summary

26

26

Summary

26

★ FGFT runs driver code as transactions★ Provides fault tolerance at incremental

performance and programmer efforts

★ Introduced device checkpoints★ Provides fast and complete recovery semantics

★ Fast device checkpoints should be explored in other domains like fast reboot, upgrade etc.

26

Questions

Asim Kadav★ http://cs.wisc.edu/~kadav★ [email protected]★ Graduating in spring!

27

http://cs.wisc.edu/~kadav

http://cs.wisc.edu/~kadav

mailto:[email protected]

mailto:[email protected]

Extra slides

★ Unlike suspend, devices continue to be accessed after a checkpoint★ Rely on drivers following ACPI specifications for

correctness

28

Latency for device checkpoint/restore

Driver Class Bus Checkpoint Times

Restore Times

8139too net PCI 33μs 62μse1000 net PCI 32μs 280msr8169 net PCI 26μs 30μspegasus net USB 0μs 4msens1371 sound PCI 33μs 111mspsmouse input serio 0μs 390ms

29

Fast checkpoint/restore using suspend/resume

29

Transforming drivers to run as FGFT

If (c==0) {.print (“Driver init”);}..

Driver with annotations

Static modifications30

30





User supplied annotations

Source transformation (adds driver transactions)

30









Main driver module

SFI driver module

SFI = software fault isolated

30




Static modifications Run-time support30





Main driver module

SFI driver module


30




Communication and recovery

support

Static modifications Run-time support30



1200 LOC



Object tracking

Marshaling/Demarshaling

Kernel undo log

Main driver module

SFI driver module


30

fine-grained fault tolerance using device...

Documents