Understanding the Propagation of Hard Errors to Software and Implications for Resilient System DesignM. Li, P. Ramachandra, S.K. Sahoo, S.V. Adve, V.S. Adve, Y. Zhou (UIUC), ASPLOS’08

Shimin ChenLBA Reading Group Presentation

Introduction Hardware reliability

Aging/wear out Infant mortality (insufficient burn-in) Soft errors (radiation) Design defects

Willing to pay 10% area overhead for reliability Industry panel discussion in SELSE II Conventional dual modular redundancy too costly

How?

Two Observations Only need to handle observable device faults

Faults that propagate through higher levels of the system and observable by software

Fault-free operation is the common case Must be optimized Willing to have increased overhead after a fault is

detected

Proposals: Cooperative HW-SW Detect high-level anomalous SW behavior

(symptoms of faults) Checkpoint/replay + diagnosis components

(For mission-critical system, may incorporate previous backup detection techniques)

Potential Advantages Generality: oblivious to numerous failure mechanisms

and microarchitectures Ignoring masked faults Optimizing for the common case Customizability: which action to take upon fault? Amortizing overhead across other system functions

Reuse online SW bug detection support

Investigation in This PaperQuestion to answer: Coverage: What HW faults produce detectable

anomalous SW behavior w/ high probability? Latency: What is the fault detection latency? Impact on OS:

How frequently is OS state corrupted by HW faults? Detection coverage and latency for such faults?

Focus on permanent faults (increasingly important)Methodology: Fault-injection study using simulations

Major Results Detection coverage: most permanent faults

that propagate to SW are easily detectable Detection latency: <= 100K instructions for

86% cases Impact on OS: often corrupt OS state

Outline SWAT System Assumptions Methodology Results Implications for Resilient System Design

SWAT (SoftWare Anomaly Treatment)

The investigation assumes the following context: Always-on SW symptom-based detection A multicore system, at least one fault-free core Checkpoint/replay mechanism

Replay when fault is detected If anomalous behavior is deterministic, this is HW fault,

recover using a fault-free core Otherwise ignore (transient)

HW has the ability to repair or reconfigure around permanent faults

Firmware controlled diagnosis and recovery hide HW errors from becoming externally visible

Outline SWAT System Assumptions Methodology Results Implications for Resilient System Design

Simulation Environment Virtutech Simics + Wisconsin GEMS micro-

architectural and memory timing simulators

SPARC V9 ISA, 6 SpecInt2000, 4 SpecFP2000

OS activity < 1% for fault-free runs

Fault Injection Timing-first approach in GEMS

Cycle-accurate GEMS timing simulator Simics functional simulator Compare and set GEMS state based on simics state (so

GEMS can skip the support for some rare instructions) Fault injection

Inject fault into GEMS timing simulator If the mismatched states are due to fault injection, corrupt

simics states Activated fault vs. architecturally masked fault:

If GEMS state mismatched simics state? OS or user mode?

Check privilege mode

Fault Model: permanent faults Stuck-at-0:

A bit is always 0

Stuck-at-1: A bit is always 1

Dominant-0 Acts like a logical-AND between adjacent faulty bits

Dominant-1 Acts like a logical-OR between adjacent faulty bits

Dominant-x a.k.a. bridging fault

Number of Injected Faults 10 benchmarks 40 random points per benchmark after initialization 4 fault models 8 micro-architectural structures

Total = 10 x 40 x 4 x 8 = 12800

Fault Detection Run 10 million instructions with detailed simulation

If no SW symptom is detected, run fast simulation to finish the benchmark and check for corruption

Fatal HW TrapTypically not thrown during a correct execution

SPARC: Data Access Exception Division by zero Illegal instruction Memory misaligned Recover Error and Debug (too many nested traps) Watchdog reset (no instruction retires in the last

65536 cycles)

Abnormal Application Exit

Application may have a seg fault or assertion failure

OS knows the exit status

In simulation, looks for OS idle loop as an indication of such an exit

Hangs During the 10 million instructions Keep a counter per observed branch PC Increment the counter for a branch If any counter exceeds 100,000 (or 1% of the total

instructions), then flag a hang

Profiling the fault-free executions and mask out a handful of branches that do not satisfy this

High OS Activity Amount of time the execution remains in OS Typically control returns to user-mode for a few 10s

of instructions except A timer interrupt after a quantum expires

(this < 10,000 instructions) System calls (could be 100K to 1 million instructions)

Detection threshold: Over 30,000 contiguous OS instructions But not in a system call

Metrics Coverage:

Masked faults: architecture + application Detection latency:

total number of instructions retired from the first architecture state corruption till the detection of the fault within 10M instructions

Outline SWAT System Assumptions Methodology Results Implications for Resilient System Design

How do faults manifest in SW?FPU

Excluded

What are masked faults?

Stuck-at faults: Register file: unused physical register RAT: unused logical register FPU: integer benchmarks

Bridging faults: Upper 32-bit in a 64-bit operations Often sign extensions: all-1 or all-0 In SW, small data size

Large number of detections in OS

Although low OS activity, over 65% of detected faults are through symptoms from the OS

Why? A fault in user-mode often results in a memory access

to a cold address, invoking a TLB miss SPARC TLB miss is software managed In OS trap, the same faulty HW OS is more control and memory intensive Often result in corrupted OS state

Fatal HW Traps Illegal instruction traps:

opcode bit changes result in illegal opcode (decoder) Watchdog timer reset:

over 65536 cycles no retired instructions ROB or RAT errors: register src, dest dependences

are messed up, resulting in some kind of indefinite wait

Misaligned accesses: Memory addresses are wrong

Red state exception: Over 4 nested traps

High-OS OS trap handling TLB miss

Permanent HW fault corrupts TLB handler, resulting in the code never returning to user-mode

Significant overlap with fatal traps and hangs High-OS detects 30% of the faults Remove it reduces coverage by 15% Many cases eventually lead to fatal traps or hangs But detecting High-OS reduces latency

Others

Application aborts: 1% coverage Hangs: 3% coverage

Mostly in application Because OS-hangs are often detected

first as High-OS E.g. loop index variable is wrong, never

terminate

Undetected Faults

All but FPU, 0.8% of injected faults result in silent data corruption

FPU: 10% of faults result in silent data corruption Why? FPU results hardly affect memory

addresses or program control

Which SW components are corrupted?

• Need to checkpoint OS

• None case: watchdog reset trap, the first instruction in ROB is blocked

Detection latency

Application state corruption OS state corruption

Latency from App State Corruption

• Some Combination of SW and HW checkpointing schemes are needed

Latency from OS State Corruption

• HW checkpointing schemes may be sufficient

Transient Faults Have Different Characteristics

94% are architecturally masked within the 10M instruction window

3.4% are detected in the 10M window 1.2% are masked by applications 1.3% eventually results in detectable

symptoms Only 0.1% of the total injections result

in silent data corruption

Outline SWAT System Assumptions Methodology Results Implications for Resilient System Design

Detection A majority of permanent faults that propagate to

SW are detectable through low-cost monitoring of simple symptoms

Preliminary experiments show that the use of value-based invariants can significantly improve latency and coverage

FPU: use more HW mechanisms

Recovery

OS recovery is necessary HW recovery mechanisms (e.g. ReVive,

SafetyNet) may be sufficient

Application recovery requires SW checkpoints