a practitioner‘s guide to software-based soft-error ... · soft-errors (transient hardware...

System Software Group

A Practitioner‘s Guide to Software-based Soft-Error Mitigation Using AN-Codes

http://www4.cs.fau.de

Peter Ulbrich 15th IEEE International Symposium on High Assurance Systems Engineering January 09, 2013

■  Soft-Errors (Transient hardware faults)!■  Caused by (cosmic) radiation!  

  !   !  

Soft Errors – A Growing Problem

Peter Ulbrich – [email protected] 2

■  Soft-Errors (Transient hardware faults)!■  Caused by (cosmic) radiation!■  PPeerrffoorrmmaannccee ((tteecchhnnoollooggyy)) vvss.. rreelliiaabbiilliittyy

  !   !  



■  Soft-Errors (Transient hardware faults)!■  Caused by (cosmic) radiation!■  PPeerrffoorrmmaannccee ((tteecchhnnoollooggyy)) vvss.. rreelliiaabbiilliittyy

■  Software-based fault-tolerance!■  Selective and resource-efficient (costs!)!■  Vital component: AArriitthhmmeettiicc eerrrroorr ccooddiinngg (AN codes)



The Combined Redundancy Approach (CoRed ) [1]

!!   !!


Encoded opera+on

Sphere of redundancy (SOR)

Isola+on domain

(1)  Ulbrich, Peter; Hoffmann, Martin; Kapitza, Rüdiger; Lohmann, Daniel; Schmid, Reiner; Schröder-Preikschat, Wolfgang: ““EElliimmiinnaattiinngg SSiinnggllee PPooiinnttss ooff FFaaiilluurree iinn SSooffttwwaarree--BBaasseedd RReedduunnddaannccyy””, EDCC 2012.

Combined Redundancy Approach CoRed


!!   !!


Encoded opera+on


Isola+on domain



Triple Modular Redundancy


!!   !!


Encoded opera+on


Isola+on domain




Arithmetic Error Coding


!!→ KKeeyy eelleemmeenntt:: CCooRReedd DDeeppeennddaabbllee VVootteerr !!


Encoded opera+on


Isola+on domain





Problem Statement

Goals: ■  Full 1-bit fault coverage ■  Get what you’re paid for

Implementation: ■  UAV Flight-Control ■  DanceOS – Safety RTOS ■  KESO Embedded JVM!

    "

 

 


CoRed

Problem Statement



Problems: ■  Experiments showed ddiissccrreeppaanncciieess "

(in line with [3]) ■  IImmpplliiccaattiioonnss on error probability?

 


CoRed

Problem Statement



Problems: ■  Experiments showed ddiissccrreeppaanncciieess "

(in line with [3]) ■  IImmpplliiccaattiioonnss on error probability?

→ Practitioners cannot blindly relyon coding theory! ""


CoRed

Get what you’re paid for

Agenda ■  Introduction!

■  Background!■  The CoRed Dependable Voter!■  Arithmetic Error Coding

■  Think Binary!■  Choosing Appropriate Keys ■  Pitfall 1: Mapping Code to Binary

■  Know Your Compiler & Architecture!

■  Pitfall 2: Inter-Instruction State ■  Pitfall 3: Undefined Execution Environment ■  Multi-Bit Faults – A Glimpse

■  Conclusions & Lessons Learned!


Encoded Voter

D

C

(

Encode Encoded Voter Replica 2

Encode Replica 1

Encode Replica 3

XC X

Y

Z

YC

ZC

{BE ,WC } Decode

W

TMR Provider AN Code Protec8on Domain TMR Consumer

Decode

Decode

W

W

The CoRed Dependable Voter – Basics

■  Complex encoded comparison operation!

■  Data-flow integrity!■  Input: Variants ( XC , YC , ZC ) ■  Output: Constant signature (BE ) and encoded winner (WC ) ■  VVaalliiddaattiioonn: Subsequent check (decode)

■  Control-flow integrity!■  SSttaattiicc ssiiggnnaattuurree (expected value): Compile-time "→ Used as return value E

■  DDyynnaammiicc ssiiggnnaattuurree (actual value): Runtime "→ Applied to winner WC


Arithmetic Error Coding – Basics

■  General coding theory!!■  Data word + redundant information = code word ■  Fault detection → ddiissttaannccee bbeettwweeeenn ccooddee wwoorrddss

  !    


Arithmetic Error Coding – Basics

■  General coding theory!!■  Data word + redundant information = code word ■  Fault detection → ddiissttaannccee bbeettwweeeenn ccooddee wwoorrddss

■  Arithmetic error codes!■  Can cope with ccoommppuuttaattiioonnaall flflaawwss ■  AArriitthhmmeettiicc ooppeerraattoorrss (+, -, ×, =, …)


vc = A • v + Bv + D

Encoded value Constant (Key) Value Signature Timestamp

p

pred

✓1A

◆

2 8 192 16 384 32 768 61 440

106

105

104

103

values of A (16-bit constant key)

p

sd

c

(res

idua

lerr

orpr

obab

ility

)

What to Expect? – Residual Error Probability

■  Silent Data Corruption (SDC)!!■  Undetectable code-to-code word mutation

■  Residual error probability!■  Chance for a SDC ■  Fundamental property for safety assessment

→ TThhee bbiiggggeerr kkeeyy A,, tthhee bbeetttteerr??!Peter Ulbrich – [email protected] 8

psdc =valid code words

possible code words≈

1A


■  Background!■  Arithmetic Error Coding ■  The CoRed Dependable Voter!






Appropriate Keys M i C d t Bi

Encoded Voter

D

C

(

Think Binary – Choosing Appropriate Keys?


■  Theory: pprriimmee nnuummbbeerrss [4]!!■  Intuitively plausible ■  Non-primes suitable as well? [3]

  !!    

  !         !

  !!




■  Practitioner’s approach: mmiinn.. HHaammmmiinngg ddiissttaannccee!!■  Distance (d ) between code words (# unequal bits) ■  d-1 bit eerrrroorr ddeetteeccttiioonn ccaappaabbiilliittiieess

  !         !

  !!

x! 1! 0! 1! 0!

y! 1! 1! 0! 0!

d = 2!





■  Brute force!■  11..44××11001144 eexxppeerriimmeennttss for all 16 bit As       !

  !!

x! 1! 0! 1! 0!

y! 1! 1! 0! 0!

d = 2!





■  Brute force!■  11..44××11001144 eexxppeerriimmeennttss for all 16 bit As

A = 58,368 dmin = 2 #errors detectable = 1     !

  !!

x! 1! 0! 1! 0!

y! 1! 1! 0! 0!

d = 2!






A = 58,368 dmin = 2 #errors detectable = 1 58,831 3 2

  !

  !!

x! 1! 0! 1! 0!

y! 1! 1! 0! 0!

d = 2!






A = 58,368 dmin = 2 #errors detectable = 1 58,831 3 2 58,659 " " " "6 " " " " " "5!

  !!

x! 1! 0! 1! 0!

y! 1! 1! 0! 0!

d = 2!






A = 58,368 dmin = 2 #errors detectable = 1 58,831 3 2 58,659 " " " "6 " " " " " "5!

→ TThhee bbiiggggeerr tthhee bbeetttteerr iiss mmiisslleeaaddiinngg!! "!!

x! 1! 0! 1! 0!

y! 1! 1! 0! 0!

d = 2!

Double Check – Implementation in the Spotlight

■  Fault-simulation → eennttiirree ffaauulltt--ssppaaccee!!■  EEaacchh aanndd eevveerryy A, v and fault pattern ■  66..55××11001166 eexxppeerriimmeennttss for 16 bit As and 1-8 bit soft errors

  !!

  !!


p

pred

✓1A

◆

2 8 192 16 384 32 768 61 440

106

105

104

103

values of A (16-bit constant key)p

sd

c

(res

idua

lerr

orpr

obab

ility

)



→ EExxcceessss ooff pprreeddiicctteedd rreessiidduuaall eerrrroorr pprroobbaabbiilliittyy""!!

  !!


p

brd

(borderline bit errors)

p

pred

✓1A

◆

2 8 192 16 384 32 768 61 440

106

105

104

103


sd

c

(res

idua

lerr

orpr

obab

ility

)



→ EExxcceessss ooff pprreeddiicctteedd rreessiidduuaall eerrrroorr pprroobbaabbiilliittyy""!!

→ MMiissmmaattcchh wwiitthh HHaammmmiinngg ddiissttaannccee eexxppeerriimmeennttss !!


p

brd

(borderline bit errors)

p

pred

✓1A

◆

2 8 192 16 384 32 768 61 440

106

105

104

103


sd

c

(res

idua

lerr

orpr

obab

ility

)

Pitfall 1: Mapping Code to Binary


■  Pitfall 1: Binary representation of code words!■  Coding theory is unaware of machine word sizes →  DDaannggeerroouuss oovveerr-- aanndd uunnddeerrflflooww ccoonnddiittiioonnss

   



■  Pitfall 1: Binary representation of code words!■  Coding theory is unaware of machine word sizes →  DDaannggeerroouuss oovveerr-- aanndd uunnddeerrflflooww ccoonnddiittiioonnss

■  EEAANN PPaattcchh:: decode(vC, A, B, D) ■  Additional range checks → Prevent code space violation

W EAN Decode








ment

Encoded Voter

D

D

C

Analysing the Assembly ■  Fault-Injection with FAIL* [5]

■  Based on Bochs simulator ■  EEaacchh aanndd eevveerryy register, flag, "

instruction and execution path ■  Fault-space pruning → Feasibility

  !       "

 

  !


h bility




■  Experimental setup!■  Implementation: C++ ■  Compiler: GCC 4.7.2-5 (IA32), -O2 ■  Footprint:"

■  RTOS: Spatial and temporal isolation

  !


h bility

), -O2

CoRed Voter Simple Voter Instructions 92 38 Memory (Bytes) 301 112




■  Experimental setup!■  Implementation: C++ ■  Compiler: GCC 4.7.2-5 (IA32), -O2 ■  Footprint:"

■  RTOS: Spatial and temporal isolation

→ VViioollaattiioonn ooff pprreeddiicctteedd ffaauulltt--ddeetteeccttiioonn ccaappaabbiilliittiieess!


h bility

), -O2

CoRed Voter Simple Voter Instructions 92 38 Memory (Bytes) 301 112

Know your Compiler and Architecture


■  Pitfall 2: Architecture specifics!■  Example: Absence of compound tteesstt--aanndd--bbrraanncchh ■  Control-flow iinnffoorrmmaattiioonn iiss ssttoorreedd iinn ssiinnggllee bbiitt →  RReedduunnddaannccyy iiss lloosstt

   

   

  !      

/* if (a == b) */ cmp eax, ebx je Lequal




■  EEAANN PPaattcchh:: apply(vC,sigDYN) ■  Malicious control-flow → Signature overflow → Additional check

   

  !      

Encoded Voter

XC

YC

ZC

{BE ,WC }






   

■  Pitfall 3: Undefined Execution Environment!■  CCoommppiilleerr llaazziinneessss leaves encoded values in registers ■  ZZoommbbiiee vvaalluueess → leaking from caller to voter function →  IIssoollaattiioonn aassssuummppttiioonnss vviioollaatteedd

Encoded Voter

XC

YC

ZC

{BE ,WC }






■  EEAANN PPaattcchh:: vote(xC, yC, zC) ■  Cleaning the local storage restores isolation

■  Pitfall 3: Undefined Execution Environment!■  CCoommppiilleerr llaazziinneessss leaves encoded values in registers ■  ZZoommbbiiee vvaalluueess → leaking from caller to voter function →  IIssoollaattiioonn aassssuummppttiioonnss vviioollaatteedd

Encoded Voter

XC

YC

ZC

{BE ,WC }


Fault-Injection Campaigns – Final Results


■  3 Fault-Injection Campaigns:!■  Instructions and ■  General purpose registers and CPU flags ■  Program counter

  !

Instructions Registers and Flags Program Counter Simple CoRed Simple CoRed Simple CoRed

OK! 784 2772 1040 3204 127 267 Detected (Code)! - 995 - 1435 - 420 Detected (Trap)! 93 246 8 41 21 241 Detected (Isolation)! 825 1834 1825 3736 2804 6240 Detected (Timeout)! 0 1 0 0 0 0 Undetected (SDC)! 450 0 807 0 152 0

Fault-Injection Campaigns – Final Results


■  3 Fault-Injection Campaigns:!■  Instructions and ■  General purpose registers and CPU flags ■  Program counter

→ CCooRReedd ddeeppeennddaabbllee vvootteerr ppeerrffoorrmmss aass EEXXPPEECCTTEEDD!! !

Instructions Registers and Flags Program Counter Simple CoRed Simple CoRed Simple CoRed

OK! 784 2772 1040 3204 127 267 Detected (Code)! - 995 - 1435 - 420 Detected (Trap)! 93 246 8 41 21 241 Detected (Isolation)! 825 1834 1825 3736 2804 6240 Detected (Timeout)! 0 1 0 0 0 0 Undetected (SDC)! 450 0 807 0 152 0

Encoded Voter

D

D

D

Multi-Bit Faults – The Good, the Bad and the …

■  2-bit Fault-injection experiments!■  Full fault space coverage ■  Triple check fault-detection capabilities

■  Distances: dgood = 6, dbad = 2!


nd the …

Good A = 58,659 Bad A = 58,368 OK! 38639 38639 Detected (Code)! 21596 21519 Detected (Trap)! 47 47 Detected (Isolation)! 60438 60438 Detected (Timeout)! 0 0 Undetected! 0 77

Multi-Bit Faults – Tighten the Rules

Peter Ulbrich – [email protected] 18 18

3-bit faults 4-bit faults 5-bit faults OK! 33.742% 33.605% 33.544% Detected (Code)! 18.209% 18.356% 18.431% Detected (Trap)! 0.001% <0.001% 0% Detected (Isolation)! 47.993% 48.030% 48.023% Detected (Timeout)! 0% 0% 0% Undetected! 0 0 0 Fault Space! 3.59 × 106 1.03 × 108 2.90 × 109 Coverage! 16.13% 0.59% 0.04%








ment

Encoded Voter

D

D

D

Conclusions & Lessons Learned


■  Soft-Errors (Transient hardware faults)!■  Caused by (cosmic) radiation!■  PPeerrffoorrmmaannccee ((tteecchhnnoollooggyy)) vvss.. rreelliiaabbiilliittyy ■  Software-based fault-tolerance!■  Selective and resource-efficient (costs!)!■  Vital component: AArriitthhmmeettiicc eerrrroorr ccooddiinngg (AN codes)

[2]


Peter Ulbrich – [email protected]

2

(Transie t h d

→  Software-based fault-tolerance !is hard to implement!→  Missing tool support!



■  Soft-Errors (Transient hardware faults)!■  Caused by (cosmic) radiation!■  Performance (technology) vs. reliability ■  Software-based fault-tolerance!■  Selective and resource-efficient (costs!)!■  Vital component: Arithmetic error coding (AN codes)

[2]



2


Soft E

2



12

■  Pitfall 1: Binary representation of code words!

■  Coding theory is unaware of machine word sizes

→  DDaannggeerroouus over-- and underflow conditions

■  EEAANN PPaattcchh:: decode(vC, A, B, D)

■  Additional range checks Prevent code space violation

W EAN'Decode'

: B

ding theory is unaware of mach

d nderflflflflflow conditions

B

f machine word sizes

→  HHuugghh iimmppaacctt ooff eennccooddiinngg ppaarraammeetteerrss!!

→  IImmpprroovveemmeenntt bbyy oorrddeerrss ooff mmaaggnniittuuddee!!!!




[2]



2




12



→  Dangerous over- and underflow conditions

■  EAN Patch: decode(vC, A, B, D)


W EAN'Decode'

→  Hugh impact of encoding parameters!

→  Improvement by orders of magnitude!!

rich ulbrich@cs fau de

Peter Ulbrich – ulbric



15

■  Pitfall 2: Architecture specifics!■  Example: Absence of compound tteesstt--aanndd--bbrraanncchh ■  Control-flow iinnffoorrmmaattiioonn iiss ssttoorreedd iinn ssiinnggllee bbiitt →  RReedduunnddaannccyy iiss lloosstt ■  EEAANN PPaattcchh:: apply(vC,sigDYN) ■  Malicious control-flow Signature overflow Additional check

■  EEAANN PPaattch: vote(xC, yC, zC) ■  Cleaning the local storage restores isolation ■  Pitfall 3: Undefined Execution Environment!■  CCoommppiler laziness leaves encoded values in registers ■  ZZoommbbiie valuess leaking from caller to voter function →  IIssoollaattiioonn aassssuummppttiioonnss vviioollaatteedd

Encoded'Voter'

XC

YC

ZC

{BE ,WC }

ow Signature overflow Addi

h:ng

ndefined Execution Environmentler laziness leaves encoded values in registersl

g ature overflow Additional check→  LLiittttllee oobbvviioouuss ssoouurrccee ooff vvuullnneerraabbiilliittiieess!!→  TTiigghhtt ffeeeeddbbaacckk lloooopp wwiitthh FFII rreeqquuiirreedd!!→  IIssoollaattiioonn aanndd OOSS--ssuuppppoorrtt mmaannddaattoorryy!!




[2]



2




12






W EAN'Decode'



rich



15

■  Pitfall 2: Architecture specifics!■  Example: Absence of compound test-and-branch ■  Control-flow information is stored in single bit →  Redundancy is lost ■  EAN Patch: apply(vC,sigDYN) ■  Malicious control-flow Signature overflow Additional check

■  EAN Patch: vote(xC, yC, zC) ■  Cleaning the local storage restores isolation ■  Pitfall 3: Undefined Execution Environment!■  Compiler laziness leaves encoded values in registers ■  Zombie values leaking from caller to voter function →  Isolation assumptions violated

Encoded'Voter'

XC

YC

ZC

{BE ,WC }

→  Little obvious source of vulnerabilities!→  Tight feedback loop with FI required!→  Isolation and OS-support mandatory!

h – [email protected] h ulbrich@cs fau de

Know your Co

Pe

15



18 18

3-bit faults 4-bit faults 5-bit faults

OK!33.742% 33.605% 33.544%

Detected (Code)! 18.209% 18.356% 18.431%

Detected (Trap)! 0.001% <0.001% 0%

Detected (Isolation)! 47.993% 48.030% 48.023%

Detected (Timeout)! 0% 0% 0%

Undetected!0 0 0

Fault Space!3.59 × 106 1.03 × 108 2.90 × 109

Coverage!16.13% 0.59% 0.04%

ode)

ap0.001% <0.001%

ol im 0 0

03 108

p) 0.0

0 m→  TToooolliinngg ssppeeeedd iiss ccrruuiiccaall!!!!




[2]



2




12






W EAN'Decode'



rich



15

■  Pitfall 2: Architecture specifics!■  Example: Absence of compound test-and-branch ■  Control-flow information is stored in single bit →  Redundancy is lost ■  EAN Patch: apply(vC,sigDYN) ■  Malicious control-flow Signature overflow Additional check

■  EAN Patch: vote(xC, yC, zC) ■  Cleaning the local storage restores isolation ■  Pitfall 3: Undefined Execution Environment!■  Compiler laziness leaves encoded values in registers ■  Zombie values leaking from caller to voter function →  Isolation assumptions violated

Encoded'Voter'

XC

YC

ZC

{BE ,WC }

→  Little obvious source of vulnerabilities!→  Tight feedback loop with FI required!→  Isolation and OS-support mandatory!

h – [email protected] h ulbrich@cs fau de



18

3-bit faults 4-bit faults 5-bit faults

OK!33.742% 33.605% 33.544%

Detected (Code)! 18.209% 18.356% 18.431%

Detected (Trap)! 0.001% <0.001% 0%

Detected (Isolation)! 47.993% 48.030% 48.023%

Detected (Timeout)! 0% 0% 0%

Undetected!0 0 0

Fault Space!3.59 × 106 1.03 × 108 2.90 × 109

Coverage!16.13% 0.59% 0.04%

→  Tooling speed is cruical!!

15Peter Ulbrich – [email protected]

18


!!→ KKeeyy eelleemmeenntt:: CCooRed Dependable Voter !


Encoded'opera+on'''

Sphere'of'redundancy'(SOR)'

Isola+on'domain'''





mbined Redundancy Approach (C

Red Dependable VoVoVoVoVoVoV ter

Encoded'opera+on'Isola+on'domain'


Arithmetic Erro Coding

Thank you!

Implementation and further experimental results:"http://www4.cs.fau.de/Research/CoRed!

a practitioner‘s guide to software-based soft-error ... · soft-errors (transient hardware...

Documents