software reliability methods

Software Reliability Methods

Sorin Lerner

Software reliability methods: issues

• What are the issues?


• What is software reliability? How to measure it?– Bug counts ? Will we ever have bug-free software?– How many 9’s ?– Service Level Agreements ?

• What is a bug?– Adherence to specifications– But what is a specification…– User unhappy: is that a bug?– Different levels of severity


• Cost of the methods for achieving reliability– Independently develop 5 versions of the software, run

them all in parallel ) less likely that they fail at the same time in the same way. But… cost… is… high

– For tools, cost of development of the tools

• Burden on the programmer– fully automated vs. semi-automated methods– allow progressive adoption


• Level of guarantee provided by the method– Hard guarantees, statistical guarantees, no formal

guarantee– What if tool is broken: trusted computing base

• When is the method used?– compile-time, link-time, load-time, run-time

• What does the tool see?– source code, assembly, the whole program or part of

the program

One way of dividing the spectrum

Compiler

if (…) { x := …;} else { y := …;}…;

010010110100101011011


CompilerCompiler

if (…) { x := …;} else { y := …;}…;

010010110100101011011

Static techniques

Testing techniques

Run-time techniques

Compiler

if (…) { x := …;} else { y := …;}…;

010010110100101011011


CompilerCompiler

if (…) { x := …;} else { y := …;}…;

010010101001011010010101001010110111011011

Static techniques

Testing Testing techniquestechniques

Run-time Run-time techniquestechniques

010010110100101011011

Testing techniques

Run-time techniques

Static techniques

Static Techniques

• Spec: says what code should and should not do

• Complete spec: specifies all behaviors (hard to formalize)

• Incomplete spec: only defines some behaviors– e.g. “no null derefs”, “requests received are eventually processed”

• Many formalisms exist for specs (Pre/Post conditions, FSMs, Temporal Logic, Abstract State Machines etc.)

if (…) { x := …;} else { y := …;}…;

Spec

«¬

$ \ rt l

Code satisfies spec?

Static Techniques

• Language Design– Clean language

design– Type Systems– Domain-specific

languages– …

if (…) { x := …;} else { y := …;}…;

Spec

«¬

$ \ rt l

Code satisfies spec?

• Program Analysis– Dataflow analysis– WP/SP– Model checking– Automated

Theorem Proving– …

Interaction between the

two

CleanLTSysDSL

DFAWP/SPMCATP

ESC/Java [Leino et al PLDI 2002]

object Foo {

//@ PRE (FORMULA) method bar(...) {

...

} //@ POST (FORMULA)}

Compute Weakest Precondition

WP(POST, bar) = weakest condition Q such that Q at entry to bar establishes POST at exist

)

• Programmer annotates code with pre- and post-conditions, tool verifies that these hold

Automated Theorem Prover

CleanLTSysDSL

DFAWP/SPMCATP

ParserCodeGen

Compiler

DSLOpt

DSLOpt

DSLOpt

DSLOpt

DSLOpt

DSLOpt

Checker Checker Checker

Rhodium [Lerner et al POPL 2005]

CleanLTSysDSL

DFAWP/SPMCATP

ParserParserCodeCodeGenGen

CompilerCompiler

DSLOpt

DSLOpt

DSLOpt

Checker

DSLOpt

Checker

DSLOpt

Checker

DSLOpt


CleanLTSysDSL

DFAWP/SPMCATP


VCGen

LocalVC

AutomaticTheoremProver

Rdm Opt

Checker

LemmaFor any Rhodium opt:

If Local VC is trueThen opt is OK

Proof

«¬

$

\ rt l

Opt-independent

Opt-dependent

CleanLTSysDSL

DFAWP/SPMCATP

ESP [Das et al PLDI 2002]

Interface usage rules in documentation

–Order of operations, data access–Resource management –Incomplete, wordy, not checked

Violated rules ) crashes–Failed runtime checks–Unreliable software

CleanLTSysDSL

DFAWP/SPMCATP


C Progra

m

Safe Not Safe

Rules

ESP

CleanLTSysDSL

DFAWP/SPMCATP


• ESP is a program analysis that keeps track of object state at each program point– e.g.: is file handle open or closed?

• Challenge: scale to large programs– One of scalability issues: merge nodes– Always analyze both sides of merge node )

exponential (or non-terminating) program analyses

• ESP has a heuristic for handling merges that– avoids exponential blow-up and runs fast in practice– maintains enough precision to verify programs

CleanLTSysDSL

DFAWP/SPMCATP

BLAST [Henzinger et al POPL 2000]

Interface usage rules in documentation

–Order of operations, data access–Resource management –Incomplete, wordy, not checked

Violated rules ) crashes–Failed runtime checks–Unreliable software


C Progra

m

SafeError Trace

Rules

BLAST


Perform “Predicate

Abstraction” C Program

Rules

start with a set of predicates

Safe

Refine set of predicates

No errors found Analyze

trace

Trace feasible

Error Trace

Trace infeasible

BLAST

error trace found

augmented set of predicates


Perform “Predicate

Abstraction” C Program

Rules

start with a set of predicates

Safe

Refine set of predicates

No errors found Analyze

trace

Trace feasible

Error Trace

Trace infeasible

BLAST

error trace found

augmented set of predicates

CleanLTSysDSL

DFAWP/SPMCATP

Type Systems

• What is a type system?

• A discipline for writing code that can be mechanically checked, and can prevent certain kinds of run-time errors

• For example, java type system prevents calling methods that don’t exists, or calling methods with parameters of the wrong type

CleanLTSysDSL

DFAWP/SPMCATP

Type Systems

• Type systems can track and provide guarantees about many other aspects of computation:– Safe explicit memory management (Crary, Walker

and Morrisett, POPL 99)– Execution time bounds (Crary and Weirich, POPL 00)– Information flow (Myers, POPL 00)– Security automata (Walker, POPL 00)

CleanLTSysDSL

DFAWP/SPMCATP

Type Systems

• MultiJava [Clifton et al 2000] adds to Java:– multi-methods: methods that dispatch symetrically on

the type of all params, not just the first– open classes: classes

• Adding these features makes modular type checking harder, and required innovations on the type system side

• Interplay between language design and type systems

CleanLTSysDSL

DFAWP/SPMCATP

software reliability methods

Documents