How Model-Checking Can Help Model Exploration

Marsha Chechik

Dept of Computer ScienceUniversity of Toronto

Joint work with Arie Gurfinkel, Benet Devereux

Overview of Automated Verification

Answer +

Counterexample

SW/HW

artifactCorrectnessproperties

Temporallogic

Model of

System

Model Extraction Translation

CheckerEngine

Correct?

Correctness properties: CTLpropositional temporal logicbranching-time logic, allowing explicit quantification over

possible futuresSyntax:

True and False are CTL formulas

if p and q are CTL formulae, then so are: p, p q, p qEX p - p is true in some next statesEF p - along some path, p is true in some future state; E[p U q] - along some path, p holds until q holds; EG p - along some path, p holds in every state

Universal quantification: AX p , AF p , A[p U q], AG p

ECTL: EX, EF, EU, EG, disjunction, negation atomic ACTL: AX, AF, AU, AG, conjunction, negation atomic

Conventional state machinesM = <S, A, s0, I , R>S is a (finite) set of statesA is a (finite) set of propositional variabless0 is a unique initial state (s0 S)I: S 2A is a labelling function that maps each state

to the set of propositional variables that hold in it R S S is a (total) transition relation

p q

r

pqr

pqr

s0

s2

s1

Models:Kripke Structures

Counterexamples and Witnesses Goal:

explain why the answer is as givencounterexample: why (s0) = F

i.e., why the property fails to holdwitness: why (s0) = T

i.e., why the property holds Counterexamples and mathematical proofs:

to disprove that holds on all elements of S, produce a single element s S s.t. holds on s.

counterexamples are restricted to universally-quantified formulas

counterexamples are paths (trees) from initial state illustrating the failure of property

Examples

WitnessesEG p

pqr

pqr

p q

r

s0

s2

s1

pqrs0

p q

rs2

CounterexamplesAG p

pqrs0

pqrs1

EX p EX qpqr

s0

p q

rs2

pqrs1

AX p AX qpqr

s0

p q

rs2

pqrs1

Witnesses/Counterexamples Counterexamples (SMV)

AG(Running AF Idle) is false long path leading to state Running followed by a counterexample for AF

EX p is falsenothing given even though it is false!

AG (Running EF Idle)no feedback given when the result is either true or false

Witnesses (SMV)similar -- just negate the properties!A witness to is a counterexample to

So, what do we want? Witnesses/counterexamples

Goodshort -- contain only as much information as necessary

correspond to the modelBad

not available in all cases

often hard to navigate to an “interesting part”

Proofs:Good:

complete

(available for all temporal properties)

all information is hereBad:

too verbose

not particularly intuitive

Where is the model?

Want proof-like counter-examples [TACAS’03]

Example: Cruise Control System (CCS)For keeping an automobile running at a certain speed

Driver accelerates to desired speed and then presses a button on steering wheel (Button = bCruise)

System maintains car speed until The driver presses the brake pedal (Break)

The driver presses the accelerator (Accel)

The driver turns the cruise control off (Button=bOff)

The driver turns the ignition off (Ignition)

The car’s speed becomes uncontrollable (Toofast)

The system can be reactivated by pressing a “resume” button (Button=bResume)

The controlled variable is Throttle

Model-Checking : Example

Proof View: Example

Proof-Like Counter-Example (part 1)

Property:

AG((CC=Cruise) -> Brake)

Proof-Like Counter-Example

(part 2)

Property:

AG((CC=Cruise) -> Brake)

Proof-Like Counter-Example (part 3)

Property:

AG((CC=Cruise) -> Brake)

Mixed Quantifier Formula: Example

Mixed Quantifier Formula: Proof View

Property: EF (AX Brake)

Mixed Quantifier Property: Example

Property:

EF (AX Brake)

The Framework

Model-checkerengine

KEGVis

SMV model(with fairness)

Temporal logic

property (CTL)

solution +proof-like witness

partial witness/counterexample

Navigation/explorationstrategies

Exploration Witness/Counterexample exploration:

dealing with large witnesses How?

how to fast-forward to “interesting” partsSpecify a starting condition, e.g., navigate by a formula

Example: AG(Running AF Idle)not interested in how to get to Running

Specify stopping conditionPick direction (forward/backward)

how to limit information given to user so that interesting cases are easier to find

Navigation Witness/Counterexample Navigation:

making the most interesting choice Sources of choices:

explicit (disjunction)which part of property to consider

Example: (EF p) (EG q)

implicit (via EX)which state to pick as a witness?

Example: EX p

p pp

s0

s1 s3

s2

Bounds in Property-Based Navigation Depth of exploration

Example: witness for AF feasible: (AF p)(s0) = (AF1 p)(s0)

unfeasible: when bound |S|

(witness is as big as the model!)

pq

pq

pq

s0

s2

s1

Can get partial knowledge using depth of explorationExample: (AF p)(s0), set depth to 3

p

p

s0

s1

s4

s2

s3what if depth is smaller than expected?

State-based navigation Pick successor in which (some propositional formula) holds

EX p

pq

p q

pq

s0

s1s3s2

Pick a state using number of successorsleast -- lineargreatest -- branching

ts

Attempt to maintain largest common prefixExample: (EX p) (EX q)

try to pick next state where p and q hold

greedy approximation

shortest witness not necessarily the most interesting!Example: (EF Good EF Error)(s0)

= (E[T U Good ]) (s0) (E[T U Error]) (s0)

= (E [T U7 Good ]) (s0) (E [T U3 Error]) (s0)

picking the shortest counter-exampleautomatically manually based on size or additional information

Strategy: A combination of navigation and exploration to ensure that some user-specified goal is met

Example: shortest counterexample

traditionally, counterexample generators always do shortest (greedy)

counterexample

Strategies Choices:

anything based on paths, states, etc.

including depth of expansion, history, longest common prefix, shortest counter-example...

decision procedure always greedy (and thus non-optimal)

if a strategy was not followed, does not mean that it could not be followed!

Examples:User has complete control.

whenever there is a choice, always ask user

Always attempt to go through Idle state.Always choose state s over t; otherwise, ask user

Part II

Finding Properties

AG(p q)

AG(p AX q)

Model-Checking Typically used for verification

Answer +

Counter-exampleCheckerEngine

Correct?

Where do the properties come from? What to do when they do not hold?So, goal is not just verification but discovery of

properties that the system should have!

Temporal

logic

property

Model of

System

(state-based)

Query Checking [Chan, CAV’00] Goal: speed-up design understanding

discover properties not known a priorip

qr

pqr

pqr

s0

s2

s1

(p q) r is strongest solution to AG ?x,

p q is strongest solution to AG ?x{p,q}

Temporal logic querytemporal logic formula with placeholders

(unknowns)e.g., AG ?x, AG (p ?x)

evaluates to strongest propositional formula that makes query true.

Some applicationsprovide partial explanation when property holds

e.g. instead of AG (a b), ask AG ?x{a, b}answer a b is stronger!

provide diagnostic information when property failse.g. if AG (req AF ack) fails - ask AG (req AF ?x)

Types of queries number of placeholders

e.g., AG ?x, AG (?x EX ?y)

positive vs negativepositive queries - placeholder under even # of negations

e.g. AG ?x

then look for strongest solutions

negative queries - placeholder under odd # of negationse.g. AG (?x p) = AG (?x p)

then look for weakest solutions

mixed queries - neither positive nor negative number of maximally strong solutions

“valid” queries - one strongest solutionarbitrary queries - several strongest solutions

p q

r

pqr

pqr

s0

s2

s1

two solutions to (EX ?x)(s0):p q r and p q r

Deciding TLQ Problem Related work on TLQ

Original definition [Chan, CAV’00]algorithm for “valid” (single strongest solution) subset of CTL

Extended by [Bruns & Godefroid, LICS’01] arbitrary temporal logic formulae

via extended alternating automata

Our work[Gurfinkel, Chechik, Devereux, FSE’02, TSE’03]

reduction to multi-valued model-checking

implementation that deals with arbitrary temporal logic formulae

… with arbitrary number of “unknowns”

answers include witnesses

Example: Cruise Control System (CCS)For keeping an automobile running at a certain speed

Driver accelerates to desired speed and then presses a button on steering wheel (Button = bCruise)

System maintains car speed until The driver presses the brake pedal (Break)

The driver presses the accelerator (Accel)

The driver turns the cruise control off (Button=bOff)

The driver turns the ignition off (Ignition)

The car’s speed becomes uncontrollable (Toofast)

The system can be reactivated by pressing a “resume” button (Button=bResume)

The controlled variable is Throttle

Specification of CCS Specified using SCR method

State changes – in response to events (changes in environment)

@T(a) WHEN bab a

@T(a) WHEN b = a b a’

Input: monitored variablesOutput: controlled variablesSystem state: modeclasses

Sets of states (modes) that partition the state space

The system is in exactly one mode of each modeclass at any point

Mode TransitionTable for CCS

Cruise@T(Button=bResume) WHEN Ignition AND Running AND NOT(Toofast) AND NOT(Brake) AND NOT(Accel)

Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT(Break) AND NOT(Accel)

Inactive@F(Running) WHEN IgnitionOff@F(Ignition)Override

Override@T(Brake) OR @T(Accel) OR @T(Button=bOff)

Inactive@T(Toofast) OR @F(Running)Off@F(Ignition)Cruise

Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel)

Off@F(Ignition)InactiveInactive@T(Ignition)OffNew ModeEventOld Mode

Initial Mode: Off WHEN NOT(Ignition)

Event Table for Throttle

Initial: Throttle=tOff

tAccelThrottle’=

@T(Inmode)when (Speed=slow)Cruise@T(Speed=slow)Cruise

EventsModes@T(Speed=ok)@T(Inmode)when (Speed=ok)

@T(Speed=fast)@T(Inmode)when (Speed=fast)

@F(Inmode)

@F(Inmode)

tMaintain tDecel tOff

Applications of TLQ

“find all possible values of Throttle EF ( (CC=Cruise) ?x {Throttle})when the system is in mode Cruise”

“what modes can follow Off” EF(CC=Off EX?x {CC})

“which pairs of modes can EF(?x{CC} EX?y {CC})follow each other”

Invariant Discovery“find the strongest invariant” AG ?x

“find invariant of mode Inactive AG ((CC=Inactive) ?x{Ignition,Running})w.r.t. Ignition and Running”

“find invariants of all modes w.r.t AG (?x{CC} ?y{Ignition,Running})Ignition and Running”

Reachability analysis “are all modes of modeclass CCreachable?”

EF(CC=Cruise)EF ?x{CC}

Query-Checking Witnesses A witness

a subtree explaining why ECTL formula holdsproduced automatically by model-checker

p q

r

pqr

pqr

s0

s2

s1

Example:query: (EX ?x{p})(s0)solutions: ?x = p

?x = ps0

s2 s1

?x = p ?x = pwitness:

Same strategies as for exploring model-checking witnesses available

TLQ Applications: Testing Current approach (Gargantini, Heitmeyer [FSE’99])

branch coverage for each mode in mode transition table, test each event at least once for each mode, test every no-change at least once

Off@F(Ignition)Cruise

Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel)

Off@F(Ignition)InactiveInactive@T(Ignition)OffNew ModeEventOld Mode

e.g. for mode Off:need @T(Ignition) and no-change

form CTL propertiesEF((CC=Off) EX(CC=Inactive))

EF((CC=Off) EX(CC=Off))

witness produced by model-checker is the test case!

Query-Checking for Testing our approach:

witness to single queryEF((CC=Off) EX ?x{CC})

Off@F(Ignition)Cruise

Cruise@T(Button=bCruise) WHEN Ignition AND Running AND NOT(Toofast) AND NOT (Brake) AND NOT (Accel)

Off@F(Ignition)InactiveInactive@T(Ignition)OffNew ModeEventOld Mode

Testing Transitions from Mode Off

Generated Witness

Property:

EF((CC=Off) EX ?x{CC})

Query Checking for Testing An even better approach:

coverage of the entire mode transition tableEF(?x{CC} EX ?y{CC})

Testing All Transitions from Table

Witness

Implementation Our framework is named XChek [CAV’02]

multi-valued model-checking engineTLQSolver [CAV’03]

query checker built on top of it KEGVis [TACAS’03,FME’03]

tool for visualization and exploration of count. ex. and witnesses

Input: XML models (based on GXL derived language)SMV-like input language (synchronous product of simple state

machines)+ fairness condition (e.g., p is true infinitely often)CTL property or query

For more infoTo obtain a prototype version

send e-mail to xchek@cs.toronto.eduhttp://www.cs.toronto.edu/~chechik/publications.html

Questions?Comments?Concerns?

Suggestions? THANKS FOR YOUR ATTENTION!

Overview of Multi-Valued Model-Checking

Answer +

Counter-example

How Correct?

SW/HW

artifactCorrectnessproperties

MV-logic

CheckerEngine

Temporallogic

MV-Model

of System

Model Extraction Translation

Multi-Valued Algebras Use additional ‘truth values’ to represent levels of

contradiction, uncertainty or anything elsee.g., True, False, Maybe, Likely, etc.

Can be defined on a (finite) distributive lattice of truth values with True at the top and False at the bottom using lattice meet as conjunction… and lattice join as disjunction

Negation is defined to preserve involution, i.e.A = A

Then get preservation of associativity, idempotency, distributivity, and De Morgan’s laws

These algebras are called quasi-boolean (optional) may also add a refinement operator

T

F

T

F

M(Maybe)

Classical logicRepresentinguncertainty

Uses: reasoning

about abstractionand partial systems

Representingdisagreement and uncertainty

Multi-Valued Algebras: ExamplesTT

TM

TF

MT

MMFT

FM MF

FF

Multi-valued state machines: Xkripke

structures Extension of conventional state machines (Kripke

structures)variables take any value from the logictransitions between states take any value from the logic

False transitions are not shown (by convention)

Example:p=TTq=FTr=TT

p=FFq=TTr=TT

p=TTq=TFr=TT

s0

s2

s1

TT

TT

FT

TFTT

TT

FF

TF FT

Partial information Algebra:

use three-valued algebra (Kleene)intermediate value represents incomplete

information or uncertainty

T

F

M

compact representation for all possible refinements of this model

if a property is True/False on the partial model, it is True/False on a refined one

initial theory developed by Bruns & Godefroid, CAV’99

p=Tq=Fr=T

p=Mq=Mr=F

p=Tq=Mr=T

s0

s2

s1

T

T

M

MT

Reasoning about Abstractiona way to overcome the state-space explosion

problem in classical model-checkingcollapses sets of concrete states into a single

abstract state… thus indicating that any differences between the concrete

states within a single abstracts state are ignored

Goal: state-wise preservation if a formula evaluates to True (False) in an abstract state, it

evaluates to True (False) in the corresponding concrete state

T

F

M

p q

r

pqr

pq

r

s0

s2

s1

s0,1 s2

p=Mq=Mr=T

p=Tq=Tr=F

M

T

M

ComplexityRunning time of the model checker is O(|S| ||

k), where |S| - size of state space || - the size of the XCTL formulak - time to compute EX

Same as classical!!!!!

Solving Query-Checkingarbitrary temporal logic formulae

not necessarily “valid”

positive, negative, mixed queries

any number of placeholdersoutput includes “reason” why the answer is as givenextend language of queries

e.g. EF ?x{p, q, r}, where p and q are not true simultaneously

Some formalismA - set of prop. variables

e.g., {p}

set of prop. formulas over A forms lattice ordered by

true

pp

false

{p, p, true}

{p, true}{p, true}

{true}{}

{false, p, p, true}

{}

{p, p}

{p}{p}

{true}

{false}

B ={c | bB s.t. b c} {p, p} = {p, p, true}

X is an upset if X=X {p, p} not an upset, {p, p, true} is

result - upset lattice, ordered by set inclusion

each upset can be represented by a set of minimal elements

{false} can represent {p, p, true, false}

So, if X is a solution to a query, all elements of X are solutions

Reasoning with Colors Given a non-temporal formula with

colors, how to evaluate in a state?e.g.: = (pq red) (pq green)

(p q yellow) (p q blue) in state where pq holds

s0

s1 pq

pq s2

{}

{p, p}

{p}{p}

{true}

{false}

answer: red Given a temporal formula, how does it

evaluate?(EX ) (s) = t succ(s) (t) e.g., evaluate (EX ) (s0):

(pq red) (pq green) (p q yellow) (p q blue)

= red green

= yellow

Encoding TLQ Encoding non-temporal formula with ?x

if pq holds in s0, then ?x{p,q}(s0) = (pq)

{}

{p, p}

{p}{p}

{true}

{false}

s0

s1 pq

pq s2

in general,?x{p} = (pp) (p (p)) = (p green) (p red)

each color is strongest possible solutionalways get exactly one color per state!

Temporal formula(EX ?x )(s) = t succ(s) ?x (t)

e.g., (EX ?x{p}) (s0)= red green

= yellow

= {p, p}

Queries with Multiple Placeholders Solution

Li - lattice of propositional formulas over i th placeholdersolution - from upset lattice over L1 … Ln

Example: ?x (EX ?x AX ?y)solution in (B L2) ((C L2) (L1 D)) = (B L2) (C D)

i.e., {(x,y) | x B (x C y D)}

p q

r

pqr

pqr

s0

s2

s1

Another example:?x{p,q} EX ?y{p,q} in state s0

[[?x]](s0) = {p q}

[[EX?y]](s0) = {p q, p q}

[[?x]](s0) = {p q} {false}

[[EX?y]](s0) = {false} {p q, p q}

solution:{p q} {p q, p q}answers: {(p q, p q), (p q, p q)}

Negation All occurrences of placeholder are either negative or

positiveExample: AG ?x

solve for AG ?x, choose from it

AG and thus AG ( ) hold

so is in solution-set for AG ?x

A given placeholder (?x) appears in both negative and positive formsreplace each positive occurrence with ?x+

replace each negative occurrence with ?x-

solvethe set of all solutions to ?x is intersection of solutions

to ?x+ and ?x-

Multi-Valued Model-Checking Multi-Valued CTL (XCTL)

introduce new constants that interact with true and falsee.g., true red = red

allow to include these constants into CTL formulas Multi-Valued Models

include colors in transition relations and/or in states XChek: symbolic model-checker

receives a lattice of colors, describing how to compose them

multi-valued model + XCTL property

fairness requirements (e.g., assume p holds infinitely often)

returnsappropriate color, counter-example/witness

Running time Naïve query-checking algorithm O(|S| || 22n)

n - number of atomic propositions of interest|S| - size of state space|| - the size of the CTL formula

Query-checking for valid queries O(|S| || 2n) Running time of our model checker O(|S| || dd)

dd - time to compute EX symbolicallydepends on:

size of lattice (number of join-irreducible elements in it)

cost of performing union and intersections

Running time of Query Checker Preliminaries

CT(n) - complexity of performing an operation on terminal nodes of ADD

n -- number of propositions restricting the placeholder

complexity of performing any operation on entire ADD is linear in CT(n) and exponential in |A|

Theorem: Complexity of solving a query with one placeholder is linear in |S|, ||, CT(n) and exponential in |A|

Theorem: Complexity of solving a query with multiple placeholders (?1(n1)…?k(nk)) is same as with single placeholder, where CT(i=1

k ni)but… how many terminal nodes are there?

Running time (Cont’d)CT(n) is quadratic in the number of strongest solutions to !

How many solutions are there in a query? Queries about states

Example: AG (q AX ?x{p})Number of solutions: O(2n < |S|)Query-checking - same complexity class as model-checking

Queries about pathsExample: EG (?x)Number of solutions in worst case [hornus02]: O(22n < 2|S|) Query-checking can be infeasible even for small problems!… but not always!

Ex: EF EG ?x{CC} in a Cruise Control System (later in the talk)

Improvements to Running Time Query-checking and model-checking

query built of some query-checking and some model-checking parts

AG(?x AF p): CTL model-checking of AF p, result used for query-

checking

Theorem: if placeholder occurs in scope of V temporal quantifiers is V Q + (| | V) M

Q - worst case complexity of query checking a formula with one temporal quantifier

M - worst case complexity of query checking a formula with one temporal quantifier

Improvements (Cont’d) Heuristics:

constructing decision diagram for ?x is $$ …but can be avoided in most queries

Transition relation is boolean so cost of performing conjunction and disjunction on terminal nodes is O(1) and does not depend on CT.