Compositional Verification of Compositional Verification of Evolving Component-based Evolving Component-based

SystemsSystems

Nishant Sinha

ECE, CMUPhD Proposal Presentation

2

OverviewOverview

Problem of Re-validating Evolving Systems Our Solution : Substitutability Check

• Containment• Compatibility

Assume-Guarantee for Simulation Related Approaches Proposed Research Timeline

3

Evolving Systems/DesignsEvolving Systems/Designs

Evolution is inevitable in any real hardware/software system:

• Changing requirements, designs• Bug fixes• Product changes (e.g., underlying platform)

• Upgrades of vendor-provided components

4

Re-validation on EvolutionRe-validation on Evolution

Assembly A

Component C (1.0)Component C’ (2.0)

P ?

addedfunctionalities

bug fixes

5

Re-validating Evolving SystemsRe-validating Evolving Systems

Component assembly re-validation• Necessary on upgrade of any component• High costs of complete global verification

For example, a software system of 25 components (~20K LOC) and 100+ properties might take up to a month of verification effort

Require efficient formal techniques for re-validation upon evolution

6

Re-validation on EvolutionRe-validation on Evolution

Pervasive problem in Software and Hardware Engineering communities

Evolution • Upgrades, Bug Fixes, Re-designs ..

Re-validation appears in numerous forms• Regression checks, Upgrade compatibility

checks, Re-design validation

Our proposed formulation/solution• Substitutability Check

7

Subst. Check - RequirementsSubst. Check - Requirements

Substitutability Check for Upgrades• Formal technique for statically re-

validating upgrades Local and Incremental nature

• Should avoid complete re-validation• Local : focus on modified components• Incremental : reuse previous validation

results

A “make” for component-based systems …

8

Subst. Check - Our SolutionSubst. Check - Our Solution

Our Substitutability Check procedure• Focus on Static Re-validation of upgrades• Local and Incremental• Supports changes in Component

Interfaces and Behaviors• Supports multiple component upgrades

9

ComponentC

Identical BehaviorsNew

BehaviorsLost

Behaviors

Upgraded Component

C’

Component UpgradeComponent Upgrade

Involves both addition and removal of behaviors

Compatibility Check(global)

Containment Check(local)

Two-phase Subst. Check• Local : Containment • Global : Compatibility

10

Substitutability CheckSubstitutability Check

11

Substitutability CheckSubstitutability Check

Containment• Detects if any useful behaviors are lost• Locally checks if behaviors (services) of the

previous component are still present in new one• C C’

Compatibility• Safety with respect to other components in

assembly• Detects if the added behaviors cause any

assembly-level violations

• Does [ C’1 || C2 ||.. || Cn P ] hold?

[FM’05]

[FM’05] Dynamic Component Substitutability Analysis, S. Chaki, E. Clarke, N. Sharygina, N. Sinha, Formal Methods Conference, 2005.

12

Subst. Check for Software ComponentsSubst. Check for Software Components

We have formalized Substitutability Check in context of Component-based Software Systems [FM’05]• Components are communicating C programs• Obtain a finite behavioral model of all components by

abstraction: State-event communicating automata

Containment: • Use both Under- and Over- approximations of

components

Compatibility: • Use Dynamic Assume-Guarantee Reasoning• Focus of proposed research

13

State-event (SE) automataState-event (SE) automata

Both state and actions labeled• <Q,,T,P,L>

Composition semantics• Synchronize on shared actions

Represents abstractions

p

!q

q

!p

C ComponentComponent SE

Automaton abstraction

Predicate Abstraction

14

Component AssemblyComponent Assembly

A set of communicating concurrent C programs • No recursion, procedures inlined• Communication is abstracted into interface actions

C1 C2 C3

M1 M2 M3

Predicate Abstraction

Component Assembly

Abstract Assembly

15

Component Assembly ExampleComponent Assembly Example

Inter-process communication module

Lock server Queue

Write_to_queue Read_from_queue

Enter_crit_section,Leave_crit_section

ReadMQWriteMQ

16

Containment CheckContainment Check

17

Containment CheckContainment Check

Goal: Check C C’• All behaviors retained after upgrade• Cannot check directly: need

approximations Idea: Use both under- and over-

approximations Solution:

• Compute M: C M• Compute M’: M’ C’• Check for M M’

C

Containment Check

Identical New Lost

C’

18

Containment (contd.)Containment (contd.)

C C’

M M’

over-approx under-approx

M M’ ?True

C C’False, CE

CE C ?False,Refine M

CE C’ ?

True,Refine M’

! ( C C’),CE provided as feedback False

True

M

C

C’

M’

19

Containment (contd.)Containment (contd.)

Computing over- (may) approximation• Conventional predicate abstraction• Amay(a,a’) =

s, s’. [ R(s,s’) a= h(s) a’ = h(s’) ]

Computing under- (must) approximation• Modified predicate abstraction • Universal abstraction• Amust(a,a’) =

s, s’. [ R(s,s’) a= h(s) a’ = h(s’) ]

20

Compatibility CheckCompatibility Check

21

Compatibility CheckCompatibility Check

Goal:• Detect system-level spec violations due

to added behavior Approach: Dynamic Compositional Reasoning

• Use compositional reasoning to obtain verification proofs for the initial assembly

• Identify parts of the proof that need to be re-validated on upgrades

• Re-validate those parts while reusing as much of the previous proof as possible

C

Compatibility Check

Identical New Lost

C’

22

Compositional ReasoningCompositional Reasoning

Assume-Guarantee Reasoning (AGR)

Automatically generate assumption A • Cobleigh et. al. at NASA Ames

Use learning algorithm for regular languages, L*

M1 || A P M2 A

M1 || M2 P

AG - Non Circular

23

L* learner

Learning Regular languages: L*Learning Regular languages: L* Proposed by D. Angluin, improved by Rivest et al.

• Learning regular sets from queries and counterexamples, Information and Computation, 75(2), 1987.

• Learns the minimal DFA corresponding to an unknown regular lang.

Minimally adequate Teacher

IsMember( trace )

IsCandidate( DFA D )

a

b

a

b

UnknownRegular Language U

±Counterexample/ Yes

Modelchecker

Yes/No

Minimum DFA

• Polynomial in the number of states and length of max counterexample

24

L* : Data StructuresL* : Data Structures

Observation table : empty initially

Rows have access traces Columns have distinguishing

traces Table entries show

membership of traces in U

Iteratively updates the table using provided counterexample samples

Finally, builds a DFA from a closed table and presents it as a candidate

a ab

a

b

aa

ab

1 0

1 1 0

1 1 0

1 1 0

0 1 0

q0 q1

a,b

a

b

[q0]

[q1]

[q1]

[q1]

[q0]

25

Automated AG Reasoning Automated AG Reasoning using L*using L*

R1: M1 || A P

R2: M2 A

trueL* Assumption

GenerationA

CE

CE AnalysisActual CE

M1 || M2 P

-CE for A

+CE for A

Teacher

M1 || M2 P

true

[Cobleigh et al.]

26

Handling Multiple ComponentsHandling Multiple Components

AG-NC can be applied recursively• (Cobleigh et al.)

R1: M1 || A PR2: M2 A

M1 || M2 P

M1 || A1 P

M2 || A2 A1 M3 A2

M2 || M3 A1

M1 || M2 || M3 P

• AG Proof Tree

•Each Ai computed by a separate L* instantiation

27

Compatibility of UpgradesCompatibility of Upgrades Suppose assumptions are available from the old assembly Re-validate an upgrade

• We propose Dynamic AGR : Incremental AGR technique• Reuse previous verification results

C

Identical New Lost

C’

• Can we reuse previous assumptions directly?• NO: upgrades may change the unknown U to be learned

• Therefore, Dynamic AGR needs Dynamic L*

M1 || A1 P M2 A1

M1 || M2 P

M’1 || A’1 P M2 A’1

M’1 || M2 P

Upgrade

Reuse?

ViolatesL* invariant

28

Dynamic L*Dynamic L*

Learn assumption A corresponding to U• Corresponding Observation Table T

Unknown language U changes to U’

Goal: Continue learning from previous model A• Start learning U’ by reusing table T

Central Idea: Re-validate T against U’ to obtain T’ • T’ is valid with respect to U’

29

Dynamic L*Dynamic L*

a ab

a

b

aa

ab

1 0

1 1 0

1 1 0

1 1 0

0 1 0

q0 q1

a,b

a

b

U’ = U\{b}

Re-validationagainst U’

a ab

a

b

aa

ab

1 0

1 1 0

0 1 0

1 1 0

0 1 0

q0 q1

a

a

b

b

Table valid for U Table valid for U’

AssumptionA

AssumptionA’

30

Dynamic AGDynamic AG

M1 || A1 P M2 A1

M1 || M2 P

M’1 || A’1 P M2 A’1

M’1 || M2 P

(Dynamic L*) Re-Validate! and Reuse

Upgrade

• Multiple Components • Heuristics for choosing order of re-validating Ai’s in proof tree

31

ImplementationImplementation

ComFoRT framework for verifying concurrent C programs

Industrial benchmark • ABB Inter-process Communication (IPC) software• 4 main components – CriticalSection, IPCQueue, ReadMQ,

WriteMQ

Evaluated on single and simultaneous upgrades • WriteMQ and IPCQueue components

Properties

• P1: Write after obtaining CS lock

• P2: Correct protocol to write to IPCQueue

32

Experimental ResultsExperimental Results

Upgrade# (Property)

Torig (msec)

(AGR from scratch)

Tug (msec)

(Dynamic AGR)

Ipc1 (P1) 2260 13Ipc1 (P2) 1694 14Ipc2 (P1) 3286 17Ipc2 (P2) 805 10Ipc3 (P1) 3624 17Ipc3 (P2) 1649 14Ipc4 (P1) 1102 24

33

Summary Till NowSummary Till Now

Automated Substitutability Checking• Containment and Compatibility• Containment is a local check• Compatibility is a global check and reuses

previous verification results• Handles multiple upgrades

Implementation• ComFoRT framework• Compatibility Check applied to IPC

example

34

AGR for Simulation AGR for Simulation ConformanceConformance

Till now during AGR• Notion of conformance was trace containment

Simulation conformance : local, non-det.

• M1 || M2 ≤ P M1 || A ≤ P; M2 ≤ A

• Counterexamples are trees

We proposed automated AGR for simulation conformance of LTSs• Learn from tree CEs instead of trace CEs• Tree language of an LTS: set of trees that it

simulates

[CAV’05]

[CAV’05] Automated Assume-Guarantee Reasoning for Simulation Conformance, S. Chaki, E. Clarke, N. Sinha, P. Thati, Conference on Computer-Aided Verification, 2005.

35

AGR for Simulation ConformanceAGR for Simulation Conformance

Key contributions• Formulate simulation check as regular tree

language containment : M ≤ N T(M) T(N)• Represent tree languages with det. tree

automata• Learning algorithm LT for an unknown tree lang.

using teacher• For AGR: Teacher uses tree language

containment to answer queries Experiments with C implementations of Secure

Socket Layer (SSL) Protocol client and server

36

ResultsResults

Direct AG GainVerif. Result

T1 Mem1

T2 Mem2

Mem1/Mem2

|A| MQ #

CQ #

Invalid

* 2146 325 207 10.4 8 265 3

Valid * 2080 309 163 12.8 8 279 3Valid * 2077 309 163 12.7 16 279 3Valid * 2076 976 167 12.4 16 770 4Valid * 2075 969 167 12.4 16 767 4Invali

d* 2074 3009 234 8.9 24 1514 5

Invalid

* 2075 3059 234 8.9 24 1514 5

Invalid

* 2072 3048 234 8.9 24 1514 5

37

Related WorkRelated Work

38

Related WorkRelated Work

Behavioral Subtyping : Liskov ,Wing [’01]• Extend OO subtyping to behaviors• Subtype Req: If x) is provable about

object x:T, then (y) must hold for y:S, where S is a subtype of T

• Add invariant and history constraints to object specifications

• Local substitution check for an object, independent of its use by the environment

39

Related WorkRelated Work

Interface automata formalism: Henzinger et al. [’01-’03]• Variant of Input/Output automata used to

represent component abstractions and interfaces• Composition and Refinement theory• Composition is well-defined only if under some

environment two automata don’t deadlock• M refines N if M allows more inputs and exhibits

less outputs than N

• Refinement is used as a notion of Substitution : does not capture environment use of a component

40

Related WorkRelated Work

Checking compatibility of Upgrades : McCamant, Ernst[’04]

• Compute observational summaries of components before upgrades using sample execution behaviors

• Internal and External summaries• Pre-, Post-conditions to component method calls• On an upgrade, check if new summaries refine old

ones using a theorem prover

• Unsound component behavior abstraction• No temporal information about component

behaviors

41

Related Work Related Work

Automated Assume-Guarantee Reasoning (AGR)

• Weakest assumption [GPB02]• Learning assumptions automatically

[CGP03]• Circular Rules for Automated AGR

[BGP03]• Predicate Abstraction and AGR [CCGP04]• Symbolically learning assumptions

[AMN04]

42

Related WorkRelated Work

Learning for Verification• Black-box model checking [PVY99,GPY02]• Learning for infinite-state model checking

[HV04, VSVA04]• Synthesizing interfaces of Java classes

[ACMN05]• Behavior-based model construction [HS04]• Inductive learning for shape analysis

[SLR05]

43

Proposed ResearchProposed Research

44

Proposed ResearchProposed Research

Substitutability Framework for SystemC

AGR for Liveness

Improving Substitutability Check

Learning-assisted CEGAR

45

SystemCSystemC

C++ library to simulate concurrency SystemC constructs are macros in C++ Being adopted for System-level Design by

industry• Modular computation structure• Has Communication primitives: Channels/Events• Executable specifications• Supports HW/SW Co-Design

Goal: Develop a Compatibility Check Framework for SystemC Designs

46

SystemC ExampleSystemC Example

47

SystemC SchematicSystemC Schematic

48

Program StructureProgram Structure

program = < processes , channels , variables , events>process =

( comm | ctrl flow )*comm =

event.notify() | event.notify_delayed() | signal.read() | signal.write() | channel.<transaction_name>()| port.read() | port.write()| port.<transaction_name>()

ctrl flow = if (exp) then <s1> else <s2> | while (exp) <s1>

variable types = bool | int | bitvector

49

Verifying SystemC Hardware DesignsVerifying SystemC Hardware Designs

Synchronous model of execution• Based on Discrete-event simulation

Verification easier than C++ • Finite number of concurrent processes• No dynamic objects

Available open-source parsers• PINAPA

50

Plan: Subst. Check for SystemCPlan: Subst. Check for SystemC

Flatten and Translate toSynch. Intermediate Language

Flatten and Translate toSynch. Intermediate Language

SystemCDesign

Build Verifier for Intermediate Language

Build Verifier for Intermediate Language

Build Compositional Verifier for Intermediate Language

Build Compositional Verifier for Intermediate Language

Perform Compatibility CheckPerform Compatibility Check

Delta-cycle semantics

Events, Channels, Shared Vars, Processes

Assumption automata over <Input,Output>

alphabet

51

AGR for LivenessAGR for Liveness

Till now, safety and simulation conformance• Checking liveness specs involves reasoning

about infinite system behaviors

Consider -regular language containment

• M1 || M2 P M1 || A P ; M2 A• “A” corresponds to an -regular language

• Weakest assmp. language exists; is -regular

52

AGR for LivenessAGR for Liveness

Develop a learning algorithm for general -regular languages • Previous algorithms target a subclass• Such languages have non-canonical finite

representations• Cannot extend L* directly; Myhill-Nerode

congruence is insufficient• Use a right-congruence theorem for -regular

languages (Maler and Staiger)

53

Improving Substitutability CheckImproving Substitutability Check

Till now, only behavioral changes allowed during upgrades• Interface ports of modules remain the

same

Consider changes in alphabet during Dynamic Check• Extend Dynamic L* to handle both

unknown language change and alphabet change

54

Learning-assisted CEGARLearning-assisted CEGAR

Predicate Abstraction

C1

C2

M1

M2

M1 || A P

M2 A

M1 || M2 P

Single Counterexample CE

Compositional Reasoning for Software Components C1 and C2

C1 || C2 P

55

Learning-assisted CEGARLearning-assisted CEGAR

Predicate Abstraction

C1

C2

M1

M2

M1 || A P

M2 A

M1 || M2 P

LearnerSpurious CE

Constrains

Provides Multiple Spurious CEs

Compositional Reasoning for Software Components C1 and C2

C1 || C2 P

Builds spurious CE language automata