PINCETTE project:Validation of changes and upgrades

in large software systemsUnique challenges and suggested solutions

Hana Chockler

2

Motivation:Challenges of validation of evolving software Large software systems are usually built incrementally:

Maintenance (fixing errors and flaws, hardware changes, etc.) Enhancements (new functionality, improved efficiency,

extension, new regulations, etc.) Changes are done frequently during the lifetime of most systems

and can introduce new software errors or expose old errors Upgrades are done gradually, so the old and new versions have to

co-exist in the same system Changes often require re-certification of the system, especially

for mission-critical systems

3

"Upgrading a networked system is similar to upgrading software of a car while the car's engine is running, and the car is moving on a highway. Unfortunately, in networked systems we don't have the option of shutting the whole system down while we upgrade and verify a part of it.“

source:

software update

4

PINCETTE Project – Validating Changes and Upgradesin Networked Software

Checking for crushes

Static Analysis Component

Black box testing

Dynamic AnalysisComponent

Front end

Methodologybook

Using function

summaries

Verifying only the change

White box testing

5

What does it mean to validate a change in a software system?

• Equivalence checking – when the new version should be equivalent to the previous version in terms of functionality• Changes in the underlying hardware• Optimizations

• No crashes – when several versions need to co-exist in the same system, and we need to ensure that the update will not crash the system• When there is no correctness specification, this is often the

only thing we can check• Checking that a specific bug was fixed

• A counterexample trace can be viewed as a specification of a behavior that needs to be eliminated in the new version

• Validation of the new functionality• If a correctness specification for the change exists, we can

check whether the new (or changed) behaviors satisfy this specification

6

Why is it validation of evolving software different from standard software validation?

• Software systems are too large to be formally verified or exhaustively tested at once

• Even if it is feasible to validate the whole system, often the process is too long and expensive and does not fit into the schedule of small frequent changes

• When validating the whole system, there is a danger of overlooking the change

How can we use the fact that we are validating evolving software?

• If the previous version was validated in some way, we can assume that it is correct and not re-validate the parts that were not changed

• If the results of previous validation exist, we can use them as a basis for the current validation – especially useful when there are many versions that differ from each other only slightly

• The previous version can be used as a specification

7

After more than two years of PINCETTE:

1. How to verify an evolving system that does not have assertions?

2. How to verify only the change without relying on results of previous verification?

8

1. How to verify an evolving system that does not have assertions?

9

Dynamically Discovering Assertionsto Support Formal Verification

Motivation: •“Gray-box” components (such as OTS components) – poor specifications, partial view of internal details•Lack of specification complicates validation and debugging•Lack of description of the correct behavior complicates integrationIdea: Analyze gray-box components by dynamic analysis techniques:•Monitor system executions by observing interactions at the component interface level and inside components• Derive models of the expected behavior from the observed events• Mark the model violations as symptoms of faults

©Leonardo Mariani, UniMiB

10

Dynamically Discovering Assertions at BCT (UniMiB tool)

• Combining dynamic analysis and model-based monitoring• Combining classic dynamic analysis techniques (Daikon) with

incremental finite state generation techniques (kBehavior) to produce I/O models and interaction models• FSA are produced and refined based on subsequent executions

• Extracting information about likely causes of failures by automatically relating the detected anomalies

• Filtering false positives in two steps:• Identify and eliminate false positives by comparing failing and

successful executions with heuristics already experienced in other contexts

• Rank the remaining anomalies according to their mutual correlation and use this information to push the related likely false positives far from the top anomalies

©Leonardo Mariani, UniMiB

11

Extracting a set of valid assertions for regression

Program V.0 Program V.1

Testing

set of traces

Analysis

Set of candidateassertions

Formalmethods

Set of assertionsthat hold in V.0

Set of invalidassertions in V.1

Formalmethods

Set of assertionsthat hold in V.0

+

Efficient and automatic regression verification

user

UniMiB+Oxford+USI

12

Parameters of the assertions discovery

• Scope of monitoring• All lines• Function invocations• Loops• Particular variables

• Output refinement• Many violations – maybe leave only those in the slice of

the change• Look for a minimal set

• How to design the search algorithm? • Can we check properties of the type “if P_1, P_2,…P_n

are true then Q_1, Q_2,..Q_m” where each property may hold at a different code location?

13

1. how2. How to verify only the change without

relying on results of previous verification?

The main idea: Verify only the changeignoring the rest of the program

Using ExpliSAT

Joint work with Sitvanit Ruah

14

What is ExpliSAT?

• A model-checker that combines explicit traversal of CFG with symbolic representation of data:• Traverses CFG• Ensure that we traverse only real paths by computing a

representative path with real data values• Builds a SAT formula for each path on the control flow graph and

invokes a SAT solver• Bugs-hunting heuristics can be added to change the order of

traversal of the graph, so that the places where bugs are more likely to occur are traversed first

The concolic approach is

widely used in testing

concrete+symbolic

Heuristics direct the search towards areas in which bugs are more likely to occur

15

The path representative – checking that the symbolic path is real

• ExpliSAT traverse the CFG and create all possible control paths

• Symbolically represent data variables of all executions that follow the same control path

• A path representative for a certain control path is a valuation of every variable in the control path that holds the path constraint.

• For every legal control path there exists a path representative

• The representative is computed iteratively

AUpdate the representative until A and look at command in A

Use the representative prefix until A to decide which branch to take

a control path with a representative is a legal control path

The other branch may be

infeasible – checked by

trying to construct a

representative

16

Ordering - Heuristically deciding in which order to traverse the control paths

11: x=input();

2: if (x>N) then

else

3: y=x;

4: y=N;

1: x=input();

2: if (x>N) then

else4: y=N;

5: assert(y>=N);

2

43

5

1

2

4

55: assert(y>=N);

The particular heuristic depends on the verification goals – for updates, we use

the update-checking heuristic

17

Extracting the part of the control-flow graph affected by a change

change

subgraph induced by the changed node

initial state

Backward reachability of the initial state from the changed node

18

ExpliSAT with update-checking

ExpliSAT with

update checking

goto-cc SAT solversyntactic

marking of changes

Oldversion

------------------

Newversion

------------------

only the new version with marking is

passed

Checks paths that go through a

changed node first

result: goto-program with the change

markingParser + translationto a goto-program

Cprover framework

19

Update-checking heuristics in ExpliSAT

Turns the model-checker to a bug

hunter

Two options:

Check only the change Check the change first, then continue checking the whole program

Very efficient New bugs will

be discovered

quickly

The whole process is complete

20

Effect of a change on functionality of the program

• Claim: in sequential programs, the algorithm from the previous slide finds all paths affected by a change

straightforward

• Observation: in concurrent programs, a change can also affect other paths – future work

Changes in global

variables

21

Experimental results

• Compared performance of ExpliSAT on the whole program with the performance of ExpliSAT on the change only

Not surprisingly, ExpliSAT on

changes only is much faster

• Real-life example: a C++ program supplied by VTT Technical Research Center of Finland, computing the velocity and acceleration of a robot used in the European ITER project – a new type of a reactor based on energy of nuclear fusion

ExpliSAT found a bug in update in several seconds

On the whole program ExpliSAT doesn’t terminate

22

VTT (Finland) - the ITER EU Project: the thermo-nuclear reactor

ITER is an international magnetic confinement fusion experimental project participated by the EU, India, Japan, China, Russia, South Korea, and the USA. The expected budget of ITER is €5B with 50% of funding provided by the EU. The reactor will be constructed in Cadarache, France, around 2016. ITER is one of the most important and complex research projects in the history of the EU.

divertor:goes at the bottom of the reactor;

should be changed periodicallyRobot that changes

the divertor cassette

23