robust test generation and coverage for hybrid systems€¦ · 1 robust test generation and...

1

Robust Test Generation and Coverage for Hybrid Systems

A. Agung Julius, Georgios E. Fainekos, Madhukar Anand, Insup Lee and George J. Pappas

Deptartments of ESE and CISUniversity of Pennsylvania

HSCC April 2007, Pisa, Italy

HSCC April 2007, Pisa, Italy 2

Hybrid System Design Cycle

Specification/RequirementsSpecification/Requirements

Mathematical Analysis

Mathematical Analysis

Model DesignModel Design

Testing/VerificationTesting/

Verification

SystemImplementation

SystemImplementation

TestingTesting


Testing VS Verification: continuous time

TestingTan, Kim, Sokolsky, Lee IRI’04

TorX [Bohnenkamp.et al]Maler, Nickovic FORMATS’04Briones, Brinksma FATES’04

van Osch FATES\RV’06Krichen, Tripakis SPIN’04

UPPAAL-TRON [Mikucionis. et al]

Systematic TestingBadban, Franzle, Peleska, Teige SOQUA’06

Cheng, Kumar WAFR’06Kim, Esposito ACC’05

Krichen, Tripakis FORMATS’04Kapinski, Krogh, Maler, Stursberg HSCC’03

Branicky, Curtiss, Levine, Morgan, Yale Workshop’05Bhatia, Frazzoli, HSCC’04


Testing of hybrid systems

UnsafeInit

L1

L2

L3

In this paper, testing for hybrid automata:1) How robust is a test trajectory?2) How can we provide a confidence level for the system correctness?


Defining the robustness of a simulation trajectory


How robust is a test trajectory?

y(t)

Sg(x(t))y(t)f(x(t))(t)x

==&

pn y I, x(0),x ℜ∈∈ℜ∈

Consider the dynamical system:

Unsafe

y(t)

)),((inf0

Unsafetydt≥

=ε

Fainekos, Girard, Pappas: Temporal Logic Verification Using Simulation, FORMATS 2006


Bisimulation functions

x1

x2

X

time

( ) ( ){ }εφεφ ≤∈= 2121 ,:, xxXxxB

A bisimulation function is nonincreasing along any two trajectories of the system.


Bisimulation functions

The function φ : X x X ö + is a bisimulation function if the following properties hold

for all x1, x2 ∈ X it is ||g(x1)-g(x2))||2 ≤ φ(x1,x2)for all x1, x2 ∈ X it is

0)(),()(),(2

2

211

1

21 ≤∂

∂+

∂∂ xf

xxxxf

xxx φφ

A. Girard & G.J. Pappas, Approximation Metrics for Discrete and Continuous Systems, IEEE TAC, to appear.


Systems with affine dynamics


Lyapunov equation

Lyapunov equation ... always has a solution for stable A.


Invariance property

x1

Xx2

x1

x1x2

x2


How robust is a test trajectory?

y(t)

Sg(x(t))y(t)f(x(t))(t)x

==&


Consider the dynamical system:

Unsafe

y(t)

)),((infinf0

ztyUnsafezt

φε∈≥

=

When φ is a metric …


How robust is a hybrid test trajectory?

A hybrid automaton is a tuple H = (X, L, E, f, g, U, Inv, Init, G, R, Unsafe)

where X is the continuous state spaceL is the set of control locations E Œ L × L is the set of control switchesInv : L Ø P(X) assigns an invariant set to each locationOut : L × Z Ø V is the control input for S’Init Œ X0 × L is the set of initial conditions G : E Ø P(bd(Inv(l))) is the guard condition that enables transition e=(l,l’)œER : E Ø Inv(l’) is the reset map for the transition e=(l,l’)œEUnsafe Œ X0 × L is the unsafe regionf, g

y(t)

S(x(t))gy(t)(x(t))f(t)x

i

i

==&



How robust is a hybrid test trajectory?

UnsafeInit

L1

L2Note: invariance sets can be different in each location.


One step of the algorithm


What about neighboring trajectories?

UnsafeInit

L1

L2

Bisimulation metric takes care of that …


Robustness implies same qualitative behavior

UnsafeInit

L1

L2

L3


We have timing guarantees, too.

g1g2

act

x0

(x ,d )min0Bφ

^g2

ξ(τ,x )0

ξ (τ+ε,x )0

ξ (τ−ε,x )0

Unsafeact

dunsafe

dout


Main result: Loop Invariance

Thus, a guarantee on the qualitative behavior and timing.


A Testing algorithm

for Hybrid Automata


Covering of the parameter space

It is impossible to cover an uncountable testing parameter space with points.

Initial Conditions


Covering with robust tests

Each test represents a (nonzero measure) neighborhood of testing parameters.

Parameters that lead to tests with the same qualitative properties are grouped together.

Initial Conditions


Each test represents a (nonzero measure) neighborhood of testing parameters.

Parameters that lead to tests with the same qualitative properties are grouped together.

Finite covering is possible!

Only if the system is robust.

Covering with robust tests


Overview of algorithm

Includesinitial

conditions

Max simulation time, max number of tests,etc

Pick a point in theparameter space

Pick a point in theparameter space

SimulatetrajectorySimulatetrajectory

ComputeRobustnessCompute

Robustness

Update parameterspace

Update parameterspace

Remove computed ellipsoid from initial conditions

Stoppingcriterion?

No

OutputResultsOutputResults

Yes

Safe?

Yes

HybridAutomaton

HybridAutomaton

InputParameters

InputParameters

SystemUnsafe

SystemUnsafe

No

Includes the computation of bisimulation functions


Coverage strategies

Randomized strategy: easy to implement, almost impossible to get 100% coverage.

Grid based strategy: easy to implement, suffers from curse of dimensionality.

Minimal dispersal: based on partitioning the parameter space with weighted Voronoipartitions.


Computing distances

linear projections(least squares when we consider the

location dynamics)

quadratic programming Unsafe

semidefiniteprogramming


Some Examples


Navigation benchmark

0 1 2 30

1

2

3

x1

x 2

Unsafe 2 4

2 3 4

2 2 Goal



0 1 2 30

1

2

3

x1

x 2Unsafe 2 4

2 3 4

2 2 Goal


Navigation benchmark 1

0 1 2 30

1

2

3

x1

x 2

Unsafe 2 4

2 3 4

2 2 Goal



With 25 runs, we cover >48% of the initial set.

Notice that there is a clear divide in the initial set, due to different transitions.

0 1 2 30

0.5

1

1.5

2

2.5

3

(a)

x 2

x1

1 1.5 2

0.8

1

1.2

1.4

1.6

1.8

2

(b)

x 2

x1


Benchmark problem 2

Verified to be safe with CHARON

0 1 2 30

1

2

3

x1

x 2

Unsafe 2 4

2 3 4

2 2 Goal


Benchmark problem 2

Safety verified after 9 tests!(All traces have the same qualitative behavior and the system is robust wrt to the unsafe set. Termination guaranteed similar to Girard & Pappas HSCC’06, Fainekos et al FORMATS 2006)

Numerically, we compute a coverage estimate of 72%.

2 2.2 2.4 2.6 2.8 3

1

1.2

1.4

1.6

1.8

2

(a)

x 2

x1

2 2.2 2.4 2.6 2.8 3

1

1.2

1.4

1.6

1.8

2

(b)

x 2

x1



0 1 2 30

1

2

3

x1

x 2

2 3 6

3 3 Goal

2 2 Unsafe



• Test generation using Voronoi with weights

• We verified unsafetywith 10 tests.

-0.5 0 0.5 1 1.5 2 2.5 3 3.5-0.5

0

0.5

1

1.5

2

2.5

3

3.5

x1

x 2


Conclusions and Discussion


Conclusions & Discussion

We have introduced : a notion of robustness for test trajectories of hybrid systemsAn algorithm that computes confidence levels for hybrid systems

A toolbox that helps the exploration of a hybrid system

Early stages of HS designThe algorithm is automatic for hybrid systems with affine dynamicsThe framework can be effectively parallelized


Future Extensions

Temporal logic testing of hybrid systemsFainekos, Girard, Pappas: Temporal Logic Verification Using Simulation, in FORMATS 2006

Probabilistic testingJulius: Approximate abstraction of stochastic hybrid automata, in HSCC 2006

Nonlinear systemsGirard, Pappas: Approximate bisimulations for nonlinear dynamical systems, in CDC 2005

Hybrid Systems with bounded input (noise)Girard, Pappas: Verification using simulation, in HSCC 2006


Thank You! Questions?

robust test generation and coverage for hybrid systems€¦ · 1 robust test generation and...

Documents