mil testing of highly configurable continuous controllers

Reza Matinnejad Shiva Nejati Lionel Briand Software Verification and Validation Group SnT Center, University of Luxembourg Thomas Bruckmann Delphi Automotive Systems, Luxembourg

MiL Testing of Highly Configurable Continuous Controllers:

Scalable Search Using Surrogate Models

Closed-loop Continuous Controllers An Example: Conveyor Belt Controller

Controller

1

+ -‐

2

Closed-loop Controller

Closed-loop Continuous Controllers An Example: Conveyor Belt Controller

Closed-loop Continuous Controllers

3

Desired Value

Controller Model

System Output

Model-based Development of Embedded Software

Plant Model

+ -

Model-in-the-Loop Stage (MiL)

Hardware-in-the-Loop Stage (HiL)

4

Software-in-the-Loop Stage (SiL)

Configuration 1

Configuration 2 Configuration 3

Controller Model

Plant Model

5

MiL Testing Continuous Controllers (1) Instantiating the Controller Model

•  Step1: Identifying an assignment of values to the configuration parameters of the controller model

Des

ired/

Act

ual V

alue

Des

ired/

Act

ual V

alue

Time Time

MiL Testing Continuous Controllers (2) Running Model Simulation

6

Test Input 1

•  Step2: Running simulations of the controller model and examining the output signals

Test Input 2 / Output 1 / Output 2

Controller Requirements (1)

Time Time

Act

ual V

alue

•  Stability: The actual value shall stabilize at the desired value

Act

ual V

alue

7

Stability Violation

Stability Violation


•  Smoothness: The actual value shall not abruptly change when it is close to the desired value

Time

Act

ual V

alue

8

Smoothness Violation


9

Time

Act

ual V

alue

Responsiveness Violation

•  Responsiveness: The controller shall respond within a certain time-limit

Stability Objective Function: Fst

•  Fst (Test Case A) > Fst (Test Case B) •  We look for test input that maximizes Fst

Test Case A Test Case B

Act

ual V

alue

Act

ual V

alue

Time Time

10

Smoothness Objective Function: Fsm

•  Fsm (Test Case A) > Fsm (Test Case B) •  We look for test input that maximizes Fsm

Act

ual V

alue

Time Time


11

Act

ual V

alue

Responsiveness Objective Function: Fr

•  Fr (Test Case A) > Fr (Test Case B) •  We look for test input that maximizes Fr

Act

ual V

alue

Time Time


Act

ual V

alue

12

Our Approach: Search-based Testing of Continuous Controllers

13

•  Step 1 : Exploration

•  Step 2 : Search

Controller-Plant Model

Controller BehaviorOver the

Input Space

1.Exploration

Plant ⌃

Controller

Worst-Case Scenarios

CriticalOperatingRegions of

the Controller

2.Search

FD

ID FD

ID

Our Earlier Work: Search-based Testing of Continuous Controllers

with fixed values for the configuration parameters

Initial Desired (ID)

Fina

l Des

ired

(FD

)

Smoothness HeatMap Smoothness Worst-case Scenario

Controller-plant model

HeatMap Diagram


List of Critical Regions

DomainExpert

1.Exploration 2.Single-statesearch

Controller Input Space

14

Fst Fsm Fr

Published in [IST Journal 2014, SSBSE 2013]


HeatMap Diagram



DomainExpert

1.Exploration 2.Single-statesearch

Challenges When Searching the Entire Configuration Space

1.  The input space of the controller is larger •  6 configuration parameters in our case study

4.  Search becomes slower •  It takes 30 sec to run each model Simulation

2.  Not all the configuration parameters matter for all the objective functions

3.  Harder to visualize the results

15

Our approach for Testing the Controller in the Entire Configuration Space

Dimensionality ReducLon to focus the exploraLon on the variables with significant impact on each objecLve funcLon

VisualizaLon of the 8-‐dimension space using Regression Trees

Surrogate Modeling to avoid running simulaLons for parts of the input space where the surrogate model is the most accurate.

16


+Configuration

parametersranges

Regression Trees



DomainExpert

1.Exploration with Dimensionality

Reduction

x<0.2

y<0.3

z<0.1

2. Search with Surrogate Modeling

Exploration with Dimensionality Reduction Elementary Effects Analysis Method

•  Goal: IdenLfying variables with a significant impact on each objecLve funcLon

17

Exploringall the

dimensions

Exploring the significant

dimensions

ProportionalGainDerivativeGain

IntegralGainIntgrResetErrLim

...

SmoothnessProportionalGainIntgrResetErrLim

...

ElementaryEffects

AnalysisMethod

...

Regression Tree

•  Goal: Dividing the input space into homogenous parLLons with respect to the objecLve funcLon values

Smoothness Regression Tree

18

All Points

ID < 0.7757

Count MeanStd Dev

Count MeanStd Dev

ID >= 0.7757Count MeanStd Dev

FD >= 0.5674 FD < 0.5674Count MeanStd Dev

Count MeanStd Dev

1000 0.007

0.0049

574 0.0059

0.004

426 0.01034250.0049919

182 0.0134

0.005

244 0.0080.003

FD < 0.1254 FD >= 0.1254Count MeanStd Dev

Count MeanStd Dev

182 0.01345550.0052883

244 0.00802060.0031751

Smoothness HeatMap

FD<0.5674

FD>=0.1254

Smoothness Critical Partition

Search With Surrogate Modeling Supervised Machine Learning

•  Goal: To predict the values of the objecLve funcLons within a criLcal parLLon and speed up the search

19

•  Surrogate model predicts Fsm with a given confidence level

A

B •  During the search, surrogate model predicts Fsm for the next point: 1.  Not confident about Fsm(B) and

Fsm(A) relation •  Run the simulation as before

2.  Confident that Fsm(B) > Fsm(A) •  Run the simulation as before

3.  Confident that Fsm(B) < Fsm(A) •  Avoid running the Simulation

Search With Surrogate Modeling Supervised Machine Learning

•  Different Machine Learning techniques: 1.   Linear Regression 2.   ExponenLal Regression 3.   Polynomial Regression (n=2) 4.   Polynomial Regression (n=3)

•  Criteria to compare different techniques: 1.   R2 ranges from 0 to 1

•  R2 shows goodness of fit 2.   Mean RelaLve PredicLon Error (MRPE)

•  MRPE shows predicLon accuracy

20

Evaluation: Research Questions

•  RQ1 (Which ML Technique): Which Machine Learning technique performed the best?

21

•  RQ2 (Effect of DR): What was the effect of Dimensionality Reduction?

•  RQ3 (SM vs. No-SM): How did the search with Surrogate Modeling perform comparing to the search without Surrogate Modeling?

•  RQ4 (Worst-Case Scenarios): How did our approach perform in finding worst-‐case test scenarios?

RQ1: Which Machine Learning Technique?

22

•  The best technique to build surrogate models for all our three objecLve funcLons is Polynomial Regression (n=3) •  PR (n=3) has the highest R2 ( goodness of fit ) •  PR (n=3) has the smallest MRPE ( predicLon error )

•  The surrogate models are accurate and predicLve enough for Smoothness and Responsiveness (R2 was close to 1, MRPE was close to 0) •  In future, we want to try other Supervised Leaning

techniques, such as SVM

RQ2: Effect of Dimensionality Reduction

23

•  Dimensionality reducLon helps generate beber surrogate models for Smoothness and Responsiveness •  Smaller MRPE: More predicLve surrogate model

Smoothness Responsiveness 0.03

0.02

0.01 No DR DR

MR

PE

0.02

0.03

0.04

MR

PE

Mean Relative Prediction Error (MRPE)

No DR DR

RQ3: Search with vs. without Surrogate Modeling

24

•  For responsiveness, the search with SM was 8 Lmes faster

0.16

0.165

0.17 H

ighe

st v

alue

of

Fr

After 300 Sec After 2500 Sec

•  For smoothness, the search with SM was much more effecLve

Hig

hest

val

ue

of F

sm

After 800 Sec After 2500 Sec

SM No SM

0.22

0.225

0.23

SM No SM 0.22

0.225

0.23

SM

No SM

SM

0.16

0.165

0.17

SM No SM

No SM

RQ4: Worst-Case Scenarios (1)

25

•  Our approach is able to idenLfy criLcal violaLons of the controller requirements that had neither been found by our earlier work nor by manual tesLng

MiL Testing Varied Configurations

(Current work)

MiL Testing Fixed Configurations

(Earlier Work)

ManualMiL Testing

(Industry Practice)

Stability

Smoothness

Responsiveness

2.2% deviation - -24% over/undershoot

170 ms response time

20% over/undershoot

80 ms response time

5% over/undershoot

50 ms response time

RQ4: Worst-Case Scenarios (2)

26

•  For example, for the industrial controller we idenLfied the following violaLon of the Stability requirement within the given ranges for the calibraLon variables

ID = 0.36

FD = 0.22

Flap

Pos

ition

Time

2.2 % Deviation

27

Conclusion

28

Reza Matinnejad PhD Candidate Software Verification and Validation Group SnT Center, University of Luxembourg Emails: [email protected] [email protected]

MiL Testing of Highly Configurable Continuous Controllers:

Scalable Search Using Surrogate Models

Alternative Heatmap

Stability Responsiveness

ü  Our approach was implemented at Technical University of Munich too

ExisLng Tool Support for TesLng Simulink/Stateflow Models

•  None of the exis,ng tools specifically test the con,nuous proper,es of Simulink output signals

Related Work

Technique Examples Limitations and Differences

Mixed discrete-continuous modeling techniques

• Timed Automata • Hybrid Automata • Stateflow

• Require model translation • Scalability issues • Verification of logical and

state-based behavior

Search-based testing techniques for Simulink models

• Path coverage or mutation testing

• Worst-case execution time

• Mainly focus on test data

generation • Only applied to discrete-event

embedded systems

Control Engineering • Ziegler-Nichols Tuning Rules

• Design/configuration

optimization • Signal analysis and generation

Commercial tools in automotive industry

• MATLAB/Simulink Design Verifier

• Reactis tool

• Combinatorial and Logical

Properties • Lack of documentation

31

CoCoTest Results

Stability

Violation

32