institute of computer and communication network engineering multi-dimensional robustness...

Institute of Computer and Communication Network Engineering

Multi-dimensional Robustness Optimization of Embedded Systems

& Online Performance Verification

Arne Hamann

Steffen Stein

Rolf Ernst


Part I:Multi-dimensional Robustness Optimization of Embedded Systems

Arne Hamann

Rolf Ernst

3

Arne Hamann, Steffen Stein, IDA, TU Braunschweig

Outline

• System property variations

• Sensitivity Analysis

• Stochastic Multi-dimensional Sensitivity Analysis

• Robustness Metrics– Hypervolume calculation– Minimum Guaranteed Robustness (MGR)– Maximum Possible Robustness (MPR)

• Experiments

4


System Property Variations

• Why do system property variations occur?– Specification changes, late feature requests,

product variants, software updates, bug-fixes

• Robustness to property variations– decreases design risk, and increases system

maintainability and extensibility

• Property variations can have severe unintuitive effects on system performance

• Sensitivity analysis: achieve robustness without on-line parameter adaptation

5


Problem Formulation

• Find fixed parameter configuration that …• … maximizes system robustness w.r.t.

changes of several properties• Robustness = the system can sustain

property variations without severe performance degradation

• Not included: dynamic parameter adaptations (ongoing work submitted to EMSOFT 2007)

6


Stochastic Sensitivity Analysis (1)

• Problem of exact sensitivity analysis approaches: computational effort grows exponentially with number of considered dimensions

• Solution: scalable stochastic analysis able to quickly bound system sensitivity

7


Stochastic Sensitivity Analysis (2)

• Sensitivity analysis formulated as multi-objective optimization problem

Pareto-front of optimization task corresponds to sought-after sensitivity front

• Use multi-criteria evolutionary algorithms to approximate sensitivity front– E.g. SPEA2 (ETH Zurich): diversified sensitivity

front approximation through Pareto-dominance based selection and density approximation

8


Creation of the Initial Population

• Creates a certain number of points representing a first approximation of sensitivity front

• Uses 1-dim sensitivity analysis– to bound the search space in each dimension

(bounding hypercube)– to generate points representing the extrema of the

sought-after sensitivity front

• Randomly place the rest of the initial points in bounding hypercube

9


6

6,5

7

7,5

8

8,5

9

9,5

10

10,5

11

9 11 13 15 17 19 21 23 25 27

6,5

10,85

26,210Property 1

Pro

pert

y 2

Bounding Box

Initial Population - Example

10


Bounding the Search Space (1)

• Idea: bound search space containing the sought-after sensitivity front– Bounding working Pareto-front F n

• evaluated Pareto-optimal working points

– Bounding non-working Pareto-front F nw

• evaluated Pareto-optimal non-working points

• Bounding Pareto-fronts can be used to derive multi-dim. robustness metrics (later)

11



• Space between bounding Pareto-fronts is called relevant region

• Variation operators use algorithm ensuring that generated offsprings (points) are situated in the relevant region– Below bounding non-working Pareto-front– Above bounding working Pareto-front

Efficiently focuses exploration effort

12



6

6,5

7

7,5

8

8,5

9

9,5

10

10,5

11

9 11 13 15 17 19 21 23 25 27

6,5

10,85

26,210Property 1

Pro

pert

y 2

Bounding Box

13


Front Convergence Mutate (1)• Heuristic operator adapted to optimization

problem• Strategy:

– Determine X closest points on opposite Pareto-front

– Choose randomly one of these points– Place offspring point randomly on straight line

connecting the parent point and the chosen random point

• Increases convergence speed of the bounding Pareto-fronts

14


Front Convergence Mutate (2)

6

6,5

7

7,5

8

8,5

9

9,5

10

10,5

11

9 11 13 15 17 19 21 23 25 27

6,5

10,85

26,210Property 1

Pro

pert

y 2

Bounding Box

15


Front Convergence Mutate (3)

6

6,5

7

7,5

8

8,5

9

9,5

10

10,5

11

9 11 13 15 17 19 21 23 25 27

6,5

10,85

26,210Property 1

Pro

pert

y 2

Bounding Box

16


Hypervolume Calculation• Hypervolume as basis of the proposed

robustness metrics• Hypervolume is defined in a given

hypercube and associated to a point set• Two different notions of hypervolume

– inner hypervolume : Volume of space Pareto-dominated by the given points inside the given hypercube

– outer hypervolume : Volume of space Pareto-dominated by all points not Pareto-dominating any of the given points

17


Hypervolume Calculation (2)• 2D-case

– inner hypervolume: lower step function– outer hypervolume: upper step function

4

6

8

10

12

14

16

18

20

14 16 18 20 22 24 26 28 30

Bounding Box[15,28]x[6,18]

(15,18)

(18,16)

(20,12)

(26,10)

(28,6)

( )= 66λ-( )= 100λ+

18


Robustness Metrics

• Given a set of properties …

• … use stochastic sensitivity analysis to derive upper and lower robustness bounds– Minimum Guaranteed Robustness (MGR)

• Defined as inner hypervolume of the bounding working Pareto-front F w

– Maximum Possible Robustness (MPR)• Defined as outer hypervolume of the bounding

non-working Pareto-front F nw

19


Robustness Metrics (2)

Obviously: MGR <= Real Robustness <= MPR

6

6,5

7

7,5

8

8,5

9

9,5

10

10,5

11

9 11 13 15 17 19 21 23 25 27

6,5

10,85

26,210Property 1

Pro

pert

y 2

Bounding Box

MGRMPR

20


Robustness Exploration

• Idea: Pareto-optimize MGR and MPR

• Advantages– Stochastic sensitivity analysis is scalable

Little computational effort necessary to reasonably bound robustness potential of given configuration

– In-depth analysis can be performed once interesting configurations are identified (i.e. high MGR or high MPR)

Perfectly suited for robustness optimization

21


Example System

• Distributed embedded system

• 4 computational resources …

• …connected via CAN bus

• 3 constrained applications– SensAct

– SinSout

– CamVout

22


Approximation Quality (1)

• Approximation after 100 evaluations (20 sec)

• MGR = 2447• MPR = 2937


• MGR = 2580• MPR = 2813

23


Approximation Quality (2)


• MGR = 2632• MPR = 2777

• Result using exact sensitivity analysis (85 sec)

• MGR = 2585• MPR = 2826

24


3D - Robustness Maximization

Original configuration

Optimized configuration

25


Integration of New Functionality

• Integration of a fourth application with lowest priorities

• What combinations WCET T9 and WCCT C6 are feasible?

• Is there optimization potential?

• Idea: initially assume WCET T9 and WCCT C6 equal zero

T9

C6

Sens2

Sink

26


Integration of New Functionality (2)

0

5

10

15

20

25

30

0 5 10 15 20 25 30 35 40

Original

Optimized

WC

CT C

6

WCET T9

• Areas below the curves represent feasible systems


Part II:Online Performance Verification

Steffen Stein

Rolf Ernst

28


Outline

• Motivation

• Framework Architecture

• In Detail: Global Analysis Layer

• System Setup

• Approach to Analysis Control

• Experimental Results

29


Future Challenges

• In-Field updates• Run-time Reconfigurations• 90% of Innovation in Software • Networked Systems

Not Manageable

at Design-Time!

0

5

10

15

20

25

30

0 5 10 15 20 25 30 35 40

Original

Optimized

Engine Control SW

Driver Assistance

Multimedia Service

30


Approach

• Generally Speaking:– Make Systems clever enough to handle Integration

Problem themselves

• Here: Timing Properties• ToDo

– Gather performance Data during runtime– Evaluate/ Optimise online– Feed Results back into running Systems

• Result: Evolving Systems

31


Architecture: Organic Computing

• Single Instance• Multiple Instances• Multiple collaborating

Instances• Layered approach

System under Observationand Control (SuOC)

observer controller

observes controls

reports

selects observation model

Goals/Design Rules

Source: Towards a generic observer/controller architecture for Organic Computing, U. Richter, M. Mnif, J. Branke, C. Müller-Schloer, H. Schmeck, INFORMATIK 2006 -- Informatik für Menschen

32


Analysis EngineAnalysis EngineObserver ControllerObserver Controller

Heterogeneous Networked Embedded System (SuOC)

Analysis Engine

Control Plane

Local Layer

Use resourcesGather data Adjust settings

Observer ControllerData

Exchange

Global Analysis Layer

Global Controller Layer

Global Observer Layer

Self-Organisation

Control Framework

Self-Organisation

33


Distributed Setup

ARMDSP

uCPPC

CAN

Real System

T2

T5

T1

T6

S2

S1

S4

S3

T8

T0

T7

T9

T3

T4

Global Model

34


Analysis Control

T1

T3

T2

T4

T1

T6

Net

wor

k T

unne

l

Distributed Analysis Control

T1

T3

T2

T4

T1

T6

Analysis Control

Dofor all not up-to-date Resources

Analyseend

Until all Resources are up to date

While (true)if Resource invalidated

analyse Resourceend

end

35


Performance of trivial Approach

0

100

200

300

400

500

600

700

800

900

1000

0 20 40 60 80 100

Live

offline

System size (# tasks)

# a

naly

sis

ru

ns (

resou

rce level)

36


Problem

T1

T4

T2

T5

S1

S2

T3

T6

T7 T8S3 T9

Exponential increase in number of necesary Analysis runs

Solution: Caching

37


Performance with caching

0,00

20,00

40,00

60,00

80,00

100,00

120,00

140,00

160,00

180,00

0,00 20,00 40,00 60,00 80,00 100,00 120,00

Live

Offline

System size (# tasks)

# a

naly

sis

ru

ns (

resou

rce level)

38


Conclusion

• Distributed Performance Analysis implemented

• Suitable as evaluator for online performance control / optimization

• Future Work: From System observations to analysable Model.

institute of computer and communication network engineering multi-dimensional robustness...

Documents

system sensitivity slide

soughtafter sensitivity

approximation of sensitivity

dim sensitivity analysis

system robustness

approximate sensitivity

diversified sensitivity

hypercube slide