Model validity, testing and analysis

Conceptual and Philosophical Foundations

• Model Validity and Types of Models– Statistical Forecasting models (black box)– Descriptive Policy models (transparent)

• Philosophical Aspects- Philosophy of Science- Logical Empiricim and Absolute Truth- Conversational justification & relative truth (‘purpose’)- Statistical significance testing

(Barlas and Carpenter 1990 and Barlas 1996)

Two aspects of model validity

• Structure Validity– Primary importance– Special place in System Dynamics

• Behavior Validity– Role in system dynamics– The special type of behavior validity in system dynamics– Ex ante versus ex post prediction

(Barlas 1996 and 1989)

Overall Nature and Selected Tests ofFormal Model Validation

Logical Sequence of Formal Steps ofModel Validation

Structure Validity

• (Simulation Verification)

• Direct Structure Tests– Crucial, yet highly qualitative and informal– Distributed through the entire modeling methodology

• Indirect Structure Tests (Structure-oriented behavior)– Crucial and partly quantitative and formal– Tool: SiS software

Indirect Structure Testing Software: SiS

• Based on automated dynamic pattern recognition

• Extreme condition pattern testing

• Also in parameter calibration and policy design

(Kanar 1999; Kanar and Barlas 1999; Bog et al 2004)

Indirect Structure Testing Software (SiS)

Basic Dynamic Patterns

Indirect Structure Testing Software (SiS)

List of dynamic behavior pattern classes

Software Implementation

Our Software (SiS)

Main

ISTS Algorithm

Simulation

Software

8

12

3

4Integrator

5

67General Picture of the Processes in Validity Testing mode

General Picture of the Processes in “Parameter Calibration” mode

Sample Model Used with SiS

Orders inProcess

orders processing testing

AwaitingActivation

activating

fraction facilities ready

fraction facilities good

Orders RequiringService

dispatching

ProcessingCapacity

TestingCapacity

DispatchingCapacity Activating

Capacity

target process delaytarget tes t delay

start orders

target activationdelay

target service delay

Orders RequiringTesting

proc adj time

dispatch adj time

test adj time

activation adj time

NewCustomers

Validity Testing with Default Parameters

0

10000000

20000000

30000000

40000000

50000000

60000000

70000000

80000000

0 1 2 3 4 5 6Simulation Output (with default base parameters)

Likelihood Values of simulation behavior correctly classified as the GR2DB pattern

Validity Testing by Setting Parameters

0

10000000

20000000

30000000

40000000

50000000

60000000

70000000

80000000

90000000

0 1 2 3 4 5 60

10000000

20000000

30000000

40000000

50000000

60000000

70000000

80000000

90000000

100000000

0 1 2 3 4 5 6

Fig1 : Simulation Output (with base parameters) Fig2 : Simulation Output (with changed parameters)

Likelihood Values of simulation behavior in Fig2 compared to the NEXGR pattern

Parameter Calibration with Specified Pattern

0

10000000

20000000

30000000

40000000

50000000

60000000

70000000

80000000

90000000

0 1 2 3 4 5

The ranges and number of values tried for each parameter

Simulation Output (with base parameters)

Result of the Parameter Calibration 

0

20000000

40000000

60000000

80000000

100000000

120000000

140000000

160000000

180000000

200000000

0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5

Best parameter set is 41Best Likelihood Result: 1.2119776136254248 Best Parameter Set: 1. advertising effectiveness: 0.252. customer sales effectiveness: 6.03. sales size: 1.0

Simulation Output as Desired (after automated parameter calibration)

Parameter Calibration with Input Data

A view of the SiS interface during parameter calibration

Result of the Parameter Calibration 

0

10000000

20000000

30000000

40000000

50000000

60000000

0 1 2 3 4 50

5000000

10000000

15000000

20000000

25000000

30000000

35000000

0 1 2 3 4 5

Best parameter set is 21Best Likelihood Result: 3.7109428620957883 Best Parameter Set: 1. advertising effectiveness: 5.02. customer sales effectiveness: 0.0

Fig1 : Simulation Output (with base parameters)

Fig2 : Simulation Output (after parameter calibration to match the input pattern)

Behavior Validity

• Two types of patterns– Steady state– Transient

• Major pattern components– Trend, periods, amplitudes, ...

Behavior Validity Testing Software: BTS II

Uses of BTS II and SiS in Model Analysis

• Analysis: Understanding the dynamic properties of the model

• BTS II can assist in quantifying, measuring and assessing dynamic pattern components

• SiS can assist in deeper structural analysis (related to qualitative pattern modes)

Uses of BTS II and SiS in Policy Design

• BTS II can assist in numerical performance improvement policies

• SiS can assist in more structural dynamic pattern improvement

• Parameter calibration can be extended to cover automated policy design

Implementation Issues

• More tools• User friendliness• More thorough (field) testing of the tools• Better integration with simulation software...

Policy Implementation Issues

• Validity of the policy recommendation(Robustness, timing, duration, transition...)

• Finally, ‘validity of the implementation’ itself– Validated model means just a reliable

laboratory; implementation validity does not automatically follow; it is a whole area in itself

Concluding Observations

• Validity as a process, rather than an outcome• Continuous (prolonged) validity testing• Validation, analysis and policy design all integrated• From validity towards quality• Quality ‘built-in versus inspected-in’• Group model building• Testing by interactive gaming

Back to philosophy...

• A gradual, continuous, multi-method, qualitative and quantitative, formal and informal process of establishing confidence in a model. We should use any formal test/tool compatible with this philosophy, but never assume that tools themselves would be sufficient without proper philosophy

DISCUSSION

Reference

Yaman BarlasBoğaziçi University

Industrial Engineering Department34342 Bebek Istanbul, Turkey

ybarlas@boun.edu.trhttp://www.ie.boun.edu.tr/~barlas

SESDYN Group: http://www.ie.boun.edu.tr/labs/sesdyn/