MIS 643

Agent-Based Modeling and Simulation

(ABMS)

Bertan Badur

badur@boun.edu.tr

Department of

Management Information Systems

Boğaziçi University

Model Analysis

• Chapter 21-23, of Agent-Based and Individual-Based Modeling: A Practical Introduction, by S. F. Railsback and V. Grimm

Outline

• Chapter 21: Introduction to Part IV• Chapter 22: Analyzing and Understanding ABMs • Chapter 23: Sensitivity, Uncertainty and

Robustness Analysis • Chapter 24:

Chapter 21: Introduction to Part IV

• 21.1 Objectives of Part IV• 21.2 Overview of Part IV

21.1 Objectives of Part IV

• Testing – checking whether a model or submodel is correctly implemented and does what itis supposed to do

• Analysing a model: trying to understand what amodel does

• Understanding not automatic• from begining of modeling cycle

– sukbmodels or simple models– POM for sturucture, theory calibration

• Full models – frase design at some point – understand how it works and behave

• not too soon• once the model

– key processes – represent real system reasonably

• version number• two or three versions is likely• Programming and testing easy• What is science?

– relation between model and real system – POM Part III– analyse throughly – what it does

• simlfy or extend by adding new elements

• formulation few days, analysing months yearns

21.2 Overview of Part IV

• Chapter 22 • general strategies of analyzing ABMs• specific to ABMs

– structural richness and realism

• through controled simulation experiments– change assuptions submodels ...

• Chapter 23– sensitivity, uncertainty and robustness

Sensitivity, Uncertainty and Robustness Analysis

Chapter 22: Analyzing and Understanding ABMs

• 22.1 Introduction• 22.2 Example Analysis: The Segregation Model• 22.3 Additional Heuristics for Understanding

ABMs• 22.4 Statistics for Understanding• 22.5 Summary and Conclusions

22.1 Introduction

• controlled experments– varying one factor at a time – efeects on results

– establishng causal relationships – understanding how the results are affected by each factor

• Scientific method – reproducable experiments– compleatly dercribing the model - lab or field

• documenting– parameter values- input data- initial conditions

– anaylsing results of experments

• controlled simulation experiments– design, test and calibrate - models

– understanding and analyzing what models do

• How to analyse– model, the system and questions addressed,

– experience and problem solving heuristics

• Heuristics or rule of tumbs – often usefull but not always

• not unscientific •

learning objectives

• Understan purpose and goals of analyzing full AMBs– finished or preliminary

• ten heuristics• statistical anaysis for ABMs

22.2 Example Analysis: The Segregation Model

• ODD• purpose• entities, state variables and scales

– turtles – households• loaction, heppyness

– houses - patches

• space 51*51• time • stop – all heppy

• Processes• if all happy stop• far all aent• if lo limit move• update heppyness• produce output

• submodels– move

– update

Analysis

Heukristic: try extream values of parameters

• model outcomes is often easy to predict or understand

• Set tolernce low• Set tolarance high

Heuristics: findtipping points in model behavior

• qualitatively diferent behavior at extream values of parameters

• vary the parameter try to find “tipping point”– the parameter range – model behavior suddenly changes

• regiems of control– process A after some point process B may dominent

Heuristics

• try different visual representations of the model– color size patches

• run the model step by srep• look at striking or strange patterns• at interesting points keep the parmeter and vary

other parameters

22.3 Additional Heuristics for Understanding ABMs

• use several “currencies” for evaluating your simulation experiments

• analyze simplified version of your model• analyze from the buttom up• explore unrealistic senarios

Heuristics: use several “currencies” for evaluating your simulation

experiments• ABMs are rich in structure• “currincies” summary statistics or observations• emprical measures in the real system• Ex: population modeling

– measure – population size wealth

– analyze time series of population size

– even mena or range

• good currincies – observation in ODD design concept

• several currincies – how sensitive they are

• statistical distributions– mean standard deviation, range – distribution – normal, exponential

• characteristics of time series– trend, autocorrolation time units to reach a state

• measures of spatical distributions– spatial autocorrelation, fractile dimension

• measures of difference among agents– how some charcetristics different, distributions

• stability properties • network characteristics

– clustering coefficient, degree

Heuristics: analyze simplified version of your model

• simplfy• ABM so many foctors affect output• reduce complexity

– undertand what mechnizms what cause what results

• make the environment constant• make space homogenuous

– all patches same over time

• reduce stocasticity– fixed initial conditions – all agent alike– insteaad of randomness use mean values

• reduce the system size• turn off some actions in model schedule• manually create simplified initail configrations

Heuristics: analyze from the buttom up

• ABMs hard to understand • behavior of its parts – agents and their behavior• first test and undertsnd these • then full model• anaysis of submodels• developing theory for agnet bahavior

Heuristic: explore unrealistic senarios

• simulate senarios – never occur in reality• to see direct effect of a process or mechanizm on

resutls – remove it• Ex 2: How investor behavior affects double –

auction markets• interesting contrast:

– models – unrealistically simple investor behavior– produce system level results not so unrealistic

• conclusion– complex agent behavior – not reasn for complex market

dynamics– market rules themselfs might be important

22.4 Statistics for Understanding

• statistics – analysis and understanding• infer causal relatinships from a limited and fixed

data• ABM –

– generates as much data aa possible– additional mechnizms

• if cannot explain – add new mechanizms – change assuptions

• purpose and mind-set of – statistics and simulation modeling

• are quite different

• summary sttistics– aggregagting model outputs - mean, standard deviation

– extream values might be importnat so outliers are usefull

• Contrasting senarios– detect and quantify differences between senarios

– assumptions may affect resutls – number of treatments

– easier to change assuptions

– t test ANOVA

• Quantifying correlative relationships– regression ANOVA

– statistical relationsships between inputs – outputs

– inputs: paramerters, initial conditions, time series

– not directly idenfy causal relations

– but idenfity relavant factors

– meta-models

• Comparing model outputs to emprical patterns

22.5 Summary and Conclusions

• combin– reasoning, strong inference, systematic anaysis, intiution and

creativity

• once build an ABM or freeze it• understand what is does – controlled simulation

experiments• heuristics• publications• heuristics in figure 22.3• add your own

Chapter 23: Sensitivity, Uncertainty and Robustness Analysis

• 23.1 Introduction and Objectives• 23.2 Sensitivity Analysis• 23.3 Uncertainty Analysis• 23.4 Robustness Analysis• 23.5 Summary and Conclusions

23.1 Introduction and Objectives

• Does an ABM reproduce observed patterns robustly

• or sensitive to change in model– parameter

– structure

• how uncertain are model results• if model reproduce patterns foır

– parameters – limited range or values

– key processes are modelsed one exact way

• unlikely to capture real mechanizm underlying hhe patterns

Basic Definitions

• Sensitivity analysis (SA) exokıres how sensitive model’s outputs are to changes in parameter values

• Uncertainty Analysis (UA) looks at how uncertainty in parameter values affect the relaibility of model results

• Robustness analysis (RA) explores robustness ofresults and conclusions of a model to changes in its structure

Learning objectives

• local SA with BehavioSpace• visualizations – SA with several parameters or

global SA• stamdard UA methods with BehaviorSpace• steps of conducting RA

23.2 Sensitivity Analysis

• to perform SA• full version of the model• “reference” parameter set• one or two key outputs• controled simulation conditions

23.2.1 Local Sensitivity Analysis

• Objective – how sensitive the model• currency seleced• parameters one at a time• usually all parameters• Steps

– range of parameter – +or-5%

– run model for reference P and p-dP p+dp – replicate

– mean C values

– calculate sensitivity – approximatins to partial derivative

Three types of parameters

• high values of S– processes imortant in the model

• high value of S and highuncertrainty in reference valus– little information to estimate their values

– special attantion as calibration

– target of emprical research to reduce uncertainty

• low values of S– relatively unimportant processes - removable

Alternatives

• only positive change• C’/C absolugtechange• distibuton of C – variance• diferent values of P

– rgression of C on P

Limitations

• linear response so parameter change is small • parameter interractios missing• around reference parameter set

23.2.2 Analysisof Parameter Interractions via Countour Plots

• contour plots – interractions of two parameters– all other parameters are kept constant

• Multi-panel contour figures – model sensitivity – many parameters at onces

23.2.3 Global Sensitivity Analysis

• vary all parameters over their full range • look at several currencies - understanding• “brute force” - analysis

– for each parameter several values

– replicaitons

– hard to measure currencies

• regression analyis – respose surface methods• design of simulation experiments

– not all combination of parameters

23.3 Uncertainty Analysis

• similar to SA but• to understand how

– the uncertainty in parameter values and– model’s sentitivity to parameters

• interract to cause uncertainty in model results• parameters – measurment errors• steps of a UA

– identify the parameters – for each parameter – define a distribution

• belief or measurment errors

– run the modelmany times – drawing from distributions– analyze distribution of model results

23.4 Robustness Analysis

• Weisberg (2006)• Whether the results depends on the

– esentials of the model or

– details of the simplfying assuptions

• study number of distinct similar models of the same phenomena

• despte different assumptions – similar results– robust theorm - free of details of the model

• modeling, POM– robust explanations of observed patterns

• A full model – frozen• two heuristics:

– analyze simplified versions

– explore unrelistic senarios

• more complex versions• General steps of RA

– start with a well tested model version

– which elements to modify

– test modified model – reproduce observed patterns

• theory development – agent behavior– testing alternative submodels

• RA– testing alternative versions

• 23.4.1 Example: Robustness Analysis of the Breeding Synchrony Model– left as an exercise

23.5 Summary and Conclusions