asking the oracle: introducing forecasting principles into agent-based modelling grasia, universidad...

Asking the Oracle: Introducing Forecasting Principles into

Agent-Based Modelling

GRASIA, Universidad Complutense de Madrid

INSISOC, Universidad de Burgos

GUESS, Universidade de Lisboa

Samer Hassan

Javier Arroyo

José M. Galán

Luis Antunes

Juan Pavón

Samer Hassan WCSS 2010 2

Contents

A recurrent issue

The Field of Forecasting

Forecasting Principles into ABM

Conclusions


A recurrent issue… in ABM

How many times have we heard…

Are accurate predictions possible within ABM?

Should predictions be the main aim of ABM?

Is ABM mature enough to make proper predictions?

Stakeholders want predictions: is ABM an answer?


A recurrent issue… in SIMSOC

“Does anyone know of a correct, model-based forecast of the impact of any social policy?” Scott Moss, SIMSOC list, April 2009, “any correct policy impact

forecasts?”


A recurrent issue… in SIMSOC

“Does anyone know of a correct, model-based forecast of the impact of any social policy?” Scott Moss, SIMSOC list, April 2009, “any correct policy impact

forecasts?”

“The response, once misunderstandings were sorted out, was several accounts of reasons why policy impacts could not be forecast. The criteria I suggested for deeming a forecast to be correct was the correct forecast of the timing and direction of change of specified indicators.” Scott Moss, SIMSOC list, June 2009, “what is the point?”


A recurrent issue… in JASSS

Joshua Epstein (2008): Prediction is one possible aim for ABM… among 16 others

• Explanation• Guiding data collection• Raise new questions• Challenge theories• …

“Explanation does not imply Prediction”• Tectonics explain earthquakes but cannot predict them



Joshua Epstein (2008): Prediction is one possible aim for ABM… among 15 others

• Explanation• Guiding data collection• Raise new questions• Challenge theories• …

“Explanation does not imply Prediction”• Tectonics explain earthquakes but cannot predict them

Thompson & Derr (2009): “Good explanations predict” An explanatory model is valid only if it predicts real behaviour



Klaus Troitzsch (2009) Epstein & Thompson discuss different “Prediction levels”:

1) Prediction of the kind of behaviour of a system, under arbitrary parameter combinations and initial conditions

o Earthquakes occur because X and Y

2) Prediction of the kind of behaviour of a system in the near future

o Region R is likely to suffer earthquakes in the following years because X and Y

3) Prediction of the state a system will reach in the near future1) Region R will suffer an earthquake of power P in expected day D with

confidence C

1) “Explanation does not imply 3rd level Prediction” “Good explanations usually imply 1st or 2nd level”


A recurrent issue

Agent-Based Modelling has multiple aims…

…but still modellers might seek prediction…

How could we help them?


Contents

A recurrent issue



Conclusions


The field of Forecasting

Forecasting A field focused on the study of prediction

• Specially aiming 3rd level

30 years experience

• Consolidated (journals, conferences)

• Formalised

• “Forecasting experiment”



Using ABM as a Forecasting tool

Setting up a forecasting experiment: Split data in two sets

“Objective” error measures

Compare the model

Fair Comparison





• Training set (in-sample): calibration

• Test set (out-of-sample): validation


Compare the model

Fair Comparison






• Error(t)= forecasted(t) - actual_value(t)

• Aggregated Error of time series:

• Root Mean Square Error

• Mean Absolute Error…

Compare the model

Fair Comparison






Compare the model

• Benchmarks: other models, not necessarily ABM

• Naïve method (at least): V’(t+1)= V(t)

Fair Comparison






Compare the model

Fair Comparison

• Representative, large sample of forecasts

• Ex-ante: forecast of (t+1) uses info available until (t)

• Out-of-sample: not include training data in comparison


Contents

A recurrent issue



Conclusions



Principles of Forecasting Armstrong (2001) with 40 authors Summarising the best practices

Selection of subset for ABM

Six topics: Modelling Process Use of data Space of solutions Stake-holders Validation Replication



Modelling Process Decompose the problem into parts

• Bottom-up approach + combination of results

Structure problems that involve causal chains• Results of a (sub)model as input for next one• More accurate than global simulation

Consider the use of adaptive forecasting models• ABM as adaptive systems



Data-driven modelling Use theory to guide the search for information on

explanatory variables

• Reduce complexity pruning design space in advance

Use diverse data sources

• Increase of data reliability

Keep forecasting method simple

• KISS

Select simple methods unless empirical evidence calls for a more complex approach

• KISS + gradual increase of complexity on demand



Space of solutions Identify possible outcomes prior to making forecasts

• Avoid biases

Design test situations to match the forecasting problem

• Put forward scenarios to rehearse policies

Adjust for events expected in the future• Expectability should guide design space exploration and

what-if questioning



Stake-holders and Policy-makers Obtain decision makers' agreement on methods

• Ideally participatory simulation

Ask unbiased experts to rate potential methods• Emphasising their role

Test the client's understanding of the methods• Including limitations of the model

Establish a formal review process to ensure that forecasts are used properly

• Policy deployment should be controlled



Validation List all the important selection criteria before

evaluating methods• Temptation of redefining criteria to fit the outcomes

Use “objective” tests of assumptions• Quantitative approach to test assumptions when possible

Use extensions of evaluations to better generalise about what methods are best for what situations

• Generalisation leads to applicability; based on what-if scenarios

Use error measures that adjust for scale in the data• Error measuring is as important as accuracy of data

Establish a formal review process for forecasting methods

• Ensure verification, replication, trust



Replication Compare track records of various forecasting methods

• The role of replication for ABM verification

Assess acceptability and understandability of methods to users

• Sharing of models & code

Describe potential biases of forecasters• From both modellers and stakeholders• How sensitive is the model to those biases?


Contents

A recurrent issue



Conclusions


Conclusions

The choice of Agent-Based Modelling implies Interest in the “what” is going to happen (Prediction) Interest in “how” the phenomenon occurs (Understanding)

Prediction (3rd level) is a hard job Financial crisis Climate change

Forecasting Principles Best practices, not a solution Helpful in seeking the “what”


Thanks for your attention!

Samer Hassan

[email protected]

Universidad Complutense de Madrid

mailto:[email protected]


Contents License

This presentation is licensed under a

Creative Commons Attribution 3.0 http://creativecommons.org/licenses/by/3.0/

You are free to copy, modify and distribute it as long as the original work and author are cited

Para ver esta película, debedisponer de QuickTime™ y de

un descompresor TIFF (sin comprimir).

asking the oracle: introducing forecasting principles into agent-based modelling grasia, universidad...

Documents