1

Andrea Saltelli,European Commission, Joint Research Centre

andrea.saltelli@jrc.it

ECOINFORMATICS meeting US Environmental Protection

Agency, Research Triangle Park,

North Carolina, April 2008

Models, Uncertainty and Sensitivity

2

On uncertainty – 1 "That is what we meant by science. That both question and answer are tied up with uncertainty, and that they are painful. But that there is no way around them. And that you hide nothing; instead, everything is brought out into the open".

Borderliners, Peter Høeg, Delta publisher, 1995

3

On uncertainty – 2

[…] EPA’s science panel found that “quantitative evidence […] must … be characterized as having high uncertainties.” What to do in the face of uncertainty is a policy question, not a scientific question. [..] The debate is about […] what kinds of uncertainty can be tolerated as a basis for decision-making.

Hazy reasoning behind clean air

David Goldston, Nature 452|3,

April 2008

‘Science alone can’t determine how regulations

are written’

4

Source: Scientific American, Jun2005, Vol. 292, Issue 6

How to play uncertainties in environmental regulation …

On uncertainty – 3

5

- Fabrication (and politicisation) of uncertainty

The example of the US Data quality act and of the OMB “Peer Review and Information Quality” which”seemed designed to maximize the ability of corporate interests to manufacture and magnify scientific uncertainty”.

6

About the OFFICE OF MANAGEMENT AND BUDGET (OMB) Proposed Risk Assessment Bulletin (January 9, 2006)

http://www.whitehouse.gov/omb/inforeg/

‘OMB under attack by US legislators and scientists’

“The aim is to bog the process down, in the name of transparency” (Robert Shull).

Source: Colin Macilwain, Safe and sound? Nature, 19 July 2006.

“Main Man. John Graham has led the White House mission to change agencies' approach to risk” ibidem in Nature

7

The critique of models and what

sensitivity analysis has to do

with it

8

Jared Diamond’s ‘Collapse’ versus Michael Crichton’s ‘State of Fear’

9

Rising sea level will threaten “ … cities of the United Kingdom (e.g. London), India Japan and the Philippines.”, p. 493.

10

Michael Crichton presents ‘adversarial’ opinion on retreating glaciers and thickness of Antarctic ice cap – and contends that sea levels are not rising.

11

“They talk as if simulation were real-world data. They ‘re not. That ‘s a problem that has to be fixed. I favor a stamp: WARNING: COMPUTER SIMULATION – MAY BE ERRONEOUS and UNVERIFIABLE. Like on cigarettes […]” p. 556

12

For sure modelling is subject toady to an unprecedented critique, which is no longer limited to post-modern philosophers but involves intellectuals and scientists of different political hues.

Have models fallen out of grace?

13

Useless Arithmetic: Why Environmental Scientists Can't

Predict the Futureby Orrin H. Pilkey and Linda

Pilkey-Jarvis

‘Quantitative mathematical models used by policy makes and government administrators to form environmental policies are seriously flawed’

14

One of the examples discussed concerns the Yucca Mountain repository for radioactive waste disposal, where a very large model called TSPA (for total system performance assessment) is used to guarantee the safe containment of the waste. TSPA is Composed of 286 sub-models.

15

TSPA (like any other model) relies on assumptions -- a crucial one being the low permeability of the geological formation and hence the long time needed for the water to percolate from the desert surface to the level of the underground disposal.

Evidence was produced which could lead to an upward revision of water permeability of 4 orders of magnitude

(The 36Cl story)

16

The narratives:

‘How bad is the modeling that supports the Department of Energy's assertions about the safety and permanency of the Yucca Mountain nuclear waste dump? Execrable, according to legendary Duke University geologist Orrin Pilkey and his geologist daughter, Linda Pilkey-Jarvis, who works for the Washington state ecology department.’

Ken Maize Power Blog

17

We just can’t predict, concludes N. N. Taleb, and we are victims of the ludic fallacy, of delusion of uncertainty, and so on. Modelling is just another attempt to ‘Platonify’ reality …

Nassim Nichola

Taleb, The Black Swan,

Penguin, London 2007

18

Many will disagree with Pilkey and Taleb.

Yet, stakeholders and media alike expect instrumental use of models, amplification or dampening of uncertainty as a function of convenience and so on.

19

The IFPRI had raised about $460,000 for the modeling, which would have provided insights to help policymakers […]

[… ] But Greenpeace’s Haerlin and others objected that the models were not “transparent”.

Source: Dueling visions for an hungry world, Erik Stokstad, 14 MARCH 2008, 319 SCIENCE

20

The critique of models

The nature of models,

after Rosen

N

Natural system

F

Formal system

Encoding

Decoding

Entailment

Entailment

21

The critique of models

After Robert Rosen, 1991, ”World” (the natural system) and “Model” (the formal

system) are internally entailed - driven by a causal structure. [Efficient, material, final for

‘world’ – formal for ‘model’]

Nothing entails with one another “World” and “Model”; the association is hence the

result of a craftsmanship.

N

Natural

system

F

Formal syste

m

Decoding

Entailment

Entailment

Encoding

22

The critique of models

George M. Hornberger

1981

Hydrogeologist Naomi

Oreskes

1994

HistorianJean

Baudrillard

1999

Philosopher…

23

Other Newspaper headlines:

Environmental institute lies and deceits

Fuss in parliament after criticism on environmental numbers

The bankruptcy of the environmental numbers

Society has a right on fair information, RIVM does not provide it

Just philosophy? Maybe not:

A title during the RIVM media scandal (1999):

“RIVM over-exact prognoses based on virtual reality of computer models”

Jeroen van der Sluijs

24

Science for the post normal age is discussed in Funtowicz and Ravetz (1990, 1993, 1999) mostly in relation to Science for policy use.

Jerry Ravetz

Silvio Funtowicz

25

Post Normal Science

Funtowicz and Ravetz, Science for the Post Normal age, Futures, 1993

Remark: on Post Normal Science diagram increasing stakes increases uncertainty

26

GIGO (Garbage In, Garbage Out)

Science - where uncertainties in inputs must be suppressed lest outputs become indeterminate

Jerry Ravetz

27

The critique of models <-> Sensitivity

Peter Kennedy, A Guide to Econometrics

One of the ten commandments of applied econometrics according to Peter Kennedy:

“Thou shall confess in the presence of sensitivity.

Corollary: Thou shall anticipate criticism ’’

28

When reporting a sensitivity analysis, researchers should explain fully their specification search so that the readers can judge for themselves how the results may have been affected.

29

Definition. The study of how uncertainty in the output of a model (numerical or otherwise) can be apportioned to different sources of uncertainty in the model input.

A related practice is `uncertainty analysis', which focuses rather on quantifying uncertainty in model output.

The two should be run in tandem.

Sensitivity

30

In sensitivity analysis:

Type I error: assessing as important a non important factor

Type II: assessing as non important an important factor

Type III: analysing the wrong problem

31

Type III in sensitivity: Examples:

•In the case of TSPA (Yucca mountain) a range of 0.02 to 1 millimetre per year was used for percolation of flux rate. Applying sensitivity analysis to TSPA could or could not identify this as a crucial factor, but this would be of scarce use if the value of the percolation flux were later found to be of the order of 3,000 millimetres per year.

32

Prescriptions for sensitivity analysis

EPA’s 2004 guidelines on modelling

Models Guidance Draft - November 2003 Draft Guidance on the Development, Evaluation, and Application of Regulatory Environmental Models Prepared by: The Council for Regulatory Environmental Modeling, http://cfpub.epa.gov/crem/cremlib.cfm

33

CREM Prescriptions for sensitivity analysis

“methods should preferably be able to

(a)deal with a model regardless of assumptions about a model’s linearity and additivity;

(b)consider interaction effects among input uncertainties; and

(c) …an so on

34

CREM prescriptions are good. We at JRC works on practices that that take them into proper account.

What these practices have in common the aspiration to tackle the curse of dimensionality.

35

Want to to know more?Buy our book!

GLOBAL SENSITIVITY ANALYSIS. The primer

John Wiley & Sons, 2008

36

Sensitivity analysis and the White House

In the US the Proposed Risk Assessment Bulletin mentioned before also

puts forward prescription for sensitivity analysis.

37

4. Standard for Characterizing Uncertainty

Influential risk assessments should characterize uncertainty with a sensitivity analysis and, where feasible, through use of a numeric distribution

[…] Sensitivity analysis is particularly useful in pinpointing which assumptions are appropriate candidates for additional data collection to narrow the degree of uncertainty in the results. Sensitivity analysis is generally considered a minimum, necessary component of a quality risk assessment report.

Source: OFFICE OF MANAGEMENT AND BUDGET

Proposed Risk Assessment Bulletin (January 9, 2006) http://www.whitehouse.gov/omb/inforeg/

38

The OMB about transparency

http://www.whitehouse.gov/omb/inforeg/

39

The primary benefit of public transparency is not necessarily that errors in analytic results will be detected, although error correction is clearly valuable. The more important benefit of transparency is that the public will be able to assess how much an agency’s analytic result hinges on the specific analytic choices made by the agency. Concreteness about analytic choices allows, for example, the implications of alternative technical choices to be readily assessed. This type of sensitivity analysis is widely regarded as an essential feature of high-quality analysis, yet sensitivity analysis cannot be undertaken by outside parties unless a high degree of transparency is achieved. The OMB guidelines do not compel such sensitivity analysis as a necessary dimension of quality, but the transparency achieved by reproducibility will allow the public to undertake sensitivity studies of interest.

Friday, February 22, 2002Graphic - Federal Register, Part IX

Office of Management and BudgetGuidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and

Integrity of Information Disseminated by Federal Agencies; Notice; Republication

http://www.whitehouse.gov/omb/inforeg/

40

Conclusions: Role of sensitivity analysis

(in a post-normal science context)

• Good practice and due diligence (e.g. test models, obtain parsimonious model representations …)

•Check if policy options are distinguishable given the uncertainties

•Contribute to the pedigree of the assessment.

•Falsify an analysis and or / make sure that you are not falsified.

41

•Falsify the analysis (Popperian demarcation):

*‘Scientific mathematical modelling should involve constant efforts to falsify the model’ (Pilkey and Pilkey Jarvis, op. cit.)

** Fight ‘the white swan syndrome’ (Nassim N. Taleb, 2007)