department of engineering and public policy carnegie mellon university 1 m. granger morgan head,...

1Department of Engineering and Public Policy

Carnegie Mellon University

M. Granger MorganHead, Department of Engineering and Public PolicyCarnegie Mellon Universitytel: 412-268-2672e-mail: [email protected]

Analyzing and Communicating

Uncertainty in the Context of Weather and

ClimateAn overview talk for the 4th UCAR/NCAR Junior Faculty Forum on Future Scientific Directions

2006 August 01



This morning I will talk about:

•Sources of uncertainty and the characterization of uncertainty.

•Two basic types of uncertainty.

•Uncertainty about coefficient values.

•Uncertainty about model functional form.

•Performing uncertainty analysis and making decisions in the face of uncertainty.

•Public perceptions and communication.

•Some summary guidance on reporting, characterizing and analyzing uncertainty.



ProbabilityProbability is the basic language of uncertainty.

I will adopt a personalistic view of probability (sometimes also called a subjectivist or Bayesian view).

In this view, probability is a statement of the degree of belief that a person has that a specified event will occur given all the relevant information currently known by that person.

P(X|i) where:

X is the uncertain event

i is the person's state of information.



The clairvoyant test

Even if we take a personalist view of probability, the event or quantity of interest must be well specified for a probability, or a probability distribution, to be meaningful.

"The retail price of gasoline in 2008" does not pass this test. A clairvoyant would need to know things such as:

•Where will the gas be purchased?

•At what time of year?

•What octane?



Does a subjectivist view mean

your probability can be arbitrary?

NO, because if they are legitimate probabilities, they must

•conform with the axioms of probability

•be consistent with available empirical data.Lots of people ask, why deal with

probability? Why not just use subjective words such as "likely" and "unlikely" to describe uncertainties? There are very good reasons not to do this.



The risks of using qualitative

uncertainty languageQualitative uncertainty language is inadequate because:

-the same words can mean very different things to different people.

-the same words can mean very different things to the same person in different contexts.

-important differences in experts' judgments about mechanisms (functional relationships), and about how well key coefficients are known, can be easily masked in qualitative discussions.



Mapping words toprobabilities

Probability that subjects associated with the qualitative description

0.00.20.40.60.81.0

Almost certain

Probable

Likely

Good chance

Possible

Tossup

Unlikely

Improbable

Doubtful

Almost impossible

range of individualupper bound estimates

range of individuallower bound estimates

range from upperto lower medianestimate

Qualitative description of uncertainty used

Figure adapted from Wallsten et al., 1986.

This figure shows the range of probabilities that people are asked to assign probabilities to words, absent any specific context.



Ex Com of EPA SAB

SAB members:

Key:

Other meeting participants:

Probability that the material is a human carcinogen

The minimum probability associated with the word "likely" spanned four orders of magnitude.The maximum probability associated with the word "not likely" spanned more than five orders of magnitude. There was an overlap of the probability associated with the word "likely" and that associated with the word "unlikely"!

Figure from Morgan, HERA, 1998.



The bottom lineWithout at least some quantification, qualitative descriptions of uncertainty convey little, if any, useful information.

The climate assessment community is gradually learning this lesson. Steve Schneider and Richard Moss have worked hard to promote a better treatment of uncertainty in the work of the IPCC.

At my insistence, U.S. national assessment synthesis team gave quantitative definitions to five probability words:



BUT, in other fields…

recommended…"against routine use of formal quantitative analysis of uncertainty in risk estimation, particularly that related to evaluating toxicology." While analysts were encouraged to provide "qualitative descriptions of risk-related uncertainty," the Commission concluded that "quantitative uncertainty analyses of risk estimates are seldom necessary and are not useful on a routine basis to support decision-making."

Slowly such views are giving way, but progress is slow.

…such as biomedical and health effects, progress has been much slower.

A concrete example of this is provided by the recommendations of Presidential/Congressional Commission on Risk Assessment and Risk Management (1997) which



We must consider two quite different kinds of

uncertainty1. Situations in which we know the

relevant variables and the functional relationships among them, but we not know the values of key coefficients (e.g., the "climate sensitivity").

2. Situations in which we are not sure what all the relevant variables are, or the functional relationships among them (e.g., will rising energy prices induce more technical innovation?).

Both are challenging, but the first is much more easily addressed than the second. I'll talk more about the second, when I talk about uncertainty analysis.



Uncertainty about quantities

From Morgan and Henrion, Uncertainty, Cambridge, 1990/99.



PDFs and CDFs

A number of examples I am about to show are in the form of probability density functions (PDFs) or cumulative distribution functions (CDFs).

Since some of you may not make regular use of PDFs and CDF's, let me take just a moment to remind you...



Probability density functionor PDF

V, Value of the

uncertain quantity

Probability density

V V+ δ


Carnegie Mellon UniversityCumulative

distribution

function

or CDF

V, Value of the uncertain quantity

Probability density

V V+ δ

0.5

1.0

Cumulative probability

p

p

0

mean

median

mode

NOTE: In asymmetric distributions with long tails, the mean may be much much larger than the median.



If I have good data......in the form of many observations of a random process, then I can construct a probability distribution that describes that process. For example, suppose I have the 145 years of rainfalldata for San Diego, and I am prepared to assume that over that period San Diego's climate has been "stationary" (that is the basic underlying processes that create the year-to-year variability have not changed)…

Source: Inman et al., Scripps, 1998.



Then if I want…

0204060

…a PDF for future San Diego annual rainfall, the simplest approach would be to construct a histogram from the data, as illustrated to the right.

If I want to make a prediction for some specific future year, I might go on to look for time patterns in the data. Even better, I might try to relate those time patterns to known slow patterns of variation in the regional climate, and modify my PDF accordingly.



In that way……I could construct a PDF and CDF for future San Diego rainfall that would look roughly like this.However, suppose that what I really care about is the probability that very large rainfall events will occur.

Since there have only been two years in the past 145 years when rainfall has been above 60 cm/yr over, I'll need to augment my data with some model or physical theory.



In summary…

…one should use available data, and well-established physical and statistical theory, to describe uncertainty whenever either or both are available.

However, often the available data and theory are not exactly relevant to the problem at hand, or they are not sufficiently complete to support the full objective construction of a probability distribution.

In such cases, we may have to rely on expert judgment. This brings us to the problem of how to "elicit" expert judgment.



Cognitive HeuristicsWhen ordinary people or experts make judgments about uncertain events, such as numbers of deaths from chance events, they use simple mental rules of thumb called "cognitive heuristics."

In many day-to-day circumstances, these serve us very well, but in some instances they can lead to bias - such as over confidence - in the judgments we make.

This can be a problem for experts too.

The three slides that follow illustrate three key heuristics: "availability," "anchoring and adjustment," and "representativeness."



Cognitive bias

from Lichtenstein et al., 1978.

Availability: probability judgment is driven by ease with which people can think of previous occurrences of the event or can imagine such occurrences.



Cognitive bias…(Cont.)

Anchoring and adjustment: probability judgment is frequently driven by the starting point which becomes an "anchor."

from Lichtenstein et al., 1978.



Cognitive bias…(Cont.)

I flip a fair coin 8 times. Which of the following two outcomes is more likely?

Outcome 1: T, T, T, T, H, H, H, H

Outcome 2: T, H, T, H, H, T, H, T

Of course, the two specific sequences are equally likely...but the second seems more likely because it looks more representative of the underlying random process.

Representativeness: people judge the likelihood that an object belongs to a particular class in terms of how much it resembles that class.



Overconfidence is an ubiquitous problem

190030000029980029960019201940196018801860299800299790299780299770299760Recommendedvalue with reporteduncertainty

Year of experiment299750For details see: Henrion and Fischhoff, “Assessing Uncertatinty in Physical Constants,” American Journal of Physics , 54, pp791-798, 1986.

0% 10% 20% 30% 40% 50% 60%

Percentage of estimates in which the true value

lay outside of the respondent’s assessed

98% confidence interval.

For details see Morgan and Henrion, Uncertatinty , Cambridge Univ. Press, 1990, pg 117.

21 different studies on questions with known answers:

Estimates of the speed of light



Expert elicitations we have done

Over the past three decades, my colleagues and I have developed and performed a number of substantively detailed expert elicitations. These have been designed to obtain experts’ considered judgments. Examples include work on:Health effects of air

pollution from coal-fired power plants.• M. Granger Morgan, Samuel C. Morris, Alan K. Meier and

Debra L. Shenk, "A Probabilistic Methodology for Estimating Air Pollution Health Effects from Coal-Fired Power Plants," Energy Systems and Policy, 2, 287-310, 1978.

• M. Granger Morgan, Samuel C. Morris, William R. Rish and Alan K. Meier, "Sulfur Control in Coal-Fired Power Plants: A Probabilistic Approach to Policy Analysis," Journal of the Air Pollution Control Association, 28, 993-997, 1978.

• M. Granger Morgan, Samuel C. Morris, Max Henrion, Deborah A.L. Amaral and William R. Rish, "Technical Uncertainty in Quantitative Policy Analysis: A Sulfur Air Pollution Example," Risk Analysis, 4, 201-216, 1984 September.

• M. Granger Morgan, Samuel C. Morris, Max Henrion and Deborah A. L. Amaral, "Uncertainty in Environmental Risk Assessment: A case study involving sulfur transport and health effects," Environmental Science and Technology, 19, 662-667, 1985 August.



Expert elicitations…(Cont.)

• M. Granger Morgan and David Keith, "Subjective Judgments by Climate Experts," Environmental Science & Technology, 29(10), 468-476, October 1995.

• Elizabeth A. Casman, M. Granger Morgan and Hadi Dowlatabadi, “Mixed Levels of Uncertainty in Complex Policy Models,” Risk Analysis, 19(1), 33-42, 1999.

• M. Granger Morgan, Louis F. Pitelka and Elena Shevliakova, "Elicitation of Expert Judgments of Climate Change Impacts on Forest Ecosystems," Climatic Change, 49, 279-307, 2001.

• Anand B. Rao, Edward S. Rubin and M. Granger Morgan, "Evaluation of Potential Cost Reductions from Improved CO2 Capture Systems,"Proceedings of the 2nd National Conference on Carbon Sequestration, Alexandria, VA, May 5-8, 2003.

• M. Granger Morgan, Peter J. Adams and David W. Keith, "Elicitation Of Expert Judgments of Aerosol Forcing," Climatic Change, xxx.

• Kirsten Zickfeld, Anders Levermann, M. Granger Morgan, Till Kuhlbrodt, Stefan Rahmstorf, and David W. Keith, "Present state and future fate of the Atlantic meridional overturning circulation

as viewed by experts," in revision for Climatic Change.

Climate science, climate impacts and mitigation technology:

• M. Granger Morgan, "The Neglected Art of Bounding Analysis," Environmental Science & Technology, 35, pp. 162A-164A, April 1, 2001.

• Minh Ha-Duong, Elizabeth A. Casman, and M. Granger Morgan, "Bounding Poorly Characterized Risks: A lung cancer example," Risk Analysis, 24(5), 1071-1084, 2004.

• Elizabeth Casman and M. Granger Morgan, "Use of Expert Judgment to Bound Lung Cancer Risks," Environmental Science & Technology, 39, 5911-5920, 2005.

Bounding uncertain health risks:



Expert elicitation…(Cont.)In all our elicitation studies, we've focused on creating a process that allows the experts to provide their carefully considered judgment, supported by all the resources they may care to use. Thus, we have:

- Prepared a background review of the relevant literatures.

- Carefully iterated the questions with selected experts and run pilot studies with younger (Post-doc) experts to distil and refine the questions.

- Conducted interviews in experts' offices with full resources at hand.

- Provide ample opportunity for subsequent review and revision of the judgments provided.

All of these efforts have involved the development of new question formats that fit the issues at hand.



Expert elicitation takes time and care

Eliciting subjective probabilistic judgments requires careful preparation and execution.

Developing and testing an appropriate interview protocol typically takes several months. Each interview is likely to require several hours.

When addressing complex, scientifically subtle questions of the sorts involved with most problems in climate change, there are no satisfactory short cuts. Attempts to simplify and speed up the process almost always lead to shoddy results.



Warming for

2x[CO2]

1

234

56789

10111213141516

Temperature response given 2x[CO ] (K)

0-5-10 5 10 15 20

4 with "surprise"

expert

2with state change

0-5-10 5 10 15 20

2Source: Morgan and Keith, ES&T, 1995.…and, lest you conclude that most of these

experts are in basic agreement…



Pole to equator

temperature gradient for

2x[CO2]

122

56789

10111213141516

Meridinal temperature gradient given 2x[CO ] (K)

3530 40

30 35 40

4

expert

na

na

na

3w/climate state change

2

Source: Morgan and Keith, ES&T, 1995.



Biomass in Northern Forestsw/ 2xCO2 climate change

123456

78

Expert

91011

A. B.

Change in soil carbon in minimally disturbed Northern Forests between 45°N and 65°N under specified 2x[CO2] climate change.

0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.40.4Change in standing biomass in minimally disturbed Northern Forests between 45°N and 65°N under specified 2x[CO2] climate change.

North AmericaEurasia


EurasiaNorth America

"trivial"

w/permafrostw/o permafrost


North America and Eurasia E of the UralsEurope west of the Urals

0.6 0.8 1.0 1.2 1.40.40.2



w/permafrostw/o permafrost

North America and Eurasia E of the UralsEurope west of the Urals


Change in standing biomassChange in soil carbon

Source: Morgan et al., Climatic Change, 2001.



Biomass in Tropical Forests w/ 2xCO2 climate change

A. B.1

2

34

56

7

8

Expert

9

10

11

0.6 0.8 1.0 1.2 1.40.4

Change in standing biomass in minimally disturbed Tropical Forests between 20°N and 20°S under specified 2x[CO2] climate change.

0.6 0.8 1.0 1.2 1.40.4

Change in soil carbon in minimally disturbed Tropical Forests between 20°N and 20°S under specified 2x[CO2] climate change.

Change in standing biomassChange in soil carbon




Radiative Forcing byAerosols

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

We were asked to do this on a rapid pace by Ron Prinn in connection with the 4th IGCC assessment.




From IPCC 3rd Assessment

QuickTime™ and aTIFF (Uncompressed) decompressor


Source: Climate Change 2001: The Scientific Basis, Working Group 1 of the Third IPCC Assessment, Cambridge University Press, 881pp., 2001.



Aerosol forcing

direct aerosol effect

(scattering and absorption

from aerosols)

Δ aerosols

first aerosol indirect

effect

( )cloud whiteness

second aerosol indirect

effect

( )cloud lifetime effect

- semi direct aerosol

effect

( change in cloud

formation due to local

heating from black

)carbon absorption

direct

effects

indirect

effects

Direct aerosol effect: change in radiative flux by scattering and absorption of unactivated aerosol particles in the absence of any other climate changes or feedbacks.

Semi-direct aerosol effect: change in radiative flux resulting from a change

in cloud formation because of local heating by black carbon aerosols. First aerosol indirect effect (brightness): change in cloud reflectivity

resulting from a change in cloud condensation nuclei holding other cloud properties constant (e.g. total liquid water and cloud cover).

Second aerosol indirect effect (lifetime): change in cloud cover/lifetime

resulting from a change in cloud condensation nuclei.



The experts who participated

Name Affiliation Andrew Ackerman NASA Ames Research

Bruce Albrecht University of Miami

Theodore Anderson University of Washington

Meinrat O. Andreae Max Planck Institute for Chemistry Mary Barth National Center for Atmospheric Research Olivier Boucher Laboratoire d’Optique Atmosphérique, CNRS

Antony Clarke University of Hawaii

William Cotton Colorado State Johann Feichter Max Planck Institute for Meteorology

Steve Ghan Pacific Northwest National Laboratory

Mark Z. Jacobson Stanford University

Ralph Kahn NASA – Jet Propulsion Laboratory

Yoram Kaufman NASA – Goddard Space Flight Center

Jeff Kiehl National Center for Atmospheric Research

Stefan Kinne Max Planck Institute of Meteorology

Ulrike Lohmann ETH Zurich

Surabi Menon Lawrence Berkeley National Laboratory Dan Murphy NOAA – Aeronomy Laboratory Athanasios Nenes Georgia Institute of Technology

Spyros N. Pandis Carnegie Mellon University

Ronald G. Prinn Massachusetts Institute of Technology

Phil Rasch National Center for Atmospheric Research

Steve Schwartz Brookhaven National Laboratory

John Seinfeld California Institute of Technology



Scattering and absorption(also called the direct aerosol

effect)0+1-1-2-31234+256789101112131415ExpertProbability that in20 years uncertaintywill have:

GrownShrunk by 0-50%Shrunk by 50-80%Shrunk by >80%

00.8

0.150.05

0.50.4890.01

0.001

0.10.80.1~ 0

0.30.30.30.1

0.050.40.4

0.15

NA

0.10.50.40

0.050.60.3

0.05

0.050.60.2

0.15

NA

0.20.20.50.1

0.30.5

0.150.05

0.010.290.680.02

1617181920210.40.30.20.1

0.10.50.30.1

0.10.20.60.1

0.050.70.2

0.05

0.20.50.20.1

0.00.50.40.1

0.10.50.30.1

2223240.050.450.450.05

0.10.7

0.180.02

0.20.50.20.1

N+A-40.10.60.20.1

0.10.60.20.1

47645557NA557Level of expertise 0 - 7772756442262


Carnegie Mellon UniversityChange in cloud formation due to local heating by black carbon(also called the Semi-Direct Effect black carbon)

0+1-1-21234+256789101112131415NA+3NA??Probability that in20 years uncertaintywill have:


NA

0.10.30.50.1

0.10.80.1~ 0

0.30.20.20.3

0.20.5

0.250.05

0.10.30.50.1

NANA

0.10.50.30.1

0.30.40.20.1

0.050.7

0.240.01

0.20.50.20.1

0.10.60.30

1617181920210.70.10.10.1

0.20.60.20

0.20.50.20.1

0.090.80.1

0.01

0.30.30.20.2

0.10.5

0.250.15

222324NANA

0.250.5

0.1250.125

0.10.7

0.180.02

0.20.40.30.1

N+A?0.4

0.150.05

5735322NA5535557533155541ExpertLevel of expertise 0 - 7



Cloud Brightness (First aerosol indirect effect)

0+1-1-2-3123456789101112131415NA-4????Probability that in20 years uncertaintywill have:


0.10.60.20.1

NA

0.20.7

~ 0.1~ 0

0.20.30.30.2

0.40.40.1

< 0.1

NA

0.10.70.20.1

00.50.40.1

0.10.7

0.150.05

0.20.70.10

0.10.80.10

0.20.60.150.05

0.050.8

0.150

0.20.50.20.1

1617181920210.60.10.20.1

0.20.50.30

0.20.50.30

0.050.550.30.1

0.20.40.20.2

00.60.30.1

222324??0.10.70.10.1

0.20.50.20.1

0.10.70.180.02

0.250.4

0.250.1

-5N+A5767373NA6646766645544447ExpertLevel of expertise 0 - 7



Cloud Lifetime(Second aerosol indirect effect)

0+1-1-2-3123456789101112131415??+2+3-4NAProbability that in20 years uncertaintywill have:


0.250.450.20.1

NA

0.20.70.1~ 0

0.30.30.30.1

NANA

0.10.70.20.1

0.30.50.20

0.180.650.150.02

0.20.60.20

0.20.60.20

0.20.450.250.1

0.20.6

0.150.05

0.050.8

0.150

0.20.50.20.1

1617181920210.30.30.30.1

0.20.60.20

0.20.60.10.1

0.010.60.340.05

0.30.30.20.2

00.60.30.1

222324??0.10.70.10.1

0.10.70.180.02

0.20.50.20.1

N+A?677366666645646ExpertLevel of expertise 0 - 7



Total aerosol

forcing(at the top of the atmosphere)

0+1-1-2-3123456789101112131415????Probability that in20 years uncertaintywill have:


0.10.350.350.2

NA

0.10.80.1~ 0

0.30.20.20.3

NA

0.10.70.20.1

0.10.50.30.1

0.10.7

0.150.05

0.10.80.10

0.20.50.20.1

0.30.50.150.05

0.020.490.49

0

-4-5-6+2-7+3??NANA

0.20.50.20.1

1617181920210.20.40.30.1

0.10.50.30.1

0.10.60.20.1

0.10.650.20.05

0.30.30.20.2

00.50.30.2

222324??0.10.70.10.1

0.20.60.180.02

0.30.50.10.1

N+A6453657576564334364ExpertLevel of expertise 0 - 7



SurfaceForcing

0+1-1-2-3123456789101112131415-4-5-6-7-9-8-10NANAto -12NA??161718192021NANANANA222324N+A4444566566742102253ExpertLevel of expertise 0 - 7



Comparison with IPCC consensus results



0+1-1-2-3123456789101112131415Total Aerosol Forcing????-4-5-6+2+3??161718192021222324??N+A6453657576564334364ExpertLevel of expertise 0 - 7

Sources: IPCC TAR WG1Morgan et al, Climatic Change, in press.



Climate impacts on the AMOCPresent state and future fate of the Atlantic meridionaloverturning circulation as viewed by experts.

K. Zickfeld, A. Levermann, T. Kuhlbrodt and S. RahmstorfPotsdam Institute for Climate Impact ResearchG. MorganCarnegie Mellon UniversityD. KeithUniversity of Calgary



Source: UNEPClimatic Change, in revision.



Strength of

the AMOC



Transient response

2xCO2

4xCO2





Strength in 2100w/ 2xCO2 w/ 4xCO2



Probability of >90%

collapse



SOX health effects…elicited subjective judgments about oxidation rates and wet and dry deposition rates for SO2 and SO4

=

from nine atmospheric science experts.

Source: G. Morgan, S. Morris, M. Henrion, D. Amaral, W. Rish, "Technical Uncertainty in Quantitative Policy Analysis: A sulfur air pollution example,” Risk Analysis, 4, 201-216, 1984.



SOX health effects…(Cont.)Then for each expert we built a separate plume model, exposing people on the ground using known population densities.




SOX health effects…(Cont.)We elicited health damage functions for sulfate aerosol from seven health experts (of whom only five were able to give us probabilistic estimates).

Finally, for each air expert, we did an analysis using the health damage function of each health expert…

Source: D. Amaral, "Estimating Uncertainty in Policy Analysis: Health effects from inhaled sulfur oxides," Ph.D. thesis, Department of Engineering and Public Policy, Carnegie Mellon, 1983.



SOX health effects…(Cont.)The results showed that while there was great discussion about uncertainty in the atmospheric science, the uncertainty about the health damage functions completely dominated the estimate of health impacts.


CDF of deaths/yr from a new (in 1984) super-critical 1GWe FGD equipped coal-fired plant in Pittsburgh. Availability factor = 73%, net efficiency = 35%.



Multiple ExpertsWhen different experts hold different views it is often best not to combine the results, but rather to explore the implications of each expert's views so that decision makers have a clear understanding of whether and how much the differences matter in the context of the overall decision.

However, sophisticated methods have been developed that allow experts to work together to combine judgments so as to yield a single overall composite judgment.

The community of seismologists have made the greatest progress in this direction through a series of very detailed studies of seismic risks to built structures (Hanks, T.C., 1997; Budnitz, R.J. et al., 1995).



Uncertainty versus variability

Variability involves random change over time or space (e.g., "the mid-day temperature in Beijing in May is variable").

Recently, in the U.S., some people have been drawing a sharp distinction between variability and uncertainty. While the two are different, and sometimes require different treatments, the distinction can be overdrawn. In many contexts, variability is simply one of several sources of uncertainty (Morgan and Henrion, 1990).

One motivation people have for trying to sharpen the distinction is that variability can often be measured objectively, while other forms of uncertainty require subjective judgment.



Recently several investigators…

…have been combining models and the use of subjective judgment to bound or estimate parameter uncertainty. I will mention 4 examples:

Andronova and Schlesinger, 2001

Forest et al., 2006

Frame et al., 2005

Stainforth et al., 2005



Andronova and Schlesinger, 2001Used the Bayesian Monte-Carlo simulation method, specified the known uncertainties in forcing with a wide prior distribution for climate sensitivity and tested model outputs of the evolution of global temperature against observed

Source: Andronova, N, and M. E. Schlesinger, "Objective Estimation of the Probability Distribution for Climate Sensitivity," J. Geophys. Res., 106 , D19, 22,605-22,612, 2001.

historic global temperature data. Each model run was weighted (for likelihood of being true and accurate) according to its "error" with respect to the data. This comparison permits the re-weighting of the prior distribution used for climate sensitivity to develop a posterior distribution consistent with our knowledge of the different forcing functions and the empirical data on temperature trends.



Forest et al., 2006Marginal posterior probability density function obtained when using uniform probability distributions across all relevant forcings and matching outputs from the ocean and atmospheric portion of the MIT IGSM model. Light contours bound the 10% and 1% significance regions.

Forest, Chris E, and Peter H. Stone and, and Andrei P. Sokolov, "Estimated PDF’s of Climate System Properties Including Natural and Anthropogenic Forcings," Geophysical Research Letters, Vol. 3333, pp xx-xx, 2006.

Similarly, the two dark contours are for an expert PDF on climate sensitivity. Dots show outputs from a range of leading GCM’s all of which lie to the right of the high-probability region, suggesting that if Forest et al. are correct, these models may be mixing heat into the deep ocean too efficiently.



Frame et al., 200510 K8 K6 K4 K2 K0 K0012123

Relationship between climate sensitivity (light contours), effective ocean heat capacity, and 20th century warming for the case of uniform sampling of climate sensitivity (not shown are similar resultsfor uniform sampling across feedback strength). The dark contour shows the region consistent with observations at the 5% level.

Frame, D.J, B. B. B. Booth, J. A Kettleborough, D.A. Stainforth, J.M. Gregory, M. Collins, and M.R. Allen, "Constraining Climate Forecasts: The role of prior assumptions," Geophysical Research Letters, 32, 2005.



Stainforth et al., 2005

Histogram of climate sensitivities found by in their simulation of 2,578 versions of the HadSM3 GCM model.

"We find model versions as realistic as other state-of-the-art climate models but with climate sensitivities ranging from less than 2K to more than 11K. Models with such extreme sensitivities are critical for the study of the full range of possible responses of the climate system to rising greenhouse gas levels, and for assessing the risks associateds with a specific target for stabilizing these levels…

The range of sensitivity across different versions of the same model is more than twice that found in the GCM’s used in the IPCC Third Assessment Report…The possibility of such high sensitivities has been reported by studies using observations to constrain this quantity, but this is the first time that GCM’s have generated such behavior."

Stainforth, D.A., T. Aina, C. Christensen, M. Collins, N. Faull, D. J. Frame, J. A. Kett;eborough, S. Knight, A. Martin, J.M. Murphy, C. Piani, D. Sexton, L. A. Smith, R. A. Spicer, A. J. Thorpe, and M. R. Allen, "Uncertainty in Predictions of the Climate Response to Rising Levels of Greenhouse Gases," Nature, 433, pp. 403-406, 2005.



Uncertainty about model form

Often uncertainty about model form is as or more important than uncertainty about values of coefficients. Until recently there had been little practical progress in dealing with such uncertainty, but now there are several good examples:

•John Evans and his colleagues at the Harvard School of Public Health (e.g. Evans et al., 1994).

•Alan Cornell and others in the seismic risk (e.g. Budnitz et al., 1995).

•Hadi Dowlatabadi and colleagues at Carnegie Mellon in Integrated Assessment of Climate Change -ICAM (e.g. Morgan and Dowlatabadi, 1996).

•Also on climate, Lempert and colleagues at RAND (e.g. Lempert, Popper, Bankes, 2003).



John Evans and colleagues….…have developed a method which lays out a "probability tree" to describe all the plausible ways in which a chemical agent might cause harm. Then experts are asked to assess probabilities on each branch.For details see: John S. Evans et al., "A distributional approach to characterizing low-dose cancer risk," Risk Analysis, 14, 25-34, 1994; and John S. Evans et al., "Use of probabilistic expert judgment in uncertainty analysis of carcinogenic potency," Regulatory Toxicology and Pharmacology, 20, 15-36, 1994.



Energy & Emissions

Atmospheric Composition &

Climate

Demographics& Economics

INTERVENTION

Impacts ofClimate Change

StructureInputs Outputs

To run the model: 1 - Double click on INPUTS to set up the scenario inputs; 2 - Double click on STRUCTURE to set up the model; 3 - Double click on OUTPUTS and evaluate the indicators.

Regional ∆T

GHGModels

AerosolModel

ForcingChange in Long Wave

Forcing

Change in Short Wave

Forcing

ElicitedClimate Model

RITS Conc

ICAMIntegrated Climate Assessment Model

See for example:Hadi Dowlatabadi and M. Granger Morgan, "A Model Framework for Integrated Studies of the Climate Problem," Energy Policy, 21(3), 209-221, March 1993.andM. Granger Morgan and Hadi Dowlatabadi, "Learning from Integrated Assessment of Climate Change," Climatic Change, 34, 337-368, 1996.

A very large hierarchicallyorganized stochasticsimulation model builtin Analytica®.



ICAM deals with...

…both of the types of uncertainty I've talked about:1. It deals with uncertain coefficients

by assigning PDFs to them and then performing stochastic simulation to propagate the uncertainty through the model.

2. It deals with uncertainty about model functional form (e.g., will rising energy prices induce more technical innovation?) by introducing multiple alternative models which can be chosen by throwing "switches."



ICAMThere is not enough time to present any details from our work with the ICAM integrated assessment model. Here are a few conclusions from that work:

•Different sets of plausible model assumptions give dramatically different results.

•No policy we have looked at is dominant over the wide range of plausible futures we’ve examined.

•The regional differences in outcomes are so vast that few if any policies would pass muster globally for similar decision rules.

•Different metrics of aggregate outcomes (e.g., $s versus hours of labor) skew the results to reflect the OECD or developing regional issues respectively.



These findings lead us......to switch from trying to project and examine the future, to using the modeling framework as a test-bed to evaluate the relative robustness, across a wide range of plausible model futures, of alternative strategies that regional actors in the model might adopt.

We populated the model's regions with simple decision agents and asked, which behavioral strategies are robust in the face of uncertain futures, which get us in trouble.

Thus, for example, it turns out that tracking and responding to atmospheric concentration is more likely to lead regional policy makers in the model to stable strategies than tracking and responding to emissions.



Our conclusionPrediction and policy optimization are pretty silly analytical objectives for much assessment and analysis related to the climate problem.

It makes much more sense to:

• Acknowledge that describing and bounding a range of futures may often be the best we can do.

• Recognize that climate is not the only thing that is changing, and address the problem in that context.

• Focus on developing adaptive strategies and evaluating their likely robustness in the face of a range of possible climate, social, economic and ecological futures.

Recent work by Robert Lempert and colleagues takes a very similar approach.



Lempert & co-workers…





…have used a similar decision making approach to identify near-term emissions mitigation policies which are robust over a wide range ofpotential futures. They generate hundreds to millions of plausible future states of the world, and then use statistical methods, supported by interactive search and visualization to help decision makers: design and choose robust strategies which perform reasonably well across a very wide range of the plausible futures; identify residual vulnerabilities of those strategies: and assess the tradeoffs involved in addressing these vulnerabilities.

See for example: Robert J. Lempert, Steven W. Popper and Steven C. Bankes, Shaping the Next One Hundred Years: NNew methods for quantitative long-Term term Policy policy analysis, RAND MR-1626-RPC, August 2003.

Source: Lempert and Schlesinger, 2002



Propagation and analysis of uncertainty

Consider a basic model of the form y = f(x 1, x

2)

The simplest form of uncertainty analysis is sensitivity analysis using the slopes of y with respect to the inputs x 1 and x 2

Source: Morgan and Henrion, 1990



Nominal Range Sensitivity




Propagation of continuous distributions


There are many software tools available such as Analytica, @Risk, and Crystal Ball.



Tools for analysis

Tools for continuous processes:• Exposure models

• Dose response functions

• etc.

Use of some of these tools used to be very challenging and time consuming. Today such analysis is facilitated by many software tools (e.g. Analytica®, @risk®, Crystal Ball®, etc.).

Expendituresto limit losses

Losses due toclimate change

Regional GDPin 1975

GrossGDP

GDP net of climate change

PopulationGrowth

ProductivityGrowth

EconomicGrowth

Regional Popsin 1975

Population

DiscountRate

TotalMitigation Costs

& Losses

DiscountedLost GDP

$ Per cap lossdue to CC

Disc. per cap GDP Lossdue to CC

Energy & Emissions

Atmospheric Composition &

Climate

Demographics& Economics

INTERVENTION

Impacts ofClimate Change

StructureInputs Outputs

To run the model: 1 - Double click on INPUTS to set up the scenario inputs; 2 - Double click on STRUCTURE to set up the model; 3 - Double click on OUTPUTS and evaluate the indicators.

Tools for discrete events:

• Failure modes and effects analysis

• Fault tree models

• etc.



All that fancy stuff……is only useful when one has some basis to predict functional relationships and quantify uncertainties about coefficient values.

When one does not have that luxury, order-of-magnitude arguments and bounding analysis (based on things like conservation laws, etc.) may be the best one can do.

The conventional decision-analytic model assumes that research reduces uncertainty:

Research $



Research and uncertainty…(Cont.)

Research $

or even grows…

Research $

Sometimes this is what happens, but often, for variables or processes we really care about, the reality is that uncertainty either remains unchanged…

When the latter happens it is often because we find that the underlying model we had assumed is incomplete or wrong.



There are formal methodsto support decision making…

Source: Henrion, Lumina Systems

…such as B-C analysis, decision analysis, analysis based on multi-attribute utility theory, etc.

Things analysts need to remember:Be explicit about decision rules (e.g., public health versus science; degree of precaution; etc.).

Get the value judgments right and keep them explicit.

Don't leave out important issues just because they don't easily fit.

Remember that many methods become opaque very quickly, especially to non-experts.

When uncertainty is high, it is often best to look for adaptive as opposed to optimal strategies.



Identify the problemGain full understandingof all relevant issuesDo researchIdentify policy optionsInplement theoptimal policySolve the problem

A classic strategy for solving problems:

When uncertainty is high (and perhaps irreducible) an iterative adaptive strategy is better:Identify a problemIdentify adaptivepolicies and choose one that currently looks best

Do researchimplement policy and observe howit works

Reassess policy in light of new understanding

Learn what you canand what you can’t know (at least now).

Continue researchRefine problem identificaton as needed



Limited domain of

model validity

Year

2075 21002050202520001975

0

1

2

3

4

5

6

Mean change in

global temperature,°C

0

1

2

3

4

5

6

Mean change in


0

1

2

3

4

5

6

Mean change in


0.00

0.10

Probability of a change

in the climate state

0.20

0.30

0.00

0.10



0.20

0.30

0.00

0.10



0.20

0.30

Year

2075 21002050202520001975

Year

2075 21002050202520001975

Expert 1

Expert 8

Expert 15

Examples of warming estimated via the ICAM model (dark curves) and probability that the associated climate forcing will induce a state change in the climate system (light curves) using the probabilistic judgments of three different climate experts.



Modelswitching

Schematic illustration of the strategy of switching to progressively simpler models as one moves into less well understood regions of the problem phase space, in this case, over time.


Carnegie Mellon UniversityIllustration

of modelswitchin

gResults of applying the model switch-over strategy to the ICAM demographic model (until about 2050) and an estimate of the upper-bound estimate of global population carrying capacity based on J. S. Cohen.

0.0

0.2

0.4

0.6

0.8

1.0Weight for ICAM model

Weight for Cohen model

2000 2050 2100 2150 2200 2250 2300

20

40

60

0

80



The traditional approach……to risk communication involves two steps:

1. Ask an expert what people should be told.

2. Get a "communications expert" to package it.

And, if you are being really fancy:

3. Run some tests to see how people like it.

However, if you think about it for a few minutes, this approach ignores two critical issues:

• What the people receiving the message already "know" about the topic.

• What they need to know to make the decisions they face.



This led us to

develop a new

approach

Cambridge University Press, 351pp., 2002



Finding out what people know…

…is not simple! If I give people a questionnaire, I have to put information in my questions.

People are smart. They will start making inferences based on the information in my questions.

Pretty soon I won't know if their answers reflect what they already knew, or the new ideas they have come up with because of the information I have given them.

I need a better, less intrusive way to learn what they know. The method we have developed to do this is called the mental model interview.



Five step process:In work done over the past decade at Carnegie Mellon, we have developed a five-step approach to risk communication, based on people's mental models of risk processes:

1. Develop an "influence diagram" to structure expert knowledge.

Number of interviews

Cumulative number of concepts

0

40

80

120

160

1 10 20 30

A

B1

B2 B3S1

S2 S3

S4

2. Conduct an open-ended elicitation of people's beliefs about a hazard, allowing the expression of both accurate and inaccurate concepts. Map the results to the expert influence diagram.



Five steps…(Cont.)3. On the basis of results from the open-

ended interviews, develop and administer a closed-form questionnaire to a much larger group in order to determine the prevalence of these beliefs.

4. Develop a draft communication based on both a decision analytic assessment of what people need to know in order to make informed decisions and a psychological assessment of their current beliefs.

5. Iteratively test successive versions of those communications using open-ended, closed-form, and problem-solving instruments, administered before, during, and after the receipt of the message.

In 1994, we applied these methods to study public understanding of climate change. While the results are now getting old, I will show you a few since they are unlikely to have changed much.



Example of opening response

in interview Interviewer: "I'd like you to tell me all about the issue of climate change."

Subject: "Climate change. Do you mean global warming?"

Interviewer: "Climate change."

Subject: "OK. Let's see. What do I know. The earth is getting warmer because there are holes in the atmosphere and this is global warming and the greenhouse effect. Um... I really don't know very much about it, but it does seem to be true. The temperatures do seem to be kind of warm in the winters. They do seem to be warmer than in the past.. and.. hmm.. That's all I know about global warming.



Another example…

Interviewer: "Tell me all about the issue of climate change."

Subject: "I'm pretty interested in it... The ice caps are melting -- the hole in the ozone layer. They think pollution from cars and aerosol cans are the cause of all that. I think the space shuttle might have something to do with it too, because they always send that up through the earth, to get out in outer space. So I think that would have something to do with it, too."



Another example…Interviewer: "Tell me all about the issue of climate change."

Subject: "Climate change? Like, what about it? Like, as far as the ozone layer and ice caps melting, water level raising, rainforest going down, oxygen going down because of that? All of that kind of stuff?"

Interviewer: "Anything else?"

Subject: "Well, erosion all over the place. Um, topsoils going down into everywhere. Fertilizer poisoning. "

Interviewer: "Anything else that comes to mind related to climate change?

Subject: "Climate change. Winter's ain't like they used to be. Nothing's as severe. Not as much snow. Nothing like that."



Studies of laypeople's understanding

We conducted both open-ended "mental model interviews," and closed-form questionnaire studies (N = 177).

Respondents regarded global warming as bad, and highly likely. Many believed substantial warming has already occurred.

1000

50

100

3020100 3020100-10

IPCCμ=.45σ=.08

Ourrespondents

μ=2.7σ=3.3

=1.9median

, °Temperature increase C

Δ :T to date Δ 10 :T in years Δ 50 :T in yearsIPCCμ=.72σ=.15

Ourrespondents

μ=3.2σ=4.3

=1.7median

Ourrespondents

μ=6.5σ=9.1

=4.2median

IPCCμ=1.8σ= .35



Overall……our respondents had a poor appreciation of the fact that:

1)if significant global warming occurs, it will be primarily the result of an increase in the concentration of carbon dioxide in the earth's atmosphere; and

2) the single most important source of carbon dioxide additions is the combustion of fossil fuels, most notably coal and oil.

For details see:Ann Bostrom, M. Granger Morgan, Baruch Fischhoff and Daniel Read, "What Do People Know About Global Climate Change? Part 1: Mental models," Risk Analysis, 14(6), 959-970, 1994.

Daniel Read, Ann Bostrom, M. Granger Morgan, Baruch Fischhoff and Tom Smuts, "What Do People Know About Global Climate Change? Part 2: Survey studies of educated laypeople," Risk Analysis, 14(6), 971-982, 1994.



A few general conclusionson risk communication

There is no such thing as an "expert" in public communication who can simply tell you what to do.

An empirical approach is essential.

Before developing a communication one must learn what the public knows and thinks.

The mental models method offers a promising strategy for doing this.

Uncertainty must be quantified.



A recent example of……of a study on public perception that could turn out to be very important for climate abatement policy.

My colleagues and I recently produced a report for the Pew Center on Climate Change which addresses the problems of the electricity industry and climate change.

Available at: http://wpweb2k.tepper.cmu.edu/ceic/publications.htm



The U.S. makes most if itselectricity from coal

Age of coal plants in years0102030405060704000800012000160000Most U.S. coalplantstoday are 20 to 50 years old.

Many coal plants are old and will soon need to be replaced.

Coal 51.2%Nuclear 19.9%Gas 16.6%Hydro 7.2%Oil 3.1%Geothermal 0.34%Wind 0.28%Solar 0.01%



CO2 Capture and Sequestration (CCS)There are several strategies.

The two closest to commercial use are:1. Post-combustion separation after combustion in air.

2. Pre-combustion separation.

powerplant

hydrogengasificationplant

CO2

To a deep geological formationor the deep ocean.

water vapor, NOx

sulfur and other wastes

electric power

coal (or oil or natural gas)

other uses

air

coal (or oil or natural gas)

powerplant

separationplant

CO2

N2, SOx, NOx, etc.

To a deep geological formation or the deep ocean.

flue gas

electric power

air


Carnegie Mellon UniversityFour examples ofexisting

facilities

Source: Statoil

Sleipner field in the Norwegian North Sea

Source: DoE

Great PlanesCoal GasificationPlant

Shady Point, Oklahoma

Source: AES Shady Point, Inc.Source: BP

Salah gas project, Algeria: BP Amoco, Statoil and Sonatrach



The U.S. already injects lots of fluid

Mt/

yea

r

1

10

100

1000

10000

FL MunicipalWastewater

OilfieldBrine

HazardousWaste

AcidGas

Natural GasStorage

CO2 forEOR

OCS water injected for EOR and

brine disposal

OCSgases

(e.g. NG)

Large quantities

Long Time

Frame

Gases

~.5

Gt

~2

Mt

~34

Mt

~28

Mt~

150M

t

~2.

7 G

t

~6M

t

~1.

2 M

t

Sub-seabed

The mass of current U.S. fluid injections is greater than the mass of current power plant CO2 emissions.

Compiled by E. Wilson with data from EPA, 2001; Deurling, 2001; Keith, 2001,; DOE, 2001.

CO2 from all U.S.

power plants

~1.

7 G

t



Study designCo-authors are Claire Palmgren, Wandi Bruine de Bruin and David Keith.

Claire and I first ran an open-form mental model interview study several years ago. On the basis of those results, we then developed a closed-form survey with a Pittsburgh sample. N = 124.

1. Background2a. Climate change 2b. General issues including climate change 3. Options for limiting CO2 4. Energy systems that dispose of CO2

5. Places to dispose of CO2

5.1 Deep rock formations5.2 Disposal of CO2 in the deep ocean6. Final evaluation7. Environmental issues (NEPS = New Ecological Paradigm Scale)8. A few questions about yourself

Claire R. Palmgren, M. Granger Morgan, Wändi Bruine de Bruin, and David W. Keith, "Initial Public Perceptions of Deep Geological and Oceanic Disposal of Carbon Dioxide," Environmental Science & Technology, 38(24), 6441-6450, 2004.



Views on climate change

The continuing release of CO2 into the earth's atmosphere during this century may result in serious climate change.

147completely disagreecompletely agreeGovernment regulation should begin to significantly limit the amount of CO2 that is released into the earth's atmosphere.

Government regulation will begin to significantly limit the amount of CO2 that is released into the earth's atmosphere at some time in the next 20 years.

4.64.74.0



OptionsOptions that will reduce CO2 emissions by 50%

from your energy consumptionRank

1 is what youwould pay the

most for.

Biomass Blend:50% from biomass, 50% from regular coalHalf of the energy comes from regular coal, and the other half comes from biomass. Biomassrefers to harvesting and burning special fast growing grasses and trees to make energy. Biomassreleases CO2 that plants have absorbed from the atmosphere so no new CO2 is produced.

Coal Blend with deep geological disposal of CO2:50% from coal with deep geological disposal of CO2, 50% from regular coalHalf of the energy comes from regular coal. The other half also comes from coal, but won't let theCO2 enter the atmosphere. Instead they will separate the CO2 and dispose of it thousands of feetunderground.

Coal Blend with deep ocean disposal of CO2:50% from coal with deep ocean disposal of CO2, 50% from regular coalHalf of the energy comes from regular coal. The other half also comes from coal, but won't let theCO2 enter the atmosphere. Instead they will separate the CO2 and dispose of it deep in the ocean.

Energy Efficiency:100% regular coalAll of the energy comes from regular coal. This option will replace appliances and lights in yourhouse with much more efficient ones so that you use half as much electricity while enjoying usingyour lights and appliances just as much as you do now. Using less electricity will result in lessCO2 being produced.

Hydro-electric Blend:50% from hydro-electric, 50% from regular coalHalf of the energy comes from regular coal, and the other half comes from hydro-electric. Hydro-electric refers to making electricity with water power from dams. Hydro-electric produces noCO2.

Natural gas Blend:100% from natural gasThe energy comes from burning natural gas in highly efficient plants. Burning natural gasefficiently produces about half as much CO2 as burning coal.

Nuclear power Blend:50% from nuclear power, 50% from regular coalHalf of the energy comes from regular coal, and the other half comes nuclear power.Nuclear power produces no CO2.

Solar power Blend:50% from solar power, 50% from regular coalHalf of the energy comes from regular coal, and the other half comes from solar. Solar powerrefers to making electricity with energy from the sun. Solar power produces no CO2.

Wind power Blend:50% from wind power, 50% from regular coalHalf of the energy comes from regular coal, and the other half comes from wind. Wind powerrefers to making electricity with energy from the wind. Wind power produces no CO2.

"Whatever your own beliefs are about climate change, imagine that the US government has decided that we must cut in half the amount of CO2 that is released by generating electric power.

Suppose that your electricity supplier has different methods of meeting the goal to reduce CO2 emissions by 50%. Some of these methods use a mixture of generation systems that produce little or no CO2, combined with regular coal-burning power plants. These methods do not all cost the same because some ways of making electricity with less CO2 are cheaper than others.

The supplier will be offering their customers a choice of how they would like them to meet this reduction. Some of the options to reduce CO2 emissions will be more expensive than the way we produce electricity today, so we would like you to tell us which method you would be willing to pay the most for."



Rank order

of options

Options used different generation mixes (base of coal) in order to reduce CO2 emissions by 50%.

Initial ranking(before information)Final ranking(after information)solar (3.4)(3.5)hydro (3.8)(3.7)wind (4.0)(4.1)natural gas (4.4)(4.3)energy efficiency(4.8)(4.9)biomass (5.4)(5.4)nuclear (5.3)geological disposal(6.9)(6.7)ocean disposal(7.0)(7.1) lower rank

number means willing to pay more



Summary evaluations

147completely opposecompletely favor147completely opposecompletely favorGeological disposal:before detailsafter detailsbefore detailsafter detailsOcean disposal:



CCSP Guidance Document

Along with several colleagues I am currently completing a guidance document for the US Climate Change Science Program.

The final slides reproduce our draft summary advice.

I've provided the meeting organizers with a draft copy. We'd welcome feedback and suggestions.





Doing a good job……of characterizing and dealing with uncertainty can never be reduced to a simple cookbook. One must always think critically and continually ask questions such as:

• Does what we are doing make sense?• Are there other important factors which are as or more

important than the factors we are considering?• Are there key correlation structures in the problem which

are being ignored?• Are there normative assumptions and judgments about

which we are not being explicit?

That said, the following are a few words of guidance…



Reporting uncertaintyWhen qualitative uncertainty words (such as likely and unlikely) are used, it is important to clarify the range of subjective probability values are to be associated with those words. Unless there is some compelling reason to do otherwise, we recommend the use of the framework shown below:

0.250.50.751.00likely greater than an even chance> 0.5 to ~ 0.8

very likely~ 0.8 to < 0.99vertually certain> 0.99about an even chance~ 0.4 to ~ 0.6unlikely less than an even chance~ 0.2 to > 0.5

vertually impossible > 0.99very unlikely> 0.99 to ~ 0.2 probability that a statement is true



Reporting uncertainty…(Cont.)

Another strategy is to display the judgment explicitly as shown:

0.25 0.5 0.75 1.00

probability that a statement is true

This approach provides somewhat greater precision and allows some limited indication of secondary uncertainty for those who feel uncomfortable making precise probability judgments.




In any document that reports uncertainties in conventional scientific format (e.g 3.5+0.7) it is important to be explicit about what uncertainty is being included and what is not, and to confirm that the range is plus or minus one standard deviation. This reporting format is generally not appropriate for large uncertainties or where distributions have a lower or upper bound and hence are not symmetric.

Care should be taken in plotting and labeling the vertical axes when reporting PDFs. The units are probability density (i.e. probability per unit interval along the horizontal axis), not probability.

Since many people find it difficult to read and correctly interpret PDFs and CDFs, when space allows it is best practice to plot the CDF together with the PDF on the same x-axis.




When many uncertain results must be reported, box plots (first popularized by Tukey, 1977) are often the best way to do this in a compact manor. There are several conventions. Our recommendation is shown below, but what is most important is to be clear about the notation.

minimum

possible

value

maximum

possible

value

0.05 0.25 0.75 0.95

median

value

mean

value

cumulative probability values

moving from left to right

X, value of the quantity of interest

Tukey, John W., Exploratory Data Analysis, Addison-Wesley, 688pp, 1977.




While there may be circumstances in which it is desirable or necessary to address and deal with second-order uncertainty (e.g. how sure an expert is about the shape of an elicited CDF), one should be very careful to determine that the added level of such complication will aide in, and will not unnecessarily complicate, subsequent use of the results.



Characterizing and analyzing uncertainty

Unless there are compelling reasons to do otherwise, conventional probability is the best tool for characterizing and analyzing uncertainty.



Characterizing and analyzing…(Cont.)

The elicitation of expert judgment, often in the form of subjective probability distributions, can be a useful way to combine the formal knowledge in a field as reflected in the literature with the informal knowledge and physical intuition of experts. Elicitation is not a substitute for doing the needed science, but if can be a very useful tool in support of research planning, private decision making, and the formulation of public policy.

HOWEVER the design and execution of a good expert elicitation takes time and requires a careful integration of knowledge of the relevant substantive domain with knowledge of behavioral decision science.




When eliciting probability distributions from multiple experts, if they disagree significantly, it is generally better to report the distributions separately then to combine them into an artificial "consensus" distribution.

There are a variety of software tools available to support probabilistic analysis using Monte Carlo and related techniques. As with any powerful analytical tool, their proper use requires careful thought and care.




In performing uncertainty analysis, it is important to think carefully about possible sources of correlation. One simple procedure for getting a sense of how important this may be is to run the analysis with key variables uncorrelated and then run it again with key variables perfectly correlated. Often, in answering questions about aggregate parameter values experts assume correlation structures between the various components of the aggregate value being elicited. Sometimes it is important to elicit the component uncertainties separately from the aggregate uncertainty in order to reason out why specific correlation structures are being assumed.



Characterizing and analyzing…(Cont.)Methods for describing and dealing with data pedigree (see for example Funtowicz and Ravetz, 1990) have not been developed to the point that they can be effectively incorporated in probabilisitic analysis. However, the quality of the data on which judgments are based is clearly important and should be addressed, especially when uncertain information of varying quality and reliability is combined in a single analysis.

While full probabilistic analysis can be useful, in many context simple parametric analysis, or back-to-front analysis (that works backwards from an end point of interest) may be as or more effective in identifying key unknowns and critical levels of knowledge needed to make better decisions.

Funtowicz, S.O. and J.R. Ravetz, Uncertainty and quality in science for policy, Kluwer Academic Publishers, 229 pp,1990.



Characterizing and analyzing…(Cont.)Scenarios analysis can be useful, but also carries risks. Specific detailed scenarios can become cognitively compelling, with the result that people may overlook many other pathways to the same end-points. It is often best to "cut the long causal chains" and focus on the possible range of a few key variables which can most affect outcomes of interest.

Scenarios which describe a single point (or line) in a multi-dimensional space, cannot be assigned probabilities. If, as is often the case, it will be useful to assign probabilities to scenarios, they should be defined in terms of intervals in the space of interest, not in terms of point values.




Analysis that yields predictions is very helpful when our knowledge is sufficient to make meaningful predictions. However the past history of success in such efforts suggests great caution (see for example Ch.s 3 and 6 in Smil, 2005). When meaningful prediction is not possible, alternative strategies, such as searching for responses or policies that will be robust across a wide range of possible futures, deserve careful consideration.

Smil, Vaclav, Energy at the Crossroads, MIT Press, 448pp, 2005.




For some problems there comes a time when uncertainty is so high that conventional modes of probabilistic analysis (including decision analysis) may no longer make sense. While it is not easy to identify this point, investigators should continually ask themselves whether what they are doing makes sense and whether a much simpler approach, such as a bounding or order-of-magnitude analysis, might be superior (see for example Casman et al., 1999).

Casman, Elizabeth A., M. Granger Morgan and Hadi Dowlatabadi, “Mixed Levels of Uncertainty in Complex Policy Models,” Risk Analysis, 19(1), 33-42, 1999.



The use of subjective probability



Lynn Johnson, Thur. Nov 10, 2005

department of engineering and public policy carnegie mellon university 1 m. granger morgan head,...

Documents