u.s. e nergy s urprises h ave b ecome m ore f requent evan sherwin with inês azevedo & max...

U.S. ENERGY SURPRISES

HAVE BECOME MORE FREQUENTEvan Sherwin

With Inês Azevedo & Max Henrion Carnegie Mellon University

Funding:

2

GOALSThe goals of this work are to: • characterize what constitutes a surprise• identify the biggest surprises in recent

decades• understand how the frequency of surprises

changed over time

3

Surprises affect long-term decisions

Projection from 1986

Source: DOE 1999, AEO 2014

Actual values

US Natural Gas Imports

3

4





Actual values


44

5






Actual values


5

% of projected consumption

LNG terminals approved

2004 projection 23% (import) 18 (import, by 2005)

Projected US natural gas imports for 2025:

5Sources: AEO 2005, AEO 2015, FERC 2015, Landfreid et al. 2005

Cost of an LNG import/export terminal: O($1bn) – O($10bn)LNG: Liquefied natural gas;5

6




Actual values







6

Projected US natural gas imports for 2025:


Cost of an LNG import/export terminal: O($1bn) – O($10bn)




2015 projection 13% (export) 9 (export)

Sources: AEO 2005, AEO 2015, FERC 2015, Landfreid et al. 2005LNG: Liquefied natural gas;6

7

GROWTH OF RETROSPECTIVE ANALYSIS1980s: First wave of retrospectives

(Huss 1985abc, and Nelson & Peck 1985, Landsberg 1985, Adam et al 1985)

1980s 1990s 2000s 2010s

1990s: Looking at the extremes

(Shlyakhter et al. 1994, Huntington 1994)2000s: Expansion of interest(Smil 2000, Craig et al. 2002, Joskow et al. 2003, Koomey et al. 2003, Auffhammer 2005, RFF 2009, Considine & Clemente 2007 [and comment by Rode & Fischbeck])

2010s: Applying what we’ve learned

(2013: Climate and Energy Decisionmaking Center leads workshop

Wara et al. 2015)

8

WE USE DATA FROM THE ANNUAL ENERGY OUTLOOK (AEO) :• Projections published from 1982 to 2014 for 31 energy related

quantities (reference case)

• US production, consumption, prices, and imports for oil, coal, natural gas, and electricity; energy consumption by economic sector; GDP, inflation, CO2 emissions

• Published every year with annual resolution by the Energy Information Administration (EIA)

• AEO is based on a large energy-economic model of the US energy system

• Collecting this data required substantial effort, and adjustments in some cases

• Commonly used as a baseline projection for the US energy system

9

Example: Natural gas production


Actual values

9

10

Example: Natural gas production

1980s projections

Actual values


11

1990s projections

Example: Natural gas productionActual values


12

2000s projections



13

2010s projections



14

WHAT IS A “SURPRISE”?First, let’s define projection error:

% projection error =

We define as surprises the largest and smallest 2.5% of all % projection errors for each quantity.

We compute surprises separately for short-term (0-5 year), medium-term (6-10 year) and long-term (11+ year) projections

Note that 5% of all projection values are surprises by definition

We perform extensive sensitivity analysis, using very different definitions of surprise

actual value(projected value – actual value)

Note: The term “error” is shorthand, used by the EIA in its retrospective reports

1515

Gas production surprise frequency

1616


1717


18

Surprises seem to become more frequent after 2004

18


19

FREQUENCY OF SURPRISES FOR ALL QUANTITIES

2020

Frequency over all quantities:Total # surprises/Total # projections

21

MAIN CHARACTERISTICS OF SURPRISE PLOTS• Surprises appear to be more frequent in the

past 10 years than in previous 10, or even 20 years

• Much of this is likely due to the recession and shale gas

• Still, the increase begins in 2005, before the recession• Probably in part attributable to the bust in natural gas-

fired electricity generating capacity

• Much of this is due to a large increase in the number of positive surprises (overprojections)• Negative surprises (underprojections) seem to occur at a

fairly constant rate over time

22

CONCLUSIONS

• The frequency of US energy surprises has increased for all energy quantities

• The US may be in a more volatile energy regime than in the period from 1985-2004

• Long-term energy-related decisions should account for this

23

Evan SherwinCarnegie Mellon University, Engineering and Public Policy

[email protected]

Acknowledgments

Funding:

Advisors:Prof. Inês Azevedo, Carnegie Mellon UniversityDr. Max Henrion, Lumina Decision Systems, Inc.

24

Future work Similar frequency analysis of international and world energy projections

Historical case studies When could surprises conceivably have been predicted? When were the earliest predictions? When did EIA catch on?

25

POLICY IMPLICATIONS• Long-term energy infrastructure projects

should proactively consider the possible effects of unlikely but conceivable surprises• The same goes for other long-term efforts that

depend on energy projections, e.g. greenhouse gas abatement policies

• Increased emphasis on project robustness in a deeply uncertain world

• Similar retrospective analysis on other sets of projections can provide valuable insights

26

CAN WE CONFIRM THE INCREASE?• We use a t-test to determine whether the frequency of

surprises has increased over the past 10 years relative to the previous 20 years• We compare 2005-2014 to 1995-2004, and 1985-1994

• We aggregate over different categories of quantities• Prices• Primary energy production• Primary energy consumption• Primary energy imports• Primary energy consumption/$GDP• Energy consumption by sector• Energy consumption by sector/$GDP

• We use Welch’s t-test for samples of unequal variance

• We assume data are uncorrelated, and treat the effects of correlation parametrically (under development)

27

THE FREQUENCY OF SURPRISES INCREASED BETWEEN ’95-’04 AND ’05-’14

°=5% significance, *=2.5% significance, **=1% significance, ***=0.1% significanceThis analysis treats all surprises as independent and identically distributed.

Category ΔFrequency(’05-’14 v. ’95-’04)

p-value Sig. Surprise standard deviation(’95-’04)

Sample size(’95-’04)

Surprise standard deviation(’05-’14)


Energy prices (constant dollars)

2.8% 0.8% ** 16% 544 23% 762

Consumption, primary energy

3.2% 3% ° 19% 408 26% 548

Production, primary energy

6.4% 0.0004% *** 15% 408 28% 549

Imports, primary energy

4.6% 0.05% *** 9% 272 22% 359

Primary energy consumption/$GDP

4.4% 0.03% *** 16% 536 26% 711

Energy consumption by sector

2.8% 2% *** 19% 544 25% 753

Energy consumption by sector/$GDP

3.0% 1% ** 18% 536 24% 712



2.8%


3.2%


6.4%


4.6%


4.4%


2.8%


3.0%


p-value Sig.


2.8% 0.8% **


3.2% 3% °


6.4% 0.0004% ***


4.6% 0.05% ***


4.4% 0.03% ***


2.8% 2% ***


3.0% 1% **

Recall, average frequency of surprises is 5%.

28

THE FREQUENCY OF SURPRISES GENERALLY INCREASED BETWEEN ’85-’94 AND ’05-’14

°=5% significance, *=2.5% significance, **=1% significance, ***=0.1% significanceThis analysis treats all surprises as independent and identically distributed.


p-value Sig. Surprise standard deviation(’95-’04)


Surprise standard deviation(’05-’14)



-2.3% 16.8% 27% 324 23% 762


4.2% 0.6% ** 17% 243 26% 548


6.5% 0.002% *** 14% 243 28% 549


-5.2% 5.4% 31% 162 22% 359


5.8% 0.0004% *** 12% 220 26% 711


2.9% 3.5% ° 19% 324 25% 753


2.7% 8.5% 19% 220 24% 712



-2.3%


4.2%


6.5%


-5.2%


5.8%


2.9%


2.7%


p-value Sig.


-2.3% 16.8%


4.2% 0.6% **


6.5% 0.002% ***


-5.2% 5.4%


5.8% 0.0004% ***


2.9% 3.5% °


2.7% 8.5%

Yellow denotes a decrease in the frequency of surprises

29

LITERATURE REVIEW• 1980s: The first wave of retrospective analysis

of energy projections• Huss 1985abc, and Nelson & Peck 1985, Landsberg 1985, Adam et al

1985

• 1990s: Looking at the extremes• Shlyakhter et al. 1994, Huntington 1994

• 2000s: Expansion of interest• Smil 2000, Craig et al. 2002, Joskow et al. 2003, Koomey et al. 2003,

Auffhammer 2005, RFF 2009, Considine & Clemente 2007 [and comment by Rode & Fischbeck]

• 2010s: Applying what we’ve learned• 2013: Climate and Energy Decisionmaking Center leads workshop• Wara et al. 2015

30

COMPUTING SURPRISE THRESHOLDS• For each quantity, we want a low and high error threshold

• If % Projection Error exceeds one of these thresholds, the projection for that year is a surprise

• Step 1: Order all % Projection Error values for the desired quantity

• Step 2: Generate cumulative density function from these errors

• Step 3: Compute the x and 1-x percentiles (default x=2.5%ile)

• Step 4: These computed percentiles are the low and high surprise thresholds for the quantity in question

• Note: We compute these thresholds separately for short-term (0-5yr), medium-term (6-10yr), and long-term (11+yr) projections

31

GAS PRODUCTION PROJECTION ERRORS

31

97.5%

2.5%

Negative threshold: -16%

Positive threshold: 19%

31

32

GAS PRODUCTION PROJECTION ERRORS

97.5%

2.5%

Negative thresholds Short-term: -16% Medium-term: -17% Long-term: -17%

Positive thresholds: Short-term: 11% Medium-term: 23% Long-term: 18%

32

33

POST-WWII ENERGY STABILITY

• The US energy system post-WWII was fairly predictable.• Due to structural economic reasons (electricity

demand grew by ~7.25% for several decades until 1973)

• Due to policy reasons (e.g. gas price regulation)• The 1970s were much more unstable.

• 2 OPEC oil embargos • Oil prices more than tripled in 1973, then more

than doubled again in 1979• Switch from exponential to linear electric power

demand growth• First major government energy projections

commissioned to inform energy decision-making

34

WHAT IS A “SURPRISE”?

First, let’s define projection error:

% projection error =actual value

(projected value – actual value)

We then define “surprises” as the values outside a specified confidence interval for the cumulative density function of the distribution of all % projection error for a quantity.

Note: The term “error” is shorthand, used by the EIA in its retrospective reports

35

COMPUTING SURPRISE THRESHOLDS• For each quantity, we want a positive and negative

error threshold• If % Projection Error exceeds one of these thresholds, the

projection for that year is a surprise

• Step 1: Order all % Projection Error values for the desired quantity

• Step 2: Generate cumulative density function from these errors

• Step 3: Compute the x and 1-x percentiles (default x=2.5%ile)

• Step 4: These computed percentiles are the positive and negative surprise thresholds for the quantity in question

u.s. e nergy s urprises h ave b ecome m ore f requent evan sherwin with inês azevedo & max...

Documents