U.S. ENERGY SURPRISES
HAVE BECOME MORE FREQUENTEvan Sherwin
With Inês Azevedo & Max Henrion Carnegie Mellon University
Funding:
2
GOALSThe goals of this work are to: • characterize what constitutes a surprise• identify the biggest surprises in recent
decades• understand how the frequency of surprises
changed over time
3
Surprises affect long-term decisions
Projection from 1986
Source: DOE 1999, AEO 2014
Actual values
US Natural Gas Imports
3
4
Surprises affect long-term decisions
Projection from 1986
Projection from 1998
Source: DOE 1999, AEO 2014
Actual values
US Natural Gas Imports
44
5
Surprises affect long-term decisions
Projection from 1986
Projection from 1998
Source: DOE 1999, AEO 2014
Projection from 2004
Actual values
US Natural Gas Imports
5
% of projected consumption
LNG terminals approved
2004 projection 23% (import) 18 (import, by 2005)
Projected US natural gas imports for 2025:
5Sources: AEO 2005, AEO 2015, FERC 2015, Landfreid et al. 2005
Cost of an LNG import/export terminal: O($1bn) – O($10bn)LNG: Liquefied natural gas;5
6
Surprises affect long-term decisions
Projection from 1986
Projection from 1998
Actual values
Source: DOE 1999, AEO 2014
Projection from 2015
% of projected consumption
LNG terminals approved
2004 projection 23% (import) 18 (import, by 2005)
US Natural Gas Imports
6
Projected US natural gas imports for 2025:
Projection from 2004
Cost of an LNG import/export terminal: O($1bn) – O($10bn)
% of projected consumption
LNG terminals approved
2004 projection 23% (import) 18 (import, by 2005)
2015 projection 13% (export) 9 (export)
Sources: AEO 2005, AEO 2015, FERC 2015, Landfreid et al. 2005LNG: Liquefied natural gas;6
7
GROWTH OF RETROSPECTIVE ANALYSIS1980s: First wave of retrospectives
(Huss 1985abc, and Nelson & Peck 1985, Landsberg 1985, Adam et al 1985)
1980s 1990s 2000s 2010s
1990s: Looking at the extremes
(Shlyakhter et al. 1994, Huntington 1994)2000s: Expansion of interest(Smil 2000, Craig et al. 2002, Joskow et al. 2003, Koomey et al. 2003, Auffhammer 2005, RFF 2009, Considine & Clemente 2007 [and comment by Rode & Fischbeck])
2010s: Applying what we’ve learned
(2013: Climate and Energy Decisionmaking Center leads workshop
Wara et al. 2015)
8
WE USE DATA FROM THE ANNUAL ENERGY OUTLOOK (AEO) :• Projections published from 1982 to 2014 for 31 energy related
quantities (reference case)
• US production, consumption, prices, and imports for oil, coal, natural gas, and electricity; energy consumption by economic sector; GDP, inflation, CO2 emissions
• Published every year with annual resolution by the Energy Information Administration (EIA)
• AEO is based on a large energy-economic model of the US energy system
• Collecting this data required substantial effort, and adjustments in some cases
• Commonly used as a baseline projection for the US energy system
9
Example: Natural gas production
Source: DOE 1999, AEO 2014
Actual values
9
10
Example: Natural gas production
1980s projections
Actual values
Source: DOE 1999, AEO 201410
11
1990s projections
Example: Natural gas productionActual values
Source: DOE 1999, AEO 201411
12
2000s projections
Example: Natural gas productionActual values
Source: DOE 1999, AEO 201412
13
2010s projections
Example: Natural gas productionActual values
Source: DOE 1999, AEO 201413
14
WHAT IS A “SURPRISE”?First, let’s define projection error:
% projection error =
We define as surprises the largest and smallest 2.5% of all % projection errors for each quantity.
We compute surprises separately for short-term (0-5 year), medium-term (6-10 year) and long-term (11+ year) projections
Note that 5% of all projection values are surprises by definition
We perform extensive sensitivity analysis, using very different definitions of surprise
actual value(projected value – actual value)
Note: The term “error” is shorthand, used by the EIA in its retrospective reports
1515
Gas production surprise frequency
1616
Gas production surprise frequency
1717
Gas production surprise frequency
18
Surprises seem to become more frequent after 2004
18
Gas production surprise frequency
19
FREQUENCY OF SURPRISES FOR ALL QUANTITIES
2020
Frequency over all quantities:Total # surprises/Total # projections
21
MAIN CHARACTERISTICS OF SURPRISE PLOTS• Surprises appear to be more frequent in the
past 10 years than in previous 10, or even 20 years
• Much of this is likely due to the recession and shale gas
• Still, the increase begins in 2005, before the recession• Probably in part attributable to the bust in natural gas-
fired electricity generating capacity
• Much of this is due to a large increase in the number of positive surprises (overprojections)• Negative surprises (underprojections) seem to occur at a
fairly constant rate over time
22
CONCLUSIONS
• The frequency of US energy surprises has increased for all energy quantities
• The US may be in a more volatile energy regime than in the period from 1985-2004
• Long-term energy-related decisions should account for this
23
Evan SherwinCarnegie Mellon University, Engineering and Public Policy
Acknowledgments
Funding:
Advisors:Prof. Inês Azevedo, Carnegie Mellon UniversityDr. Max Henrion, Lumina Decision Systems, Inc.
24
Future work Similar frequency analysis of international and world energy projections
Historical case studies When could surprises conceivably have been predicted? When were the earliest predictions? When did EIA catch on?
25
POLICY IMPLICATIONS• Long-term energy infrastructure projects
should proactively consider the possible effects of unlikely but conceivable surprises• The same goes for other long-term efforts that
depend on energy projections, e.g. greenhouse gas abatement policies
• Increased emphasis on project robustness in a deeply uncertain world
• Similar retrospective analysis on other sets of projections can provide valuable insights
26
CAN WE CONFIRM THE INCREASE?• We use a t-test to determine whether the frequency of
surprises has increased over the past 10 years relative to the previous 20 years• We compare 2005-2014 to 1995-2004, and 1985-1994
• We aggregate over different categories of quantities• Prices• Primary energy production• Primary energy consumption• Primary energy imports• Primary energy consumption/$GDP• Energy consumption by sector• Energy consumption by sector/$GDP
• We use Welch’s t-test for samples of unequal variance
• We assume data are uncorrelated, and treat the effects of correlation parametrically (under development)
27
THE FREQUENCY OF SURPRISES INCREASED BETWEEN ’95-’04 AND ’05-’14
°=5% significance, *=2.5% significance, **=1% significance, ***=0.1% significanceThis analysis treats all surprises as independent and identically distributed.
Category ΔFrequency(’05-’14 v. ’95-’04)
p-value Sig. Surprise standard deviation(’95-’04)
Sample size(’95-’04)
Surprise standard deviation(’05-’14)
Sample size(’05-’14)
Energy prices (constant dollars)
2.8% 0.8% ** 16% 544 23% 762
Consumption, primary energy
3.2% 3% ° 19% 408 26% 548
Production, primary energy
6.4% 0.0004% *** 15% 408 28% 549
Imports, primary energy
4.6% 0.05% *** 9% 272 22% 359
Primary energy consumption/$GDP
4.4% 0.03% *** 16% 536 26% 711
Energy consumption by sector
2.8% 2% *** 19% 544 25% 753
Energy consumption by sector/$GDP
3.0% 1% ** 18% 536 24% 712
Category ΔFrequency(’05-’14 v. ’95-’04)
Energy prices (constant dollars)
2.8%
Consumption, primary energy
3.2%
Production, primary energy
6.4%
Imports, primary energy
4.6%
Primary energy consumption/$GDP
4.4%
Energy consumption by sector
2.8%
Energy consumption by sector/$GDP
3.0%
Category ΔFrequency(’05-’14 v. ’95-’04)
p-value Sig.
Energy prices (constant dollars)
2.8% 0.8% **
Consumption, primary energy
3.2% 3% °
Production, primary energy
6.4% 0.0004% ***
Imports, primary energy
4.6% 0.05% ***
Primary energy consumption/$GDP
4.4% 0.03% ***
Energy consumption by sector
2.8% 2% ***
Energy consumption by sector/$GDP
3.0% 1% **
Recall, average frequency of surprises is 5%.
28
THE FREQUENCY OF SURPRISES GENERALLY INCREASED BETWEEN ’85-’94 AND ’05-’14
°=5% significance, *=2.5% significance, **=1% significance, ***=0.1% significanceThis analysis treats all surprises as independent and identically distributed.
Category ΔFrequency(’05-’14 v. ’95-’04)
p-value Sig. Surprise standard deviation(’95-’04)
Sample size(’95-’04)
Surprise standard deviation(’05-’14)
Sample size(’05-’14)
Energy prices (constant dollars)
-2.3% 16.8% 27% 324 23% 762
Consumption, primary energy
4.2% 0.6% ** 17% 243 26% 548
Production, primary energy
6.5% 0.002% *** 14% 243 28% 549
Imports, primary energy
-5.2% 5.4% 31% 162 22% 359
Primary energy consumption/$GDP
5.8% 0.0004% *** 12% 220 26% 711
Energy consumption by sector
2.9% 3.5% ° 19% 324 25% 753
Energy consumption by sector/$GDP
2.7% 8.5% 19% 220 24% 712
Category ΔFrequency(’05-’14 v. ’95-’04)
Energy prices (constant dollars)
-2.3%
Consumption, primary energy
4.2%
Production, primary energy
6.5%
Imports, primary energy
-5.2%
Primary energy consumption/$GDP
5.8%
Energy consumption by sector
2.9%
Energy consumption by sector/$GDP
2.7%
Category ΔFrequency(’05-’14 v. ’85-’94)
p-value Sig.
Energy prices (constant dollars)
-2.3% 16.8%
Consumption, primary energy
4.2% 0.6% **
Production, primary energy
6.5% 0.002% ***
Imports, primary energy
-5.2% 5.4%
Primary energy consumption/$GDP
5.8% 0.0004% ***
Energy consumption by sector
2.9% 3.5% °
Energy consumption by sector/$GDP
2.7% 8.5%
Yellow denotes a decrease in the frequency of surprises
29
LITERATURE REVIEW• 1980s: The first wave of retrospective analysis
of energy projections• Huss 1985abc, and Nelson & Peck 1985, Landsberg 1985, Adam et al
1985
• 1990s: Looking at the extremes• Shlyakhter et al. 1994, Huntington 1994
• 2000s: Expansion of interest• Smil 2000, Craig et al. 2002, Joskow et al. 2003, Koomey et al. 2003,
Auffhammer 2005, RFF 2009, Considine & Clemente 2007 [and comment by Rode & Fischbeck]
• 2010s: Applying what we’ve learned• 2013: Climate and Energy Decisionmaking Center leads workshop• Wara et al. 2015
30
COMPUTING SURPRISE THRESHOLDS• For each quantity, we want a low and high error threshold
• If % Projection Error exceeds one of these thresholds, the projection for that year is a surprise
• Step 1: Order all % Projection Error values for the desired quantity
• Step 2: Generate cumulative density function from these errors
• Step 3: Compute the x and 1-x percentiles (default x=2.5%ile)
• Step 4: These computed percentiles are the low and high surprise thresholds for the quantity in question
• Note: We compute these thresholds separately for short-term (0-5yr), medium-term (6-10yr), and long-term (11+yr) projections
31
GAS PRODUCTION PROJECTION ERRORS
31
97.5%
2.5%
Negative threshold: -16%
Positive threshold: 19%
31
32
GAS PRODUCTION PROJECTION ERRORS
97.5%
2.5%
Negative thresholds Short-term: -16% Medium-term: -17% Long-term: -17%
Positive thresholds: Short-term: 11% Medium-term: 23% Long-term: 18%
32
33
POST-WWII ENERGY STABILITY
• The US energy system post-WWII was fairly predictable.• Due to structural economic reasons (electricity
demand grew by ~7.25% for several decades until 1973)
• Due to policy reasons (e.g. gas price regulation)• The 1970s were much more unstable.
• 2 OPEC oil embargos • Oil prices more than tripled in 1973, then more
than doubled again in 1979• Switch from exponential to linear electric power
demand growth• First major government energy projections
commissioned to inform energy decision-making
34
WHAT IS A “SURPRISE”?
First, let’s define projection error:
% projection error =actual value
(projected value – actual value)
We then define “surprises” as the values outside a specified confidence interval for the cumulative density function of the distribution of all % projection error for a quantity.
Note: The term “error” is shorthand, used by the EIA in its retrospective reports
35
COMPUTING SURPRISE THRESHOLDS• For each quantity, we want a positive and negative
error threshold• If % Projection Error exceeds one of these thresholds, the
projection for that year is a surprise
• Step 1: Order all % Projection Error values for the desired quantity
• Step 2: Generate cumulative density function from these errors
• Step 3: Compute the x and 1-x percentiles (default x=2.5%ile)
• Step 4: These computed percentiles are the positive and negative surprise thresholds for the quantity in question