renewable energy & electricity markets

Renewable energy & Electricity markets

Adam Wierman, CaltechJoint work with Sachin Adlakha, Subhonmesh Bose, Desmond Cai, John Ledyard, Steven Low, and Jayakrishnan Nair.

Be careful what you wish for

MW

WorldwideWind:

MW

Europe

AmericasChina

Solar PV:

Renewable energy is coming!

here!

Renewable energy is coming!…but incorporation into the grid isn’t easy

They are typically Uncontrollable (not available “on demand”) Intermittent (large fluctuations) Uncertain (difficult to forecast)

Each line is wind generation over 1 day

here!

Key Constraint: Generation = Load(at all times)

low uncertainty

Today’s grid

GenerationLoad

Key Constraint: Generation = Load(at all times)

low uncertaintycontrollable(via markets)

Today’s grid

GenerationLoad

Key Constraint: Generation = Load

less controllablehigh uncertainty

(at all times)

Tomorrow’s grid

low uncertainty

Key Constraint: Generation = Load

less controllablehigh uncertainty

low uncertainty

(at all times)

1) Huge price variability, leading to generators opting out of markets!2) More conventional reserves needed, countering sustainability gains!

“ON JUNE 16th something very peculiar

happened in Germany’s electricity market. The

wholesale price of electricity fell to minus €100

per megawatt hour (MWh). That is, generating

companies were having to pay the managers of

the grid to take their electricity.”

“Energiewende has so far

increased, not decreased,

emissions of greenhouse

gases.”

What can be done?Reduce the uncertainty

Design for the uncertainty

•Better prediction• “Aggregation” … in time (storage) … in space (distributed generation) … in generation (heterogeneous mix)

•Redesign electricity markets• Increase amount of demand response

our focus at Caltech

This talk: Two electricity market design challenges1) How many markets should there be? and when should they occur?2) The nasty economic consequences of Kirchhoff's laws

The newsvendor problemNetworked Cournot competitionstochastic

networks

Forget about energy for a second…This section is really about the role of uncertainty in newsvendor problems

Forget about energy for a second…This section is really about the role of uncertainty in newsvendor problems

Estimate demand,

Purchase,

Demand is realized

lost revenue wasted inventory

uncertainty

“You have to decide today how many newspapers you want to sell tomorrow…”

Forget about energy for a second…

“You have to decide today how many newspapers you want to sell tomorrow…”seasonal productsperishable goods

compute instancesenergy

…

This section is really about the role of uncertainty in newsvendor problems

timeint. /day

ahead

realtime

longterm

Utility buys power to

meet demand

Electricity marketsmarkets

timeint. /day

ahead

realtime

longterm

markets

PIRP

timeint. /day

ahead

realtime

longterm

markets

What is the impact of long term wind contracts?As renewable penetration increases: 1)Should markets be moved closer to real-

time? 2)Should markets be added?

4 hr market

How should utilities procure electricity in the presence of renewable energy?First step:

As renewable penetration increases: 1)Should markets be moved closer to real-

time? 2)Should markets be added?

What is the impact of long term wind contracts?

int. /day

ahead

realtime

longterm

price↑

𝑝𝑙𝑡 𝑝𝑖𝑛 𝑝𝑟𝑡

int. /day

ahead

realtime

longterm

price volatility↑

𝑝𝑙𝑡 𝑝𝑖𝑛 𝑝𝑟𝑡

𝑞𝑙𝑡 𝑞𝑖𝑛 𝑞𝑟𝑡

𝐸 [𝑝𝑖𝑛 ]>𝑝𝑙𝑡 𝐸 [𝑝𝑟𝑡|𝑝𝑖𝑛 ]>𝑝𝑖𝑛

int. /day

ahead

realtime

longterm

price↑

𝑝𝑙𝑡 𝑝𝑖𝑛 𝑝𝑟𝑡

𝑞𝑙𝑡 𝑞𝑖𝑛 𝑞𝑟𝑡

�̂�𝑙𝑡 �̂�𝑖𝑛 𝑤

𝜀1=�̂�𝑙𝑡− �̂�𝑖𝑛 𝜀2=�̂�𝑖𝑛−𝑤

Assumption: and are independent(A generalization of the martingale model of forecast evolution)

wind uncertainty ↓

𝑞𝑙𝑡+𝑞𝑖𝑛+𝑞𝑟𝑡+𝑤≥𝑑Key Constraint: Generation = Load

int. /day

ahead

realtime

longterm

price↑wind uncertainty ↓

𝑝𝑙𝑡 𝑝𝑖𝑛 𝑝𝑟𝑡

𝑞𝑙𝑡 𝑞𝑖𝑛 𝑞𝑟𝑡

�̂�𝑙𝑡 �̂�𝑖𝑛 𝑤

(we ignore network constraints for now)

int. /day

ahead

realtime

longterm

price↑wind uncertainty ↓

𝑝𝑙𝑡 𝑝𝑖𝑛 𝑝𝑟𝑡

𝑞𝑙𝑡 𝑞𝑖𝑛 𝑞𝑟𝑡

�̂�𝑙𝑡 �̂�𝑖𝑛 𝑤

Utility goal:min𝐸 [𝑝𝑙𝑡 𝑞𝑙𝑡+𝑝𝑖𝑛𝑞𝑖𝑛+𝑝𝑟𝑡𝑞𝑟𝑡 ]Subject to causality constraints

int. /day

ahead

realtime

longterm

𝑝𝑙𝑡 𝑝𝑖𝑛 𝑝𝑟𝑡

𝑞𝑙𝑡 𝑞𝑖𝑛 𝑞𝑟𝑡

�̂�𝑙𝑡 �̂�𝑖𝑛 𝑤

Utility goal:min𝐸 [𝑝𝑙𝑡 𝑞𝑙𝑡+𝑝𝑖𝑛𝑞𝑖𝑛+𝑝𝑟𝑡𝑞𝑟𝑡 ]Subject to causality constraintsVariant of the newsvendor problem

[Arrow et. al. ’51], [Silver et. al. ’98], [Khouja ’99], [Porteus ’02], [Wang et. al. ’12].

Theorem:The optimal procurement strategy is characterized by reserve levels and such that

where

and uniquely solves

int. /day

ahead

realtime

longterm �̂�𝑙𝑡 �̂�𝑖𝑛 𝑤

𝜀1=�̂�𝑙𝑡− �̂�𝑖𝑛 𝜀2=�̂�𝑖𝑛−𝑤

baseline, e.g., average output of a wind farm scale, e.g., number of wind farms

Scaling regime

aggregation, e.g., degree of correlation between wind farms

𝒘 𝒍𝒕 (𝜸 )=𝜸𝜶 𝜺𝟐 (𝜸 )=𝜸𝜽𝜺𝟐𝜺𝟏 (𝜸 )=𝜸𝜽𝜺𝟏

baseline, e.g., average output of a wind farm scale, e.g., number of wind farms

Scaling regime

aggregation, e.g., degree of correlation between wind farms

Theorem:

Procurement with zero uncertainty

Extra procurementdue to uncertainty

baseline, e.g., average output of a wind farm scale, e.g., number of wind farms

Scaling regime

aggregation, e.g., degree of correlation between wind farms

Theorem:

Depends on markets & predictions - prices - forecasts

Depends on wind aggregation - =1/2 (independent) - =1 (correlated)

baseline, e.g., average output of a wind farm scale, e.g., number of wind farms

Scaling regime

aggregation, e.g., degree of correlation between wind farms

Theorem:

This form holds more generally than the model studied here:

-- more than three markets: [Bitar et al., 2012]-- when prices are endogenous: [Cai & Wierman, 2014]-- when small-scale storage is included: [Hayden, Nair, & Wierman, Working paper]

timeint. /day

ahead

realtime

longterm

markets

Electricity markets

As renewable penetration increases: 1)Should markets be moved closer to real-

time? 2)Should markets be added?

No!

What is the impact of long term wind contracts?

timeint. /day

ahead

realtime

longterm

markets

Electricity markets

As renewable penetration increases: 1)Should markets be moved closer to real-

time? 2)Should markets be added?

4 hr ahead marke

t?What is the impact of long term wind contracts?

realtime

longterm v/s int. real

timelongterm

What happens to if a market is added?

What happens to if a market is added?

6 6.5 7 7.5 8 8.5 9 9.5 10

int. /day

ahead

realtime

longterm

𝜀2 Gaussian

𝑝𝑙𝑡=6 6<𝑝𝑖𝑛<10 𝑝𝑟𝑡=10

𝑝𝑖𝑛

]

2 markets

3 markets

3 markets are always better!

When does this happen?

Theorem:If is increasing for , decreasing for , and satisfies:

is decreasing for is decreasing for

then the expected procurement is lower with 3 markets than with 2 markets.

Satisfied by the Gaussian distribution

int. /day

ahead

realtime

longterm

𝜀2 Weibull

𝑝𝑙𝑡=6 6<𝑝𝑖𝑛<10 𝑝𝑟𝑡=10

6 6.5 7 7.5 8 8.5 9 9.5 10𝑝𝑖𝑛

]

2 markets

3 markets

3 markets can be worse!

When does this happen?

Theorem:If satisfies the condition:

=0 , then there exist prices such that the expected procurement is higher with 3 markets than with 2 markets.

Estimation errors are heavy-tailed(specifically, long-tailed)

timeint. /day

ahead

realtime

longterm

markets

As renewable penetration increases: 1)Should markets be moved closer to real-

time? 2)Should markets be added?

No! It depends, Gaussian or heavy-tailed?

4 hr market

What is the impact of long term wind contracts?

timeint. /day

ahead

realtime

longterm

marketsmarkets

PIRP

How should wind be incorporated into the markets?

What is the impact of long term wind contracts?

This talk: Two electricity market design challenges1) How many markets should there be? and when should they occur?2) The nasty economic consequences of Kirchhoff's laws

The newsvendor problemNetworked Cournot competition

Forget about energy for a second…This section is really about intermediaries & competition in networked markets

Forget about energy for a second…This section is really about intermediaries & competition in networked markets

Rarely is competition in a single, well defined market… firms typically compete across a variety of markets

Firms Markets

Gas pipelines in the US

Rarely is competition in a single, well defined market… firms typically compete across a variety of markets

Examples: gas, airlines, construction, … , energy

Forget about energy for a second…This section is really about intermediaries & competition in networked markets

Key Constraint: Generation = Load(at all times)

G

G

G

GG

(at all times)

L L

L

Key Constraint: Generation = Load

controllable(via markets)

G

G

G

GG

L L

L

cost

quantity

Market run by the Independent System Operator (ISO)Determines the quantity to procure and price to charge each generator in order to meet the load s.t. network constraints.

A toy example

𝐺1

𝐺2

Load = 6

capa

city

= 1

cost

quantity

cost

quantity

𝐺1

𝐺2

Load = 6

capa

city

= 1

2

1

3

𝐺1

𝐺2

Load = 6

capa

city

= 1

2

1

3

3 2

1

But what if is strategic?

𝐺1

𝐺2

Load = 6

capa

city

= 1

cost

quantity

2

1

3

Kirchhoff's laws create a hidden monopoly!

“…supply-demand imbalance, flawed market design and inconsistent rules made possible significant market manipulation” -- FERC

Kirchoff’s laws can have nasty market consequences…

“JPMorgan Chase & Co. will pay $410 million to settle U.S. Federal Energy Regulatory Commission allegations that the bank manipulated power markets, enriching itself at the expense of consumers in California and the Midwest from 2010 to 2012.”

Kirchoff’s laws can have nasty market consequences…

“Energy Capital Partners … paid $650 million last year to acquire three generating plant complexes, including the second largest electric power plant in New England, Brayton Point. Five weeks after the deal closed, Energy partners moved to shutter Brayton Point. Why would anyone spend hundreds of millions of dollars to buy the second largest electric power plant in New England and then quickly take steps to shut it down?

Kirchoff’s laws can have nasty market consequences…

How can “market power” be identified and quantified? Can markets be designed to mitigate market power?

G

G

G

GG

L

L

cost

quantity

Market run by the Independent System Operator (ISO)

Networked Cournot competition

Determines the quantity to procure and price to charge each generator in order to meet the load s.t. network constraints.

Market maker / Intermediary (ISO)

GeneratorsBid: quantityQuadratic Costs: Profit: Load

Linear demand function

Networked Cournot competition

Stochastic?

Determines the quantity to procure and price to charge each generator in order to meet the load s.t. network constraints.

ISO behavior is typically regulatedOften forced to maximize one of :1) Social welfare: Consumers’ utility – generation costs2) Residual social welfare: Consumers’ utility – generator profits3) Consumer surplus: Consumers’ utility – consumer payments

Market maker / Intermediary (ISO)Determines the quantity to procure and price to charge each generator in order to meet the load s.t. network constraints.

Choose “rebalancing quantities” to

Shift factor matrix line constraints(Kirchhoff’s Laws)

Market maker / Intermediary (ISO)Determines the quantity to procure and price to charge each generator in order to meet the load s.t. network constraints.

Choose “rebalancing quantities” to &

GeneratorsBid: quantityQuadratic Costs: Profit: Load

Linear demand function

Market maker / Intermediary (ISO)

[Barquin & Vasquez 2005, 2008], [Iklic 2009], [Neuhoff et at, 2005], [Yao, Oren, Adler, 2005, 2007] …

Networked Cournot competition

Existence?

TheoremA generalized Nash equilibrium always exists when the ISO maximizes social welfare or residual social welfare.

However, a generalized Nash equilibrium may not exist if the ISO maximizes consumer surplus.

very susceptible to market power manipulations

A toy example: “Path 15”

linear demand

]

𝐺1

𝐺2

𝐿2

𝐿1

A toy example: “Path 15”

linear demand

quadratic cost

quadratic cost 𝑐1=𝑐2

𝑐1=𝑐2

linear demand

]

𝐺1

𝐺2

𝐿2

𝐿1

A toy example: “Path 15”

linear demand

quadratic cost

quadratic cost 𝑐1=𝑐2

𝑐1=𝑐2

consumer surplus

social welfareresidual social welfare 𝐺1

𝐺2

𝐾Profi

t

Even without line constraints the 2-node network is not

equivalent to an aggregated market!

Line expansion has very different impact depending

on the market objective

linear demand

𝑟 ∈(−∞ ,∞)

𝐺1

𝐺2

𝐿2

𝐿1

A toy example: “Path 15”

linear demand

quadratic cost

TheoremA generalized Nash equilibria exist for all three objectives, but the equilibria differ considerably:- For social welfare, - For residual social welfare, .- For consumer surplus,

quadratic cost 𝑐1=𝑐2

𝑐1=𝑐2

Even without line constraints the 2-node network is not

equivalent to an aggregated market!

Line expansion has very different impact depending

on the market objective

How can “market power” be identified and quantified? Can markets be designed to mitigate market power?

What is the “right” market objective?

This talk: Two electricity market design challenges1) How many markets should there be? and when should they occur?2) The nasty economic consequences of Kirchhoff's laws

The newsvendor problemNetworked Cournot competition

Many other rich, challenging stochastic networks problems…

Renewable Energy & Electricity MarketsAdam Wierman, CaltechJoint work with Sachin Adlakha, Subhonmesh Bose, Desmond Cai, John Ledyard, Steven Low, and Jayakrishnan Nair.

Be careful what you wish for

Subhonmesh Bose, Desmond Cai, Steven Low and Adam Wierman. “The role of a market maker in networked Cournot competition.” Under submission.

Chenye Wu, Subhonmesh Bose, Adam Wierman and Hamed Mohsenian-Rad. “A unifying approach for assessing market power in deregulated electricity markets.” Proceedings of IEEE PES General Meeting, 2013. ``Best Paper on System Operations and Market Economics'' award recipient.

Jayakrishnan Nair, Sachin Adlakha and Adam Wierman. “Energy procurement strategies in the presence of intermittent sources.” Proceedings of ACM Sigmetrics, 2014.

Desmond Cai and Adam Wierman. “Inefficiency in Forward Markets with Supply Friction.” Proceedings of IEEE CDC, 2013.

This talk: 3 electricity market design challenges1) How many markets should there be? and when should they occur?2) The nasty economic consequences of Kirchhoff's laws3) Who should have control: the engineer or the economist?

the newsvendor problemnetworked Cournot competitionshadow pricing vs. VCG

Key Constraint: Generation = Load

less controllablehigh uncertainty

low uncertainty

(at all times)

Tomorrow’s grid

Demand must follow Generation(to some extent)

Grid needs huge growth in Demand Response

Demand must follow Generation(to some extent)

News articles

[ADD REFS TO DEMOS, ETC]

The big debate for demand response: The economist vs. The engineer

Prices to devices, a.k.a. “let there be markets”Send nodal price signals to consumers and let consumer devices respond

The big debate for demand response: The economist vs. The engineer

Prices to devices, a.k.a. “let there be markets”Send nodal price signals to consumers and let consumer devices respond

+ Prices can be designed so that, at equilibrium, social optimality is achieved- Consumer response is uncertain- Markets do not equilibrate instantaneously, and convergence is likely unstable

[ADD CITATIONS]

The big debate for demand response: The economist vs. The engineer

[ADD REFS TO DEMOS, ETC]

Direct control, a.k.a. “Hand over the keys”Give the utility control over consumer devices

The big debate for demand response: The economist vs. The engineer

Direct control, a.k.a. “Hand over the keys”Give the utility control over consumer devices

+ Response is fast and guaranteed- Computational demands on utility are extreme- Utility does not know customer preferences, so control is not socially optimal

The big debate for demand response: The economist vs. The engineer

How can we combine these perspectives?

“Mechanisms for control”Idea: price control policies rather than consumption

𝑟 ∈[−𝐾 ,𝐾 ]

G

G

L

L

A toy example: “Path 15”

consumerswith utility

generators w/ quadratic cost

generators w/ quadratic cost

consumerswith utility

Social objective:

“Mechanism for control”1. Consumers report 2. Utility computes allocation and prices

Social objective:

The challenges:1. Communication: Can the consumers describe their utilities?2. Incentives: Will the consumer be truthful?3. Computation: Can the utility compute the prices efficiently?