ÖGOR - IHS WorkshopMathematische Ökonomie und Optimierung in der Energiewirtschaft

September 23, 2010Vienna

Optimal Day-Ahead Bidding of Electricity Storage using Approximate Dynamic Programming

Nils Loehndorf David Wozabal Stefan Minner

Department of Business AdministrationUniversity of Vienna, Austria

Outline

Dispatch of Electricity Storage

Optimal dispatch is a stochastic-dynamic decision problem

Bidding decisions on the day-ahead real-time market

Decision about storage level

Risk aversion or price response to avoid excessive use of balancing market

Method

Stochastic dynamic programming (SDP)

Instantaneous profit calculated by two stage dynamic stochastic problem

Sampling based approach

Solved using Approximate Dynamic Programming using post decision logic

Numerical Results

2

Decision Process

Inter-day decision process

Daily decisions

Optimize storage levels over time

State transition is a Markov process

State variables: daily wind production, mean temperatures and weekday

Approximate Dynamic Programming (ADP)

Intra-day decision problem

Hourly decisions

Optimize day-ahead bidding decisions

Day-ahead & real-time price uncertainty dependent on state

Stochastic Programming

3

Decision Process

4

real-timeprice scenarios

day-aheadprice scenarios

day-aheadmarket is closed

final reservoirlevel

submit bids

day-ahead pricerealization

balancing and storage operation

state transition

state = {day, wind, temperature, reservoir}

inter-day decision

reservoir levelat 100%

reservoir levelat 0%

reservoir levelat 50%

Bidding Curves

Bids to day ahead (DA) market are submitted one day in advance

Bidding curves are piecewise constant

Random prices random bids random storage levels

Excess bidding on DA market is balanced on real time (RT) market

5

Dynamic Programming Formulation

6

Define rt as the function that maps every vector of price realizations to a final storage state. We write

where π is the instantaneous profit, x is the intraday decision and

Proposition 1. The function R(.,.) is concave in rt and rt+1.

Proposition 2. V(St,.) is concave.

Algorithmic Strategy

Partition the state space into multi-dimensional grid, i.e.

where the Gk are the disjoint grid cells.

Define function G(S) which maps every state into grid cell

Approximate V(S, .) by piecewise linear concave function V(G(S), .)

Iteratively update the sampled post decision value function

7

Approximate Value Iteration (Inter-Day)

8

Generate a sample of state transitions: S1,...,ST

Define a set of initial reservoir levels as breakpoints: B1,...,BL

For t=2 to T do

Sample price realizations dependent on St

For l=1 to L do

End

End

Return value function

0

10

20

30

40

50

60

70

0 480 960 1440 1920 2400Reservoir Level (in MWh)

Expected Profit (in $1,000)

update the post-decision value

of the previous state

Stochastic Programming Formulation (Intra-Day)

9

Day-ahead price response:

real-time value functionday-ahead

Real-time price response:

Objective: maximize risk adjusted expected profit

realized day-ahead bids

real-time balancing decisions

realized reservoir levels

discount factor

estimated slope coefficients

day-ahead price scenarios

Subject to... Min./max. capacity constraints (binary) Market and reservoir balance equations Price dependent, piecewise-constant bidding curves

Ps

Bidding Curve

10

-120

-80

-40

0

40

80

120

20 30 40 50 60 70

Bid

din

g V

olu

me

Price

realized day-ahead bid

bidding curve coefficient

pre-defined price segment

day-ahead price scenario

Linear formulation Price segments are given by samples Equal number of scenarios per price segment price segments with flexible width price segments with equal probability

Piecewise-constant bidding curve*:

price scenarios

*See Fleten and Kristoffersen (2007) for a piece-wise linear formulation.

Bounding the Value Function

11

PJM Market

12

Day ahead market Daily clearing Bidding curves Low volatility High market depth

Balancing market Intra day trading High volatility Low market depth

Econometric Model

13

Day of the Year

Wind Power Day LengthTemperature

LoadR2=(0.75,0.82,0.88)

Hour of the Day Day of the Week(Weekday, Weekend)

Day-Ahead PriceR2=(0.74,0.86,0.94)

Real-Time PriceR2=(0.24,0.6,0.91)

State Transition Model

Electricity Price Model

48 semi-log models Stepwise backward-forward

regression t-distributed error terms intra-day price scenarios

Trigonometric regression Detrended VAR(1) model inter-day state transition

Markov process of wind power, temperature and day length

State

State Transition Model

14

Data Sources: PJM Energy Market, Wolfram Mathematica WeatherData

Tem

pe

rature

(in

F)Cap

acit

y (i

n G

W)

Days of 2009

0

25

50

75

100

0

30

60

90

120

150

180

1 31 61 91 121 151 181 211 241 271 301 331 361

Wind Generation Wind Forecast Temperature Temperature Forecast

Electricity Price Model

15

Data Sources: PJM Energy Market, Wolfram Mathematica WeatherData

Load

(in G

W)

Day

-Ah

ead

Pri

ce (

in $

)

06/29/2009 – 07/05/2009

0

20

40

60

80

100

0

10

20

30

40

50

60

1 25 49 73 97 121 145

Prices Forecast Price Demand Forecast Demand

Value Function Comparison

Parameters 10,000 training iterations

20 day ahead price scenarios for the intraday problem

Bidding functions with 10 segments

γ = 0.999, ρ = 0.05, 11 breakpoints for value functions

Evaluated over 100,000 state transitions Simulate intra-day prices scenarios

Execute bids based on the value function

Evaluate decision

Calculate average profit

Lower Bound: deterministic optimization based on average prices

16

Benchmark Policies

Clairvoyant: future state transitions and prices are known

Parametric policy

Optimal parameters found with CMA-ES optimizer (Hansen 2006)

17

Result: Value Function Comparison

18

Approximation Quality

Maximal relative differences between upper and lower bound

19

Iterations 3 5 11 21 52

100 22.7 13.4 1.8 0.5 0.01

1000 18.6 9.6 1.1 0.3 0.0004

10000 16.7 6.9 0.8 0.2 0.003

Convergence Speed

20

Balancing: Price Response vs Risk Measure

21

Conclusion

Model, Method & Results

We solve a complex stochastic-dynamic decision process

Training a value function with ADP methods

Using a stochastic program as immediate profit function

Applied the model to the PJM market of 2009

Diminishing returns to sophistication for value function approximation

ADP better than simple heuristics and deterministic approaches

Need risk measure or price response to obtain realistic bids on the RT market

Future Work

Multiple (connected) storage facilities

More careful estimation of price response function on the real time market

Application to other markets (EEX, Nordpool)

22

References

Hansen, N. 2006. The cma evolution strategy: A comparing review. Towards a New Evolutionary Computation. Studies in Fuzziness and Soft Computing.

Fleten, S.E., T.K. Kristoffersen (2007), Stochastic programming for optimizing bidding strategies of a Nordic hydropower producer, European Journal of Operational Research 181:916-928.

Nascimento, J.M., W.B. Powell (2009), An optimal approximate dynamic programming algorithm for the energy dispatch problem with grid-level storage, working paper, Princeton University.

Powell, W. B. 2007. Approximate dynamic programming. Solving the curses of

dimensionality. Wiley Series in Probability and Statistics

Pritchard, G., A. B. Philpott, P. J. Neame. 2005. Hydroelectric reservoir optimization in a

pool market. Mathematical Programming

Sioshansi, R., P. Denholm, T. Jenkin, J. Weiss (2009), Estimating the value of electricity storage in PJM: Arbitrage and some welfare effects, Energy Economics 31:269-277 .

23

Bidding Curve, 11h

24