nested dynamic programming algorithm

1

Nested Dynamic Programming (nDP) algorithm

for multipurpose reservoir optimizationBlagoj Delipetrev

Andreja JonoskiDimitri P. Solomatine

HIC, NYC August 17 – 21, 2014

Overview nested Dynamic programming algorithm

The nDP algorithm is built from two algorithms: 1) dynamic programming (DP) and 2) nested optimization algorithm implemented with Simplex and quadratic Knapsack.

The novel idea is to include a nested optimization algorithm into the DP transition that lowers the starting problem dimension and alleviates the DP curse of dimensionality.

The nDP can solve multi-objective optimization problems, without significantly increasing the algorithm complexity and the computational expenses.

Computationally, the nDP is very efficient and it can handle dense and irregular variable discretization

It is coded in Java as a prototype application and has been successfully tested with eight objectives at the Knezevo reservoir, located in the Republic of Macedonia.

2

3

Introduction

Optimal reservoir operation methods include:Dynamic programming (DP)Stochastic dynamic programming (SDP)


“curse of dimensionality” “curse of modelling”

Successive approximations, incremental dynamic programming and differential dynamic programming

the computational complexity with the state – decision space dimension

Dynamic programming

4

ttttt erqss 1 Reservoir model

111,,min ttttttt sVassgsVBellman equation

(for stages t=T-1,T-2,…1)

20

15

10

5

20

15

10

5

20

15

10

5

20

15

10

5

20

15

10

5

20

15

10

5

Inflow

Release

20

15

10

5

20

15

10

5


5

Research question


How to enhance DP algorithm and develop new algorithm (methodology) that is flexible with including additional objectives like cities water demand, agriculture water demand, ecology water demand, hydro power production and etc., alleviate as much as possible the curse of dimensionality and computational cost?

Hydropower

production

Minimum and

maximum

reservoir targets

6

Main ideanested Dynamic programming (nDP)


One step of the DP algorithm. One step of the nDP algorithm

rt

St

qt

g(st,st+1,at)+minV(St+1)

St+1

rt

Nested optimal allocation algorithm

minV(St+1)

r1t

U1t

r2t rnt…

U2t

Unt

rt

st

qt

g(st,st+1,at)+minV(st+1)

st+1

7

Nested optimal allocation algorithmsSimplex – Quadratic Knapsack


n

iititit rdw

1

min

tnttt rrrr ...21

ntnttttt drdrdr ..,, 2211

0..,,, 21 ntttt dddr

n

iititit rdw

1

2min

Simplex Quadratic Knapsack

nDP demonstration Zletovica river basin

8

Zletovica river basin is in the eastern part of the Republic of Macedonia.


9

Objective function


28

11,, it

iittttt Dwassg

The optimization problem has eight objectives and six decision variables.

weighted sum of squared deviations over the entire time horizon

,

,0

t1t1

1t1 hdifdh

hdifD

tt

tt

tttt

tt2 ,

,0FlifFlFlif

D t

where d1t and d2t are the minimum and maximum level targets and ht is the reservoir level height at time step t

Objective function

Water demand users1) the towns of Zletovo and Probishtip (one intake), d3t, 2) the upper agricultural zone, d4t, 3) the towns of Shtip and Sveti Nikole (one intake), d5t, 4) the lower agricultural zone, d6t, 5) the minimum environmental flow, d7t.

10

itititit

ititit rdifrd

rdifD

, ,0

where ht is the hydropower demand and pt is the hydropower production

tttt

tt pdifpd

pifD

88

8t8 ,

d ,0


11

Results


Experiments w1 w2 w3 w4 w5 w6 w7 w8nDP-L1 and nDP-Q1

25000 25000 0.2 0.1 0.2 0.1 0.26 0.04

nDP-L2 and nDP-Q2

25000 25000 0.15 0.1 0.25 0.1 0.26 0.04

• 55-year monthly data (1951-2005), with 660 time-steps.• The reservoir operation volume 23 million m3 is discretized in 73 equal

levels (300 103 m3 each). • The minimum reservoir level target was set at 1021.5 [amsl],• and the maximum reservoir level target at 1060 [amsl].• The water supply, irrigation, and hydropower are set to their monthly

demands

nDP-L1 optimal reservoir level in 55 years

12

Resuts


13

Results


Experiments w1 w2 w3 w4 w5 w6 w7 w8nDP-L1 and nDP-Q1

25000 25000 0.2 0.1 0.2 0.1 0.26 0.04

nDP-L2 and nDP-Q2

25000 25000 0.15 0.1 0.25 0.1 0.26 0.04

Comparison of nDP with other DP algorithms

14

Classical DP model of reservoir operation

aggregated water demand (AWD) DP algorithmttttt wwwwww

t 76543'

3

tttttt dddddd 76543'3

7

3

22'3

'3

'

3

iititittt rdwRdw

t


Conclusions

The nDP algorithm has the following advantages: It effectively alleviates the curse of dimensionality in optimal

reservoir operation. It has better optimization capabilities compared to the DP

aggregated water demand approach and can solve problems that are more complex where the DP aggregated water demand approach is not feasible.

Computationally, it is very efficient and runs fast on standard personal computers. The presented case study optimization was executed in less than five minutes.

The algorithm allows for employing dense and variable discretization on the reservoir volume and release.

It supports using a variable weight at each time step for every objective function.

15


Conclusion It provides a framework for including many objectives in the nested

optimization algorithm without a significant change in the source code or an increase in the computational expenses.

Different optimization algorithms can be used in the nesting for water allocation, however, since nested optimization has to be repeated multiple times (for each transition of DP) the algorithm used for this purpose needs to be fast.

The method presented can be applied to stochastic dynamic programming, reinforcement learning and other similar algorithms.

16

The nDP main idea is to include nested optimization algorithm inside the DP transition, which lowers the problem dimension. With this method, it is possible to solve optimization problems that are currently unsolvable with classical methods, rapidly decrease the optimization time and improve the result that was demonstrated in this paper.


17

Email [email protected]

[email protected], NYC August 17 – 21, 2014

rt

St

qt

g(st,st+1,at)+minV(St+1)

St+1

rt

Nested optimal allocation algorithm

minV(St+1)

r1t

U1t

r2t rnt…

U2t

Unt

https://github.com/deblagoj/DP-3Objectives

mailto:[email protected]

mailto:[email protected]

nested dynamic programming algorithm

Data & Analytics