Optimal Decision Making for Electric Vehicles Providing Electric Grid Frequency Regulation:

A Stochastic Dynamic Programming Approach

Jonathan Donadee Ph.D. Student, ECE, Carnegie Mellon University

jdonadee@andrew.cmu.edu

Marija Ilic IEEE Fellow, Professor of ECE and EPP, Carnegie Mellon University

milic@ece.cmu.edu 9th International Conference on Computational Management Science

Imperial College London, UK April 20th, 2012

Support for this research was provided by Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology)

through the Carnegie Mellon Portugal Program

Outline

Problem Introduction

Deterministic Equivalent Problem Model

Stochastic Dynamic Programming Algorithm

Simulation Results

Conclusions

Questions

2

Electrical Grid Frequency Regulation

Electrical Grid AC frequency must be maintained within ± 0.05 Hz

60Hz (USA)

50Hz (Europe)

Demand < Supply, f

Demand > Supply, f

Some generators follow “AGC” or “Regulation” control signal on second to minute basis

Generators bid capacity (MW) into hourly markets

Graphic Source: PJM ISO Training

Power Supply and Demand Balance

Regulation is on seconds to minutes timescale

3

One Day’s AGC Signal

0 5 10 15 20 25-1.5

-1

-0.5

0

0.5

1

Hours

No

rma

lize

d S

ign

al

1 Day of PJM AGC Signal

4

Motivation

In the future, greater need for fast responding electrical grid resources Compensate renewable resource forecast errors

Mitigate trend of decreasing system inertia smaller imbalances causing larger frequency deviations

Integration of Electric Vehicles Minimize EV charging cost

Deterministic models and methods are not well suited for managing uncertain resources

5

Electric Vehicle Participation in Frequency Regulation EVs can adjust charge rate to follow AGC signal

We’ll focus on charging only strategies, no discharging

Specify preferred charge rate, Pavg

Specify capacity for regulating, B Adjust according to negative of AGC signal scaled by B

6

0 ≤ 𝐵 ≤ 𝑃𝑎𝑣𝑔

0 ≤ 𝑃𝑎𝑣𝑔 ≤ 𝑃𝑚𝑎𝑥 0 ≤ 𝐵 ≤ 𝑃𝑚𝑎𝑥 − 𝑃𝑎𝑣𝑔

Smart Charging Scenario

Driver arrives at home and plugs in vehicle Inputs time of departure

Inputs inconvenience cost for not finishing on time ($/hr)

Smart charger optimizes Pavg, B decisions for each hour

Smart charger can participate in markets directly, without delay

Picture Source: Ford.com

7

A Stochastic Model is Needed

State of charge can take any non-decreasing path

t1 t2 t3 tf

t

e0

Emax

Average charge rate to hit Emax at tf

Bound of Possible State of Charge

Ebatt

8

A Stochastic Model is Needed

t1 t2 t3 tf

t

e0

Emax

Average charge rate to hit Emax at tf

Bound of Possible State of Charge

Ebatt

State of charge can take any non-decreasing path

Regulation contract is violated if charging finishes early

Regulation Bid Violated

9

A Stochastic Model is Needed

t1 t2 t3 tf

t

e0

Emax

Average charge rate to hit Emax at tf

Bound of Possible State of Charge

Ebatt

State of charge can take any non-decreasing path

Regulation contract is violated if charging finishes early

Driver inconvenienced if vehicle is not charged on time

Regulation Bid Violated

Driver

Inconvenienced

10

A Stochastic Model is Needed

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80

2

4

6

8

10

12

14

16

18

Integrated Hourly Energy

Co

un

ts

Histogram of July 2011 PJM AGC Signal Energy

t1 t2 t3 tf

t

e0

Emax

Average charge rate to hit Emax at tf

Bound of Possible State of Charge

Regulation Bid Violated Ebatt

Driver

Inconvenienced

11

A Stochastic Model is Needed

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80

2

4

6

8

10

12

14

16

18

Integrated Hourly Energy

Co

un

ts

Histogram of July 2011 PJM AGC Signal Energy

t1 t2 t3 tf

t

e0

Emax

Average charge rate to hit Emax at tf

Bound of Possible State of Charge

Regulation Bid Violated Ebatt

Driver

Inconvenienced

Providing regulation makes future battery state of charge uncertain

Literature ignores effect of regulation or optimizes considering expected value

Stochastic Model needed to: value risk of regulation contract violation (pro-rated by time)

value risk of inconveniencing EV driver

Optimize choice of average charge rates and regulation contracts size under uncertainty 12

Dynamic Programming Solution

Solve many hour long optimization problems in a backwards recursion

Minimize Expected Future Cost given current time and state of charge Find a single decision to minimize average cost over all future

outcomes 𝜔𝑖,ℎ ∈ Ω ℎ

𝑉ℎ 𝐸𝑖,ℎ is a Stochastic Deterministic Equivalent Problem

𝑉ℎ 𝐸𝑖,ℎ = min𝑃𝑖,ℎ ,𝐵𝑖,ℎ

𝔼𝜔 𝐽ℎ 𝐸𝑖,ℎ, 𝑃𝑖,ℎ , 𝐵𝑖,ℎ, 𝑅ℎ𝜔 + 𝑉ℎ+1 𝑒𝑡𝑓

𝜔

13

DEP and Optimal Value Function

1618

2022

24

Hour 5

Hour 6

0

0.05

0.1

0.15

0.2

0.25

Energy (kWh)

Time

Op

tima

l Va

lue

Fu

nct

ion

Co

st (

$)

Path Bounds

(from Regulation Bid)

Energy in sample ω

𝑉 6

Energy at Pavg

Solving for V5(16.8) Using 30 Sample Regulation Signals

14

Sample Path Generation

Each DEP uses 30, hour long, AGC signals, 𝑅ℎ𝜔

Sample historical data using crude monte carlo

𝜔𝑖,ℎ ∈ Ω ℎ

Integrate signal over 5 minutes

Becomes normalized energy for discrete state equations

𝑅𝐻𝜔 is a correlated 12 dimensional vector

Assume AGC independent across hours

15

State Equations

𝒆𝒕𝝎 = 𝒆𝒕−𝟏

𝝎 + ∆𝒕 ∙ 𝑷 − 𝑩 ∙ ∆𝒕 ∙ 𝑹𝒕−𝟏𝝎 − 𝒔𝒕−𝟏

𝝎 , 𝒕 ≥ 𝟐, ∀𝝎

𝑒𝑡𝜔 = 𝐸𝑖,ℎ , 𝑡 = 1, ∀𝜔

𝑒𝑡𝜔 ≥ 𝑒𝑡−1

𝜔 , ∀𝑡, ∀𝜔

𝑒𝑡𝜔 ≤ 𝐸𝑚𝑎𝑥, ∀𝑡, ∀𝜔

𝑠𝑡𝜔 ≥ 0, ∀𝑡, ∀𝜔

Energy Actually

Consumed

≠

𝑃 ∙ ∆𝑡

T0 1Hr16

17

18

19

20

21

22

23

24

En

erg

y (

kW

h)

Time

Example State Dynamics

16

Regulation Contract Risk

If the battery reaches full state of charge, cannot provide regulation

Payment is pro-rated, time-based Regulation contract violation indicator, 𝑢

Allows penalty to be a function of time, not energy 𝑢 is a binary variable

𝑢𝑡𝜔 ≥ 𝑢𝑡−1

𝜔 , 𝑡 ≠ 𝑡𝑓, ∀𝜔

𝑢𝑡𝜔 ∙ 𝑃𝑚𝑎𝑥 ∙ ∆𝑡 ≥ 𝑠𝑡

𝜔, 𝑡 ≠ 𝑡𝑓, ∀𝜔

𝑒𝑡+1𝜔 ≥ 𝑢𝑡

𝜔 ∙ 𝐸𝑚𝑎𝑥, 𝑡 ≠ 𝑡𝑓, , ∀𝜔

t

e0

Emax

Average charge rate to hit Emax at tf

Bound of Possible State of Charge

Ebatt

𝑢1 = 0 𝑢2 = 0 𝑢3 = 1

17

Driver Inconvenience Risk When not fully charged by

unplug time Charge at 𝑃𝑚𝑎𝑥 until battery is

full

𝑇𝜔 =𝐸𝑚𝑎𝑥−𝑒𝑡𝑓

𝜔

𝑃𝑚𝑎𝑥 time late on

sample path ω

𝐿 Driver’s inconvenience cost ($/hr)

𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

t1 t2 t3 tf

t

e0

Emax

Average charge rate to hit Emax at tf

Bound of Possible State of Charge

Ebatt

Driver

Inconvenienced

𝑇𝜔

18

Objective Function (if final decision)

𝜃 =1

𝑁 𝑐𝐻 −𝐵𝑖,𝐻 ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,𝐻 ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉𝐻 𝐸𝑖,𝐻 = min𝑃𝑖,𝐻 ,𝐵𝑖,𝐻 𝑐𝐻𝑃𝑖,𝐻 − 𝑟𝐻𝐵𝑖,𝐻 + 𝜃

19

Objective Function (if final decision)

𝜃 =1

𝑁 𝑐𝐻 −𝐵𝑖,𝐻 ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,𝐻 ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉𝐻 𝐸𝑖,𝐻 = min𝑃𝑖,𝐻 ,𝐵𝑖,𝐻 𝑐𝐻𝑃𝑖,𝐻 − 𝑟𝐻𝐵𝑖,𝐻 + 𝜃

Baseline

Energy Cost

20

Objective Function (if final decision)

𝜃 =1

𝑁 𝑐𝐻 −𝐵𝑖,𝐻 ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,𝐻 ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉𝐻 𝐸𝑖,𝐻 = min𝑃𝑖,𝐻 ,𝐵𝑖,𝐻 𝑐𝐻𝑃𝑖,𝐻 − 𝑟𝐻𝐵𝑖,𝐻 + 𝜃

Baseline

Energy Cost

Regulation Contract

Revenue

21

Objective Function (if final decision)

𝜃 =1

𝑁 𝑐𝐻 −𝐵𝑖,𝐻 ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,𝐻 ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉𝐻 𝐸𝑖,𝐻 = min𝑃𝑖,𝐻 ,𝐵𝑖,𝐻 𝑐𝐻𝑃𝑖,𝐻 − 𝑟𝐻𝐵𝑖,𝐻 + 𝜃

Energy Cost

Adjustement

Baseline

Energy Cost

Regulation Contract

Revenue

22

Objective Function (if final decision)

𝜃 =1

𝑁 𝑐𝐻 −𝐵𝑖,𝐻 ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,𝐻 ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉𝐻 𝐸𝑖,𝐻 = min𝑃𝑖,𝐻 ,𝐵𝑖,𝐻 𝑐𝐻𝑃𝑖,𝐻 − 𝑟𝐻𝐵𝑖,𝐻 + 𝜃

Energy Cost

Adjustement

Contract

Violation Cost

Baseline

Energy Cost

Regulation Contract

Revenue

23

Objective Function (if final decision)

𝜃 =1

𝑁 𝑐𝐻 −𝐵𝑖,𝐻 ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,𝐻 ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉𝐻 𝐸𝑖,𝐻 = min𝑃𝑖,𝐻 ,𝐵𝑖,𝐻 𝑐𝐻𝑃𝑖,𝐻 − 𝑟𝐻𝐵𝑖,𝐻 + 𝜃

Energy Cost

Adjustement

Driver

Inconvenience Cost

Contract

Violation Cost

Baseline

Energy Cost

Regulation Contract

Revenue

24

Objective Function (if final decision)

𝜃 =1

𝑁 𝑐𝐻 −𝐵𝑖,𝐻 ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,𝐻 ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐𝐻+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉ℎ 𝐸𝑖,𝐻 = min𝑃𝑖,𝐻 ,𝐵𝑖,𝐻 𝑐𝐻𝑃𝑖,𝐻 − 𝑟𝐻𝐵𝑖,𝐻 + 𝜃

Energy Cost

Adjustement

Driver

Inconvenience Cost

Contract

Violation Cost

Baseline

Energy Cost

Regulation Contract

Revenue

Future Energy

Purchases

25

Objective Function (not final decision)

𝜃 =1

𝑁 𝑐ℎ −𝐵𝑖,ℎ ∙ 𝑅𝑡

𝜔 ∙ ∆𝑡 − 𝑠𝑡𝜔

𝑡𝑓−1

𝑡=1

+ 𝑄 ∙ ∆𝑡 ∙ 𝐵𝑖,ℎ ∙ 𝑢𝑡𝜔

𝑡

+ 𝐿 ∙ 𝑇𝜔 + 𝑐ℎ+1(𝐸𝑚𝑎𝑥 − 𝑒𝑡𝑓𝜔)

𝜔∈Ω𝑁

𝑉ℎ 𝐸𝑖,ℎ = min𝑃𝑖,ℎ ,𝐵𝑖,ℎ 𝑐ℎ𝑃𝑖,ℎ − 𝑟ℎ𝐵𝑖,ℎ + 𝜃

Energy Cost

Adjustement

Contract

Violation Cost

Baseline

Energy Cost

Regulation Contract

Revenue

+ 𝑉ℎ+1 𝑒𝑡𝑓𝜔

Future Optimal Value Function

26

Exact Linearization of Contract Violation Cost 𝑄 ∙ ∆𝑡 ∙ 𝐵 ∙ 𝑢𝑡

𝜔𝑡 is nonlinear

Replace with 𝑄 ∙ ∆𝑡 ∙ 𝑥𝑡

𝜔𝑡

Add constraints When 𝑢 = 0, 𝑥=0 When 𝑢 = 1, 𝑥= 𝐵

𝑥𝑡𝜔 ≤ 𝐵

𝑥𝑡𝜔 ≤

𝑃𝑚𝑎𝑥

2 𝑢𝑡

𝜔

𝑥𝑡𝜔 ≥ 𝐵 −

𝑃𝑚𝑎𝑥

2 1 − 𝑢𝑡

𝜔

27

Stochastic Dynamic Programming

Ebatt

E5

Time (Hours)

Unplug Time

E1

Emax

𝑉5 𝐸5

h1 h2 h3 h4 h5

28

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

h1 h2 h3 h4 h5

E1

E5

𝑉5 𝐸5

29

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

h1 h2 h3 h4 h5

E1

E5

𝑉5 𝐸5

30

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

h1 h2 h3 h4 h5

E1

E5

𝑉5 𝐸5

31

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

h1 h2 h3 h4 h5

E1

E5

𝑉5 𝐸5

32

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

h1 h2 h3 h4 h5

E1

E5

𝑉5 𝐸5

33

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

h1 h2 h3 h4 h5

E1

E5

𝑉5 𝐸5

34

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

h1 h2 h3 h4 h5

E1

E5

𝑉5 𝐸5

35

Find state, cost points on the convex hull of all points Andrews Monotone Chain Algorithm

Basically compares slopes

Future Cost - 𝑉𝟓 𝑒𝑡𝑓𝜔

𝑉5 𝐸5

E5 36

Find state, cost points on the convex hull of all points Andrews Monotone Chain Algorithm

Basically compares slopes

Future Cost - 𝑉𝟓 𝑒𝑡𝑓𝜔

On the hull

Not on the hull 𝑉5 𝐸5

E5 37

Future Cost - 𝑉𝟓 𝑒𝑡𝑓𝜔

Create inequalities from points on the convex hull

Add new inequality constraints to DEP 𝑉4 𝐸𝑖,4

𝑉5 𝐸5

Cut j

𝑉ℎ+1 𝑒𝑡𝑓𝜔 ≥ 𝐼𝑛𝑡𝑒𝑟𝑐𝑒𝑝𝑡𝑗 − 𝑆𝑙𝑜𝑝𝑒𝑗 ∗ 𝑒𝑡𝑓

𝜔 , ∀𝑗

E5 38

Stochastic Dynamic Programming

1618

2022

24

Hour 5

Hour 6

0

0.05

0.1

0.15

0.2

0.25

Energy (kWh)

Time

Op

tim

al V

alu

e F

un

ctio

n C

ost ($

)

4

5

39

𝑉ℎ+1 𝑒𝑡𝑓𝜔

𝑒𝑡𝑓1

Stochastic Dynamic Programming

Repeat backwards recursion until the current state is reached

Ebatt

Time (Hours)

Unplug Time

Emax

h1 h2 h3 h4 h5

E1

40

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

Repeat backwards recursion until the current state is reached

h1 h2 h3 h4 h5

E1

41

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

Repeat backwards recursion until the current state is reached

h1 h2 h3 h4 h5

E1

42

Stochastic Dynamic Programming

Emax

Ebatt

Time (Hours)

Unplug Time

Repeat backwards recursion until the current state is reached

h1 h2 h3 h4 h5

E1

43

Implementation

Calculate Optimal Value Functions, Vh

At initial state, time Solve one DEP for P1, B1

Implement decision and wait 1hr, see what happens

Given new state, Optimize decision,

implement, wait

At unplug time If not full, charge at Pmax

Else, Done!

1 2 3 4 5 6 7 8

12

14

16

18

20

22

24

26

Ba

tte

ry S

tate

of C

ha

rge

(kW

h)

Simulation Timestep (hr)

Simulation Results

Energy Bounds

(from Regulation Bid)

Actual Energy

State

Energy at Pavg

44

Forward Simulation for Comparison

Simulate 150 different, 7 hour long realizations of AGC Signal

Each trial uses the same Optimal Value Functions

Initial state

Deterministic prices

Set of samples in DEPs

Compare with an expected value formulation 1 sample, using expected value of AGC signal

45

Results- 150 forward Simulations

0 0.5 1 1.5 2 2.5 3 3.5 4 4.50

20

40

60

80

100

120Histogram of Expected Value Formulation Costs

Cost($)

Cou

nts

0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.30

5

10

15

20

25

30

35

Cost($)

Counts

Histogram of Stochastic Formulation Costs

$20/hr Inconvenience cost

Stochastic

Model

Expected Value

Model

μ $ 0.23 $ 0.44

Σ2 5.0 E-4 0.35

Late trials 0% 29%

$200/hr Inconvenience cost

Stochastic

Model

Expected Value

Model

Μ $ 0.23 $ 2.29

Σ2 5.0 E-4 35.58

Late trials 0% 29%

46

Observations

Expected Value formulation often inconveniences driver, while Stochastic formulation is robust

For final decision, P,B are chosen such that driver is not inconvenienced on any sample path

Cost of uncharged energy ÷ 30 > All hourly Energy Prices

Vast majority of DEP solutions are on the CH

good approximation of 𝑉ℎ

If Charging, regulation contract size, B is on upper

bound , but decisions are dependent

47

Future Work

Method Improvement/Evaluation Bias Estimation and Correction Number of AGC Samples Number of Discretizations Parallelize

Model Expansion Investigate AGC signal properties Uncertain Prices- ARIMA or GARCH Form CH in 4 dimensions with QuickHull

Fleet Aggregation Apply method to other technologies (flywheels) Integrate into broader Smart Distribution Network model

48

Conclusions

Stochastic models are necessary for demand side frequency regulation

We have accurately modeled risks of providing frequency regulation

Our method is tractable and parallelizable

49

Thank you!

1 2 3 4 5 6 7 8

12

14

16

18

20

22

24

26

Ba

tte

ry S

tate

of C

ha

rge

(kW

h)

Simulation Timestep (hr)

Simulation Results

0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.30

5

10

15

20

25

30

35

Cost($)

Counts

Histogram of Stochastic Formulation Costs

50

Support for this research was provided by Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology)

through the Carnegie Mellon Portugal Program