domestic water heaters power optimization using fuzzified rl
TRANSCRIPT
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
1/24
Domestic Water HeatersPower Optimization Using
Fuzzified RLBy: Khalid Al-jabery
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
2/24
Outlines
Problem Description Previous Research
Dynamic Programing (DP) and Adaptive DP The Proposed Approach Algorithm Results Discussion
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
3/24
Problem Description (1)
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
4/24
Definition How to control the operation of water heaters in
order to minimize peak grid load demand andmaintain worm water with temperature aboveor equal to threshold delivered to the client?
Therefore; There are 3 variables defining thesystem:
1. The temperature of the Water supplied.
2. The current Grid demand3. The instantaneous rate of hot waterconsumption. (User Demand)
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
5/24
Some Previous Research Demand Side Management, by M.H. Nehrir et. al. 1998.
Applied on a block of dwellings. Need connections among users. Try to reduce the power consumed regardless of the user satisfaction
Control Strategy for Domestic WHs during Peak Periods, by AlanMoreau , 2011.
Based on timing to control the operation of the WHs. Highly depends on the size of the WH tank. Try to reduce the power consumed regardless of the user satisfaction
Demand Side Management using BPSO, by Sepulveda et. al. 2010. Required communications and synchronization among dwellings.
According to the simulation results it doesnt show any improvementin power optimization.
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
6/24
Dynamic Programing (DP) and Adaptive DP
DP ADP
Adaptive DynamicProgramming ,
, , [ , , + ]| |=
Dynamic Programming , 1 , +
[ , , + ( , )
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
7/24
The Proposed Approach
1. Define the system states by the linguisticdescription of the control variables. This wasachieved by using Fuzzy Logic.
2. Derive Markov chain process based on the states
and the available actions.3. Using Q-learning with discounted reward to trainthe system in order to reach the near optimalpolicy.
4. The Training algorithm is designed to providebalance between power optimization andcustomer satisfaction.
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
8/24
Algorithm1. Initialize Q-factors (Ns,Na)=0;2. Initialize state (S) and action (a);3. Repeat (4 to 9) until Stopping condition; 4. Read System Variables ;5. Select new Action =a ;6. Calculate Fuzzy membership ;7. Determine next State =s ;8. Calculate immediate reward ;9. Update Q(s,a) according to Q-learning algorithm;
10. Find Suboptimal policy;
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
9/24
Sample code O/P
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
10/24
Result-1Itrs. S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18
10 1 2 2 1 1 2 1 2 2 1 2 2 1 2 2 1 2 2
10 2 2 2 1 2 2 1 2 1 2 2 2 2 2 2 2 2 2
30 2 2 2 1 2 2 1 1 2 2 2 2 2 2 2 1 2 2
30 1 2 2 1 2 1 1 1 2 2 2 2 2 2 2 1 2 2
100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 2 2 2
100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 1 2 2
States where the grid load is Low States where the grid load is High
Th L M H L M H L M H L M H L M H L M H
W L L L M M M H H H L L L M M M H H H
GL L L L L L L L L L H H H H H H H H H
Actions =0.85
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
11/24
Results-2Itrs. S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18
10 1 2 2 1 2 2 1 2 2 2 2 2 2 2 2 1 2 2
10 2 2 2 1 1 2 1 2 2 2 2 2 1 2 2 1 2 2
30 2 2 2 1 1 2 1 1 2 1 2 2 2 2 2 1 2 2
30 2 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2
100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2
100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2
States where the grid load is Low States where the grid load is High
Th L M H L M H L M H L M H L M H L M H
W L L L M M M H H H L L L M M M H H H
GL L L L L L L L L L H H H H H H H H H
Actions =0.9
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
12/24
The Effect of ( )
Itrs.S1
S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18
10 1 2 2 1 1 2 1 2 2 1 2 2 1 2 2 1 2 2 0.85
10 2 2 2 1 1 2 1 2 2 2 2 2 1 2 2 1 2 2 0.9
10 2 2 2 1 2 2 1 2 1 2 2 2 2 2 2 2 2 2 0.85
10 2 2 2 1 1 2 1 2 2 2 2 2 1 2 2 1 2 20.9
30 2 2 2 1 2 2 1 1 2 2 2 2 2 2 2 1 2 2 0.85
30 2 2 2 1 1 2 1 1 2 1 2 2 2 2 2 1 2 2 0.9
30 1 2 2 1 2 1 1 1 2 2 2 2 2 2 2 1 2 2 0.85
30 2 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2 0.9
100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 2 2 2 0.85
100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2 0.9
100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 1 2 2 0.85
100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2 0.9
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
13/24
System Output Graph
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
14/24
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
15/24
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
16/24
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
17/24
System Variables (2)
Hot water demand Energy Distribution
Water Temperature
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
18/24
Fuzzy Memberships
0
0.20.4
0.6
0.8
1
1.2
90 95 100 105 110 115 120 125 130 135 140 145 150 155 160
Temperature in F
Water Temperature
LOW
Medium
High
0
0.2
0.4
0.6
0.8
1
1.2
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
Samples Every 30 Mins.
Grid Load
LOW
High
0
0.2
0.4
0.6
0.8
1
1.2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Gallons /sample
Water Consumption Rate
LOW
Medium
High
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
19/24
States RepresentationS1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18
L M H L M H L M H L M H L M H L M H
L L L M M M H H H L L L M M M H H H
L L L L L L L L L H H H H H H H H H
L= Low, M=Medium , H= High: e.g: S9 is the state when the water temperature high,the user hot water demand is high and the Grid load is low.
Water Consumption
Temperature
Grid Demand
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
20/24
Markov Chain for Action1
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
S16
S17
S18
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
21/24
Transition Rewards (off action)
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
22/24
Updating Q The update is according to the eqn.: , 1 , + [ , , +
max , ] Where:S: system old state,a : action selected at S,
: learning rate, here = log(k)/K : is the discount factor,
Q : is the q-factor that associated with each(s,a) pair.
R (s,a,s ) : is the reward for moving from state s to state s using action a.fm : is the fuzzy membership value of the state preferred property *
b: is the action selected at state S and has the max Q-factor.
* The proff ered proper ty means the property that defi ne whether the system is going to be rewar ded orpunished wil l be explain ed in TRM s Der ivation.
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
23/24
References1. Control Strategy for Domestic Water Heaters during Peak Periods and its
Impact on the Demand for Electricity By A. Moreau, Canada. 2011. 2. Measurement of Domestic Hot Water Consumption in Dwellings, DEFRA
Department for Environment Food and Rural affairs UK.
3. Handbook of Intelligent Control By D. Sofga.
4. Simulation Based Optimization A. Gosavi.
5. Dynamic Programming and Optimal Control, D. Bertskas.
6. A Novel Demand Side Management Program using Water Heaters andParticle Swarm Optimization, A. Sepulveda, 2010.
7. A Reinforcement Learning-Based Architecture for Fuzzy Logic Control,
Hamid R. Berenji.8. A customer-interactive electric water heater demand-side management
strategy using fuzzy logic., M.H. Nehrir, 1998.
-
8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL
24/24