baseline analysis cbp, amp, and dbp steve braithwait, dan hansen, and dave armstrong christensen...
TRANSCRIPT
![Page 1: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/1.jpg)
Baseline AnalysisCBP, AMP, and DBP
Steve Braithwait, Dan Hansen, and Dave Armstrong
Christensen Associates Energy Consulting
DRMEC Spring Workshop
May 7, 2014
May 2014 1
![Page 2: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/2.jpg)
May 2014 2
Presentation Outline
Objectives Methodology
Data Performance measures
Aggregator program (CBP and AMP) results Demand Bidding Program (DBP) results
![Page 3: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/3.jpg)
May 2014 3
Objective: Assess Performance of Alternative Baseline Types
For each Utility and Notice type: All customers, with BL adjustment as chosen All customers, simulated with universal selection
of the BL adjustment Sum of individual BL vs. portfolio BL (constructed
from aggregated customer loads), for AMP and CBP only
Examine unadjusted and day-of adjustments with 20%, 30%, 40%, 50% caps, and uncapped
![Page 4: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/4.jpg)
May 2014 4
Analysis Details
For actual program event days The “true” baseline is the estimated reference
load from the ex post evaluation For event-like non-event days
The “true” baseline is the observed load
![Page 5: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/5.jpg)
May 2014 5
Performance Measures (1)Percentage Baseline Error
Percentage BL error for each customer/portfolio-event day is: Percentage error = (LP
d – LAd) / LA
d
LAd = actual, or “true” baseline load on day d
LPd = “predicted” baseline to be evaluated
Positive value = over-estimated baseline (implies over-stated program load impact)
Negative value = under-estimated baseline (implies under-stated program load impact)
![Page 6: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/6.jpg)
May 2014 6
Performance Measures (2) Accuracy
Accuracy is measured as the median absolute percentage error (MAPE) Calculate the absolute value of the percentage error for
each customer/event-day Calculate the median of values across customer/event-
days (mean can be misleading due to extreme values) Higher values correspond to larger baseline errors
![Page 7: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/7.jpg)
May 2014 7
Performance Measures (3)Bias
Bias is measured by the median percentage error, without taking the absolute value
Positive values indicate upward bias (i.e., the program baseline tends to over-state the “true” baseline)
Negative values indicate downward bias (i.e., the program baseline tends to under-state the “true” baseline)
![Page 8: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/8.jpg)
Nominated Customers by Choice of BL Adjustment – CBP and AMP
May 2014 8
![Page 9: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/9.jpg)
May 2014 9
Accuracy (Median Abs. % Error)PG&E CBP-DO
![Page 10: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/10.jpg)
May 2014 10
Bias (Median % Error)PG&E CBP-DO
![Page 11: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/11.jpg)
May 2014 11
Percentiles of % Errors – PG&E CBP-DOActual Events, by Adjustment Cap
![Page 12: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/12.jpg)
May 2014 12
Percentiles of % Errors – PG&E CBP-DOSimulated Events, by Adjustment Cap
![Page 13: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/13.jpg)
Summary: Accuracy & Bias (Aggregated Indiv.; Universal Adj.; 40% cap)
May 2014 13
![Page 14: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/14.jpg)
Summary: Percentiles of % Errors(Aggregated Indiv.; Universal Adj.; 40% cap)
May 2014 14
![Page 15: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/15.jpg)
May 2014 15
Summary of Findings Accuracy and bias measures vary by utility, program
and notice type Suggests that factors other than baseline type and
adjustment caps may be most important, such as types of customers (e.g., highly variable load) and event-day characteristics (e.g., event on isolated hot day)
Day-of adjustment often improves accuracy and reduces bias, but level of cap is less important Largest errors typically occur for Unadjusted BL and
Unlimited cap BL with small median error (e.g., 1%) can have >10%
errors in 20 percent of cases
![Page 16: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/16.jpg)
16May 2014
DBP Results:PG&E Distribution of % Errors
-100.0%
-90.0%
-80.0%
-70.0%
-60.0%
-50.0%
-40.0%
-30.0%
-20.0%
-10.0%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
Unadj. 20% 30% 40% 50% No Cap
% B
as
eli
ne
Err
or
Baseline Method
5th %ile
10th %ile
25th %ile
Median
75th %ile
90th %ile
95th %ile
![Page 17: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/17.jpg)
17May 2014
DBP Results:SCE Distribution of % Errors
-40.0%
-30.0%
-20.0%
-10.0%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
Unadj. 20% 30% 40% 50% No Cap
% B
ase
line
Err
or
Baseline Method
5th %ile
10th %ile
25th %ile
Median
75th %ile
90th %ile
95th %ile
![Page 18: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/18.jpg)
Summary
Day-of adjustments tend to improve baseline accuracy and reduce bias
The analysis provides support for making the day-of adjustment the default option
The effectiveness of the day-of adjustment is not very sensitive to the level of the cap
May 2014 18
![Page 19: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/19.jpg)
May 2014 19
Questions?
Contact – Steve Braithwait or Dan Hansen, Christensen Associates Energy ConsultingMadison, Wisconsin [email protected] [email protected] 608-231-2266
![Page 20: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/20.jpg)
Appendix
SCE – CBP DO SDG&E – CBP DO PG&E – AMP DO SCE – AMP DO
May 2014 20
![Page 21: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/21.jpg)
May 2014 21
Accuracy (Median Abs. % Error)SCE CBP-DO
![Page 22: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/22.jpg)
May 2014 22
Bias (Median % Error)SCE CBP-DO
![Page 23: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/23.jpg)
May 2014 23
Percentiles of % Errors – SCE CBP-DOActual Events, by Adjustment Cap
![Page 24: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/24.jpg)
May 2014 24
Percentiles of % Errors – SCE CBP-DOSimulated Events, by Adjustment Cap
![Page 25: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/25.jpg)
May 2014 25
Accuracy (Median Abs. % Error)SDG&E CBP-DO
![Page 26: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/26.jpg)
Accuracy – Med. Abs. Err. (MW)SDG&E CBP DO
May 2014 26
![Page 27: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/27.jpg)
May 2014 27
Bias (Median % Error)SDG&E CBP-DO
![Page 28: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/28.jpg)
May 2014 28
Accuracy (Median Abs. % Error)PG&E AMP-DO
![Page 29: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/29.jpg)
May 2014 29
Bias (Median % Error)PG&E AMP-DO
![Page 30: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/30.jpg)
May 2014 30
Accuracy (Median Abs. % Error)SCE AMP-DO
![Page 31: Baseline Analysis CBP, AMP, and DBP Steve Braithwait, Dan Hansen, and Dave Armstrong Christensen Associates Energy Consulting DRMEC Spring Workshop May](https://reader036.vdocument.in/reader036/viewer/2022070408/56649e665503460f94b61053/html5/thumbnails/31.jpg)
May 2014 31
Bias (Median % Error)SCE AMP-DO