desc model smoothing evaluation 09 101109€¦ · 08/09-07/08 (rms=7.1, 7.2 excl. 09b) smoothed...
TRANSCRIPT
Model Smoothing - Evaluation
Supporting Document: DESC_Model Smoothing Review09_101109.pdf
DESC 10th November 2009
Model Smoothing Model Smoothing -- EvaluationEvaluation
Supporting Document: Supporting Document: DESC_ModelDESC_Model Smoothing Review09_101109.pdfSmoothing Review09_101109.pdf
DESC 10DESC 10thth November 2009November 2009
Model Smoothing
� Model smoothing: Averaging of 3 years models (including the current /
most recent data sets) to derive new parameters
� First undertaken 1999/00 and applied to all subsequent years based on similar methodology
� This current analysis is the first full assessment of model smoothing results since Autumn 2007 and has been carried out along the same lines
� Inform decisions on approach and application of model smoothing for
Spring 2010
� Supporting document gives full commentary and detailed analysis
Principles of Model Smoothing
� Introduced to address year on year volatility in EUC models
� Averaging 3 years models
� Greater stability and removal of year on year volatility rather than improving model predictability (‘accuracy’)
� Two objectives of stability and accuracy are not necessarily consistent
� Underlying change in customer behaviour
� May lead to changes in model characteristics
� BUT: May then be stabilised by model smoothing
� BUT: Underlying change or trend of single year? (Modelling annual trends)
� Consider: predictive, volatility and trend analysis - single to smoothed year comparisons to assess appropriateness
Principles of Model Smoothing Evaluation
� Analysis compares actual 2008 / 09 consumption data with:
� 2007/08 Models – Single Year model that would have been applied
� Smoothed models for 2005/06, 2006/07 and 2007/08 – Smoothed Models that
were applied
� Using (Consumption Band & WAR Bands)
� CWV Intercepts – Point the line crosses the x-axis (weather sensitivity): predictive ability of single or smoothed compared to actual
� Root Mean Squared (RMS) - Variance of smoothed or single year from actual
� Year on Year Volatility - Change in CWV intercept values each year in single year
and smoothed models
� Trend Analysis - Identification of trends in single year CWV intercepts and change
in load factor values
Analysis 1: Predictive Analysis - Purpose
� Compares CWV intercept values for:
� Most recent actual data set 2008/09 gas year against
� Single year model from 2007/08 that would have applied to 2008/09
� Smoothed model (05/06, 06/07, 07/08) that was applied
� Compare variance of actual intercept to single year model and smoothed year
model
� Includes RMS value for smoothed & single model
� Highlights the variance of the model from the actual
Predictive Analysis: Consumption Bands – Small NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models
FIGURE 1: SMALL NDM (<2,196,000) CONSUMPTION BAND EUCs PREDICTIVE ABILITY:
Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept
0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
<-1.75 -1.75 to -1.25 -1.25 to -0.75 -0.75 to -0.25 -0.25 to 0.25 0.25 to 0.75 0.75 to 1.25 1.25 to 1.75 >1.75
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=0.6) 08/09-SMOOTHED (05/06-07/08) (RMS=0.5)
Smoothed Model
Similar CWV
Intercept Differences
Lower RMS Values
Similar Predictive
Ability
Predictive Analysis: Consumption Bands – Large NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models
FIGURE 2: LARGE NDM (>2,196,000)CONSUMPTION BAND EUCs - PREDICTIVE ABILITY:
Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept
0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
< -7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=6.5, 6.6 excl. 09B) 08/09-SMOOTHED (05/06-07/08) (RMS=5.3, 5.9 excl. 09B)
Smoothed Model
Better ‘spread’ of
CWV Intercept
Differences
Lower RMS Values
Better Predictive
Ability
Predictive Analysis: All EUC Bands – Small NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models
FIGURE 3: SMALL NDM EUCs PREDICTIVE ABILITY:
Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept
0
10
20
30
40
50
60
70
80
90
100
110
120
<-7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=2.7) 08/09-SMOOTHED (05/06-07/08) (RMS=2.1)
Smoothed Model
Similar CWV
Intercept Differences
Lower RMS Values
Better Predictive
Ability
Predictive Analysis: All EUC Bands – Large NDMCWV Intercepts: Predictive Accuracy - Smoothed and Single Year Models
FIGURE 4: LARGE NDM EUCs PREDICTIVE ABILITY:
Actual Consumption Model Intercept - Single or Smoothed Year Model Intercept
0
20
40
60
80
100
120
140
160
180
200
< -14 -14 to -10 -10 to -6 -6 to -2 -2 to 2 2 to 6 6 to 10 10 to 14 >14
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=7.1, 7.2 excl. 09B) 08/09-SMOOTHED (05/06-07/08) (RMS=6.1, 6.3 excl. 09B)
Smoothed Model
Similar CWV
Intercept Differences
Lower RMS Values
Better Predictive
Ability
Analysis 1: Predictive Analysis - Conclusions
� Comparing single & smoothed year model CWV intercepts to actual
consumption models:
� Consumption band analysis highlights smoothed model is slightly better in
predictive ability overall to the single year model
� Consumption + WAR band analysis (all EUCs) highlights smoothed model
has an improved spread of CWV intercepts compared to the single year
model and RMS values are lower for smoothed model
� Overall, comparisons are not overwhelming evidence as to the superior predictive capability of smoothed models. However there is a small
improvement
� Main driver for using a smoothed model is the mitigation of year on year
volatility rather than predictive capability.
� Further analysis required to confirm model smoothing appropriateness…..
Analysis 2: Year on Year Volatility Analysis - Purpose
� Compares year on year volatility reduction of each model type (smoothed
and single year)
� AIM: To reduce differences in between each year
� Compare 09/10 applied smoothed model (06/07, 07/08, 08/09)
� TO
� Applied smoothed model for 08/09 (05/06, 06/07, 07/08)
� Compare 08/09 single year model (that would have been applied to 09/10)
� TO
� Single year model for 07/08 (that would have been applied to 08/09)
� Using variations in CWV intercept and RMS values to identify level of
volatility between model types and years
FIGURE 7: SMALL NDM CONSUMPTION BAND EUCs YEAR ON YEAR VOLATILITY:
08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model
0
5
10
15
20
25
30
35
40
<-1.75 -1.75 to -1.25 -1.25 to -0.75 -0.75 to -0.25 -0.25 to 0.25 0.25 to 0.75 0.75 to 1.25 1.25 to 1.75 >1.75
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=0.6) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=0.2)
Volatility Analysis: Consumption Bands – Small NDM08/09 - 07/08 Single Year Model : 09/10 - 08/09 Smoothed Model
Smoothed Model
Smaller CWV
Intercept Differences
Lower RMS Values
Less Volatility
FIGURE 8: LARGE NDM CONSUMPTION BAND EUCs YEAR ON YEAR VOLATILITY:
08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model
0
5
10
15
20
25
30
35
40
< -7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=6.5, 6.6 excl. 09B) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=1.6, 1.7 excl. 09B)
Volatility Analysis: Consumption Bands – Large NDM08/09 - 07/08 Single Year Model : 09/10 - 08/09 Smoothed Model
Smoothed Model
Smaller CWV
Intercept Differences
Lower RMS Values
Less Volatility
Volatility Analysis: All EUC Bands – Small NDM08/09 - 07/08 Single Year Model : 09/10 - 07/08 Smoothed Model
FIGURE 5: SMALL NDM EUCs YEAR ON YEAR VOLATILITY:
08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model
0
20
40
60
80
100
120
140
160
<-7 -7 to -5 -5 to -3 -3 to -1 -1 to 1 1 to 3 3 to 5 5 to 7 >7
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=2.7) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=0.6)
Smoothed Model
Smaller CWV
Intercept Differences
Lower RMS Values
Less Volatility
Volatility Analysis: All EUC Bands – Large NDM08/09 - 07/08 Single Year Model : 09/10 - 07/08 Smoothed Model
FIGURE 6: LARGE NDM EUCs YEAR ON YEAR VOLATILITY:
08/09 - 07/08 Single Year Model COMPARED TO 09/10 - 08/09 Smoothed Model
0
25
50
75
100
125
150
175
200
225
250
< -14 -14 to -10 -10 to -6 -6 to -2 -2 to 2 2 to 6 6 to 10 10 to 14 >14
CWV INTERCEPT DIFFERENCE
FR
EQ
UE
NC
Y
08/09-07/08 (RMS=7.1, 7.2 excl. 09B) SMOOTHED (06/07-08/09) - SMOOTHED (05/06-07/08) (RMS=1.7, 1.8 excl. 09B)
Smoothed Model
Smaller CWV
Intercept Differences
Lower RMS Values
Less Volatility
Analysis 2: Volatility Assessment - Conclusions
� Results as expected
� Smoothed models show less year on year volatility
� Smaller CWV intercept differences between smoothed models
� Better RMS values for all Small and Large NDM EUCs
� Analysis continues to support principles of model smoothing
� 3rd aspect of model smoothing appropriateness needs to be considered...
Analysis 3: Trend Analysis - Purpose
� Identification of trends occurring
� Appropriate: Model averaging or annual model extrapolation
� Extrapolation of annual trend (no smoothing) has been deemed as only
appropriate if a clear trend emerging over recent years could be detected
� Applying single year models without such evidence = higher levels of
volatility in ALP, DAF and load factors each year
� Analysis: Compares ‘trends’ in CWV intercept value for the 3 single year
models constituting the 09/10 smoothed model
� 2006 / 07
� 2007 / 08
� 2009 / 10
� 5 possible outcomes…
Trend Analysis - Outcomes
2008/092007/08
2006/07
UP/UP (UU)
2008/09
2007/08
2006/07
UP/DOWN (UD)
2006/07
2005/062004/05
DOWN/DOWN (DD)
2008/09
2007/082006/07
DOWN/DOWN (DD)
2006/072004/05
FLAT (F)
2005/06
2008/092006/07
FLAT (F)
2007/08
2006/07
2005/06
2004/05
DOWN/UP (DU)
2008/09
2007/08
2006/07
DOWN/UP (DU)
2008/092007/08
2006/07
UP/UP (UU)
2008/092007/08
2006/07
UP/UP (UU)
2008/09
2007/08
2006/07
UP/DOWN (UD)
2008/09
2007/08
2006/07
UP/DOWN (UD)
2006/07
2005/062004/05
DOWN/DOWN (DD)
2008/09
2007/082006/07
DOWN/DOWN (DD)
2006/07
2005/062004/05
DOWN/DOWN (DD)
2008/09
2007/082006/07
DOWN/DOWN (DD)
2006/072004/05
FLAT (F)
2005/06
2008/092006/07
FLAT (F)
2007/08
2006/072004/05
FLAT (F)
2005/062005/06
2008/092006/07
FLAT (F)
2007/08
2006/07
2005/06
2004/05
DOWN/UP (DU)
2008/09
2007/08
2006/07
DOWN/UP (DU)
2006/07
2005/06
2004/05
DOWN/UP (DU)
2008/09
2007/08
2006/07
DOWN/UP (DU)
Trend Analysis – 3 Year Analysis Results
� Over the three years there are limited number of instances of specific EUCs and LDZs where a trends occur to a notable extent
� Supporting document Table 1, 2 and 3 highlight full results
� Most frequently observed pattern: UP / UP (129 of 429: 30%)
� DOWN / DOWN: 37 of 429 (9%)
� DOWN / UP & UP / DOWN: 224 of 429 (52%)
� Trends this year are not consistent with previous years
� UP / UP instances 11% in Autumn 2008
� DOWN / DOWN instances higher at 21% in Autumn 2008
� No obvious trend developing, further investigation supports this……..
Trend Analysis – 4 Year Analysis Results
� Inclusion of 4th year (05/06) to extend analysis
� 356 of 429 cases indicate no consistent trend (83%)
� Similar to previous years analysis (352 in 2008 and 353 in 2007)
� EUCs indicating a consistent trend are small
� 18 of 429 instances indicate a decreasing CWV intercept trend
� 16 of 429 instances indicate an increasing CWV intercept trend
� Worst case is EUC xxE0905W01 which shows upward trend for 4 years
(in 6 of 13 LDZs) – LFs show upward trend in 3 of them (representing
0.19% of aggregate NDM load).
� Conclusion is further supported when observing the single year model
load factors
� Overriding evidence shows the predominant effect is one of no consistent
trend (see document)
Model Smoothing - Conclusion
� Principles of model smoothing
� Reduce year on year volatility (remove modelling just annual trends)
� Not necessarily improve model prediction
� Necessary to review and assess if emerging trends are identified
� Current analysis consistent with results from previous analysis
� Model smoothing overall does reduce year on year volatility
� Model smoothing highlights slightly better level of predictability
� No signs of emerging trends of sufficient clarity have been identified
� Transporters view current methodology to use model smoothing over 3 years to be appropriate and fit for purpose
� Recommend model smoothing is applied for 2010/11 analysis