NEURAL NETWORK MODELS FOR ELECTRICITY MARKET FORECASTING Z. Xu*, Z. Y. Dong*, W. Q. Liu* *

*School of Information Technology and Electrical Engineering The University of Queensland, St Lucia, QLD 4072, Australia

** School of Computing, Curtin University of Technology Perth, GPO 1987 WA 6845, Australia

1 ABSTRACT

For increasingly deregulated electricity market, accurate forecasting of electricity demand and spot price has become crucial for the independent system operators, generators and consumers. However, the exclusive features of electricity present a number of challenges for this task. This paper describes the author’s continual effort in power system forecasting studies. By using various new techniques, such as wavelet, neural network and support vector machine, different models for electricity load and price forecast have been developed, which are able to forecast at one or more time steps ahead. Case studies, using actual data from Australian National Electricity Market, have been carried out and the results are presented. In addition, comparisons with other forecasting models have shown that the proposed new methods can provide more stable and reliable performance even under the chaotic market conditions.

2 INTRODUCTION

Since last century, the electrical power industry world wide has been continuously deregulated and reconstructed from a centralized monopoly to a competitive market. To date, it has been well understood that the competitive market is a more proper mechanism for energy supply to free the customer’s choices and improve overall social welfare [1].

In the open market, power companies and consumers submit their generation or consumption bids and corresponding prices to the market operator (MO), who will then use a market-clearing tool to clear the market. The clearing process is normally based on a single round auction to determine the spot price for the corresponding time interval. Short-term load and price forecasting aims at predicting the system demand and market clearing price with a leading time from half or one hour to several days, which is essential for adequate scheduling and operations of the market participants as well as the managements. However, forecasting is a rather complex task as the electricity possesses many exclusive features. Unlike other commodities, the electricity can not be stored and power system stability requires consistent balance between supply and demand; besides, there are many physical and technical constraints for the delivery of energy. Moreover, unpredictable and uncontrollable contingencies in power system increase the complexities. As such, electricity market exhibits a high volatile nature and is probably the most volatile market. Proper price and demand forecasts are essential to risk management plans for the participating companies. In Victoria, Australia, generators and retailers can hedge against pool

price volatility by vesting contracts and contestable contracts [2]. If the electricity market clearing price and demand can be predicted properly, generators and retailers can reduce their risks and maximize their interests respectively.

In terms of statistics, forecasting is an approximation of future data. Traditional linear models, such as Auto-regressive (AR) and Auto-regressive Moving Average (ARMA), have been used in time series forecast in practice. These models are straightforward for implementation, but fail in general to give satisfactory performance when dealing with the non-linear and non-stationary signals, such as the electricity price signals.

Recent researches have seen a number of new techniques appearing in the area, such as the neural network (NN), chaos analysis and fuzzy logic. These new methods have been proven to provide superior performance than traditional ones [3]-[4]. Among those, the models using NNs, such as the recurrent and Back-propagation neural networks, have gained their importance. In the machine learning area, the latest development of Support Vector Machine (SVM) has proven its excellent abilities in both classification and regression with a fast learning speed [5]-[6]. It therefore motivates our implementation of these new techniques in this research.

We develop new forecasting models for electricity market based on our previous research [7]. Several new techniques have been applied in developing these models, including time-invariant or Non-decimated Wavelet Transform (NWT), neural network, SVM as well as data mining. The wavelet transform is introduced to pre-process the load and price data in order to enhance the accuracy of forecasting. The coefficients obtained form NWT, which are well localized in both time and frequency domains, are used in NN or SVM forecasting at the subsequent stage. Specially, the wavelet packet transform (WPT) has been implemented in price forecasting due to its decomposition capability over a wider frequency range. In load forecasting, the weather (temperature), which has significant impacts on energy consumption, has been used in addition to the historical load data. In case studies, it has been shown that new approaches are able to forecast the price and load of Australian National Electricity Market (NEM) with adequate accuracy. In addition, the introduction of WPT in the model has been justified through comparisons. Moreover, comparisons with other forecasting methods have shown the new models can provide more stable and reliable performance even under extremely chaotic market conditions.

3 OVERVIEW OF THE WPT-SVM FORECAST MODEL

As depicted in Fig. 1, the WPT-SVM forecast model for electricity price forecasting uses the same overall structure as that of [7]. There are p delay stages in the forecast model. The input is the electricity price series u[t], where t = 1, 2,…, N. The output of the system is the forecasted price data at n steps ahead, i.e. û[t+n]. If n =1, the whole system becomes a single step prediction model. The prediction approach involves three stages, namely the discrete wavelet packet decomposition of the input data, multi-resolution forecasting using SVMs, and wavelet reconstruction of the forecasted data.

3.1 Wavelet packet transform of electricity price data Initially, the present and p previous samples of the input data are fed into the discrete WPT. Consequently, the wavelet coefficients of different time and frequency

resolutions are obtained. Fig 1 shows the WPT process, where cAi and cDi represents the approximation and detail coefficients at decomposition level k. The number p can be considered as the order of the nonlinear forecast system, and its value is decided through empirical observation. Only three-level decomposition has been considered for WPT since too many decomposition levels imply excessive computational needs and time. The mother wavelets considered in the approach are those from the Daubechies family, since this family of wavelets excels in extracting features from signals and the discrete wavelet transform algorithm is available for this family [8]-[9].

Fig. 1: The multi-resolution forecast model

Comparing to the discrete wavelet algorithm (Mallat algorithm), the WPT extends the analysis in the high frequency range, and thus offers greater flexibility for signal analysis. This attribute has made the WPT more suitable than discrete wavelet transform (DWT) in our case studies for electricity price forecasting, since the price data contains much noise of high frequency. The classical DWT algorithm was developed by Mallat in 1988 [9]. However, due to the decimation employed, this algorithm is not a time-invariant transform and drops historical data in process. To keep the important historical information, as well as the time invariance for the transform, the redundant or NWT has been applied in our model instead of the Mallat algorithm. In the developed forecast model, the à trous algorithm is applied to achieve such time-invariant or non-decimated wavelet packets transform [10]. Details about DWT and WPT will be covered in section III.

3.2 Multi-resolution forecasting using SVMs In the second stage, a number of forecastors are allocated to predict the wavelet coefficients obtained form the WPT. The support vector machines are applied as the basic forecasting units. The SVMs have been successfully applied to classification tasks and more recently also to regression [6]. The perfect nonlinear generalization ability of SVM makes it ideal for forecasting the chaotic electricity price. Specifically, the SVM is based on the statistical learning theory that provides the SVM with

excellent ability to generalize well on not only training data but also unseen data [5], [6]. This property is definitely essential in any machine learning and NN applications, such as the time series prediction in this study. More details about the SVM and its application in the forecast model will be covered in Section 5.

3.3 Wavelet reconstruction of forecasted price data The forecasted wavelet coefficients are to be used for reconstruction of the forecasted data. As in the decomposition process, the à trous algorithm is applied for reconstruction, and details are shown in the following section.

4 THE WAVELET TRANSFORM

4.1 Discrete Wavelet and Wavelet Packet Transform The discrete wavelet transform (DWT) of a discrete signal f(k) is defined as [9], [11],

��

�

�

��

�

� −=� m

m

m

K

k

nkgkfnmDWT

2

2

2

1)(),( (1)

where m is the scaling constant (stretching, or decomposition level), and m =1, 2,…, mmax. mmax is given by Km ≤max2 , where K is length of the discrete signal f(k). n is

the shifting (translating) constant and is an integer. The ���

����

� −m

mnkg

2

2 is the scaled,

shifted wavelet function (baby wavelet), given the mother wavelet function as g(k).

Fig. 2 illustrates the Mallat algorithm for discrete wavelet transform, which is a classical filter bank scheme [8], [9]. H and L and H’ and L’ are the low-pass and high-pass filters for decomposition and reconstruction respectively. These filters are derived from the mother wavelet g(k). By feeding the signal (or the approximation coefficients) into the scheme successively, the DWT decomposes the signal into coefficients with deferent resolutions of time and frequency. The discrete wavelet packet transform is an extension of DWT. The WPT extends the decompositions to not only approximation (cA) but also detail (cD), and therefore it can analyse signals over wider band and with more flexibilities. The mother wavelets used here are from the Daubechies family for which the DWT algorithm is available [9].

Fig. 2: The classical DWT (Mallat algorithm)

As indicated by (↓) in Fig. 2, the Mallat algorithm “down sample” the original data in the resultant coefficients by discarding every other data coming out of the filters in the decomposition [8]-[9]. This decimation process facilitates a faster computational process. Nevertheless, it also incurs the losses of historical information and time-invariance of the transform. The losses are especially undesirable for the forecast

models, since complete and accurate historical data is essential in forecasting future data. To avoid the losses while keeping the advantages of wavelet, the à trous algorithm is applied for the WPT. Basically; the à trous algorithm is similar to that of Mallat, except there is no decimation step. More details of the à trous algorithm can be found in [11].

4.2 Best basis selection As shown in Fig 3 the coefficients obtained from a WPT of three-level decomposition form a tree-like structure, known as WPT tree. Since there are many nodes in the tree, and any combination of them can be used in forecasting as long as they satisfy the wavelet reconstruction conditions [8]. However, there is the need to select the decomposition components (nodes) in WPT tree that best ‘characterize’ the original input signal in order to enhance the forecast performance. The selected components will then be used in forecasting and reconstruction of the predicted electricity price. Proposed by Coifman and Wickerhauser, the ‘best basis algorithm’ is chosen in our approach to select the optimal coefficients for forecasting [12].

Fig. 3: The WPT tree of three-level decomposition

Based on the information theory, the best basis method calculates and compares a cost function according to each node in WPT tree. The optimal nodes are those generating the minimum sum of costs, provided they satisfy the condition of orthogonality for wavelet reconstruction [12]. There are quite a few cost functions available, among which the Shannon Entropy is usually used. Equation (2) defines the (non-normalized) Shannon Entropy of a signal s [9], [12].

0)0log(0)(log)(2

2

2 =−= withsH ss (2)

According to the Fig. 1, since price signals of p +1 length are consecutively decomposed in the forecast model at every time step, a dataset of these signals is obtained eventually. To select the best basis for all data in the dataset, the global entropy is calculated. Equation (3) defines the global entropy, where M denotes the number of samples for SVM training. The training data for SVMs forms the ensemble of price signals for this selection procedure. Nodes from (3, 0) to (3, 7) comprise of the best basis for the training data in the study.

�=M

iiglobal sH

MH )(

1 (3)

5 SUPPORT VECTOR MACHINE REGRESSION

The SVM is a nonlinear generalization of the Generalized Portrait algorithm developed in sixties [5]-[6]. The algorithm is firmly grounded on the statistical learning theory, or Vapnik-Chervonenkis (VC) theory, which enables the leaning machine not only perform well on training data but also generalize well on unseen data, namely testing data. This property is certainly attractive for any machine learning applications, and it is also one motivation for our implementation.

The SVM hasn’ t been applied in regression and time series prediction until recently [6]. According to equation (4), the SVM regression is to map the data X into a high dimensional feature space Γ via a nonlinear mapping function φ to do linear regression in this space, where b is the threshold.

Γ∈Γ→+⋅= ωφφω ,:))(()( nRwithbxxf (4)

For ε-SVM regression, the goal is to find a f(x) that has at most ε-deviation from actually obtained Yi for all training data { (X1, Y1), (X2, Y2), …,(Xm, Ym)} ⊂ X×R, and is as flat as possible simultaneously [5]-[6]. The regression thus forms an optimization problem by minimizing a risk function,

2

2][][ ωλ+= fRfR empreg (5)

where the Remp[f] is a cost function determining how we will penalize the prediction

errors based on the empirical data X. The term 2

2ωλ is actually a flatness index of

f(x) [6]. The regression quality is controlled by ε, the error cost function and the mapping function φ and the flatness index. There are many types of mapping function, among which the radial basis function is most preferred in regression due to its excellent performance and has been implemented in our approach as well. More details about the SVM regression and the program used in the research can be found in [5], [6] and [13].

6 TEMPERATURE DEPENDENT WAVELET NEURAL NETWORK

FORECAST MODEL

The Short Term Load Forecast (STLF) model is another model proposed. Similar to the price forecast model, it has three stages, including data pre-processing using NWT, feed-forward NN forecasting and data post-processing by NWT. Fig 4 shows the overall structure of the STFL model.

6.1 Data pre-processing Both the historical data of electricity load and temperature are treated as time series signals. à trous transform, which is an example of NWT, is used as the pre-signal

processor in the proposed model. The hidden patterns of both electricity demand and temperature are extracted by this algorithm in our approach.

6.2 Neural network forecasting In this stage, the wavelet coefficients obtained from NWT decomposition are fed into neural networks to predict future data at one more time steps ahead. A set of feed-forward NNs are allocated to forecast the wavelet at different resolution levels. These networks contain only one hidden layer, which is adequate to approximate functions of any complexities [14]. The Scaled Conjugate Gradient algorithm (SCG) is used in training the NNs, due to its advantage of fast computational time for a large NN size.

6.3 Data post-processing For post signal processing, the same wavelet technique and resolution level as mentioned in pre-processing stage is used. In this stage, the outputs from the signal predictors (NNs) are combined to form the final predicted output.

Fig. 4: The temperature dependent forecast model

7 NUMERICAL RESULTS AND ANALYSIS

In this section, the forecast models are to be tested using actual data from the Australian NEM. Performance of the developed models will be compared with other methods. The results show the developed models are able to provide superior forecast performance for periods of different demand intensity.

7.1 Introduction to Australian National Electricity Market The Australian power industry has been deregulated into a competitive market since 1995. The Australian NEM was formally launched in December 1998. Fig. 5 shows the Australian NEM, which consists of the electricity markets of the states of New South Wales, Victoria, South Australia, Queensland and possibly Tasmania [15]. The market clearing of NEM is conducted every half hour. As part of NEM, New South Wales (NSW) electricity market is the largest one and is selected to test our price forecasting model. The NSW electricity market has a maximum peak demand of 11572 MW (recorded in 2000), annual average price varies from $28.88 (99/00) to $38.36 (00/01), with high volatility, especially in winter or summer time.

The proposed STLF model is tested with historical data of Queensland Market in January 2001. The data used in our case studies is obtained from (Australian)

Commonwealth Bureau of Meteorology and National Electricity Market Management Company Limited (NEMMCO) [15].

Fig. 5: The Australian National Electricity Market

(Arrows indicate interregional links of power transmission)

7.2 Case study-Electricity Price Forecast Two sets of New South Wales market data are to be studied. One is the first two weeks of July 2001 (from days July 1 to 15), which is winter time and usually a low demand period. The other is the first two weeks of December 2001 (from days Dec 1 to 15), which is summer time and usually a high demand period. Because of the seasonal nature of the electricity market, the first week of each dataset will be used to train the forecast model, and the rest will be used for testing accordingly. In case studies, it should be noted that the price data used is prior to the appearance of dramatic swing period, known as outlier in time series forecasting literature. These unusual price spikes are due to unexpected and uncontrollable events in the power system. To detect and forecast the outlier is a challenging task and requires a better understanding and refined modelling of the price series. Additional information about outlier detection can be found in [16], [17].

Figs 6 and 7 show the testing results of the two cases as well as the original data respectively. The simulation is based on a single step prediction. For performance evaluation of the model, the Root Mean Square (RMS) error (ErrRMS) on the testing data set between the forecasted and real price is calculated using Equation (6) below,

NErr

N

nRMS

nunu� −== 1

2

))()(ˆ( (6)

where û[n] is the predicted price at step n; u[n] is the actual price at sep n; and N is the total forecast steps. The absolute percentage error (APE) is another performance measure used. The APE at prediction step n is defined as,

%100)(

)()()( ×

−=

∧

nu

nunnAPE u (7)

Fig. 6: Single step forecast of July week in NSW market (Errrms= 1.2226)

Fig. 7: Single step forecast of December week in NSW market (Errrms= 1.0072)

The RMS errors of the July and December weeks are 1.2226 and 1.0072 respectively. The results demonstrate that the SVM and WPT approach is able to forecast the market price with a reasonable accuracy. Especially, the model presents a quite reliable performance even for the fairly chaotic December week in Fig 7. The APE of December week prediction is presented in Fig 8, where the average of the APE is 3%.

Fig. 8: The APE of December week forecast in NSW market (APEaverage= 3%)

7.3 Case study - Load forecast Eight sets of historical data containing the electricity load and temperature data for the month of January 2001 are used in testing temperature dependent STFL model with one set of electricity demand data of Queensland Market and 7 sets of temperature data from 7 different locations of higher power consumptions. The load data is on half-hour basis. The reason for using 7 sets of temperature data is a compromise due to the unavailability of a general set of average temperature data for Queensland. The forecasting performance is measured by the mean absolute percentage error (MAPE),

%100*1

1�

=���

����

� −=

N

i i

ii

x

yx

NMAPE (8)

where N is the number of points measured, xi is the actual values and yi is the predicted values. The model is trained with one week (336 points) of combined demand and temperature data for fifty cycles. The performance of the forecast model was evaluated and the results are as shown in Figs 9 to 11.

Fig. 9: 7-day forecasting (336 points)

The MAPE values were calculated based on the forecasted values and the original time series - Table 1.

Table 1: Summary of MAPEs

No. of forecasted points MAPE (%)

336 (7 days) 1.4134

384 (8 days) 1.6198

432 (9 days) 2.0048

Fig. 10: 8-day forecasting (384 points)

Fig. 11: 9-day forecasting (432 points)

7.4 Comparisons The developed price forecasting model has been compared to a scheme without the wavelet decomposition and another scheme using DWT and feed-forward neural network approach [7]. Fig 12 shows the basic structure of the model without wavelet decomposition. To make the comparison, the three models have the same p system

delays and decomposition level for wavelet. Besides the data used in the previous cases studies, more typical high and low demand weeks are added to make in the comparison dataset. The added high demand weeks include those from January, and low demand weeks are from August.

Fig.12: The normal neural network forecast model

Table 2 summarizes the performance of the three forecasting models for comparisons. For single step prediction, it is observed that the developed WPT-SVM model generally presents a more stable performance for both high and low demand weeks. The DWT-MLP (multilayer perceptron) model can give a better accuracy over the low-demand weeks, as 0.8355 and 1.5851 in July and August weeks respectively, however, it has shown a quite poor performance for the high demand weeks. The SVM alone has presented very poor performance for each week and it therefore can be concluded that the WPT pre-processing is essential in the developed scheme to enhance forecasting accuracy. An interesting observation is made in comparing forecasts of the January weeks. Fig 13 shows the real and forecasted prices by the three models, where an outlier obviously appeared in the original data. As shown, the WPT-SVM model presents the best performance in this extremely chaotic case. The RMS error of the WPT-SVM model is 12.9397, which is much less than 25.8096 of the DWT-MLP model and 23.3757 of SVM alone. In brief, the comparisons have shown that the WPT-SVM model is able to provide a more stable and reliable forecasting even under extremely chaotic market conditions, such as the January weeks.

Fig. 13: Single step forecast of the November weeks in NSW market

Table 2: Single step forecast performance (RMS Error)

Season Weeks WPT-SVM

model DWT-MLP

model SVM

12.9397 25.8096 23.3757 Summer

Jan December 1.0072 4.7417 3.3745

1.2226 0.8355 5.0318 Winter

July August 1.6169 1.5851 4.6184

8 DATA MINING APPROACH TO ELECTRCITY MARKET PRICE SPIKE

FORECASTING

From the case studies, it can be seen that the proposed models provide a reasonable accuracy in electricity price and load forecasting, provided they have been trained thoroughly. Comparing to load forecasting, the fairly chaotic electricity price is more complex to be predicted. The high volatility of electricity price is attributed to the many exclusive features of the electricity and the power systems. Especially, the outliers or spikes in the price series have made the forecasting extremely difficult as in Fig 13, although market participants rely much on this information for financial management. With the models proposed in this paper, we have further proposed an electricity price forecasting scheme. Basic forecasting function will be achieved using the models developed in this paper, while the data mining technique is employed to account for outlier prediction.

From the data set (see Fig 14), it can be seen that the total electricity demands (TED) are more or less periodical. The price spikes in regional reference price (RRP) are correlated to the peak time of TED. As a result the price forecasting can be constrained by employing data mining for prediction of the time of the price spikes by correlation with the demand series in the following steps:

1. Calculate the correlation coefficient r between TED and RRP:

2)((

(

Yyx

Yyxr

iXi

iXi

−

−=��

−

−

2)

)( ) (9)

2. For a given time series database D, at time t, with the window size w, we take a sub-series s in the time period t-w, to search D for a match s’ that satisfies the minimum pre-defined Euclidean distance d between s and s’ .

3. For s’ the RRP( t) × r will be the predicted price.

This approach is particularly in predicting the electricity market price spikes in correlation to the electricity demand. Combining the data mining technique with the forecasting model developed, the price forecasting scheme with outliers detection is illustrated in Fig 15.With the proposed outline, further research on electricity price forecasting with appearance of outliers is underway.

0

1000

2000

3000

4000

5000

6000

7000

1 15 29 43 57 71 85 99 113

127

141

155

169

183

197

211

225

239

253

TOTALDEMAND

RRP

Fig. 14: The correlation between total demands and price (RRP)

Fig. 15: Proposed price forecasting scheme with outlier detection

9 CONCLUSIONS

In this paper, time series models for short-term electricity price and load forecasting has been developed based on techniques such as wavelet analysis, support vector machine, neural network and adaptive data mining. Cases studies, using actual data from Australian NEM, have shown the developed models provide adequate accuracy in forecasting the electricity price. Through comparisons, the necessity of introducing wavelet in the forecasting model has been confirmed. Furthermore, the excellent capability of the developed model has been further demonstrated in forecasting the extremely chaotic market with an outlier. To further improve forecasting accuracy, the detection and forecasting the outlier in electricity price series with the proposed data mining model has been identified as the relevant research topic in future.

10 REFERENCES

[1] F. F. Wu and P. Varaiya, “Coordinated multilateral trades for electric power networks: theory and implementation 1” , Electrical Power and Energy Systems 21, pp.75-102, 1999

[2] F. A. Wolak, “Market design and price behavior in restructured electricity markets: an international comparison” , http://www. stanford.edu/~wolak

[3] T. Niimura, H. S. Ko and K. Ozawa, “A day-ahead electricity price prediction based on a fuzzy-neuro autoregressive model in a deregulated electricity market” Proc. the 2002 International Joint Conference on Neural Networks, vol.2, pp. 1362 -1366, 2002.

Price series

Pre-processing

Demand series

Base Series NN-Wavelet Forecast

Module

Correlation Processing Module

Final Processing

Price Spikes

Base Series Final forecasted price signal

[4] B. Amir Geva, “ScaleNet - Multiscale neural-network architecture for time series prediction” , IEEE Trans. Neural Networks, Vol. 9, No. 5, pp. 1471-1482, 1998.

[5] V. Vapnik, “The nature of statistical learning theory” , Springer, NY, 1995. [6] A. J. Smola and B. Schölkopf. “A tutorial on support vector regression” ,

NeuroCOLT Technical Report NC-TR-98-030, Royal Holloway College, University of London, UK, 1998.

[7] Z. Xu and Z. Y. Dong, “A multi-resolution neural network model for electricity market price forecasting” , Proceeding of 4th Australian Workshop on Signal Processing Applications, Brisbane, Australia, 2002.

[8] I. Daubechies, “Ten Lectures on Wavelets” , SIAM, Philadelphia, PA, 1992. [9] M. Misiti, Y. Misiti, G. Oppenheim, and J. M. Poggi, “Wavelet toolbox for use

with MATLAB” , The Math Works Inc., March 1996. [10] G. Zheng et al. “The wavelet transform for filtering financial data streams,”

Journal of Computational Intelligence in Finance, vol. 7, no. 3, May/June 1999. [11] M. J. Shensa, “The discrete wavelet transform: wedding the à trous and Mallat

algorithms”, IEEE Transactions on Signal Processing, vol. 40 iss. 10, pp. 2464 -2482, Oct. 1992.

[12] R. R. Coifman and M.V Wickerhauser, “Entropy-based algorithms for best basis selection” , IEEE Transactions on Information Theory, vol. 38 issue 2, part 2, pp. 713 -718, March 1992.

[13] S. Rüping, “mySVM-Manual” , University of Dortmund, Lehrstuhl Informatik 8. Available:http://www-ai.cs.uni-dortmund.de/SOFTWARE/ MYSVM/.

[14] S. Haykin, “Neural Networks” , Macmillan College Publishing Co. Inc. 1994. [15] (Australian) National Electricity Market Management Company Limited

(NEMMCO). Available: http: //www.nemmco.com.au/. [16] A. M. Azoff, “Neural network time series forecasting of financial markets” , John

Wiley & Sons, 1994. [17] A. Pankratz, “Forecasting with Dynamic Regression Models” , New York: Wiley,

1991.