Application of Application of Back-Propagation neural Back-Propagation neural

network in data forecastingnetwork in data forecastingLe Hai Khoi, Tran Duc MinhLe Hai Khoi, Tran Duc Minh

Institute Of Information Technology – VASTInstitute Of Information Technology – VAST

Ha Noi – Viet NamHa Noi – Viet Nam

AcknowledgementAcknowledgement

The authors want to Express ourThe authors want to Express our 　　 thankfulnthankfulness to Prof. Junzo WATADA who read and ess to Prof. Junzo WATADA who read and gave us worthy comments.gave us worthy comments.

AuthorsAuthors

CONTENTCONTENT

IntroductionIntroduction

Steps in data forecasting modeling Steps in data forecasting modeling using neural networkusing neural network

Determine network’s topologyDetermine network’s topology

ApplicationApplication

Concluding remarksConcluding remarks

IntroductionIntroduction•　 Neural networks are “Universal Approximators”

•　 To find a suitable model for the data forecasting problem is very difficult and　 in reality, it might be done only by trial-and-error

•　 We may take the data forecasting problem for a kind of data processing　 problem

Data collecting and analyzing

Neural Networks

Post-processing

Pre-processing

Figure 1: Data Processing.

Steps in data forecasting modeling using neural networkSteps in data forecasting modeling using neural network

The works involved in are:

* Data pre-processing: 　 determining data interval: daily, weekly, monthly or quarterly; data type:　 technical index or basic index; method to normalize data: max/min or　mean/standard deviation.* Training: 　 determining the learning rate, momentum coefficient, stop condition, maximum　 cycles, weight randomizing, and size of training set, test set and verification　 set.* Network’s topology: 　 determining number of inputs, hidden layers, number of neurons in each layer,　 number of neurons in output layer, transformation functions for the layers and　 error function

Steps in data forecasting modeling using neural networkSteps in data forecasting modeling using neural network

The major steps in design the data forecasting model is as follow:

１ .　 Choosing variables２． Data collection３． Data pre-processing４． Dividing the data set into smaller sets: training, test and verification５． Determining network’s topology: number of hidden layers, number of　　 neurons in each layer, number of neurons in output layer and the　　 transformation function.６． Determining the error function７． Training８． Implementation.

In performing the above steps, it is not necessary to perform steps sequentially. We could be back to the previous steps, especially in training and choosing variables steps. The reason is because in the designing period, if the variables chosen gave us unexpected results then we need to choose another set of

variables and bring about the training step

Choosing variables and Data collection

Determining which variable is related directly or indirectly to the data that we need to forecast.

• If the variable does not have any affect to the value of data that we need to forecast then we should wipe it out of consider.

• Beside it, if the variable is concerned directly or indirectly then we should take it on consider.

Collecting data involved with the variables that are chosen

Data pre-processingAnalysis and transform values of input and output data to emphasize the important features, detect the trends and the distribution of data.

Normalize the input and output real values into the interval between max and min of transformation function (usually in [0, 1] or [-1, 1] intervals). The most popular methods are following:

SV = ((0.9 - 0.1) / (MAX_VAL - MIN_VAL)) * (OV - MIN_VAL)Or:

SV = TFmin + ((TFmax - TFmin) / (MAX_VAL - MIN_VAL)) * (OV - MIN_VAL)where:

SV: Scaled ValueMAX_VAL: Max value of dataMIN_VAL: Min value of dataTFmax: Max of transformation functionTFmin: Min of transformation functionOV: Original Value

Dividing patterns setDivide the whole patterns set into the smaller sets:

(1) Training set

(2) Test set

(3) Verification set.

The training set is usually the biggest set employed in training the network. The test set, often includes 10% to 30% of training set, is used in testing the generalization. And the verification set is set balance between the needs of enough patterns for verification, training, and testing.

Determining network’s topology

This step determines links between neurons, number of hidden layers, number of neurons in each layer.

1. How neurons in network are connected to each other.

2. The number of hidden layers should not exceed two

3. There is no method to find the most optimum number of neurons used in　 　 hidden layers.

=> Issue 2 and 3 can only be done by trial and error since it is depended on the problem that we are dealing with.

Determining the error function

• To estimate the network’s performance before and after training process.

• Function used in evaluation is usually a mean squared errors. Other functions may be: least absolute deviation, percentage differences, asymmetric least squares etc.

Performance index

F(x) = E[eTe] = E [ ( t - a )T ( t - a ) ]

Approximate Performance index

F(x) = eT(k)e(k)] = (t(k) - a(k) )T ( t(k) - a(k))

• The lastest quality determination function is usually the Mean Absolute Percentage Error - MAPE.

TrainingTrainingTraining tunes a neural network by adjusting the weights and biases

that is expected to give us the global minimum of performance index or error function.

When to stop the training process ?1. It should stop only when there is no noticeable progress of the error

function against data based on a randomly chosen parameters set?2. It should regularly examine the generalization ability of the network by

checking the network after a pre-determined number of cycles?3. Hybrid solution is having a monitoring tool so we can stop the training

process or let it run until there is no noticeable progress.4. The result after examining of verification set of a neural network is most

persuadable since it is a directly obtained result of the network after training.

Implementation

This is the last step after we determined the factors related to network’s topology, variables choosing, etc.

1. Which environment: Electronic circuits or PC2. The interval to re-train the network: might be depended on the times and also other factors related to our problem.

Determine network’s topologyDetermine network’s topology

Multi-layer feed-forward neural networks

S2x1

S1x1

n1

1

S1xR1

R1　x1

W1

b1

f1

S1x1

S1x

1

a1

S2x1

n2

1

S2xS1

W2

b2

f2

S2x1

a2

P

Figure 2: Multi-layer feed-forward neural networks

where:P: input vector (column vector)Wi: Weight matrix of neurons in layer i. (SixRi: Si rows (neurons), Ri columns

(number of inputs))bi: bias vector of layer i (Six1: for Si neurons)ni: net input (Six1)fi: transformation function (activate function)ai: net output (Six1): SUM functioni = 1 .. N, N is the total number of layers.

a2 = f2( W2 f1 (W1p + b1) + b2)

Determine training algorithm and network’s topologyDetermine training algorithm and network’s topology

Output

x １

x2

…

xn

bias

biaswij

Input layer Hidden layers Output layer

…

… …

wjk

wkl

Transfer function is a sigmoid or any squashing function that is differentiable

ƒ(x) =1

1 + e-δx

and ƒ’(x) = ƒ(x) { 1 - ƒ(x) }

1

1

Figure 3: Multi-layered Feed-forward neural network layout

Back-propagation algorithmBack-propagation algorithm

Step 1: Feed forward the inputs through networks:a0 = p

am+1 = fm+1 (Wm+1 am + bm+1), where m = 0, 1, ..., M – 1.a = aM

Step 2: Back-propagate the sensitive (error):

atnFs

MM

M 2

11

mTmmm

m sWnFs

where m = M – 1, ..., 2, 1.

Step 3: Finally,　weights　and　biases　are　updated　by　following　formulas:

mmm

Tmmmm

kk

kk

sbb

asWW

1

1 1

.

(Details on constructing the algorithm and other related issues should be found on text book Neural Network Design)

at the output layer

at the hidden layers

Using Momentum Using Momentum

This is a heuristic method based on the observation of training results. The standard back-propagation algorithm will add following item to the weight as the weight changes:

∆Wm(k) = - sm (am – 1) T, ∆bm(k) = - sm .

When using momentum coefficient, this equation will be changed as follow:

∆Wm(k) = ∆Wm(k – 1) – (1 – ) sm (am – 1)T,∆bm(k) = ∆bm(k – 1) – (1 – ) sm .

ApplicationApplication

Arrow: inheritance relationRhombic antanna arrow: Aggregate relationNEURAL NET class includes the components that are the instances of Output Layer and Hidden Layer.Input Layer is not implemented here since it does not do any calculation on the input data.

Arrow: inheritance relationRhombic antanna arrow: Aggregate relationNEURAL NET class includes the components that are the instances of Output Layer and Hidden Layer.Input Layer is not implemented here since it does not do any calculation on the input data.

NEURAL NETclass

Output layer

Hidden layer

LAYERclass friend

ApplicationApplication

ApplicationApplication

ApplicationApplication

Concluding remarksConcluding remarks

The determination of the major works is important and realistic. It will help The determination of the major works is important and realistic. It will help develop more accuracy data forecasting systems and also give the develop more accuracy data forecasting systems and also give the researchers the deeper look in implementing the solution using neural researchers the deeper look in implementing the solution using neural networksnetworks

In fact, to successfully apply a neural network, it is depended on three major In fact, to successfully apply a neural network, it is depended on three major factors:factors:

First, the time to choose the variables from a numerous quantity of data as well as performFirst, the time to choose the variables from a numerous quantity of data as well as perform

pre-processing those data; pre-processing those data;

Second, the software should provide the functions to examine the generalization ability,Second, the software should provide the functions to examine the generalization ability,

help find the optimal number of neurons for the hidden layer and verify with many input sets; help find the optimal number of neurons for the hidden layer and verify with many input sets;

Third, the developers need to consider, examine all the possible abilities in each timeThird, the developers need to consider, examine all the possible abilities in each time

checking network’s operation with various input sets as well as the network’s topologies checking network’s operation with various input sets as well as the network’s topologies

so that the chosen solution will exactly described the problem as well as give us the mostso that the chosen solution will exactly described the problem as well as give us the most

accuracy forecasted data. accuracy forecasted data.

THANK YOU FOR THANK YOU FOR ATTENDING!ATTENDING!

AuthorsAuthors

Kytakyushu 03/2004Kytakyushu 03/2004