Neural NetworksNeural Networks

& Cases& Cases

ByBy

Jinhwa KimJinhwa Kim

2

Neural Computing is a problem solving methodology that attempts to mimic how human brain function

Artificial Neural Networks (ANN)

Machine Learning

Neural Computing: The BasicsNeural Computing: The Basics

3

Computing technology that mimic certain processing capabilities of the human brain

Knowledge representations based on Massive parallel processing Fast retrieval of large amounts of information The ability to recognize patterns based on

historical casesNeural Computing = Artificial Neural Networks

(ANNs) Purpose of ANN is to simulate the thought

process of human brain Inspired by the studies of human brain and the

nervous system

Neural ComputingNeural Computing

4

The Biology AnalogyThe Biology Analogy

Neurons: brain cells Nucleus (at the center) Dendrites provide inputs Axons send outputs

Synapses increase or decrease connection strength and cause excitation or inhibition of subsequent neurons

Figure 15.1

5

A model that emulates a biological neural network

Software simulations of the massively parallel processes that involve processing elements interconnected in a network architecture

Originally proposed as a model of the human brain’s activities

The human brain is much more complex

Artificial Neural Networks (ANN)Artificial Neural Networks (ANN)

6

Biological ArtificialSoma NodeDendrites InputAxon OutputSynapse WeightSlow speed Fast speedMany neurons Few neurons (Billions) (Dozens)

Artificial Neural Networks (ANN)Artificial Neural Networks (ANN)

Three Interconnected Artificial Neurons

7

Components and Structure“A network is composed of a number of processing elements organized in different ways to form the network structure”

Processing Elements (PEs) – Neurons Network

Collection of neurons (PEs) grouped in layers Structure of the Network

Topologies / architectures – different ways to interconnect PEs

ANN FundamentalsANN Fundamentals

Figure 15.3

8

ANN FundamentalsANN Fundamentals

Figure 15.4

9

Processing Information by the Network Inputs Outputs Weights Summation Function

Figure 15.5

ANN FundamentalsANN Fundamentals

10

Transformation (Transfer) Function Computes the activation level of the neuron Based on this, the neuron may or may not produce an

output Most common: Sigmoid (logical activation) function

AIS 15.3

ANN FundamentalsANN Fundamentals

11

Learning in ANNLearning in ANN

1. Compute outputs2. Compare outputs with

desired targets3. Adjust the weights

and repeat the process

Figure 15.6

12

Neural NetworkNeural NetworkApplication DevelopmentApplication Development

Preliminary steps Requirement determination Feasibility study Top management champion

ANN Application Development Process1. Collect Data2. Separate into Training and Test Sets3. Define a Network Structure4. Select a Learning Algorithm5. Set Parameters, Values, Initialize

Weights6. Transform Data to Network Inputs7. Start Training, and Determine and

Revise Weights8. Stop and Test9. Implementation: Use the Network

with New Cases

13

Collect data and separate it into Training set (60%) Testing set (40%)

Make sure that all three sets represent the population: true random sampling

Use training and cross validation cases to adjust the weights

Use test cases to validate the trained network

Data Collection and PreparationsData Collection and Preparations

14

Neural Network ArchitectureNeural Network Architecture

There are several ANN architectures Figure 15.9

15

Neural Network ArchitectureNeural Network Architecture

Feed forward Neural Network Multi Layer Perceptron, - Two, Three,

sometimes Four or Five Layers

16

Step function evaluates the summation of input values

Calculating outputs Measure the error (delta) between outputs

and desired values Update weights, reinforcing correct resultsAt any step in the process for a neuron, j, we

getDelta = Zj - Yj

where Z and Y are the desired and actual outputs, respectively

How a Network LearnsHow a Network Learns

17

Updated Weights areWi (final) = Wi (initial) + alpha × delta × X1

where alpha is the learning rate parameter

Weights are initially random The learning rate parameter, alpha, is set low Delta is used to derive the final weights, which

then become the initial weights in the next iteration (row)

Threshold value parameter: sets Y to 1 in the next row if the weighted sum of inputs is greater than 0.5; otherwise, to 0

How a Network LearnsHow a Network Learns

18

How a Network LearnsHow a Network Learns

19

Continue

BackpropagationBackpropagation

Backpropagation (back-error propagation)

Most widely used learning Relatively easy to implement Requires training data for conditioning

the network before using it for processing other data

Network includes one or more hidden layers

Network is considered a feedforward approach

20

1. Initialize the weights2. Read the input

vector3. Generate the output4. Compute the error

Error = Out - Desired5. Change the weights

Drawbacks: A large network can take a very long time to

train May not converge

BackpropagationBackpropagation

21

Test the network after training Examine network performance: measure the

network’s classification ability Black box testing Do the inputs produce the appropriate outputs? Not necessarily 100% accurate But may be better than human decision makers Test plan should include

Routine cases Potentially problematic situations

May have to retrain

TestingTesting

22

ANN Development ToolsANN Development Tools

NeuroSolutions Statistica Neural Network Toolkit Braincel (Excel Add-in) NeuralWorks Brainmaker PathFinder Trajan Neural Network Simulator NeuroShell Easy SPSS Neural Connector NeuroWare

23

Benefits of ANNBenefits of ANN

Pattern recognition, learning, classification, generalization and abstraction, and interpretation of incomplete and noisy inputs

Character, speech and visual recognition Can provide some human problem-solving

characteristics Can tackle new kinds of problems Robust Fast Flexible and easy to maintain Powerful hybrid systems

24

Limitations of ANNLimitations of ANN

Lack explanation capabilities Limitations and expense of

hardware technology restrict most applications to software simulations

Training time can be excessive and tedious

Usually requires large amounts of training and test data

25

ANN DemonstrationANN Demonstration

www.roselladb.com

NeuroSolutions http://www.nd.com/neurosolutions/products/ns/nnandnsvideo.html by NeuroDimentions, Inc. www.nd.com

DMWizard By Knowledge Based Systems, Inc. Funded by US Army

www.roselladb.com

26

Business ANN ApplicationsBusiness ANN Applications

Accounting Identify tax fraud Enhance auditing by finding irregularities

Finance Signatures and bank note verifications Foreign exchange rate forecasting Bankruptcy prediction Customer credit scoring Credit card approval and fraud detection* Stock and commodity selection and trading Forecasting economic turning points Pricing initial public offerings* Loan approvals …

27

Business ANN ApplicationsBusiness ANN Applications

Human Resources Predicting employees’ performance and behavior Determining personnel resource requirements

Management Corporate merger prediction Country risk rating

Marketing Consumer spending pattern classification Sales forecasts Targeted marketing, …

Operations Vehicle routing Production/job scheduling, …

28

Bankruptcy Prediction with ANNBankruptcy Prediction with ANN

Based on a paper Published in Decision Support Systems, 1994 By Rick Wilson and Ramesh Sharda

ANN Architecture Three-layer (input-hidden-output) MLP Backpropagation (supervised) learning network

Training data Small set of well-known financial ratios Data available on bankruptcy outcomes

Moody’s industrial manual (between 1975 and 1982)

29

Application Design Specifics Five Input Nodes

X1: Working capital/total assets X2: Retained earnings/total assetsX3: Earnings before interest and taxes/total assetsX4: Market value of equity/total debtX5: Sales/total assets

Single Output Node: Final classification for each firm

Bankruptcy or Nonbankruptcy

Development Tool: NeuroShell

Bankruptcy Prediction with ANNBankruptcy Prediction with ANN

30

Bankruptcy Prediction with ANNBankruptcy Prediction with ANN

31

Bankruptcy Prediction with ANNBankruptcy Prediction with ANN Training

Data Set: 129 firms Training Set: 74 firms; 38 bankrupt, 36 not Ratios computed and stored in input files for:

The neural network A conventional discriminant analysis program

Parameters Number of PEs Learning rate and Momentum

Testing Two Ways

Test data set: 27 bankrupt firms, 28 nonbankrupt firms

Comparison with discriminant analysis

32

Results The neural network correctly predicted:

81.5 percent bankrupt cases 82.1 percent nonbankrupt cases

ANN did better predicting 22 out of the 27 cases discriminant analysis predicted only 16 correctly

Error Analysis Five bankrupt firms misclassified by both

methods Similar for nonbankrupt firms

Accuracy of about 80 percent is usually acceptable for this problem domain

Bankruptcy Prediction with ANNBankruptcy Prediction with ANN