Ko, Youngjoong

Dept. of Computer Engineering,

Dong-A University

Introduction of Multilayer Perceptron in Python

1. Non-linear Classification

2. XOR problem

3. Architecture of Multilayer Perceptron (MLP)

4. Forward Computation

5. Activation Functions

6. Learning in MLP with Back-propagation Algorithm

7. Python Code and Practice

2

Contents

Many Impossible Cases to be classified linearly

Linear model: perceptron

Non-linear model: decision tree, nearest neighbor models

Explore to find a non-linear learning model from perceptron

3

Non-linear Classification

Limitation of performance of perceptrons in XOR problem

75% Accuracy

Overcome this limitation by using two perceptrons

4

XOR Problem

Two Steps for Solution

Mapping an original feature space into a new space

Classify in the new space

5

XOR Problem

Example of Multilayer Perceptron as a solution

6

XOR Problem

Multilayer Perceptron (MLP) in Neural Network

To chain together a collection of perceptrons

Two layers (not three layers)

Don’t count the inputs as a real layer

Two layers of trained weights

Each edge corresponds to a different weight

Input -> hidden, hidden ->output

7

Architecture of Multilayer Perceptron

Multilayer Perceptron (MLP) in Neural Network

Input layer, Hidden layer and Output layer

Weights: u and v

8

Architecture of Multilayer Perceptron

Functions in MLP

9

Forward Computation

Other Understanding of MLP Forward Propagation

10

Forward Computation

11

Activation Functions

Hyperbolic tangent function

Popular link function

Differential: its derivative is 1-tanh2(x)

Sigmoid functions

12

Activation Functions

Simple Two-layer MLP

13

Activation Functions

Small Two-layer Perceptron to solve the XOR problem

14

Activation Functions

-1 1

-1

15

Learning in MLP

16

Learning in MLP

zj

17

Learning in MLP

Back-propagation Algorithm

18

Learning in MLP

19

Learning in MLP

zj

20

Learning in MLP

xj

Understanding back-propagation on a simple example

Two layers MLP and No activation function in the output layer

21

Learning in MLP

f3(s)

x1

x2

f4(s)

f5(s)

f2(s)

v1

v2

w1

w2

w3

f1(s)

Understanding back-propagation on a simple example

22

Learning in MLP

Understanding back-propagation on a simple example

23

Learning in MLP

f3(s)

x1

x2

f4(s)

f5(s)

f2(s) v2

w1

w2

w3

f1(s)

e1

e2

e3 e3 = v13e1 + v23e2

Understanding back-propagation on a simple example

24

Learning in MLP

f3(s)

x1

x2

f4(s)

f5(s)

f2(s) v2

w1

w2

w3

f1(s)

e1

e2

e3 e4 = v14e1 + v24e2

e4

Understanding back-propagation on a simple example

25

Learning in MLP

f3(e)

x1

x2

f4(e)

f5(e)

f2(e) v2

w2

w3

f1(e)

e1

e2

e3

e4

e5

Understanding back-propagation on a simple example

26

Learning in MLP

f3(e)

x1

x2

f4(e)

f5(e)

f2(e) v2

w1

w2

w3

f1(e)

e1

e2

e3

e4

e5

Back-propagation

Algorithm

27

Learning in MLP

Simple Example in MLP Learning

28

Learning in MLP

29

Learning in MLP

Back-propagation

Line 8:

Line 9:

Line 10:

30

Learning in MLP

Back-propagation

Line 11:

Line 12: Line 13:

31

Learning in MLP

32

Learning in MLP

Typical complaints

# of layers

# of hidden units per layer

The gradient descent learning rate

The initialization

the stopping iteration or weight regularization

Random Initialization

Small random weights (say, uniform between -0.1 and 0.1)

By training a collection of networks, each with a different random

initialization, we can often obtain better solutions

33

More Considerations

significant

Initialization Tip

34

More Considerations

out

When is the proper number of iteration for early stopping?

35

More Considerations

36

Python Code and Practice

You should install Python 2.7 and Numpy

Download from: http://nlpmlir.blogspot.kr/2016/02/multilayer-

perceptron.html

Homework

37

• 오일석. 패턴인식. 교보문고.

• Sangkeun Jung. “Introduction to Deep Learning.” Natural Language Processing Tutorial,

2015.

• http://ciml.info/

References

Thank you for your attention!

http://web.donga.ac.kr/yjko/

고 영 중