START OF DAY 4Reading: Chap. 3 & 4

Project

Topics & Teams

• Select topics/domains• Select teams• Deliverables

– Description of the problem– Selection of objective(s)– Description of the methods used

• Data preparation• Learning algorithms used

– Description of the results• Project presentations

Perceptron

Neural Networks

• Sub-symbolic approach:– Does not use symbols to denote objects– Views intelligence/learning as arising from the

collective behavior of a large number of simple, interacting components

• Motivated by biological plausibility

Natural Neuron

Artificial Neuron

Captures the essence of the natural neuron

• (Dendrites) Input values Xi from the environment or other neurons

• (Synapses) Real-valued weights wi associated with each input

• (Soma’s chemical reaction) Function F({Xi},{wi}) computing activation as a function of input values and weights

• (Axon) Activation value that may serve as input to other neurons

Feedforward Neural Networks

• Sets of (highly) interconnected artificial neurons (i.e., simple computational units)– Layered organization

• Characteristics– Massive parallelism– Distributed knowledge representation (i.e., implicit in

patterns of interactions)– Graceful degradation (e.g., grandmother cell)– Less susceptible to brittleness– Noise tolerant– Opaque (i.e., black box)

There exist other types of NNs

FFNN Topology

• Pattern of interconnections among neurons: primary source of inductive bias

• Characteristics– Number of layers– Number of neurons per layer– Interconnectivity (fully connected, mesh, etc.)

Perceptrons (1958)

• Simplest class of neural networks• Single-layer, i.e., only one set of connection

weights between inputs and outputs• Boolean activation (aka step function)

x1

xn

x2

w1

w2

wn

z

Learning for Perceptrons

• Algorithm devised by Rosenblatt in 1958• Given an example (i.e., labeled input pattern):– Compute output– Check output against target– Adapt weights

Example (I)

.8

.3

z

.4

-.2

.1

net = .8*.4 + .3*-.2 = .26

=1

x1 x2 t

0

1

.1

.3

.4

.8

Output matches target

Example (II)

.8

.3

z

.4

-.2

.1

net = .4*.4 + .1*-.2 = .14

=1

x1 x2 t

0

1

.1

.3

.4

.8

Output does not match target

Learn-Perceptron

• When should weights be changed?– Output does not match target: (ti-zi)

• How should weights be changed?– By some fixed amount (learning rate): c(ti-zi)– Proportional to input value: c(ti-zi)xi

• Algorithm:– Initialize weights (typically random)– For each new training example

• Compute network output• Change weights: Dwi = c(ti – zi)xi

– Repeat until no change in weights

What About Θ?

1 0 1 -> 01 0 0 -> 1

Augmented Version1 0 1 1 -> 01 0 0 1 -> 1

• Treat threshold like any other weight. Call it a bias since it biases the output up or down

• Since we start with random weights anyways, ignore - notion; just think of the bias as an extra available weight

• Always use a bias weight

Example• Assume 3-input perceptron (plus bias, outputs 1 if net > 0, else 0) • Assume c=1 and initial weights all 0• Training set: 0 0 1 -> 0

1 1 1 -> 11 0 1 -> 10 1 1 -> 0

PatternTarget Weight Net OutputDW____

0 0 1 1 0 0 0 0 0 00 0 0 0 0

1 1 1 1 1 0 0 0 0 00 1 1 1 1

1 0 1 1 1 1 1 1 1 31 0 0 0 0

0 1 1 1 0 1 1 1 1 31 0 -1 -1 -1

0 0 1 1 0 1 0 0 0 00 0 0 0 0

1 1 1 1 1 1 0 0 0 11 0 0 0 0

1 0 1 1 1 1 0 0 0 11 0 0 0 0

0 1 1 1 0 1 0 0 0 00 0 0 0 0

Dwi = c(ti – zi)xi

Another Example• Assume 2-input perceptron (plus bias, outputs 1 if net > 0, else 0) • Assume c=1 and initial weights all 0• Training set: 0 0 -> 0

1 0 -> 11 1 -> 00 1 -> 1

PatternTarget Weight Net OutputDW _

0 0 1 0 0 0 0 00 0 0 0

1 0 1 1 0 0 0 00 1 0 1

1 1 1 0 1 0 1 21 0 -1 0

0 1 1 1 0-1 0 -1 00 0 1

0 0 1 0 0 0 1 11 0 0 0

1 0 1 1 0 0 0 00 1 0 1

1 1 1 0 1 0 1 21 0 -1 0

0 1 1 1 0-1 0 -1 00 0 1

Dwi = c(ti – zi)xi

What is happening? Why?

Decision Surface

• Assume 2-input perceptron– z=1 if w1x1+w2x2≥Θ

– z=0 if w1x1+w2x2<Θ

• Decision boundary: w1x1+w2x2=Θ– A line with slope –w1/w2 and intercept Θ/w2

– No bias line goes through origin• In general: hyperplane (i.e., a linear surface)

Linear Separability

Generalization:noise vs. exception

Limited functionality?

The Plague of Linear Separability

• The good news is:– Learn-Perceptron is guaranteed to converge to a correct

assignment of weights if such an assignment exists

• The bad news is:– Such an assignment exists only for linearly separable tasks

• The really bad news is:– There is a large number of non-linearly separable tasks

• Let d be the number of inputs

– Too many tasks escape the algorithm

Are We Stuck?• So far we have used

• What if we preprocessed the inputs in a non-linear way and did

• To the perceptron algorithm it would look just the same, except with different inputs

• For example, for a problem with two inputs x and y (plus the bias), we could also add the inputs x2, y2, and x·y

• The perceptron would just think it is a 5-dimensional task, and it is linear in those 5 dimensions– But what kind of decision surfaces would it allow for the 2-d input space?

Quadric Machine Example• All quadratic surfaces (2nd order)

– ellipsoid– parabola– Etc.

• For example:

-3 -2 -1 0 1 2 3 f1

-3 -2 -1 0 1 2 3

f2

f1

• Perceptron with just feature f1 cannot separate the data

• Assume we add another feature to our perceptron f2 = f12

Quadric Machine

• All quadratic surfaces (2nd order)– ellipsoid– parabola– etc.

• That significantly increases the number of problems that can be solved, but there are still many problems that are not quadrically separable

• Could go to 3rd and higher order features, but number of possible features grows exponentially

• Multi-layer neural networks will allow us to discover high-order features automatically from the input space

Backpropagation

Towards a Solution

• Main problem:– Learn-Perceptron implements discrete model of error (i.e.,

identifies the existence of error and adapts to it, but not the magnitude of the error – since step function)

• First thing to do:– Allow nodes to have real-valued activations (amount of error

= difference between computed and target output)

• Second thing to do:– Design learning rule that adjusts weights based on error

• Last thing to do:– Use the learning rule to implement a multi-layer algorithm

Real-valued Activation

• Replace the threshold unit (step function) with a linear unit, where:

For instance d:

Defining Error

• We define the training error of a hypothesis, or weight vector, by:

• Goal: minimize E– Find direction of steepest ascent of E (aka, gradient)– Move in the opposite direction (i.e., to decrease E)

Minimizing the Error

The Delta Rule

• Gradient descent on the error surface:

• Initialize weights to small random values• Repeat until no progress

– Initialize each wi to 0 – For each training example <x,t>

• Compute output o for x• For each weight wi

– wi wi + c(t – o)xi

– For each weight wi

• wi wi + wi

Stochastic version• For each weight wi

– wi wi + c(t – o)xi

Better?

Note change in sign since we minimize E

Discussion

• Gradient-descent learning (with linear units) requires more than one pass through the training set

• The good news is: – Convergence is guaranteed if the problem is solvable

• The bad news is:– Still produces only linear functions– Even when used in a multi-layer context (composition of

linear functions is a linear function)

• Needs to be further generalized!

Non-linear Activation

• Introduce non-linearity with a sigmoid function:

1. Differentiable (required for gradient-descent)2. Most unstable in the middle

Derivative of the Sigmoid

You need only compute the sigmoid; its derivative comes for free!

Multi-layer Feed-forward NN

i

i

i

i

j

k

k

k

Backpropagation Learning

• Repeat– Present a training instance d– Compute error k of output units– For each hidden layer• Compute error j using error from next layer

– Update all weights: wpq wpq + wpq

where • Until stopping criterion

Note that BP is stochastic

Setting Up the Derivation

Output Units

Hidden Units

Putting it all togetheri

i

i

i

jk

k

k

Max when sigmoid is most unstable

Example (I)

• Consider a simple network composed of:– 3 inputs: x, y, z– 1 hidden node: h– 2 outputs: q, r

• Assume c=0.5, all weights are initialized to 0.2 and weight updates are incremental

• Consider the training set:– 1 0 1 – 0 1– 0 1 1 – 1 1

• 4 iterations over the training set

Example (II)

Local Minima

• FFNN can get stuck in local minimum– More common for small networks– For most large networks (many weights), local minima

rarely occur in practice

Many dimensions of weights unlikely to be in a minima in every dimension simultaneously – almost always a way down (e.g., water running down a high-dimensional surface)

If needed, can use momentum or train several NNs

Momentum

• Simple speed-up modification

• Weight update maintains momentum in the direction it has been going– Faster in flats– Could leap past minima (good or bad)– Significant speed-up, common value ≈ .9– Effectively increases learning rate in areas where the

gradient is consistently the same sign (a common approach in adaptive learning rate methods)

Learning Parameters

• Connectivity: typically fully connected between layers• Number of hidden nodes:

– Too many nodes make learning slower, could overfit– Too few will underfit

• Number of layers: usually 1 or 2 hidden layers which are usually sufficient, attenuation makes learning very slow – 1 most common

• Momentum: (.5 - .99)• Most common method to set parameters: a few trial and error runs

(CV)• All of these could be set automatically by the learning algorithm and

there are numerous approaches to do so

Backpropagation Summary• Most common neural network approach

– Many other different styles of neural networks (RBF, Hopfield, etc.)• Excellent empirical results• Scaling – the pleasant surprise

– Local minima very rare as problem and network complexity increase• User defined parameters usually handled by multiple experiments• Many variants, such as

– Regression – Typically linear output nodes, normal hidden nodes– Adaptive parameters, ontogenic (growing and pruning) learning algorithms– Many different learning algorithm approaches– Recurrent networks– Deep networks– Still an active research area

END OF DAY 4Homework: Decision Tree Learning