What I Did On My Summer Vacation: Undergraduate

Research Internships, Neural Networks, & Airport Security

J. McLean Sloughter

“Soon after the electrical current became known many attempts were made by the older physiologists to explain nervous impulses in terms of electricity. The analogy between the nerves of the body and a system of telephone or telegraph wires was too striking to be overlooked.”(from Studies in Advanced Physiology, Louis J. Rettger, A.M., 1898, p. 443)

2

An Extremely Over-Simplified Explanation The brain is made up of interconnected

neurons Neurons are binary – either fire or don’t fire As a neuron receives signals from other

neurons, it will start firing if the total signal reaches some threshhold

How

the

Bra

in W

orks

3

How

the

Bra

in W

orks

4

Just like that, except way more complicated Actually a lot more neurons involved Frequency of firing is also important

But let’s ignore those details for now…

How

the

Bra

in W

orks

5

Putting a philosophy degree to work

Warren McCulloch, a psychologist and philosopher, postulated that thought is discrete

Suggested a “psychon” – the smallest unit of thought

Thought that an individual neuron firing or not firing might be a psychon

Recommended developing a “calculus of ideas” to describe neural activity

His

tory

– 1

940s

6

Philosophy + Math = Fame

McCulloch teamed up with Walter Pitts, a math prodigy

Together they published “A Logical Calculus of the Ideas Immanent in Nervous Activity”

This paper introduced the idea of a “nervous network,” the first artificial neural model of cognitionH

isto

ry –

194

0s

7

Enter von Neumann

Von Neumann became an early proponent of their work

However, he criticized it as being overly simplistic

Based on some of von Neumann’s suggestions, McCulloch & Pitts proposed a system using a large number of neurons

This allows for robustness – an ability, for example, to recognize a slightly deformed square as still being essentially a square

His

tory

– 1

940s

8

Best Mathematician Name Ever

Norbert Weiner (“The Father of Cybernetics”) proposed a more involved system

Weighted inputs – one neuron can be more influential than another

Memory = learning weights Did not propose how this learning takes

place, dismissed that as a problem for engineers to deal with

His

tory

– 1

940s

9

In which not a whole lot happened

Marvin Minsky introduced a system based on behavioural conditioning

Neurons had probabilities of sending signals

When they produced the correct output, probabilities were increased

When the produced the wrong output, probabilities were decreased

And nobody really seemed to care (they were all busy becoming computer programmers)

His

tory

– 1

950s

10

Perceptrons

In 1960, Rosenblatt published a proof of the capabilities of what he named the “perceptron”

The perceptron acted much like the nervous network, but with weighted signals

The major advance was a learning algorithm Rosenblatt was able to prove that, using his

learning algorithm, any possible configuration of the perceptron could be learned, given the proper training data

His

tory

– 1

960s

11

Perceptron function

Consider a simple case where nodes A and B are each sending signals to node B

Node B has some threshold, T, which it needs to receive to be activated

A, B, and C are all binary – 0 or 1 W1 and W2 are the weights between A and C

and B and C Then, if A*W1 + B*W2 > T, C = 1 Otherwise, C = 0

His

tory

– 1

960s

12

Perceptron learning

Initialize weights randomly Set threshold to some arbitrary value (why does it not

matter what value the threshold is set to?) Randomly select one set of inputs Find the result based on current weights Subtract result from desired result = error term Look at each initial node individually

Multiply input value by error term by “learning coefficient” (between 0 and 1, controls amount of change you’ll allow at each iteration)

Add result to weight previously associated with that node to get a new weight

Pick a new set of inputs, repeat until convergence

His

tory

– 1

960s

13

Adaline

Widrow and Hoff created a system called Adaline – “Adaptive linear element”

Very similar to perceptrons (though with a slightly different learning algorithm)

Major changes were the use of -1 instead of 0 for no signal, and a “bias” term – a node that always fires

These were significant because they had no basis in neurophysiology, and were added purely because they could improve performance

His

tory

– 1

960s

14

The Wrath of Minksy

In 1969, Minsky again entered the world of neural networks, this time co-authoring the book “Perceptrons” with Seymour Papert

His

tory

– 1

960s

15

Xor

Minsky and Papert showed, among other critiques of perceptrons, that they weren’t capable of learning an exclusive OR (can you see why?)

An exclusive OR could be made by combining multiple other networks – have A and B feed into both an OR and a NAND, and then AND the results

But learning rules only worked with a single layer network – Minskey and Papert suggested researching whether learning rules could be developed for multi-layered networks

His

tory

– 1

960s

16

The Problem

Minsky & Papert put their critique of perceptrons at the front of the book

They put their suggestions for research into multi-layered perceptrons at the back of the book, after a few hundred pages of rather dense math

People didn’t seem to read that far Research on perceptrons died

His

tory

– 1

960s

17

Nothing important happenedH

isto

ry –

197

0s

18

The Multi-Layer Perceptron

Rumelhart, Hinton, and Williams created a learning algorithm for multi-layer perceptrons

Requires differentiation of functions, and thus the hard threshold had to be replaced by a sigmoid function

His

tory

– 1

980s

19

MLP function

Net input to a node:

Output from a node:

His

tory

– 1

980s

j

1

ijxw

n

j

I

eI

If

1

1)(

20

MLP learning

Change weight as follows:

Where is the learning coefficient, and E is the error term:

where

His

tory

– 1

980s

)(wij IEf

actualdesiredoutput yyE

n

j

outputj

imiddlei E

dI

IdfE

1

ijw)(

))(1)(()(

IfIfdI

Idf

21

The Problem

Metal detectors only detect things that are, well, metal (and even then only sometimes)

Lots of bad things aren’t metal – plastic explosives, ceramic guns, plastic flare guns

An x-ray could potentially see these objects, but submitting people to x-rays every time they fly isn’t an especially good ideaA

irpo

rt S

ecur

ity

22

The Solution

Scientists at Pacific Northwest National Laboratory developed a millimeter wave camera

Millimeter waves are not harmful like x-rays They can penetrate clothing, but are reflected

by skin Plastics and ceramics show up with a

distinctive speckled pattern, as they only partially reflect the waves

Air

port

Sec

urit

y

23

The New Problem Caused by the Solution Scientists at a

government lab just made a camera that can take pictures of you through your clothes

Implementing this in airports would have every passenger go through a virtual strip-search

Air

port

Sec

urit

y

24

The Solution to the Problem Caused by the Solution to the Other Problem Rather than have a human operator look at the

pictures, we can have a computer look at them for us The computer can identify suspicious areas and

provide a non-naughty picture to the security officer

Air

port

Sec

urit

y

25

In Practice

This technology is now in use by SafeView, a company spun off from this project

It is being used in airports, government buildings, border crossings, and other locations around the world

Air

port

Sec

urit

y

26

Student Research Opportunities I was involved in this project while a student intern at Pacific

Northwest National Lab Information about PNNL’s student internship programs can be

found online at http://science-ed.pnl.gov/students/ One of my summers on this project, I applied through the

Department of Energy’s internship program, which includes opportunities at a number of other national labs

Information on DOE internship programs is available at http://www.scied.science.doe.gov/scied/erulf/about.html

Res

earc

h In

tern

ship