Neural Networks

Dr. Randa Elanwar

Lecture 1

Lecture Content

• Introduction and a historical review:

– History of neurocomputing.

– Overview of neurocomputing.

2Neural Networks Dr. Randa Elanwar

History of Neurocomputing

• Alexander Bain (1873) claimed that both thoughts and body activity resulted from interactions among neurons within the brain.

• For Bain, every activity led to the firing of a certain set of neurons. When activities were repeated, the connections between those neurons strengthened. According to his theory, this repetition was what led to the formation of memory.

• The general scientific community at the time was doubting Bain’s theory because it required what appeared to be an excessively large number of neural connections within the brain.

3Neural Networks Dr. Randa Elanwar

History of Neurocomputing

• It is now apparent that the brain is exceedingly complex and that the same brain “wiring” can handle multiple problems and inputs.

• Neural network theory has served both to:

– better identify how the neurons in the brain function and

– provide the basis for efforts to create artificial intelligence.

4Neural Networks Dr. Randa Elanwar

Overview of Neurocomputing

• The brain is said to be composed of natural neural network, I.e., mass of highly non-linear parallel inter-connected computational units called neurons.

• Each neuron is connected to many other neurons

• Neurons transmit signals to each other

• Whether a signal is sent, depends on the strength of the bond between two neurons

• Whether a signal is transmitted is an all-or-nothing event (the electrical potential in the cell body of the neuron is thresholded: fired up/ activated)

5Neural Networks Dr. Randa Elanwar

Brain vs. Digital Computers

- Computers require hundreds of cycles to simulate a firing of a neuron.

- The brain can fire all the neurons in a single step. Parallelism

- Serial computers require billions of cycles to perform some tasks but the brain takes less than a second.

e.g. Face Recognition

6Neural Networks Dr. Randa Elanwar

Overview of Neurocomputing

7Neural Networks Dr. Randa Elanwar

Overview of Neurocomputing

Neural Networks Dr. Randa Elanwar 8

Overview of Neurocomputing

What are Neural Networks

• Neural Networks (NNs) are networks of neurons, for example, as found in real (i.e. biological) brains.

• Artificial Neurons are crude approximations of the neurons found in brains. They may be physical devices, or purely mathematical constructs.

• Artificial Neural Networks (ANNs) are networks of Artificial Neurons, and hence constitute crude approximations to parts of real brains. They may be physical devices, or simulated on conventional computers.

Neural Networks Dr. Randa Elanwar 9

Overview of Neurocomputing

What are Neural Networks

• From a practical point of view, an ANN is just a parallel computational system, consisting of many simple processing elements, connected together in a specific way in order to perform a particular task, which is difficult to traditional (serial) computers.

• One should never lose sight of how crude the approximations are, and how over-simplified our ANNs are compared to real brains.

Comparison between brain verses ANN

Neural Networks Dr. Randa Elanwar 10

Brain ANN

Speed Few ms. Few nano sec. massive parallel

processing

Size and

complexity

1011 neurons & 1015

interconnections

Depends on designer

Storage

capacity

Stores information in its

interconnection (synapse)

No Loss of memory

Contiguous memory locations

loss of memory may happen

sometimes.

Tolerance Has fault tolerance Limited fault tolerance:

Information gets disrupted when

interconnections are disconnected

Control

mechanism

Complicated involves

chemicals in biological neuron

Simpler in ANN

Neural Networks Dr. Randa Elanwar 11

NON-LINEARITYIt can model non-linear systems

INPUT-OUTPUT MAPPINGIt can derive a relationship between a set of input & output responses

ADAPTIVITYThe ability to learn allows the network to adapt to changes in the surrounding environment

EVIDENTIAL RESPONSEIt can provide a confidence level to a given solution

Advantages of NN

Neural Networks Dr. Randa Elanwar 12

CONTEXTUAL INFORMATIONKnowledge is presented by the structure of thenetwork. Every neuron in the network ispotentially affected by the global activity of allother neurons in the network. Consequently,contextual information is dealt with naturally inthe network.

FAULT TOLERANCEDistributed nature of the NN gives it fault tolerant capabilities

NEUROBIOLOGY ANALOGYModels the architecture of the brain

Advantages of NN

Who uses NN

13Neural Networks Dr. Randa Elanwar

Neuro products and application areas

14Neural Networks Dr. Randa Elanwar

Neural Networks Dr. Randa Elanwar 15

How does the nervous system works?

• The human nervous system can be broken down into three stages that may be represented in block diagram form as:– The receptors collect information from the environment – e.g. photons on the retina.– The effectors generate interactions with the environment – e.g. activate muscles.– The flow of information/activation is represented by arrows – feed forward and

feedback.

Overview of Neurocomputing

Overview of Neurocomputing

The structure of a biological neuron

Neural Networks Dr. Randa Elanwar 16

•A biological neuron has three types of main components; dendrites, soma (or cell body) and axon.

•Dendrites receives signals from other neurons.

•The soma, sums the incoming signals. When sufficient input is received, the cell fires; that is it transmit a signal over its axon to other cells.

17

Neural network: Definition

• Neural network: information processing paradigm inspiredby biological nervous systems, such as our brain

• Structure: large number of highly interconnectedprocessing elements (neurons) working together

• Like people, they learn from experience (by example)

Neural Networks Dr. Randa Elanwar

Artificial Neural Network: Definition

• The idea of ANN: NNs learn relationship between cause and effect or organize large volumes of data into orderly and informative patterns.

• Definition of ANN: “Data processing system consisting of a large number of simple, highly interconnected processing elements (artificial neurons) in an architecture inspired by the structure of the cerebral cortex of the brain”

(Tsoukalas & Uhrig, 1997).

18Neural Networks Dr. Randa Elanwar

Why Study Artificial Neural Networks?

• Artificial Neural Networks are powerful computational systems consisting of many simple processing elements connected together to perform tasks analogously to biological brains.

• They are massively parallel, which makes them efficient, robust, fault tolerant and noise tolerant.

• They can learn from training data and generalize to new situations.

• They are useful for brain modeling and real world applications involving pattern recognition, function approximation, prediction, …

19Neural Networks Dr. Randa Elanwar

What are Artificial Neural Networks Used for?

• Brain modeling– Models of human development – help children with developmental problems– Simulations of adult performance – aid our understanding of how the brain

works– Neuropsychological models – suggest remedial actions for brain damaged

patients

• Real world applications– Financial modeling – predicting stocks, shares, currency exchange rates– Other time series prediction – climate, weather, airline marketing tactician– Computer games – intelligent agents, backgammon, first person shooters– Control systems – autonomous adaptable robots, microwave controllers– Pattern recognition – speech recognition, hand-writing recognition, sonar

signals– Data analysis – data compression, data mining– Noise reduction – function approximation, ECG noise reduction– Bioinformatics – protein secondary structure, DNA sequencing

20Neural Networks Dr. Randa Elanwar