introduction cs/cmpe 537 – neural networks. cs/cmpe 537 - neural networks (sp 2004/2005) - asim...
Post on 19-Dec-2015
228 views
TRANSCRIPT
![Page 1: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/1.jpg)
Introduction
CS/CMPE 537 – Neural Networks
![Page 2: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/2.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 2
Biological Inspiration
The brain is a highly complex, nonlinear, and parallel computer Simple processing units called neurons Cycle times in milliseconds Massive number of neurons (estimated at 1011 in humans) Massive interconnection (estimated at 6012 connections)
The brain can perform certain computations (e.g. pattern recognition, perception, etc) many times faster than the fastest digital computers available today
![Page 3: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/3.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 3
Comparison of the Brain and Digital Computer
Functionality comparison
Brain Digital computer
Fault tolerant Intolerant to errors
Adaptive Preprogrammed
Learn-ability Preprogrammed
Trained Designed
Nonlinear Linear, primarily
![Page 4: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/4.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 4
Comparison of the Brain and Digital Computer
Structural comparison
Brain Digital computer
Simple processing unit Complex processing unit
Large number of units Lesser number of units
Massive interconnection Little or no interconnection
Dynamic Static
Slow switching Faster switching
![Page 5: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/5.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 5
What is an Artificial Neural Network? (1)
Conceptual definitions An artificial neural network (neural network) is a
machine that is designed to model the way in which the brain performs a particular task or function of interest; the network is either implemented in hardware or simulated in software on a digital computer.
A neural network mimics the brain or nervous system. In what sense? In structure (simple processing units, massive
interconnection, etc) In functionality (learning, adaptability, fault tolerance, etc)
![Page 6: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/6.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 6
What is an Artificial Neural Network? (2)
A pragmatic definition A neural network is a massively parallel distributed
computing system that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: Knowledge is acquired by the network through a learning
process (called training) Interneuron connection strengths known as synaptic weights
are used to store the knowledge Other names for neural networks
Neurocomputers Connectionist networks Parallel distributed processors
![Page 7: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/7.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 7
Significance of Artificial Neural Networks
Why study neural networks? To develop artificial systems that posses human-like
characteristics (thought and action) Neural computation
To understand the working of the brain Computational neuroscience
These two fields are not mutually exclusive. Knowledge gained in each helps developments in the other.
Our primary concern will be neural computation.
![Page 8: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/8.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 8
Characteristics of Neural Networks
Nonlinearity Input-output mapping Adaptivity Evidential response Contextual information Fault tolerance VLSI implementability Uniformity of analysis and design Neurobiological analogy
![Page 9: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/9.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 9
Structural Levels of Organization in the Brain
Conceptual organization of the nervous system Receptors Neural network Effectors
![Page 10: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/10.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 10
Structural Levels in the Brain
![Page 11: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/11.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 11
Biological Neuron
![Page 12: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/12.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 12
Biological Neuron
Cell body – simple processing unit Axon – output link Dendrite – input link Synapse – connection between neuron
![Page 13: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/13.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 13
Neuron Model (1)
Bias bk
vk
![Page 14: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/14.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 14
Neuron Model (2)
Properties A set of signals x1 to xp
A set of connection weights w associated with each of the p connecting links (synapses)
A processing unit (or neuron) k that performs the weighted summation of the inputs (also known as adder)
An activation function φ(.) that limits the amplitude of the output, usually to the range {0, 1} or {-1, 1} (also known as squashing or transfer function)
A bias bk that has the effect of lowering or increasing the net input to the activation function
![Page 15: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/15.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 15
Neuron Model (3)
Mathematically…(using a threshold)
uk = Σj=1p wkjxj (linear combiner)
yk = φ(uk + bk)
Mathematically…(using a bias unit)
vk = Σj=0p wkjxj (linear combiner)
yk = φ(vk)
where vk = uk + bk
and x0 = +1 and wk0 = bk
![Page 16: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/16.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 16
Activation Functions (1)
Threshold function
![Page 17: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/17.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 17
Activation Functions (2)
Piecewise linear function
![Page 18: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/18.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 18
Activation Functions (3)
Sigmoid function
![Page 19: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/19.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 19
Signal-Flow Graph Representation
A cleaner (simpler) representation that captures the various elements of a neural network and the flow of information (signals) Directed graph – network of
arrows and nodes
Rules for signal transmission
![Page 20: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/20.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 20
Signal-Flow Graph Representation of Neuron
![Page 21: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/21.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 21
Neural Net Definition – Signal-Flow Rep.
A neural net is a directed graph consisting of nodes with interconnecting synaptic and activation links, and which is characterized by four properties: Each neuron is represented by a set of linear synaptic links,
an externally applied threshold, and a nonlinear activation link
The synaptic links of a neuron weight their respective input signals
The weighted sum of the input signals defines the total internal activity level of the neuron in question
The activity link squashes the internal activity level of the neuron to produce an output
![Page 22: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/22.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 22
Architectural Graph Representation
A signal-flow graph that omits details of flow inside the neuron A partially complete directed graph
![Page 23: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/23.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 23
Classification of Neural Networks (1)
Number of layers Single layer network Multilayer networks
Direction of information (signal) flow Feedforward Recurrent
Connectivity Fully connected Partially connected
![Page 24: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/24.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 24
Classification of Neural Networks (2)
Activation function Threshold networks Linear networks Nonlinear networks Radial-basis function networks
Learning methodology Supervised Unsupervised Reinforcement
Training algorithm Static Dynamic
![Page 25: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/25.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 25
Example
![Page 26: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/26.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 26
Example
![Page 27: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/27.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 27
Example
![Page 28: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/28.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 28
Example
![Page 29: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/29.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 29
Example
![Page 30: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/30.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 30
Design Decisions
Architecture No. of layers, units, links; connectivity Activation functions
Learning Supervised, unsupervised, competitive
Training algorithm Iterative, static, dynamic, epoch-based, sample-based
![Page 31: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/31.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 31
Knowledge Representation (1)
Neural networks encode and make available knowledge for information processing (knowledge representation)
What is knowledge? Information or models used to interpret, predict and
appropriately respond to the outside world (environment) How can knowledge be represented?
Explicit – what is made explicit Physically encoded – how can is physically encoded
The quality of knowledge representation generally translates into the quality of the solution –better representation -> better solution
![Page 32: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/32.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 32
Knowledge Representation (2)
Knowledge is encoded in the free parameters of the network. Assuming the architecture is fixed, the free parameters are…. Weights and bias values
Knowledge representation in neural nets is complex Few theories exist that relate a given weight, for example, to
a particular piece of information
Hmm, so neural nets are worthless? Nope!
![Page 33: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/33.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 33
Learning
Learning: to acquire and maintain knowledge of interest
Knowledge of interest: knowledge of environment that will enable the machine (neural net) to achieve its goals Prior information Current information
Knowledge can be built into neural networks from input-output examples by a automatic process of learning (commonly known as training) Training algorithms
![Page 34: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/34.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 34
Some General Knowledge Rep. Rules
Rule 1: Similar inputs from similar classes should usually produce similar representations inside the network, and should therefore be classified as belonging to the same category
Rule 2: Inputs to be characterized as separate classes should be given widely different representations in the network
Rule 3: If a particular feature is important, then there should be a large number of neurons involved in the representation of that item in the network
Rule 4: Prior knowledge and invariances should be built into the design of the network, thus simplifying learning
![Page 35: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/35.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 35
Building-in Prior Knowledge
Advantages Specialized structure Benefits of having a specialized structure – biologically
plausible, less complication, fewer free parameters, faster training, fewer examples needed, better generalization, etc
How to build-in prior knowledge? No hard and fast rules In general, use domain knowledge to reduce complexity of
neural network based on what we know about their performance characteristics
![Page 36: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/36.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 36
Building-in Invariance
Invariance? Fault tolerance Immunity to transformations
Invariance by structure Invariance by training Invariant feature space
![Page 37: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/37.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 37
Building-in Current Knowledge
Learning and training algorithms Subject of next 2 lectures (chapter 2)
![Page 38: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/38.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 38
An Example
A neural net for signature verification Prior knowledge
Architecture
Current knowledge Input-output examples
Learning Modification of the free parameters (weights, biases, etc)
Generalization Using the trained network for prediction
![Page 39: Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly](https://reader035.vdocument.in/reader035/viewer/2022062221/56649d375503460f94a105dc/html5/thumbnails/39.jpg)
CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 39
Application Areas of Neural Networks
Model estimation Interpolation Extrapolation
Pattern classification Lots of examples
Signal processing Noise reduction, enhancement, echo removal
Optimization Lots of examples