artificial neural networks - ucsbyfwang/courses/cs290i_prann/pdf/ann.pdf · artificial neural...
TRANSCRIPT
![Page 1: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/1.jpg)
Artificial Neural Networks
![Page 2: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/2.jpg)
2PR , ANN, & ML
Artificial Neural Networks
Connectionist, PDP, etc. models
A biologically-inspired approach for
intelligent computing machines
massive parallelism
distributed computing
learning, generalization, adaptivity
Tolerant of fault, uncertainty, imprecise info
![Page 3: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/3.jpg)
3PR , ANN, & ML
Von Neumann computer
Biological neural systems
Processor complex, high speed, few
simple, low speed, many
Memory separate from processor,
non-content addressable
integrated into processor, content addressable
Computing centralized, sequential stored programs
distributed, parallel self-learning
Reliability vulnerable fault tolerant
Compared to Von Neumann
![Page 4: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/4.jpg)
4PR , ANN, & ML
Anatomy and Pathway
Brain
Cerebrum
Frontal, parietal, temporal and occipital lobes
Cerebellum
Brainstem
Spinal Cord
Housed in vertebral column
Receptors to afferent neurons to brain (neocortex)
to efferent neurons to effectors
![Page 5: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/5.jpg)
5PR , ANN, & ML
Central Nervous System
![Page 6: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/6.jpg)
6PR , ANN, & ML
CNS
Neocortex
Gray matter (surface layer of cerebrum)
White matter (connection fiber)
A crumbled paper analogy (folded and refolded many times to fit)
Regions (lobes) for different functions
It is the
Activities of neurons (resting or depolarization)
Topology of the connection
Strength and direction (forward & backward) of connection
In the cortex that defines intelligence
![Page 7: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/7.jpg)
7PR , ANN, & ML
Biological Neural Networks soma (cell body)
dendrites (receivers)
axon (transmitters)
synapses (connection points, axon-soma, axon-dendrite, axon-axon)
Chemicals (neurotransmitters)
neurons
each makes about connections
with an operating speed of a few milliseconds
one-hundred-step rule
1011
10 103 4~
Axon hillock
![Page 8: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/8.jpg)
8PR , ANN, & ML
Different Neuron Shapes
Bipolar
E.g., found in eyes (retinal cells)
Pseudounipolar
Two axons one to spinal cord one to skin and
muscle
Multipolar
Axons + dendrites
Anaxonic
![Page 9: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/9.jpg)
9
Different Neuron Functions
PR , ANN, & ML
![Page 10: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/10.jpg)
10PR , ANN, & ML
Glia Support cells
Clean up brain debris
Transport nutrients to neurons
Hold neurons in place
Digest dead neurons
Regulate content of extracellular space
Insulation (myelin)
Difference
Star shaped – no axons
No action potential
No synapses
A lot more (10 to 50 times more)
![Page 11: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/11.jpg)
11PR , ANN, & ML
Signal Generation Resting potential
Charge difference across neuron membrane approximately –70mV
Graded potential
Stimulus across synapses of post-synaptic neuron
Action potential
If accumulation of graded potential across neuron membrane over a short period of time is higher than ~15mV, action potential is generated and propagated across axon
Same form and amplitude regardless of stimulus, signal by frequency rather than amplitude
![Page 12: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/12.jpg)
12PR , ANN, & ML
Signal Generation
15mm
![Page 13: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/13.jpg)
13PR , ANN, & ML
Signal Generation Resting
A-: protein
K+: potassium
Can go out
Cl-: cloride
Na+: sodium
Cannot go out
Excited
•Think about electrical circuit
•Flow one way due to charge (concentration) difference
•Flow the other way using cell’s ion pumps (battery)
![Page 14: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/14.jpg)
14
Summary
Two states of a neuron
Resting: negative
Firing: positive (depolarize)
Depolarization triggered by threshold potential
Opening of sodium channels to let outside Na+ in
Opening of Potassium channels to let inside K+ out
Ion pumps on the membrane will then
rebalance (energy – ATP – required)
3 Na+ out for 2 K+ in PR , ANN, & ML
![Page 15: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/15.jpg)
15PR , ANN, & ML
Signal Flow in Dendritic Trees and Axons
Flow is usually one dimensional
Longitudinal flow (little transverse flow) with no loss (active transmission line)
Myelindated
Wrapped in sheath of myelin, 100m/s
Unmyelinated
1m/s
![Page 16: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/16.jpg)
16
synapses
Electrical
Fixed, simultaneous
control
Chemical
Pattern and strength
can be learned and
adjusted
PR , ANN, & ML
![Page 17: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/17.jpg)
17PR , ANN, & ML
Signal Flow Across Synapses
Action potential causes release of
neurotransmitters from synaptic vesicles
~100 different neurotransmitters, e.g.,
dopamine, serotonin, and acetylcholine
The release, diffusion and reception of
neurotransmitters cause delay of 0.5 to 2ms
Synaptic plasticity: either facilitate or inhibit
chemical synapses
![Page 18: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/18.jpg)
18PR , ANN, & ML
Signal Flow Across Synapses
![Page 19: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/19.jpg)
19PR , ANN, & ML
Connection Patterns
Divergent (fan-out):
parallel processing, afferent neurons
Convergent (fan-in):
Efferent neurons to effectors
Chain and loop
![Page 20: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/20.jpg)
20PR , ANN, & ML
Connection Patterns
![Page 21: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/21.jpg)
21PR , ANN, & ML
A single neurons with thousands of synapses (light yellow dots)
![Page 22: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/22.jpg)
22PR , ANN, & ML
Computational Neuron Model
(McCulloch and Pitts)
x2
xn
x1 w1
w2
wnO w x ui i
i
n
( )1
u
11),tanh()(
10,1
1)(
xx
ex
x
- time dependency?
- frequency response?
![Page 23: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/23.jpg)
23PR , ANN, & ML
Computational Network Architecture
Single-layer Perceptron Multi-layer Perceptron
Feed-Forward Networks
Self-organizaing Map Hopfield Network ARTmodels
Recurrent Networks
Neural networks
1 2
![Page 24: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/24.jpg)
24PR , ANN, & ML
ANN Formulation
Learning rules – Basic “workhorse”
mechanism in adjusting weights of neurons
Error-correcting learning (gradient descent)
Memory-based learning (nearest neighbor)
Hebbian learning (mutual excitation)
Competitive learning (winner-take-all)
Boltzmann learning (statistical mechanics)
![Page 25: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/25.jpg)
25PR , ANN, & ML
ANN Formulation (cont.)
Learning paradigms – the big picture
Supervised (learning w. a teacher): correct I/O
association is provided
Unsupervised (learning w/o. a teacher): discover
similarity, inherent structure, and interesting patterns
Delayed (learning w. a critic):
Theory
capacity: how many patterns can be stored
sample complexity: how many training patterns
computational complexity: training time
![Page 26: Artificial Neural Networks - UCSByfwang/courses/cs290i_prann/pdf/ANN.pdf · Artificial Neural Networks ... PR , ANN, & ML 3 Von Neumann computer Biological neural systems Processor](https://reader030.vdocument.in/reader030/viewer/2022040609/5ecd25b45f998d2c8838df76/html5/thumbnails/26.jpg)
26PR , ANN, & ML
Relation to Pattern Recognition
Supervised mode
Single & multi-layer perceptrons for learning
complicated decision boundaries
Unsupervised mode
Competitive and self-organization maps for
constructing clusters