![Page 1: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/1.jpg)
Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and Robert Zucker
Human Visual System Neural Network
![Page 2: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/2.jpg)
The Visual System
• Hubel and Wiesel• 1981 Nobel Prize for work in early 1960s
• Cat’s visual cortex
• cats anesthetized, eyes open with controlling muscles paralyzed to fix the stare in a specific direction
• thin microelectrodes measure activity in individual cells
• cells specifically sensitive to line of light at specific orientation
• Key discovery – line and edge detectors in the visual cortex of mammals
![Page 3: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/3.jpg)
The Study
• Compare Two Neural Networks• One without vertical and horizontal line detectors
• One with vertical and horizontal line detectors
• Objective• Show that the neural network with line detectors
is superior to the one without on the six vertical-horizontal line-segment letters E, F, H, I, L, T
• Also, experiment with the full alphabet• Without line detectors
![Page 4: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/4.jpg)
Uppercase 5x7 Bit-map AlphabetHorizontal-vertical line-segment letters are E, F, H, I, L, T
![Page 5: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/5.jpg)
Neural Network Without Line Detectors
![Page 6: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/6.jpg)
Neural Network SpecificationWithout Line Detectors
Layers
1. Input layer: 20x20 retina of binary units
2. Hidden layer: 50 units (other numbers explored)
3. Output layer: 6 units for letters E, F, H, I, L, T
Weights
• 20,000 (400x50) between input and hidden layer
• 300 (50x6) between hidden and output layer
• Total of 20,300 variable weights, no fixed weights
![Page 7: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/7.jpg)
Neural Network With Line Detectors
![Page 8: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/8.jpg)
Neural Network SpecificationWith Vertical and Horizontal Line Detectors
Layers
1. Input layer: 20x20 retina of binary units
2. 576 simple vertical and horizontal line detectors
3. 48 complex vertical and horizontal line detectors
4. Hidden layer: 50 units (other numbers explored)
5. Output layer: 6 units for letters E, F, H, I, L, T
Weights
• 6336 (576x11) fixed weights from input to simple detectors
• 576 fixed weights from simple and complex detectors
• 2400 (48x50) variable weights from complex detectors to hidden layer
• 300 (50x6) variable weights from hidden to output layer
• Total of 6912 fixed weights and 2700 variable weights
![Page 9: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/9.jpg)
DETECTORS OVERLAP COVERING EACH POSSIBLE RETINAL POSITION FOR A TOTAL OF 288 (18x16) VERTICAL LINE DETECTORS
EACH DETECTOR HAS 5 EXCITATORY AND 6 INHIBITORY INPUTS (11 FIXED WEIGHTS),WITH A THRESHOLD OF 3
Vertical Line Detectors
Horizontal Line Detectors are Similar
![Page 10: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/10.jpg)
Retina Image – Letter “E” in Upper Left Area
Region of possible upper-left corners is shown in green.
![Page 11: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/11.jpg)
Retina Image – Letter “E” in Upper Right Area
Region of possible upper-left corners is shown in green.
![Page 12: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/12.jpg)
Retina Image – Letter “E” in Lower Right Area
Region of possible upper-left corners is shown in green.
![Page 13: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/13.jpg)
Example of Vertical Line Detectoron Line Segment of “E” – Detector Activated
![Page 14: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/14.jpg)
Example of Shifted Vertical Line Detectoron Letter “E” – Detector Not Activated
![Page 15: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/15.jpg)
Example of Shifted Vertical Line Detectoron Letter “E” – Detector Not Activated
![Page 16: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/16.jpg)
24 Vertical Complex Line Detector RegionsAny Simple Line Detector in a RegionActivates the Complex Line Detector
![Page 17: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/17.jpg)
24 Horizontal Complex Line Detector RegionsAny Simple Line Detector in a RegionActivates the Complex Line Detector
![Page 18: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/18.jpg)
Complex Horizontal and Vertical Line Detector Matrix
The Corresponding 48 Complex Horizontal and Vertical Line Detectors
![Page 19: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/19.jpg)
Experiments
Experiment 1o 6 Line-Segment Letters without Line Detectorso 26 Letters without Line Detectors
Experiment 2o 6 Line-Segment Letters with Line Detectors
![Page 20: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/20.jpg)
Experimental Parameter Combinations• Epochs:
50
100
200
400
800
1600
32000 (occasionally)
• Hidden Layer Units:
10
18*
50
100
200 *
300*
500*
*Selected cases
• Noise:
0%
2%
5%
10%
15%
20%
![Page 21: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/21.jpg)
Experiment 1
Simulation View – Peltarion’s Synapse Product
![Page 22: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/22.jpg)
• Weight Layer: Forward Rule: No rule Back Rule: Levenberg-
Marquardt Propagator: Weight Layer
• Function Layer:
• Function: Tanh Sigmoid Forward Rule: No rule Back Rule: Levenberg-
Marquardt Propagator: Function Layer
Simulation Settings Experiment 2 – Line Detectors
6 Line-Segment Letters: E, F, H, I, L, T
![Page 23: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/23.jpg)
6 Letters: no line detectors
50 Hidden layer units
50 Epochs
0% noise
Exp 1 – 6 Letters, No Line Detectors – 35.42% Accuracy
![Page 24: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/24.jpg)
6 Letters: no line detectors
50 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 6 Letters, No Line Detectors – 36.25% Accuracy
![Page 25: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/25.jpg)
6 Letters: with line detectors
50 Hidden layer units
50 Epochs
0% noise
Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy
![Page 26: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/26.jpg)
6 Letters: with line detectors
50 Hidden layer units
50 Epochs
0% noise
Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy
![Page 27: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/27.jpg)
6 Letters: no line detectors
10 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 6 Letters, No Line Detectors – 27.69% Accuracy
![Page 28: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/28.jpg)
6 Letters: with line detectors
10 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 6 Letters, With Line Detectors – 82.5% Accuracy
![Page 29: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/29.jpg)
6 Letters: with line detectors
10 Hidden layer units
1600 Epochs
0% noise
Exp 2 – 6 Letters, With Line Detectors – 82.5% Accuracy
![Page 30: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/30.jpg)
26 Letters: no line detectors
50 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy
![Page 31: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/31.jpg)
26 Letters: no line detectors
50 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy
![Page 32: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/32.jpg)
Percent Noise
Epochs 0% 2% 5% 10% 15% 20%
50 35.42 20.83 24.17 20 20.42 20.83
100 36.25 21.67 25 18.75 20.83 20.42
200 35.42 21.67 23.75 18.75 20.83 20.83
400 36.25 22.50 23.75 20.42 20.42 20.83
800 36.67 22.50 23.75 20.42 20.00 20.83
1600 36.25 18.33 23.75 20.42 20.42 20.83
Exp 1 – 6 Letters, No Line DetectorsEpochs versus Percent Added Noise
![Page 33: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/33.jpg)
Exp 1 – 26 Letters, No Line DetectorsEpochs versus Percent Added Noise
Percent Noise
Epochs 0% 2% 5% 10% 15% 20%
50 23.08 8.17 7.21 7.50 6.25 6.83
100 23.56 9.04 7.69 7.31 6.15 4.62
200 25.00 9.81 7.60 5.58 6.06 4.52
400 27.12 9.81 6.92 5.02 5.87 4.62
800 28.37 8.96 6.92 5.48 5.58 5.19
1600 27.69 10.10 7.31 5.67 7.21 6.15
![Page 34: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/34.jpg)
Exp 2 – 6 Letters, With Line DetectorsEpochs versus Percent Added Noise
Epochs 0% 2% 5% 10% 15% 20%
50 67.50 58.75 46.25 56.67 16.67 42.50
100 72.92 59.17 46.25 56.67 17.08 28.00
200 77.08 59.58 50.00 62.50 48.75 43.75
400 75.83 58.33 50.00 62.92 34.58 28.33
800 78.33 60.42 52.08 59.17 32.92 57.08
1600 83.33 59.17 56.67 60.00 35.00 32.08
3200 85.00 66.67 56.67 57.92 52.50 62.92
![Page 35: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/35.jpg)
Comparison of Line / No-Line Detector Networks6 letters, 50 hidden layer units, 1600 epochs, no noise
Experiment PerformanceFixed
WeightsVariable Weights
Total Weights
No Line Detectors 36.3% 0 20,300 20,300
Line Detectors 83.3% 6,912 2,700 9,612
![Page 36: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/36.jpg)
• Character recognition performance and efficiency of the neural network using Hubel-Wiesel-like line detectors in the early layers is superior to that of a network using adjustable weights directly from the retina
• Recognition performance more than doubled
• Line detector network was much more efficient
• order of magnitude fewer variable weights and half as many total weights
• training time decrease of several orders of magnitude
Main Conclusion
![Page 37: Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and](https://reader030.vdocument.in/reader030/viewer/2022032703/56649d2f5503460f94a06dd7/html5/thumbnails/37.jpg)
• Increasing the number of hidden layer units does not translate to better accuracy, it actually reduces it.
• Increasing the number of epochs increased the accuracy but not always
• For Experiment 2 (6 letters with line detectors) we can achieve perfect training accuracy and very good validation accuracy
• Training time varied from a few minutes to many hours with Experiment 1 – 26 Letters taking the longest out of all, i.e. for 500 hidden layer units it required up to 9 hours.
• When noise is added to the retina image the accuracy of the system drops significantly, even for Experiment 2 with the line detectors
Additional Conclusions