![Page 1: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/1.jpg)
Convolutional Neural Networks
Dr. Xiaowei Huang
https://cgi.csc.liv.ac.uk/~xiaowei/
![Page 2: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/2.jpg)
Up to now,
• Traditional Machine Learning Algorithms
• Deep learning • Introduction to Deep Learning
• Functional view and features
• Backward and forward computation (including backpropogation and chain rule)
![Page 3: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/3.jpg)
Topics
• convolutional neural networks (CNN) • Fully-connected• Convolutional Layer• Advantage of Convolutional Layer• Zero-padding Layer• ReLU Layer• Pooling• Softmax• Preprocessing data
• Example• LeNet
![Page 4: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/4.jpg)
Convolutional layer
ReLU layerConvolutional layerReLU layerMaxpooling layer
Dropout layer: for regularisation
Flatten layer: from convolutional to fully-connected
Fully-connected layer
![Page 5: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/5.jpg)
Fully-connected
![Page 6: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/6.jpg)
![Page 7: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/7.jpg)
Convolution
![Page 8: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/8.jpg)
Convolutional neural networks
• Strong empirical application performance
• Convolutional networks: neural networks that use convolution in place of general matrix multiplication in at least one of their layers
for a specific kind of weight matrix 𝑊
![Page 9: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/9.jpg)
Convolutional layer illustrated
![Page 10: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/10.jpg)
Illustration 1
a b c d e f
x y z
![Page 11: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/11.jpg)
Illustration 1
a b c d e f
x y z
xa+yb+zc
![Page 12: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/12.jpg)
Illustration 1
a b c d e f
x y z
xa+yb+zc xb+yc+zd
![Page 13: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/13.jpg)
Illustration 1
a b c d e f
x y z
xa+yb+zc xb+yc+zd xc+yd+ze
![Page 14: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/14.jpg)
Illustration 1
a b c d e f
x y z
xa+yb+zc xb+yc+zd xc+yd+ze xd+ye+zf
![Page 15: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/15.jpg)
Illustration 1: boundary case
a b c d e f
x y z
ya+zb
![Page 16: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/16.jpg)
Illustration 1: boundary case
a b c d e f
x y z
xa+yb+zc xb+yc+zd xc+yd+ze xd+ye+zfya+zb xe+yf
![Page 17: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/17.jpg)
Illustration 1: as matrix multiplication
ya+zb
xa+yb+zc
xb+yc+zd
xc+yd+ze
xd+ye+zf
xe+yf
![Page 18: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/18.jpg)
Illustration 2: two dimensional case
![Page 19: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/19.jpg)
Illustration 2
![Page 20: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/20.jpg)
Illustration 2
What’s the shape of the resulting matrix?
![Page 21: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/21.jpg)
A closer look at spatial dimensions:
![Page 22: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/22.jpg)
A closer look at spatial dimensions:
![Page 23: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/23.jpg)
A closer look at spatial dimensions:
![Page 24: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/24.jpg)
A closer look at spatial dimensions:
![Page 25: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/25.jpg)
A closer look at spatial dimensions:
![Page 26: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/26.jpg)
A closer look at spatial dimensions:
![Page 27: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/27.jpg)
A closer look at spatial dimensions:
![Page 28: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/28.jpg)
A closer look at spatial dimensions:
![Page 29: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/29.jpg)
A closer look at spatial dimensions:
![Page 30: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/30.jpg)
A closer look at spatial dimensions:
![Page 31: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/31.jpg)
A closer look at spatial dimensions:
![Page 32: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/32.jpg)
Advantage of Convolutional Layer
![Page 33: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/33.jpg)
Advantage: sparse interaction
![Page 34: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/34.jpg)
Advantage: sparse interaction
![Page 35: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/35.jpg)
Advantage: sparse interaction
![Page 36: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/36.jpg)
Advantage: parameter sharing/weight tying
![Page 37: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/37.jpg)
Advantage: equivariant representations
• Equivariant: transforming the input = transforming the output
• Example: input is an image, transformation is shifting
• Convolution(shift(input)) = shift(Convolution(input))
• Useful when care only about the existence of a pattern, rather than the location
![Page 38: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/38.jpg)
Zero-Padding
![Page 39: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/39.jpg)
Zero-Padding
0 0 0 0 0 0
0 a b c d 0
0 e f g h 0
0 i j k l 0
0 0 0 0 0 0
a b c d
e f g h
i j k l
What’s the shape of the resulting matrix?
w x
y z
filter
Input
![Page 40: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/40.jpg)
ReLU
![Page 41: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/41.jpg)
ReLU (rectified linear unit)
• rectifier is an activation function defined as the positive part of its argument
f(x) = max(0,x)
• A smooth approximation to the rectifier is the analytic function
f(x) = log(1+ex)
![Page 42: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/42.jpg)
Pooling
![Page 43: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/43.jpg)
Pooling
• Pooling layer is frequently used in convolutional neural networks with the purpose to progressively reduce the spatial size of the representation to reduce the amount of features and the computational complexity of the network.
![Page 44: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/44.jpg)
Terminology
![Page 45: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/45.jpg)
Pooling
• Summarizing the input (i.e., output the max of the input)
![Page 46: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/46.jpg)
Advantage
• Induce invariance
![Page 47: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/47.jpg)
Example: Max-pooling
1 3 4 5
7 2 9 1
9 2 4 7
4 3 6 2
7 9
9 7
max{1,3,7,2} = 7
max{4,7,6,2} = 7
max{4,5,1,9} = 9
max{9,2,3,4} = 9
![Page 48: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/48.jpg)
Motivation from neuroscience
• David Hubel and Torsten Wiesel studied early visual system in human brain (V1 or primary visual cortex), and won Nobel prize for this
• V1 properties• 2D spatial arrangement
• Simple cells: inspire convolution layers
• Complex cells: inspire pooling layers
![Page 49: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/49.jpg)
Softmax
![Page 50: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/50.jpg)
Softmax
• Recall that logistic regression produces a decimal between 0 and 1.0. For example, a logistic regression output of 0.8 from an email classifier suggests an 80% chance of an email being spam and a 20% chance of it being not spam. Clearly, the sum of the probabilities of an email being either spam or not spam is 1.0.
• Softmax extends this idea into a multi-class world. That is, Softmaxassigns decimal probabilities to each class in a multi-class problem. Those decimal probabilities must add up to 1.0. This additional constraint helps training converge more quickly than it otherwise would.
![Page 51: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/51.jpg)
Softmax
![Page 52: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/52.jpg)
Preprocessing data
![Page 53: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/53.jpg)
Preprocessing data
![Page 54: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/54.jpg)
Preprocessing data
![Page 55: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/55.jpg)
Preprocessing data
e.g. consider CIFAR-10 example with [32,32,3] images
• Subtract the mean image (e.g. AlexNet) (mean image = [32,32,3] array)
• Subtract per-channel mean (e.g. VGGNet) (mean along each channel = 3 numbers)
![Page 56: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/56.jpg)
Example: LeNet
![Page 57: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/57.jpg)
LeNet-5
• Proposed in “Gradient-based learning applied to document recognition” , by Yann LeCun, Leon Bottou, Yoshua Bengio and Patrick Haffner, in Proceedings of the IEEE, 1998
• Apply convolution on 2D images (MNIST) and use backpropagation
• Structure: 2 convolutional layers (with pooling) + 3 fully connected layers • Input size: 32x32x1
• Convolution kernel size: 5x5
• Pooling: 2x2
![Page 58: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/58.jpg)
LeNet-5
![Page 59: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/59.jpg)
LeNet-5
![Page 60: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/60.jpg)
LeNet-5 Pooling: 2x2, stride: 2
Filter: 5x5, stride: 1x1, #filters: 6
![Page 61: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/61.jpg)
LeNet-5 Filter: 5x5x6, stride: 1x1, #filters: 16
Filter: 5x5, stride: 1x1, #filters: 6
![Page 62: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/62.jpg)
LeNet-5 Pooling: 2x2, stride: 2
Filter: 5x5, stride: 1x1, #filters: 6
![Page 63: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/63.jpg)
LeNet-5 Weight matrix: 400x120
Filter: 5x5, stride: 1x1, #filters: 6
![Page 64: Convolutional Neural Networks - University of Liverpoolxiaowei/ai_materials...Convolutional Neural Networks Dr. Xiaowei Huang xiaowei/ Up to now, •Traditional Machine Learning Algorithms](https://reader034.vdocument.in/reader034/viewer/2022042219/5ec501bced46f61a801af1de/html5/thumbnails/64.jpg)
LeNet-5 Weight matrix: 120x84
Filter: 5x5, stride: 1x1, #filters: 6
Weight matrix: 84x10