![Page 1: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/1.jpg)
Orthogonal Convolutional Neural Networks
Jiayun Wang Yubei Chen Rudrasis Chakraborty Stella X. Yu
![Page 2: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/2.jpg)
Filter similarity increases with depth
![Page 3: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/3.jpg)
A typical conv layer has highly irregular spectrum
![Page 4: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/4.jpg)
Kernel orthogonality is widely used as a regularization
Dorobantu et al. 2016
Rodriguez et al. 2017
Bansal et al. 2018
Saxe et al. 2014
…
![Page 5: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/5.jpg)
5
OCNN can do even better
![Page 6: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/6.jpg)
Filter diversity improvement with OCNN
![Page 7: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/7.jpg)
Convolution is an efficient matrix-vector multiplication
Convolution:
Matrix-vector form:
![Page 8: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/8.jpg)
Convolution is an efficient matrix-vector multiplication
Convolution:
Matrix-vector form:
![Page 9: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/9.jpg)
Convolution is an efficient matrix-vector multiplication
Convolution:
Matrix-vector form:
![Page 10: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/10.jpg)
Orthogonal convolution or orthogonal kernel?
Convolution:
Matrix-vector form:
![Page 11: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/11.jpg)
Orthogonal convolution or orthogonal kernel?
Convolution:
Matrix-vector form:
![Page 12: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/12.jpg)
A fast algorithm for orthogonal convolution
- Convolutional Orthogonality:
- Kernel Orthogonality:
Same # parameters and test time, only 9% more training time
![Page 13: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/13.jpg)
Universal improvements
Task Metric Gain
Image Classificatio
n
CIFAR100 classification accuracy 3%ImageNet classification accuracy 1%
semi-supervised learning classification accuracy 3%
Feature Quality
ine-grained image retrieval kNN classification accuracy 3%
unsupervised image inpainting PSNR 4.3image generation FID 1.3
deep metric learning NMI 1.2Robustness black box attack attack time 7x less
![Page 14: Orthogonal Convolutional Neural Networksstellayu/... · Convolution is an efficient matrix-vector multiplication Convolution: Matrix-vector form: Convolution is an efficient matrix-vector](https://reader035.vdocument.in/reader035/viewer/2022062607/6054150e95b6b014fe087784/html5/thumbnails/14.jpg)
Thanks