Upload File

a brief review of non-neural-network approaches to deep learning

  • Home
  • Documents
  • A brief review of non-neural-network approaches to deep learning
16
A brief review of non-neural- network approaches to deep learning Naiyan Wang

Upload: ratana

Post on 23-Feb-2016

41 views

Category:

Documents


0 download

Report

Tags:

DESCRIPTION

A brief review of non-neural-network approaches to deep learning. Naiyan Wang. Outline. Non-NN Approaches Deep Convex Net Extreme Learning Machine PCAnet Deep Fisher Net (Already presented before) Discussion. Deep convex net. Each module is a two- layer convex network. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: A brief review of non-neural-network approaches to deep learning

A brief review of non-neural-network approaches to deep learning

Naiyan Wang

Page 2: A brief review of non-neural-network approaches to deep learning

Outline

• Non-NN Approaches– Deep Convex Net– Extreme Learning Machine– PCAnet– Deep Fisher Net (Already presented before)

• Discussion

Page 3: A brief review of non-neural-network approaches to deep learning

Deep convex net

• Each module is a two- layer convex network.

• After we get the prediction from each module, we concatenate it with the original input, and send it to a new module.

Page 4: A brief review of non-neural-network approaches to deep learning

Deep Convex Net

• For each module– We minimize– – U has a closed form solution:– Learning of W relies on gradient descent:

– Note that no global fine tuning involved, so it can stack up to more than 10 layers. (Fast Training!)

Page 5: A brief review of non-neural-network approaches to deep learning

Deep Convex Net

• A bit wired of why this works.• The learned features in mid-layers are NOT

representative for the input.• Maybe learn the correlation between

prediction and input could help?• Discussion?

Page 6: A brief review of non-neural-network approaches to deep learning

Deep Convex Net

Page 7: A brief review of non-neural-network approaches to deep learning

Deep Convex Net

Page 8: A brief review of non-neural-network approaches to deep learning

Extreme Learning Machine

• It is also a two layer networks:– The first layer performs random projection of

input data.– The second layer performs OLS/Ridge regression

to learn the weight.• After that, we could take the transpose of the

learned weight as the projection matrix, and stack several ELM into a deep one.

Page 9: A brief review of non-neural-network approaches to deep learning

Extreme Learning Machine

• Extremely fast learning• Note that even with simple random projection

and linear transformation, the results still can be improved!

Page 10: A brief review of non-neural-network approaches to deep learning

PCANet

• In the first two layers, use patch PCA to learn the filters.

• Then it binarizes the output in second layer, and calculate the histogram within a block.

Page 11: A brief review of non-neural-network approaches to deep learning

PCANet

• To learn the filters, the authors also proposed to use Random initialization and LDA.

• The results are acceptable in a wide range of datasets.

Page 12: A brief review of non-neural-network approaches to deep learning

Summary

• Most of the paper (except deep Fisher Net) report their results on relatively toy data. We cannot draw any conclusion about their performance.

• This could enlighten us some possible research directions.

Page 13: A brief review of non-neural-network approaches to deep learning

Discussion• Why deep architectures always help? (We don’t concern

about overfitting now)– The representation power increases exponentially as more

layers add in.– However the number of parameters increases linearly as more

layers add in.• Given a fixed budget, this is a better way to organize the

model.• Take PCA net as an example, if there are m, n neurons at

first and second layer, then there exists an equivalent m*n single layer net.

Page 14: A brief review of non-neural-network approaches to deep learning

Discussion• Why CNN is so successful in image classification?

– Data abstraction– Locality! (The image is a 2D structure with strong local

correlation.)• The convolution architecture could propagate local

information to a broader region– 1st: m * m, 2nd : n * n, then it corresponds to (m + n - 1) * (m + n

- 1) in the original image.• This advantage is further expanded by spatial pooling.• Other ways to concern about these two issues

simultaneously?

Page 15: A brief review of non-neural-network approaches to deep learning

Discussion

• Convolution is a dense architecture. It induces a lot of unnecessary computation.

• Could we come up a greedy or more clever selection in each layer to just focus on those discriminative patches?

• Or possibly a “convolutional cascade”?

Page 16: A brief review of non-neural-network approaches to deep learning

Discussion• Random weights are adopted several times, and it yields

acceptable results.• Pros:

– Data independent– Fast

• Cons:– Data independent

• So could we combine random weights and learned weights to combat against overfitting?

• Some work have been done on combining deterministic NN and stochastic NN.


Decoupled Deep Neural Network for Semi-supervised Semantic Segmentationpapers.nips.cc/paper/5858-decoupled-deep-neural-net… ·  · 2015-12-18Decoupled Deep Neural Network for Semi-supervised

A brief review of non-neural- network approaches to deep learning Naiyan Wang

Neural and Non-neural Approaches to Authorship Attributionetheses.whiterose.ac.uk/21415/1/FinalThesis_Yunita.pdf · 2018-09-04 · Neural and Non-neural Approaches to Authorship Attribution

Convolutional Neural Networks and rTansfer Learning Based ......246 Krstinic D., Braovic M., Bozic-Stulic D.: Convolutional Neural Networks ... 2 Related work Deep learning based approaches

Deep Convolutional Neural Network for Image …papers.nips.cc/paper/5485-deep-convolutional-neural...Deep Convolutional Neural Network for Image Deconvolution Li Xu ∗ LenovoResearch

Introduction to Deep Neural Networks - Deep Learning · Introduction to Deep Neural Networks 0. Logistics Spring 2020 1. Neural Networks are taking over! •Neural networks have become

Contextually Constrained Deep Networks for Scene …Deep learning approaches, such as multi-layer neural networks, leverage the amount of avail-able data to learn representations:

Deep Convolutional Neural Networks on Multichannel Time ... · Deep Convolutional Neural Networks On Multichannel Time Series ... as a common input for the neural network ... Deep

ImageNet Classification with Deep Convolutional Neural …kriz/imagenet_classification_with_deep_convolutional.pdfImageNet Classification with Deep Convolutional Neural Networks

Deep neural networks

Deep reinforcement learning for de novo drug design · and reinforcement learning (RL) approaches, ReLeaSE integrates two deep neural networks—generative and predictive—that are

ImageNet Classification with Deep Deep Convolutional …rita/visual_recog_course/talks/ImageNet... · ImageNet Classification with Deep Deep Convolutional Convolutional Neural Neural

Deep Learning in Neural Networks: An Overviejuergen/DeepLearning28May2014.pdf · Deep Learning in Neural Networks: An Overview ... 1 Introduction to Deep Learning ... Neural Fitted

DEEP LEARNING NEURAL NETWORK APPROACHES TO LAND …Deep Neural Network (DNN)-based traffic prediction model was developed using the City of Calgary’s datasets. Specifically, the

Deep convolutional neural networks - Computer Science- …web.cs.ucdavis.edu/.../lee_lecture19_deeplearning_notes.pdf · Deep convolutional neural networks ... “Deep” architecture

Practical Deep Neural Networks - DGYrt.dgyblog.com/res/dlworkshop/introduction.pdf · Practical Deep Neural Networks ... ' CS231n Convolutional Neural Networks for Visual Recognition

Pattern Recognition Letterswebia.lip6.fr/~cord/pdfs/publis/2018prlcord.pdf · Deep convolutional neural networks Learning analysisusing aprivileged approaches,information. propose

Deep Multi-State Dynamic Recurrent Neural Networks ...papers.nips.cc/paper/9594-deep-multi-state-dynamic-recurrent-neural... · Deep Multi-State Dynamic Recurrent Neural Networks

Deep Learning Binary Neural Network on an FPGA · Deep Learning Binary Neural Network on an ... deep neural networks have attracted lots of attentions ... brain has a deep architecture

Introduction to Deep Neural Networksbhiksha/courses/deep... · Introduction to Deep Neural Networks 0. Logistics Spring 2020 1. Neural Networks are taking over! •Neural networks

Deep Neural Networks - uchile.cl

How to Evaluate Efficient Deep Neural Network Approaches · n T.-J. Yang, V. Sze, “Design Considerations for Efficient Deep Neural Networks on Processing-in-Memory Accelerators,”

Computer Vision...Computer Vision From traditional approaches to deep neural networks Stanislav Frolov München, 27.02.2018

Deep Neural Networks - dmi.unibas.ch

Special Topic - University of Georgiacobweb.cs.uga.edu/~khaled/MLcourse/Deep_Learning_Lecture.pdf · Deep Neural Networks Deep (but not that deep) Neural Network 6. Deep Neural Networks

Deep Neural Decision Forests

A Deep Non-Negative Matrix Factorization Neural Network€¦ · A Deep Non-Negative Matrix Factorization Neural Network Jennifer Flenner Blake Hunter 1 Abstract Recently, deep neural

Neural Networks - cs.cmu.edubhiksha/courses/deep... · about neural networks (a.k.a deep learning) This guy learned about neural networks (a.k.a deep learning) Objectives of this

Troubleshooting Deep Neural Networks - josh-tobin.comjosh-tobin.com/assets/pdf/troubleshooting-deep-neural-networks-01... · Josh Tobin. January 2019. josh-tobin.com/troubleshooting-deep-neural-networks

Parallelized Deep Neural Networks

DATAMINING2 (Deep) Neural Networks

Partial Differential Equations Approaches to Optimization ......Partial Differential Equations Approaches to Optimization and Regularization of Deep Neural Networks Celebrating 75

Heterogeneous Gaussian Mechanism: Preserving Differential ...ix.cs.uoregon.edu/~dou/research/papers/ijcai2019_PP.pdfvate deep neural networks, compared with baseline approaches, under

Deep Neural Network Approach for the Dialog State Tracking ...people.cs.pitt.edu/~huynv/research/deep-nets/Deep Neural Network Approach for the...Deep Neural Network Approach for the

Deep Q-Learning with Recurrent Neural Networkscs229.stanford.edu/...DeepQLearningWithRecurrentNeuralNetwords... · Deep Q-Learning with Recurrent Neural Networks ... rent neural network