Learning Invariances and Hierarchies
Pierre BaldiUniversity of California, Irvine
Two Questions
1. “If we solve computer vision, we have pretty much solved AI.”
2. A-NNs vs B-NNs and Deep Learning.
If we solve computer vision…
If we solve computer vision…
• If we solve computer audition,….
If we solve computer vision…
• If we solve computer audition,….
• If we solve computer olfaction,…
If we solve computer vision…
• If we solve computer audition,….
• If we solve computer olfaction,…
• If we solve computer vision, how can we build computers that can prove Fermat’s last theorem?
Invariances
• Invariances in audition. We can recognize a tune invariantly with respect to: intensity, speed, tonality, harmonization, instrumentation, style, background.
• Invariances in olfaction. We can recognize an odor invariantly with respect to: concentrations, humidity, pressure, winds, mixtures, background.
Non-Invariances
• Invariances evolution did not care about (although we are still evolving!...)
– We cannot recognize faces upside down.– We cannot recognize tunes played in reverse.– We cannot recognize stereoisomers as such.
Enantiomers smell differently.
A-NNs vs B-NNs
Origin of Invariances• Weight sharing and translational invariance.• Can we quantify approximate weight sharing?• Can we use approximate weight sharing to improve
performance?• Some of the invariance comes • from the architecture. • Some may come from the • learning rules.
Learning Invariances
EHebbsymmetric connections
wij=wji
111
11-1
1-11
Acyclic orientation of the Hypercube O(H)
Isometry
Isometry
HebbHebb
O(H)
H
I(O(H))
I(H)
Deep Learning ≈ Deep Targets
Training set: (xi,yi) or i=1, . . ., m
?
Deep Target Algorithms
Deep Target Algorithms
Deep Target Algorithms
Deep Target Algorithms
Deep Target Algorithms
• In spite of the vanishing gradient problem, (and the Newton problem) nothing seems to beat back-propagation.
• Is backpropagation biologically plausible?
Mathematics of Dropout (Cheap Approximation to Training Full Ensemble)
Two Questions
1. “If we solve computer vision, we have pretty much solved AI.”
2. A-NNs vs B-NNs and Deep Learning.