lecture 16: spectral algorithms for gms · backpropagation: reverse-mode differentiation 12....
TRANSCRIPT
![Page 1: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/1.jpg)
CS839:ProbabilisticGraphicalModels
Lecture16:SpectralAlgorithmsforGMsTheoRekatsinas
1
![Page 2: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/2.jpg)
Overview
2
• AnoverviewoftheDLcomponents• Historicalremarks:earlydaysofneuralnetworks• Modernbuildingblocks:units,layers,activationsfunctions,lossfunctions,etc.• Reverse-modeautomaticdifferentiation(akabackpropagation)Distributedrepresentations
• SimilaritiesanddifferencesbetweenGMsandNNs• Graphicalmodelsvs.computationalgraphs• SigmoidBeliefNetworksasgraphicalmodels• DeepBeliefNetworksandBoltzmannMachines
• CombiningDLmethodsandGMs• UsingoutputsofNNsasinputstoGMs• GMswithpotentialfunctionsrepresentedbyNNs• NNswithstructuredoutputs
![Page 3: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/3.jpg)
History- Motivation
3
![Page 4: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/4.jpg)
PerceptronandNeuralNetworks
4
![Page 5: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/5.jpg)
ThePerceptronLearningAlgorithm
5
![Page 6: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/6.jpg)
ThePerceptronLearningAlgorithm
6
![Page 7: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/7.jpg)
NeuralNetworkModel
7
![Page 8: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/8.jpg)
Combinedlogisticmodels
8
![Page 9: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/9.jpg)
Combinedlogisticmodels
9
![Page 10: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/10.jpg)
Combinedlogisticmodels
10
![Page 11: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/11.jpg)
Notreally,notargetforhiddenunits...
11
![Page 12: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/12.jpg)
Backpropagation:Reverse-modedifferentiation
12
![Page 13: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/13.jpg)
Backpropagation:Reverse-modedifferentiation
13
![Page 14: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/14.jpg)
Modelbuildingblocks
14
![Page 15: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/15.jpg)
Modelbuildingblocks
15
![Page 16: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/16.jpg)
Buildingblocksofdeepnetworks
16
![Page 17: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/17.jpg)
Buildingblocksofdeepnetworks
17
![Page 18: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/18.jpg)
Hand-craftedfeatures
18
![Page 19: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/19.jpg)
Hand-craftedfeatures
19
![Page 20: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/20.jpg)
UsingDNNsforhierarchicalrepresentations
20
![Page 21: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/21.jpg)
21
![Page 22: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/22.jpg)
GraphicalmodelsvsDeepnets
22
![Page 23: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/23.jpg)
GraphicalmodelsvsDeepnets
23
![Page 24: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/24.jpg)
GraphicalmodelsvsDeepnets
24
![Page 25: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/25.jpg)
GraphicalmodelsvsDeepnets
25
![Page 26: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/26.jpg)
GraphicalmodelsvsDeepnets
26
![Page 27: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/27.jpg)
RestrictedBoltzmannMachines
27
![Page 28: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/28.jpg)
RestrictedBoltzmannMachines:LearningandInference
28
![Page 29: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/29.jpg)
RestrictedBoltzmannMachines:LearningandInference
29
![Page 30: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/30.jpg)
RestrictedBoltzmannMachines:LearningandInference
30
![Page 31: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/31.jpg)
SigmoidBeliefNetworks
31
![Page 32: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/32.jpg)
RBMsareinfinitebeliefnetworks
32
![Page 33: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/33.jpg)
RBMsareinfinitebeliefnetworks
33
![Page 34: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/34.jpg)
RBMsareinfinitebeliefnetworks
34
![Page 35: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/35.jpg)
RBMsareinfinitebeliefnetworks
35
![Page 36: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/36.jpg)
Deepbeliefnetworks:layer-wisepre-training
36
![Page 37: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/37.jpg)
DeepBoltzmannMachines
37
![Page 38: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/38.jpg)
DeepBoltzmannMachines
38
![Page 39: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/39.jpg)
39
GraphicalmodelsvsDeepnets
![Page 40: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/40.jpg)
40
CombiningsequentialNNsandGMs[Gravesetal.2013]
![Page 41: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/41.jpg)
41
CombiningsequentialNNsandGMs[Gravesetal.2013]
![Page 42: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/42.jpg)
42
HybridNNsandconditionalGMs
![Page 43: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/43.jpg)
43
HybridNNsandconditionalGMs
![Page 44: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/44.jpg)
44
HybridNNsandconditionalGMs
![Page 45: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/45.jpg)
45
Dealingwithstructuredprediction[Domke 2012]
![Page 46: Lecture 16: Spectral Algorithms for GMs · Backpropagation: Reverse-mode differentiation 12. Backpropagation: Reverse-mode differentiation 13. Model building blocks 14. Model building](https://reader035.vdocument.in/reader035/viewer/2022071100/5fd8e09740b3bb1519063f74/html5/thumbnails/46.jpg)
Summary
46
• DL&GM:thefieldsaresimilarinthebeginning(structure,energy,etc.),andthendivergetotheirownsignaturepipelines• DL:mosteffortisdirectedtocomparingdifferentarchitecturesandtheircomponents(basedonempiricalperformanceonadownstreamtask)• DLmodelsaregoodatlearningrobusthierarchicalrepresentationsfromthedataandsuitableforsimplereasoning(“low-levelcognition”)
• GM:lotsofeffortsaredirectedtoimprovinginferenceaccuracyandconvergencespeed• GMsarebestforprovablycorrectinferenceandsuitableforhigh-levelcomplexreasoningtasks(“high-levelcognition”)
• Convergenceofbothfieldsisverypromising!