getting creative with autoencoders · 2020. 1. 8. · autoencoders •training objective:...
TRANSCRIPT
![Page 1: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/1.jpg)
Gettingcreativewithautoencoders
Tijmen Tieleman,CEO&CTOatminds.aiWiSSAP 2019,Trivandrum
![Page 2: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/2.jpg)
Overview
• Thisisafairlygeneraltalk• Application-agnostic• Generalideas,ratherthanrecentresearch
• Thisisafairlyspecializedtalk• It’saboutDeepLearning• Morespecifically:representationlearning• Morespecifically:unsupervisedrepresentationlearning• Morespecifically:autoencoders
![Page 3: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/3.jpg)
Autoencoders
• Trainingobjective:reconstructtheinput• Components:encoderanddecoder• Usualpurpose:representationlearning• Generativeview:variationalautoencoders• Components:encoderanddecoder generator
• Flexible• Getcreativewithencoderand/orgenerator• Interpretablecodescanbearranged
• Codeunitsareonlypartiallyhiddenunits
![Page 4: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/4.jpg)
Clustering
• “kmeans”• Kmeansasan(odd)autoencoder• Encoder(notNN)• Generator(constantforeachcluster)
• Implicitassumption• Morepowerfulgenerator:non-trivialcovariance• Differentimplicitassumption
![Page 5: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/5.jpg)
ClusteringautoencoderWithfreecodeunitsNaïve
Withfreecodeunitsanddiscretization
![Page 6: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/6.jpg)
Intermission:afewminutesforquestionsaboutthepastfewslides.
![Page 7: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/7.jpg)
Gettingcreativewithautoencoders
• Clusteringautoencoder• Ignoringautoencoder• Binarizingautoencoder• Biasedautoencoder• Computergraphicsautoencoder• Independentfeaturesautoencoder• Etc• Feelfreetocombine
![Page 8: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/8.jpg)
Ignoringautoencoder
• Goal:codethatignoressomeinformation• E.g.ignoringspeakerinformation• E.g.ignoringthespokentext
• Application:focusingonessentials• Application:domainadaptation• Application:anti-discrimination• Application:reconstructingwithdifferentinformation• Method1:providethatinformationtothedecoder• Method2:insistthattheto-be-hiddeninformationcannotbereconstructed
![Page 9: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/9.jpg)
Binarizingautoencoder
• Goal:binarycode• Application:”semantichashing”• Application:extremecompression(lossy)• Method:usediscretizationnoiseonabankoflogisticcodeunits
![Page 10: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/10.jpg)
Biasedautoencoder
• Goal:codesthattendtobeofcertainkinds• E.g.asoftmax with2commonlyusedunitsand10rarelyusedones.• E.g.binarizingandpreferringsparsity
• Method:simplypushintheobjective
![Page 11: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/11.jpg)
Intermission:afewminutesforquestionsaboutthepastfewslides.
![Page 12: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/12.jpg)
Computergraphicsautoencoder
• Moregenerally:domain-specificgenerator• E.g.audiosynthesisautoencoder
• Goal:highlyinterpretablecodes• Goal:helptheencoderunderstandthenatureofimages• Application:reverserendering,for“imageparsing”oraspreparationforgeneratingimagevariants• Application:usingthebetterencoderasagoodfeaturizer• Method:replacethegenerator
![Page 13: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/13.jpg)
Computergraphicsasgenerator
• Problem:ithastobedifferentiable• Solution1:implementareducedCGsysteminTensorFlow• Weakness:“reduced”isunfortunate• Nice:itcancontainlearnables
• Solution2:learnasecondNNtoapproximatearealCGsystem;backpropagatethroughtheapproximation• Weakness:approximationsareapain• Nice:ithasthepotentialforfullCG• Thecodesstillhavetobecontinuous
![Page 14: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/14.jpg)
PoC:graphicsforMNIST
• Uses“capsules”• Learnedconstant:smallimagetemplates• Learnedfunction:freecodeunitvaluesà affinetransformationforthetemplates• Hard-codedfunction:applyaffinetransformationstothetemplates• Affinetransformedtemplatesareaddedtogethertocreatefinaloutput
![Page 15: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/15.jpg)
PoC:graphicsforMNIST
• Uses“capsules”• Learnedconstant:smallimagetemplates• Learnedfunction:freecodeunitvaluesà affinetransformationforthetemplates• Hard-codedfunction:applyaffinetransformationstothetemplates• Affinetransformedtemplatesareaddedtogethertocreatefinaloutput
![Page 16: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/16.jpg)
ClusteringMNIST
• Kmeansclusteringwouldn’tworkwell:thegeneratoristooprimitive• Weneedamorepowerfulgenerator• Ithastounderstandthatashiftedimageisstillaboutthesameimage• Let’susethiscomputergraphicsgenerator
• Unsupervisedclusteringinto25clusters• Goodclassificationaccuracywithjust25labeledcases
![Page 17: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/17.jpg)
![Page 18: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/18.jpg)
Independentfeaturesautoencoder
• Goal:codecomponentsthataresomewhatindependentlyinterpretable• Application:identifyingthepatternsinthedata• Application:generatingdatavariantswithahigherchanceofmeaningfulresults• Method:applydropouttothecodecomponents
![Page 19: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/19.jpg)
Independentfeatures
• Thisfocusesononeimage(seethecentralcolumn)• Thisshowsdatavariantsbyvaryingonefeatureatatime• The8featuresareclearlysomewhatindependent• Thevariantsaremeaningful
ß Featurevalueà
ßFeatureindexà
![Page 20: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/20.jpg)
Toolsforcraftingspecializedautoencoders
• Imposesomestructureonthecode• E.g.clusteringautoencoder,computergraphicsautoencoder
• Applydeterministiceditingtothecode• E.g.ignoringautoencoder
• Applyrandomeditingtothecode• E.g.binarizingautoencoder,clusteringautoencoder,independentfeaturesautoencoder
• Applyanobjectivefunctionadditiontothecode• E.g.sparsebinarizingautoencoder
• Hard-code(partof)thegenerator• E.g.computergraphicsautoencoder
![Page 21: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/21.jpg)
Conclusion
• Creatingcustomspecializedautoencodersisdoable• Thereisacollectionofbasictoolsforthis• Thiscanimproveinterpretabilityandlearningpower
• NextstageforDL:combininglearnableandhard-codedcomponents• Learnablecomponentsprovide“intuition”• Hard-codedcomponentsprovidethepowerofcomputation• Eitheraloneisquitelimited• AlphaGohashard-codedasmain()andlearnableassubroutine• TheCGautoencoderhaslearnableasmain()andhard-codedassubroutine
![Page 22: Getting creative with autoencoders · 2020. 1. 8. · Autoencoders •Training objective: reconstruct the input •Components: encoder and decoder •Usual purpose: representation](https://reader035.vdocument.in/reader035/viewer/2022071109/5fe38f6a3067e045aa0aaa38/html5/thumbnails/22.jpg)
Thankyou