![Page 1: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/1.jpg)
Adaptive)Deep)Learning)for)VisualUnderstanding
Kate)SaenkoBoston)University
Harvard&University,&January&22&2019
![Page 2: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/2.jpg)
Boston&University Slideshow*Title*Goes*Here
Prof.*Kate*Saenko
Boston&University Slideshow*Title*Goes*Here
AIR :*AI*Research*at*BU*
AIR
![Page 3: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/3.jpg)
Boston&University Slideshow*Title*Goes*Here
Prof.*Kate*Saenko
Boston&University Slideshow*Title*Goes*Here
Goal:*teach*machines*to*see,$talk,$actResearch*in*my*lab*********************************************************************************************************** Kate*Saenko
A baseball game in progress with the batter up to plate
A man is riding a bicycle
Find*“window*upper*right”
A: skateboard
Q:*What*is*the*child*standing*
on?
Find*the*moment*when*“girl*looks*
up*at*the*camera*and*smiles”
pick*up*what*you*see
A
I
Vision Lang&uage
Action
“Go*out*of*the*bedroom,*down*
the*stairs,*turn*left*and*stop*at*
the*dining*table”
![Page 4: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/4.jpg)
deep$learning$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$human$learning
?
![Page 5: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/5.jpg)
adaptiveexplainablemodular
![Page 6: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/6.jpg)
train test
HUMAN
● learns/from/a/single/example● generalizes/knowledge
adaptive,/explainable,/modular
![Page 7: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/7.jpg)
train test
SUPERVISED0DEEP0LEARNING
● learns0from010000’s0examples● fails0on0new0domains
adaptive,0explainable,0modular
imagenet.org
![Page 8: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/8.jpg)
HUMAN
● can*explain*decisions● grounds*language*in*world
adaptive,*explainable,*modular
why*is*this*a*dog?tail,*four*legs,*...
![Page 9: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/9.jpg)
DEEP$LEARNING
● cannot$explain$decisions● black$box
adaptive,$explainable,$modular
why$is$this$a$dog?
![Page 10: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/10.jpg)
HUMAN
● disentangles0properties● compositional
adaptive,0explainable,0modular
dog0running cat0sitting cat0running
![Page 11: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/11.jpg)
adaptive,)explainable,)modular
dog)running cat)sitting ??
DEEP)LEARNING
● entangles)properties● not)compositional
![Page 12: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/12.jpg)
adaptiveexplainablemodular
this2talk
![Page 13: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/13.jpg)
Kate%Saenko,%Boston%University
Domain'shift
“Dataset'Bias”“Domain'Shift”
What%your%net%is%trained%on What%it’s%asked%to%label
![Page 14: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/14.jpg)
Kate%Saenko,%Boston%University
Example(shift
train test
from+simulation++++++++++++++++++++++++++++++++++++++++++++++++to+reality
![Page 15: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/15.jpg)
Kate%Saenko,%Boston%University
Shift&from&simulation&to&reality
road
sidewalk
people
treesca
r
car
sidewalk
sidewalk
road
trees
Input1Image True1Segmentation
Model1Output
grass
![Page 16: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/16.jpg)
Kate%Saenko,%Boston%University
Solution:)Domain)Adaptation
road
sidewalk
people
trees
car
car
sidewalk
sidewalk
road
trees
Adapted/Model/Output
Input/Image True/Segmentation
Model/Output
grass
![Page 17: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/17.jpg)
Kate%Saenko,%Boston%University
Applications+of+Domain+AdaptationFrom+dataset+to+dataset
From+simulated+to+real+control
From+RGB+to+depth
From+CAD+models+to+real+images+
![Page 18: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/18.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Encoder'CNN Classifierloss
Adversarial*domain*adaptation
Classifier
![Page 19: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/19.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Adversarial*domain*adaptation
Classifier
![Page 20: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/20.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Adversarial*domain*adaptation
Adversarial'loss
Classifier
Discriminator
![Page 21: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/21.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Classifierloss
Adversarial*domain*adaptation
Adversarial'loss
ClassifierEncoder'CNN
Encoder'CNNDiscriminator
Classifier
![Page 22: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/22.jpg)
Kate%Saenko,%Boston%University
Results'on'Digits'Classification
● Domain(Confusion(Loss((Tzeng(2015)
● Adversarial(Domain(Alignment((ADDA)((Tzeng(2017)
Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation." ICML 2015Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep Transfer Across Domains and Tasks” ICCV 2015Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." CVPR 2017.
Domain(Confusion(Loss((Tzeng(2015)GradReversal((Ganin(&(Lempitsky(2015)
ADDA((Tzeng(2017)
![Page 23: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/23.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Encoder'CNN Classifierloss
Problem:)ambiguous)features
Classifier
![Page 24: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/24.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Problem:)ambiguous)features
Classifier
![Page 25: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/25.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Encoder'CNN
Encoder'CNN Classifierloss
Problem:)ambiguous)features
Adversarial'loss
Classifier
Discriminator
![Page 26: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/26.jpg)
Kate%Saenko,%Boston%University
Source'Data'+'Labelsbackpack chair
Unlabeled'Target'Data
?
Classifierloss
Problem:)ambiguous)features
Adversarial'loss
ClassifierEncoder'CNN
Encoder'CNNDiscriminator
![Page 27: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/27.jpg)
Kate%Saenko,%Boston%University
Source' Target'
Before'Adaptation Adapted
Problem:)ambiguous)features
![Page 28: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/28.jpg)
Kate%Saenko,%Boston%University
Source' Target'
Goal:&avoid&generating&ambiguous&featuresBefore'Adaptation Adapted
Adversarial'Dropout'Regularization,'Kuniaki'Saito, Yoshitaka'Ushiku, Tatsuya'Harada, Kate'Saenko,'ICLR'2018
![Page 29: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/29.jpg)
Kate%Saenko,%Boston%University
Solution:)use)the)decision)boundaryTrain&a&critic&(C)&(=discriminator)&that&can&detect&target&samples&near&decision&boundaryTrain&a&generator&(G)&that&can&fool&the&critic
Slightly&change&the&boundary&and&measure&the&change&of&p(y|x)!&&(sensitivity)
Samples&near&the&boundary&have&larger&sensitivity
Original&Boundary
Adversarial&Dropout&Regularization,&Kuniaki&Saito, Yoshitaka&Ushiku, Tatsuya&Harada, Kate&Saenko,&ICLR&2018
![Page 30: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/30.jpg)
Kate%Saenko,%Boston%University
Data FeatureClassifiers
Sampling2by2dropout
Predictions
Measure2Sensitivity
Fix G"and2train C"to2maximize2d(p1,p2)"for2target2samples.Train G"and C"to2minimize2Cross2Entropy2for2source2samples.For2k2=1:nFix2C"and2train G"to2minimize2d(p1,p2)"for"target.
Adversarial"Dropout"Regularization
Adversarial2Dropout2Regularization,2Kuniaki2Saito, Yoshitaka2Ushiku, Tatsuya2Harada, Kate2Saenko,2ICLR22018
![Page 31: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/31.jpg)
Kate%Saenko,%Boston%University
Results'on'Digits'Classification
● Adversarial+Dropout+Regularization+(ADR)+(Saito+et+al+2018)
Adversarial+Dropout+Regularization,+Kuniaki+Saito, Yoshitaka+Ushiku, Tatsuya+Harada, Kate+Saenko,+ICLR+2018
Domain+Confusion+Loss+(Tzeng+2015)GradReversal+(Ganin+&+Lempitsky+2015)
ADDA+(Tzeng+2017)
![Page 32: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/32.jpg)
Kate%Saenko,%Boston%University
Adaptation)for)semantic)segmentation
Adapted
Ground,TruthInput,image
Source5only,model
![Page 33: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/33.jpg)
Kate%Saenko,%Boston%University
Adaptation)for)semantic)segmentation
Adapted
Ground,TruthInput,image
Source5only,model
![Page 34: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/34.jpg)
Kate%Saenko,%Boston%University
Pixel&to&pixel*adaptation
![Page 35: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/35.jpg)
Kate%Saenko,%Boston%University
Strong'weak,feature,alignment
Strong'Weak,Distribution,Alignment,for,Adaptive,Object,Detection,,Saito,,Ushiku,,Harada,,Saenko,,arxiv,2018
![Page 36: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/36.jpg)
Kate%Saenko,%Boston%University
Strong'weak,feature,alignment
Strong'Weak,Distribution,Alignment,for,Adaptive,Object,Detection,,Saito,,Ushiku,,Harada,,Saenko,,arxiv,2018
![Page 37: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/37.jpg)
Kate%Saenko,%Boston%University
ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline
Domain%shift:%from%PASCAL%VOC%to%CLIPART
![Page 38: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/38.jpg)
Kate%Saenko,%Boston%University
Evidence(for(target(domain(label((GradCAM)(shows(that(the(feature(extractor(seems(to(deceive(the(domain(classifier(in(regions(with(car.
Domain(shift:(from(Grand(Theft(Auto(game(to(Cityscapes
target
source
![Page 39: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/39.jpg)
Kate%Saenko,%Boston%University
ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline
Domain%shift:%from%Cityscapes%to%Foggy%Cityscapes
![Page 40: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/40.jpg)
adaptiveexplainablemodular
![Page 41: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/41.jpg)
CNNs learn to predict pneumonia by detecting hospital which took the image
Variable(generalization(performance(of(a(deep(learning(model(to(detect(pneumonia(in(chest(radiographs:(A(cross8sectional(study.(Zech%JR1,%Badgeley%MA2,%Liu%M2,%Costa%AB3,%Titano%JJ4,%Oermann%EK3.%https://www.ncbi.nlm.nih.gov/pubmed/30399157
● Study%on%detecting%pneumonia%using%158,323%chest%radiographs
● CNNs%robustly%identified%hospital%system%and%department%within%a%hospital
● CNN%has%learned%to%detect%a%metal%token%that%radiology%technicians%place%on%the%patient%in%the%corner%of%the%image%field%of%view%at%the%time%they%capture%the%image
![Page 42: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/42.jpg)
RISE:&randomly&mask&input,&measure&output
42
P[shark]
shark
RISE:&Randomized&Input&Sampling&for&Explanation&of&Black:box&Models,&Petsiuk,&Das,&Saenko,&BMVC&2018
![Page 43: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/43.jpg)
adaptiveexplainablemodular
![Page 44: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/44.jpg)
Explainable*Neural*Computation*via*Stack*Neural*Module*Nets*Networksinput:*There*is*a*small*gray*block;*are*there*any*spheres*to*the*left*of*it?
Hu,$Andreas,$Darrell,$Saenko,$Explainable$Neural$Computation$via$Stack$Neural$Module$Networks,$ECCV’18
input*image
![Page 45: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/45.jpg)
Disentangling*properties*via*generation
A"Two&Stream"Variational"Adversarial"Network"for"Video"Generation
generated"video
![Page 46: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/46.jpg)
Disentangling*properties*via*generation
A*Two2Stream*Variational*Adversarial*Network*for*Video*Generation,*arxiv*2018
Disentangling*properties*via*generation
![Page 47: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/47.jpg)
Disentangling*properties*for*domain*transfer
![Page 48: Adaptive Deep Learning for Visual Understanding › wp-content › uploads › 2018 › 06 › Ada… · Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep](https://reader034.vdocument.in/reader034/viewer/2022042407/5f21fea95cc7dc0a4102d3da/html5/thumbnails/48.jpg)
adaptiveexplainablemodular