adaptive deep learning for visual understanding › wp-content › uploads › 2018 › 06 ›...

Adaptive)Deep)Learning)for)VisualUnderstanding

Kate)SaenkoBoston)University

Harvard&University,&January&22&2019

Boston&University Slideshow*Title*Goes*Here

Prof.*Kate*Saenko

Boston&University Slideshow*Title*Goes*Here

AIR :*AI*Research*at*BU*

AIR

Boston&University Slideshow*Title*Goes*Here

Prof.*Kate*Saenko

Boston&University Slideshow*Title*Goes*Here

Goal:*teach*machines*to*see,$talk,$actResearch*in*my*lab*********************************************************************************************************** Kate*Saenko

A baseball game in progress with the batter up to plate

A man is riding a bicycle

Find*“window*upper*right”

A: skateboard

Q:*What*is*the*child*standing*

on?

Find*the*moment*when*“girl*looks*

up*at*the*camera*and*smiles”

pick*up*what*you*see

A

I

Vision Lang&uage

Action

“Go*out*of*the*bedroom,*down*

the*stairs,*turn*left*and*stop*at*

the*dining*table”

deep$learning$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$human$learning

?

adaptiveexplainablemodular

train test

HUMAN

● learns/from/a/single/example● generalizes/knowledge

adaptive,/explainable,/modular

train test

SUPERVISED0DEEP0LEARNING

● learns0from010000’s0examples● fails0on0new0domains

adaptive,0explainable,0modular

imagenet.org

HUMAN

● can*explain*decisions● grounds*language*in*world

adaptive,*explainable,*modular

why*is*this*a*dog?tail,*four*legs,*...

DEEP$LEARNING

● cannot$explain$decisions● black$box

adaptive,$explainable,$modular

why$is$this$a$dog?

HUMAN

● disentangles0properties● compositional

adaptive,0explainable,0modular

dog0running cat0sitting cat0running

adaptive,)explainable,)modular

dog)running cat)sitting ??

DEEP)LEARNING

● entangles)properties● not)compositional

adaptiveexplainablemodular

this2talk

Kate%Saenko,%Boston%University

Domain'shift

“Dataset'Bias”“Domain'Shift”

What%your%net%is%trained%on What%it’s%asked%to%label

Kate%Saenko,%Boston%University

Example(shift

train test

from+simulation++++++++++++++++++++++++++++++++++++++++++++++++to+reality

Kate%Saenko,%Boston%University

Shift&from&simulation&to&reality

road

sidewalk

people

treesca

r

car

sidewalk

sidewalk

road

trees

Input1Image True1Segmentation

Model1Output

grass

Kate%Saenko,%Boston%University

Solution:)Domain)Adaptation

road

sidewalk

people

trees

car

car

sidewalk

sidewalk

road

trees

Adapted/Model/Output

Input/Image True/Segmentation

Model/Output

grass

Kate%Saenko,%Boston%University

Applications+of+Domain+AdaptationFrom+dataset+to+dataset

From+simulated+to+real+control

From+RGB+to+depth

From+CAD+models+to+real+images+

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Encoder'CNN Classifierloss

Adversarial*domain*adaptation

Classifier

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Unlabeled'Target'Data

?

Encoder'CNN

Encoder'CNN Classifierloss

Adversarial*domain*adaptation

Classifier

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Unlabeled'Target'Data

?

Encoder'CNN

Encoder'CNN Classifierloss

Adversarial*domain*adaptation

Adversarial'loss

Classifier

Discriminator

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Unlabeled'Target'Data

?

Classifierloss

Adversarial*domain*adaptation

Adversarial'loss

ClassifierEncoder'CNN

Encoder'CNNDiscriminator

Classifier

Kate%Saenko,%Boston%University

Results'on'Digits'Classification

● Domain(Confusion(Loss((Tzeng(2015)

● Adversarial(Domain(Alignment((ADDA)((Tzeng(2017)

Ganin, Yaroslav, and Victor Lempitsky. "Unsupervised domain adaptation by backpropagation." ICML 2015Eric Tzeng, Judy Hoffman, Trevor Darrell, Kate Saenko. “Simultaneous Deep Transfer Across Domains and Tasks” ICCV 2015Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." CVPR 2017.

Domain(Confusion(Loss((Tzeng(2015)GradReversal((Ganin(&(Lempitsky(2015)

ADDA((Tzeng(2017)

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Encoder'CNN Classifierloss

Problem:)ambiguous)features

Classifier

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Unlabeled'Target'Data

?

Encoder'CNN

Encoder'CNN Classifierloss

Problem:)ambiguous)features

Classifier

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Unlabeled'Target'Data

?

Encoder'CNN

Encoder'CNN Classifierloss

Problem:)ambiguous)features

Adversarial'loss

Classifier

Discriminator

Kate%Saenko,%Boston%University

Source'Data'+'Labelsbackpack chair

Unlabeled'Target'Data

?

Classifierloss

Problem:)ambiguous)features

Adversarial'loss

ClassifierEncoder'CNN

Encoder'CNNDiscriminator

Kate%Saenko,%Boston%University

Source' Target'

Before'Adaptation Adapted

Problem:)ambiguous)features

Kate%Saenko,%Boston%University

Source' Target'

Goal:&avoid&generating&ambiguous&featuresBefore'Adaptation Adapted

Adversarial'Dropout'Regularization,'Kuniaki'Saito, Yoshitaka'Ushiku, Tatsuya'Harada, Kate'Saenko,'ICLR'2018

Kate%Saenko,%Boston%University

Solution:)use)the)decision)boundaryTrain&a&critic&(C)&(=discriminator)&that&can&detect&target&samples&near&decision&boundaryTrain&a&generator&(G)&that&can&fool&the&critic

Slightly&change&the&boundary&and&measure&the&change&of&p(y|x)!&&(sensitivity)

Samples&near&the&boundary&have&larger&sensitivity

Original&Boundary

Adversarial&Dropout&Regularization,&Kuniaki&Saito, Yoshitaka&Ushiku, Tatsuya&Harada, Kate&Saenko,&ICLR&2018

Kate%Saenko,%Boston%University

Data FeatureClassifiers

Sampling2by2dropout

Predictions

Measure2Sensitivity

Fix G"and2train C"to2maximize2d(p1,p2)"for2target2samples.Train G"and C"to2minimize2Cross2Entropy2for2source2samples.For2k2=1:nFix2C"and2train G"to2minimize2d(p1,p2)"for"target.

Adversarial"Dropout"Regularization

Adversarial2Dropout2Regularization,2Kuniaki2Saito, Yoshitaka2Ushiku, Tatsuya2Harada, Kate2Saenko,2ICLR22018

Kate%Saenko,%Boston%University

Results'on'Digits'Classification

● Adversarial+Dropout+Regularization+(ADR)+(Saito+et+al+2018)

Adversarial+Dropout+Regularization,+Kuniaki+Saito, Yoshitaka+Ushiku, Tatsuya+Harada, Kate+Saenko,+ICLR+2018

Domain+Confusion+Loss+(Tzeng+2015)GradReversal+(Ganin+&+Lempitsky+2015)

ADDA+(Tzeng+2017)

Kate%Saenko,%Boston%University

Adaptation)for)semantic)segmentation

Adapted

Ground,TruthInput,image

Source5only,model

Kate%Saenko,%Boston%University

Adaptation)for)semantic)segmentation

Adapted

Ground,TruthInput,image

Source5only,model

Kate%Saenko,%Boston%University

Pixel&to&pixel*adaptation

Kate%Saenko,%Boston%University

Strong'weak,feature,alignment

Strong'Weak,Distribution,Alignment,for,Adaptive,Object,Detection,,Saito,,Ushiku,,Harada,,Saenko,,arxiv,2018

Kate%Saenko,%Boston%University

Strong'weak,feature,alignment

Strong'Weak,Distribution,Alignment,for,Adaptive,Object,Detection,,Saito,,Ushiku,,Harada,,Saenko,,arxiv,2018

Kate%Saenko,%Boston%University

ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline

Domain%shift:%from%PASCAL%VOC%to%CLIPART

Kate%Saenko,%Boston%University

Evidence(for(target(domain(label((GradCAM)(shows(that(the(feature(extractor(seems(to(deceive(the(domain(classifier(in(regions(with(car.

Domain(shift:(from(Grand(Theft(Auto(game(to(Cityscapes

target

source

Kate%Saenko,%Boston%University

ours%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%baseline

Domain%shift:%from%Cityscapes%to%Foggy%Cityscapes

adaptiveexplainablemodular

CNNs learn to predict pneumonia by detecting hospital which took the image

Variable(generalization(performance(of(a(deep(learning(model(to(detect(pneumonia(in(chest(radiographs:(A(cross8sectional(study.(Zech%JR1,%Badgeley%MA2,%Liu%M2,%Costa%AB3,%Titano%JJ4,%Oermann%EK3.%https://www.ncbi.nlm.nih.gov/pubmed/30399157

● Study%on%detecting%pneumonia%using%158,323%chest%radiographs

● CNNs%robustly%identified%hospital%system%and%department%within%a%hospital

● CNN%has%learned%to%detect%a%metal%token%that%radiology%technicians%place%on%the%patient%in%the%corner%of%the%image%field%of%view%at%the%time%they%capture%the%image

RISE:&randomly&mask&input,&measure&output

42

P[shark]

shark

RISE:&Randomized&Input&Sampling&for&Explanation&of&Black:box&Models,&Petsiuk,&Das,&Saenko,&BMVC&2018

adaptiveexplainablemodular

Explainable*Neural*Computation*via*Stack*Neural*Module*Nets*Networksinput:*There*is*a*small*gray*block;*are*there*any*spheres*to*the*left*of*it?

Hu,$Andreas,$Darrell,$Saenko,$Explainable$Neural$Computation$via$Stack$Neural$Module$Networks,$ECCV’18

input*image

Disentangling*properties*via*generation

A"Two&Stream"Variational"Adversarial"Network"for"Video"Generation

generated"video

Disentangling*properties*via*generation

A*Two2Stream*Variational*Adversarial*Network*for*Video*Generation,*arxiv*2018

Disentangling*properties*via*generation

Disentangling*properties*for*domain*transfer

adaptiveexplainablemodular