constrained semi-supervised learning using attributes and comparative attributes

CONSTRAINED SEMI-SUPERVISED LEARNING

USING ATTRIBUTES AND COMPARATIVE ATTRIBUTESAbhinav Shrivastava, Saurabh Singh, Abhinav Gupta

The Robotics InstituteCarnegie Mellon University

2

SUPERVISION

SUPERVISED

ACTIVELEARNING

BIG-DATA

[von Ahn and Dabbish, 2004], [Russell et al., IJCV 2009][Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007][Qi et al., CVPR 2008] [Joshi et al., CVPR 2009] [Siddiquie and Gupta, CVPR 2010]

3

SUPERVISION

SUPERVISED

UNSUPERVISED

ACTIVELEARNING

[Russell et al., CVPR 2006] [Kang et al., ICCV 2011]

4

SUPERVISION

SUPERVISED

UNSUPERVISED

ACTIVELEARNINGSEMI-

SUPERVISED

[Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009]

5

SUPERVISION

SUPERVISED

UNSUPERVISED

ACTIVELEARNINGSEMI-

SUPERVISEDBOOTSTRAPPINGLabeled Seed

ExamplesAmphitheatre

Unlabeled Data

Select Candidates

TrainModels

Add to Labeled Set

RetrainModels

Amphitheatre

6

SUPERVISION

SUPERVISED

UNSUPERVISED

ACTIVELEARNINGSEMI-

SUPERVISEDBOOTSTRAPPINGRetrainModels

Labeled Seed Examples

Amphitheatre

Unlabeled Data

Select Candidates

Add to Labeled Set

Amphitheatre

25th Iteration

[Curran et al., PACL 2007]

Semantic Drift

Amphitheatre + Auditorium

7

SUPERVISION

SUPERVISED

UNSUPERVISED

ACTIVELEARNINGSEMI-

SUPERVISEDGRAPH-BASED METHODS

[Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]

8

Amphitheatre Amphitheatre

OUR APPROACHAmphitheatre

Auditorium

Amphitheatre

Auditorium

9

OUR APPROACHAmphitheatre

Auditorium

Amphitheatre

Auditorium

Joint Learning

[Carlson et al., NAACL HLT Workshop on SSL for NLP 2009]

Share Data

AmphitheatreAmphitheatre

AuditoriumAuditorium

BanquetHall

BanquetHall

Conference Room

Conference Room

BINARY ATTRIBUTES (BA)

Indoor Man-madeTables and Chairs Large Seating CapacityIndoor Man-madeTables and Chairs Large Seating Capacity

10[Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]

BINARY ATTRIBUTES (BA)Tables and Chairs

Conference Room

BanquetHall

Auditorium

Amphitheatre

Indoor

Large Seating Capacity

Man-made

[Patterson and Hays, CVPR 2012]

Tables and Chairs

Conference Room

BanquetHall

Auditorium

Amphitheatre

Indoor

Large Seating Capacity

Man-made

12

AuditoriumIndoor Has Seat Rows

✗

13

SHARING VIA DISSIMILARITY

Amphitheatre Auditorium

Has Larger Circular Structures

[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

14

Amphitheatre AuditoriumHas Larger

Circular Structures

?

15

✗

Amphitheatre AuditoriumHas Larger

Circular Structures

16

DISSIMILARITY

Has Larger Circular Structures

[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

COMPARATIVE ATTRIBUTES

17

DISSIMILARITYCOMPARATIVE ATTRIBUTES

Has Larger Circular

Structures

[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

……

……

……

……

Features• GIST• RGB (Tiny Image)• Line Histogram of:

Length Orientation• LAB histogram

[Hays and Efros, SIGGRAPH 2007][Oliva and Torralba, 2006][Torralba et al., PAMI, 2008]

18

……

……

DISSIMILARITYCOMPARATIVE ATTRIBUTES

[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

……

…… Has Larger

Circular Structures

ClassifierBoosted Decision Tree[Hoiem et al., IJCV 2007]

✗or

Has Larger Circular

Structures

19

COMPARATIVE ATTRIBUTES

[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]

Amphitheatre > BarnAmphitheatre > Conference Room

Desert > Barn

Is More Open

Church (Outdoor) > CemeteryBarn > Cemetery

Has Taller Structures

20

Amphitheatre

Auditorium

Amphitheatre

Auditorium

Labeled Seed Examples Bootstrapping

Selected Candidates

21

Labeled Seed Examples

Amphitheatre

Auditorium

Amphitheatre

Auditorium

Bootstrapping

Amphitheatre

Auditorium

Our Approach (Constrained Bootstrapping)

Selected Candidates

Indoor

Has Seat Rows

Attributes

Has Larger Circular

Structures

ComparativeAttributes

22

Banq

uet

Bedr

oom

Labeled Data

Unlabeled Data

has more space

has larger structures

Training Pairwise Data

Promoted InstancesConference Room Banquet Hall

[Gupta and Davis, ECCV 2008]

Comparative Attribute Classifiers

mor

e sp

ace

larg

er st

ruct

ures

Attribute Classifiersin

door

has g

rass

Scene Classifiers

bedr

oom

banq

uet h

all

23

Conference Room

Itera

tion

1Ite

ratio

n 40

Seed

Ex

ampl

es

Introspection

24

Boot

stra

ppin

gBA

Con

stra

ints

AmphitheatreO

ur A

ppro

ach

Seed

Imag

es

26

EXPERIMENTAL EVALUATION

27

CONTROL EXPERIMENTS# Images

(SUN Database)

15 Scene Classes 2 (seed)

Black-box Binary Attributes (BA) 25 (separate)

Black-box Comparative Attributes (CA) 25 (separate)

Unlabeled Dataset 18,000(Distractors) (9,500)Test Set 50

SUN Database : [Xiao et al., CVPR 2010]

28

CONTROL EXPERIMENTS

29

CONTROL EXPERIMENTS

30

CONTROL EXPERIMENTS

31

CONTROL EXPERIMENTS

32

CONTROL EXPERIMENTS

\\

33

CONTROL EXPERIMENTS

\\

Eigen Functions: [Fergus et al., NIPS 2009]

34

1

40

Banquet Hall

10

Itera

tions

Seed

Imag

es

35

CONTROL EXPERIMENTS# Images

(SUN Database)

15 Scene Classes 2 (seed)

Black-box Binary Attributes (BA) 25 (separate)

Black-box Comparative Attributes (CA) 25 (separate)

Unlabeled Dataset 18,000(Distractors) (9,500)Test Set 50

SUN Database : [Xiao et al., CVPR 2010]

36

CO-TRAINING (SMALL SCALE)# Images

(SUN Database)

15 Scene Classes 2 (seed)

Black-box Binary Attributes (BA) 15x2 (seed)

Black-box Comparative Attributes (CA) 15x2 (seed)

Unlabeled Dataset 18,000(Distractors) (9,500)Test Set 50

SUN Database : [Xiao et al., CVPR 2010]

37

Itera

tion-

1Ite

ratio

n-60

Boot

stra

ppin

gO

ur A

ppro

ach

Itera

tion-

1Ite

ratio

n-60

Seed

Imag

esBedroom

38

SCENE CLASSIFICATION

Eigen Functions: [Fergus et al., NIPS 2009]

39

CO-TRAINING (LARGE SCALE)• 15 Scene Categories

25 Seed images / category

• Unlabeled Set 1Million (SUN Database + ImageNet) >95% distractors

SUN Database: [Xiao et al., CVPR 2010]ImageNet: [Deng et al., CVPR 2009]

Improve 12 out of 15 scene classifiers

40

CONCLUSION• Sharing via Dissimilarities

• Constrained Bootstrapping

AuditoriumAmphitheatre

Has Larger Circular

Structures

Labeled Seed Examples

Amphitheatre

Auditorium

Amphitheatre

Auditorium

Bootstrapping

Amphitheatre

Auditorium

Constrained Bootstrapping

41

Banq

uet

Bedr

oom

Labeled Data

Unlabeled Data

has more space

has larger structures

Training Pairwise Data

Promoted InstancesConference Room Banquet Hall

Comparative Attribute Classifiers

mor

e sp

ace

larg

er st

ruct

ures

Attribute Classifiersin

door

has g

rass

Scene Classifiers

bedr

oom

banq

uet h

all

42

43

Unary

Binary

44

EIGEN FUNCTIONSSettings from Fergus et al. NIPS, 2009• GIST descriptor• 32D space using PCA– k=64 eigen functions

45

FEATURES• 960D GIST• 75D RGB (image is resized to 5x5)• 30D histogram of line lengths• 200D histogram of orientation of lines• 784D 3D-histogram Lab color space (14x14x4)

• Total 2049

[Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]

46

CATEGORIES

47

BINARY ATTRIBUTE RELATIONSHIP

48

COMPARATIVE ATTRIBUTE RELATIONSHIPS

49

CLASSIFIERS• Boosted Decision Trees– From [Hoiem et al., IJCV 2007]

• Scene– 20 Trees, 8 Nodes

• Binary Attribute– 40 Trees, 8 Nodes

• Comparative Attribute• 20 Trees, 4 Nodes• Differential features as in [Gupta and Davis, ECCV 2008]

50

SUPERVISION

SUPERVISED

ACTIVELEARNING

[Torralba et al., PAMI 2008]

51

SUPERVISION

SUPERVISED

UNSUPERVISED

ACTIVELEARNINGSEMI-

SUPERVISEDGRAPH-BASED METHODS

Train Bus

52

MUTUAL EXCLUSION (ME)

53

EXPERIMENTAL EVALUATION

Experiments• Control• Small-scale• Large-scale

Evaluation Metrics• Scene Classification– Mean AP

• Purity of Labels

Datasets• SUN Database• ImageNet

15 Scene Categories

54

BASELINES• Bootstrapping (Self-learning)

- Binary Classifiers- Multi-class Classifiers

• Eigen Function based Graph Laplacian

55

SCENE CLASSIFICATION

Eigen Functions: [Fergus et al., NIPS 2009]

56

PURITY OF LABELING

Eigen Functions: [Fergus et al., NIPS 2009]

57

58

1

40

Banquet Hall

90

10Itera

tions

Seed

Imag

es

59

60

PURITY OF LABELING

Eigen Functions: [Fergus et al., NIPS 2009]

61

SCENE CLASSIFICATION

Mean