relative attributes - university of texas at...

51
Relative Attributes Devi Parikh and Kristen Grauman ICCV 2011 Experiment Presentation by Aishwarya Padmakumar

Upload: others

Post on 04-Aug-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative AttributesDevi Parikh and Kristen Grauman

ICCV 2011

Experiment Presentation by Aishwarya Padmakumar

Page 2: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Motivation

Image Source: https://filebox.ece.vt.edu/~parikh/relative.html

Sometimes it is hard to make a binary decision on whether an image satisfies an attribute. Comparisons are easier.

Page 3: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Sources

Code and data for following experiments obtained from :https://filebox.ece.vt.edu/~parikh/relative.html#code

Page 4: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Pubfig dataset

→ Public Figures Face Database - Images of famous public figures

→ Images of the same person are grouped into categories→ Attributes - Male, White, Young, Smiling, Chubby,

VisibleForehead, BushyEyebrows, NarrowEyes, PointyNose,BigLips, RoundFace

Page 5: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Ground Truth - Ordering of categories for each attribute

Attr1 Attr2 Attr3 Attr4

Class1 1 2 3 3

Class2 2 1 1 1

Class3 3 3 2 2

Page 6: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Evaluating Ranking

→ Obtain training pairs using category ranking→ Train SVM to get a score for each image→ Average scores of images in category to get score of

category→ Rank categories accoridng to this score→ Compare to ground truth ranking

Page 7: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Ranking PredictionC

lass

es

Attributes Rank difference

0

1

2

3

4

5

6

7

Page 8: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Varying the amount of training data

Page 9: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

With original amount of training dataC

lass

es

Attributes Rank difference

0

1

2

3

4

5

6

7

Page 10: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

With 80% of original training dataAttributes

Cla

sses

Page 11: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

With 60% of original training dataAttributes

Cla

sses

Page 12: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

With 40% of original training dataAttributes

Cla

sses

Page 13: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

What sort of mistakes does it make?

Page 14: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Failure cases - Who has bigger lips?

Page 15: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Failure cases - Who has bigger lips?

Supposed to be the person on the right but certain expressions don’t indicate this.

Page 16: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Failure cases - Who looks younger?

Page 17: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Failure cases - Who looks younger?

Supposed to be the girl on the right. Other images are more obvious but expression, make-up and lighting affect this.

Page 18: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Instance Level Supervision vs Category Level Supervision

Page 19: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

WhittleSearch Dataset

→ Collected by: Adriana Kovashka, Devi Parikh, and Kristen Grauman

→ Publication: Whittle Search: Image Search with Relative Attribute Feedback, in CVPR 2012

→ Dataset of shoes: 14658 images→ Provide extracted features (GIST and colour - LAB

histogram), accounting for image orientation

Page 20: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

→ Category level ordering and instance level comparisons for each attribute

→ Attributes: pointy-at-the-front, open, bright-in-color, covered-with-ornaments, shiny, high-at-the-heel, long-on-the-leg, formal, sporty, feminine

→ Instance level ordering obtained via MTurk

WhittleSearch Dataset

Page 21: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Examples

Women’s Boots

Women’s flatsWomen’s stiletto

Page 22: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Instance level supervision vs category level supervisionA

ccur

acy

Page 23: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Why does training on category level information do well sometimes?→ List of categories - athletic_shoes, boots, clogs, flats,

high_heels, pumps, rain_boots, sneakers, stiletto, wedding_shoes

→ Pointy, ornaments, long on the leg seem to be attributes that we can associate with some of these classes

→ Possibly category level labels result in more well-separated data, making it easier to train an SVM (SVMs are sensitive to noise near the support vectors)

Page 24: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Some cases where the prediction differs

More Sporty: Ground truth - left; Instance based - left; Category based - right

Page 25: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Some cases where the prediction differs

More Sporty: Ground truth - left; Instance based - right; Category based - left

Page 26: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Some cases where the prediction differs

More Sporty: Ground truth - equal; Instance based - right; Category based - left

Page 27: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Usefulness of different types of features

Page 28: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Comparing GIST and colour features - instance comparisons

Page 29: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Comparing GIST and colour features - category comparisons

Page 30: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Where GIST and the combination differ

More Bright: Ground truth - right; GIST - equal; Both - right

Page 31: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Where GIST and the combination differ

More Bright: Ground truth - left; GIST - equal; Both - right

Page 32: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Where colour and the combination differ

More Bright: Ground truth - right; Colour - equal; Both - right

Page 33: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Where colour and the combination differ

More Bright: Ground truth - equal; Colour - equal; Both - left

Page 34: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions

Page 35: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions

Page 36: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - Pointy at the front

Page 37: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - open

Page 38: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - bright in colour

Page 39: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - covered in ornaments

Page 40: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - shiny

Page 41: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - high at the heel

Page 42: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - long on the leg

Page 43: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - formal

Page 44: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - sporty

Page 45: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Relative Descriptions - feminine

Page 46: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Varying the number of images in between

Page 47: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Leave 1/16 of the images in between (formal)

Page 48: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Leave 1/8 of the images in between (formal)

Page 49: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Leave 1/4 of the images in between (formal)

Page 50: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Leave 1/2 of the images in between (formal)

Page 51: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman

Thank You!