Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, Aude Oliva

High-level attributes of images: How memorable is an image?

Motivations

How to measure memorability?

What content makes an image memorable?

Prediction algorithms

Applications and future directions

Prediction algorithm: SVM Regression with non-linear kernels on following features:

“Aquarium, indoors” 1) Scene annotationsscene categories of entire image

= 0.42

Predicting image memorability

Database

Memory Game

...Vigilance repeat

Memory repeat

100

1-7 back

91-109 back time

! ! ! ! !

665 participants on Amazon’s Mechanical Turk.

200 1000 1800

40%

50%

60%

70%

80%

90%

100%

Image rank N, according to specified group

Average % memorability, according to

Group 1, of 25 images

centered about rank N

Group 1

Group 2

Chance

= 0.75Memorable

Average

Forgettable

What content makes an image memorable?

Object score = (prediction when object included in image’s feature vector) - (prediction when object removed)

! 0.15 + 0.090

...

person sittingbuildingmountain personfloorskytree seats

natural lake (52%)

broadleaf forest (52%)

art studio (81%)

campus (53%)

bedroom (76%)

bakery shop (81%)

botanical garden (52%)

bathroom (84%)

...

Objects shaded according to object score (computed per

image)

Objects ranked according

to object score (averaged

across images)

Scenes ranked according to their average

memorability

Database: 2222 photographs from SUN database (Xiao et al. 2010).

Memorability = probability of correctly detecting a repeat after a single view of an image in a long stream.

Wide range of memorabilities and high inter-subject consistency

enclosed_space < 0.47

face_visible < 0.49

single_focus < 0.20 peaceful < 0.75 peaceful < 0.75

face_visible < 0.21

recognize < 0.55 recognize < 0.450.78

0.56

0.64

0.62 0.69 0.73

0.65 0.61 0.83

U

M

corr: 0.11

A

M

corr: 0.28

Aesthetics versus memorability

Unusualness versus memorability

Understanding memorability

2 4 6 80.360.38

0.40.420.440.460.48

# Features

Ran

k co

rr

TestingTraining

2) Object annotationsnumber, size, and rough position of each object class

= 0.49

“Funny, peaceful, eye contact”

3) Attribute annotations = 0.53Understandable attributes describ-ing layout, aesthetics, emotions, ac-tions, and appearche of people

5) All features = 0.59All annotations and global image features

GIST

SIFT HOG SSIM

Pixels4) Global image features

pixel histograms, GIST, spatial pyramids of SIFT, HOG, SSIM

= 0.47

Understanding memorability Applications

Predicted memorable

Predicted average

Predicted forgettable

Automatic predictions from global image features

Information theoretic feature selection

Retrieve better images

from search

Make an image morememorable

Understandhuman memory

Diagnose memory

problems

Summarize photo album

or video

Design mnemonic

aids

! !

“lourds”

“heavy”