Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, Aude Oliva
High-level attributes of images: How memorable is an image?
Motivations
How to measure memorability?
What content makes an image memorable?
Prediction algorithms
Applications and future directions
Prediction algorithm: SVM Regression with non-linear kernels on following features:
“Aquarium, indoors” 1) Scene annotationsscene categories of entire image
= 0.42
Predicting image memorability
Database
Memory Game
...Vigilance repeat
Memory repeat
100
1-7 back
91-109 back time
! ! ! ! !
665 participants on Amazon’s Mechanical Turk.
200 1000 1800
40%
50%
60%
70%
80%
90%
100%
Image rank N, according to specified group
Average % memorability, according to
Group 1, of 25 images
centered about rank N
Group 1
Group 2
Chance
= 0.75Memorable
Average
Forgettable
What content makes an image memorable?
Object score = (prediction when object included in image’s feature vector) - (prediction when object removed)
! 0.15 + 0.090
...
person sittingbuildingmountain personfloorskytree seats
natural lake (52%)
broadleaf forest (52%)
art studio (81%)
campus (53%)
bedroom (76%)
bakery shop (81%)
botanical garden (52%)
bathroom (84%)
...
Objects shaded according to object score (computed per
image)
Objects ranked according
to object score (averaged
across images)
Scenes ranked according to their average
memorability
Database: 2222 photographs from SUN database (Xiao et al. 2010).
Memorability = probability of correctly detecting a repeat after a single view of an image in a long stream.
Wide range of memorabilities and high inter-subject consistency
enclosed_space < 0.47
face_visible < 0.49
single_focus < 0.20 peaceful < 0.75 peaceful < 0.75
face_visible < 0.21
recognize < 0.55 recognize < 0.450.78
0.56
0.64
0.62 0.69 0.73
0.65 0.61 0.83
U
M
corr: 0.11
A
M
corr: 0.28
Aesthetics versus memorability
Unusualness versus memorability
Understanding memorability
2 4 6 80.360.38
0.40.420.440.460.48
# Features
Ran
k co
rr
TestingTraining
2) Object annotationsnumber, size, and rough position of each object class
= 0.49
“Funny, peaceful, eye contact”
3) Attribute annotations = 0.53Understandable attributes describ-ing layout, aesthetics, emotions, ac-tions, and appearche of people
5) All features = 0.59All annotations and global image features
GIST
SIFT HOG SSIM
Pixels4) Global image features
pixel histograms, GIST, spatial pyramids of SIFT, HOG, SSIM
= 0.47
Understanding memorability Applications
Predicted memorable
Predicted average
Predicted forgettable
Automatic predictions from global image features
Information theoretic feature selection
Retrieve better images
from search
Make an image morememorable
Understandhuman memory
Diagnose memory
problems
Summarize photo album
or video
Design mnemonic
aids
! !
“lourds”
“heavy”
