cluster based landmark and event detection for tagged photo collections
DESCRIPTION
A simple presentation of the article: "Cluster-based landmark and event detection for tagged photo collections" on the IEEE MultiMedia magazine. http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5611558TRANSCRIPT
Cluster-based Landmark and Event Detection on Tagged Photo Collections
Symeon Papadopoulos, Christos Zigkolis,
Yiannis Kompatsiaris, Athena Vakali
user generated content creates new opportunities
real-world depicted in users’ online collections
need new tools for content organization
potential for many insights into what people see, do and like
image clustering
event
landmark
clusters landmarks + events
the framework
+ +
photos tags geo
1 2
event
landmark
3
landmark
4
overview
step 1: create photo similarity graph
1 2
event
landmark
3
landmark
4
tag similarity
visual similarity casa mila, la pedrera
co-occurrence
latent semantic indexing
SIFT
SURF
step 2: use graph to cluster the photos
1 2
event
landmark
3
landmark
4
v
neighborhood of node v + node itself = structure of node v
v v
N(v) v Γ(v)
the concept of node structure
u v
Γ(v) Γ(u) ∩
Γ(v) Γ(u)
structural similarity between nodes v and u
the concept of structural similarity (1)
C
B A
high structural similarity
low structural similarity
photo cluster 1
photo cluster 2
the concept of structural similarity (2)
O (km m)
average node degree
# edges
graph-based clustering
k-means clustering O (I C n D)
hierarchical agglomerative clustering
O (n2 log n)
# iterations
# clusters
# nodes
# dimensions
complexity
step 3: detect landmarks & events
event
landmark landmark
1 2
3 4
#users / #photos
duration
[1 day, 2 users / 10 photos]
[2 years, 50 users / 120 photos]
Quack et al., CIVR 2008
baseline features
Event Tags
Landmark Tags additional features
step 4: post-process landmark clusters
event
landmark landmark
1 2
3 4
cluster merging based on proximity
low frequency tags
helado tropical
park güell jaume oller
barcelona spain cielos
park
field
sclupture el beso
generic tags
CLUSTER TAGS
cluster tag filtering
results
207,750 photos
7,768 users
33,959 unique tags
compare graph-based vs. k-means clustering
user study geospatial coherence
high geospatial coherence
low geospatial coherence
user study
precision recall κ-statistic
graph-based
k-means
VISUAL
1.000
0.806
0.110
0.324
1.000
0.226
precision recall κ-statistic
graph-based
k-means
TAG
0.950
0.848
0.182
0.307
0.820
0.564
geospatial coherence
radius std. deviation
graph-based
k-means
VISUAL
357 m
2.4 km
1.18 km
1.73 km
graph-based
k-means
TAG
456 m
767 m
1.15 km
1.76 km
357 m 1.18 km
456 m 1.15 km
classification performance
16% - 23% improvement thanks to tag features
sagrada familia, cathedral, catholic 15.2m
la pedrera, casa mila 31.8m
parc guell 9.6m
boqueria, market, mercado, ramblas 82.1m
camp nou, fc barcelona, nou camp 18.7m
landmark localization accuracy
music, concert, gigs, dj 43.1%
conference, presentation 6.5%
local traditional, parades 4.6%
racing, motorbikes, f1 3.3%
event category composition
clusttour
www.clusttour.gr
twitter.com/clusttour facebook.com/clusttour