multimodal synchronization of image galleries
TRANSCRIPT
MULTIMODAL SYNCHRONIZATION OF IMAGE GALLERIES
Maia Zaharieva Michael Riegler Manfred Del Fabro
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
GENERAL IDEA
• Cluster image collections using visual features
• Synchronize time based on cluster membership
• Cluster (again) for sub-event detection
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
AHC-BASED APPROACH
• Explore AHC at different hierarchy levels
• MPEG-7 Color Structure (CS) descriptor
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
AHC-BASED APPROACH• Image synchronization @ lowest hierarchy level
• Aim: find a transitive list of entry points to all galleries • sort image pairs by dissimilarity • two images are identical if:
• different galleries
• dissimilaritythreshold
➡ entry point for the gallery
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
AHC-BASED APPROACH• Sub-event detection @ higher hierarchy level
• fixed threshold: Ward method • reduce potential over-segmentation using time
information: merge two clusters if: • share common
gallery • min time
difference below a threshold
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
XMEANS-BASED APPROACH
• Visual features • Modification of LIRE framework
• 13 global features • Feature selection: information gain • Feature combination: late fusion
• Best-performing feature: ➡ Joint Composite Descriptor (JCD)
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
XMEANS-BASED APPROACH
• Time synchronization ➡ average deviation of the reference image
timestamps to all other images of a collection
• Sub-event detection ➡ XMeans + Time/JCD
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
RESULTS: DEVELOPMENT SET• 304 images, 10 galeries, 59 sub-events
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
galery AHC-based approach
XMeans-based approach1 1 0
2 337 75603 -15 12604 380 3605 0 06 -16 -8407 380 64208 -1250 -1809 382 696010 -14 624average deviation in sec: 18.5 2216.4
• Time offset:
RESULTS: DEVELOPMENT SET
• Sub-event detection:
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
C R P F1 NMITime-based 98 0.4738 0.8862 0.6363 0.8696AHC + MPEG7-CS 91 0.4426 0.7412 0.5543 0.8179AHC + MPEG7-CS + Time 45 0.7571 0.5399 0.6303 0.7927XMeans + JCD 89 0.4600 0.5800 0.5123 0.7812XMeans + Time 100 0.5000 0.6700 0.5731 0.8231
• 304 images, 10 galeries, 59 sub-events
RESULTS: TEST SET
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain
Vancouver LondonC RI F1 C RI F1
(1) + AHC + MPEG7-CS 379 0.9787 0.1012 368 0.9842 0.2614(1) + Time 709 0.9782 0.0505 709 0.9873 0.1687(2) + XMeans + JCD 91 0.9619 0.1087 91 0.9760 0.1331(2) + XMeans + Time 81 0.9687 0.0890 81 0.9797 0.1653(1) + XMeans + Time 98 0.9727 0.1079 98 0.9797 0.1653
Vancouver LondonP A P A
1 AHC + MPEG7-CS 0.9412 0.7919 0.4722 0.87462 XMeans + JCD 0.5882 0.5701 0.3611 0.4676
Time offset:
Sub-event detection:
QUESTIONS?
maia.zaharieva@[tuwien|univie].ac.at
MediaEval Workshop, October 16-17, 2014, Barcelona, Spain