presentation on mediaeval search & linking task 2013
DESCRIPTION
Presentation on the set-up of the MediaEval Search & Linking task 2013TRANSCRIPT
![Page 1: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/1.jpg)
Set-‐up of the Search & Hyperlinking Task
in the Series
Evalua'on of Video Hyperlinking
Roeland Ordelman -‐ Audiovisual Content Access
University of Twente, The Netherlands Netherlands Ins@tute for Sound and Vision
Cross-‐Media Interac@on
![Page 2: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/2.jpg)
Video hyperlinking
• Topic with increasing aEen@on • Relate video (fragments) to other mul@media sources • Various perspec@ves and approaches, e.g.,: – Recommenda@on/Detail-‐on-‐demand – Seman@c/Event Web – Storytelling/narra@ves – Summariza@on/Clustering
• Link up communi@es by means of special sessions, workshops and benchmarks – Mul@media Modeling (MMM 2014) – Dublin. Special session on
Mul@media Hyperlinking and Retrieval – CfP due 01 August 2013 hEp://mmm2014.org/special-‐sessions/
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 3: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/3.jpg)
MediaEval
• a mul$media benchmarking ini@a@ve running several tasks
• evaluates new algorithms for mul@media access and retrieval linked to a use case
• emphasis on the "mul$" in mul@media: audio, speech, visual content, tags, users, context.
• new evalua@on paradigms involving users
h3p://www.mul:mediaeval.org
![Page 4: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/4.jpg)
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 5: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/5.jpg)
Video linking scenario
• Audiovisual archive explora$on scenario • Video-‐to-‐video linking • Start with archive search:
assuming an “informa$on need” • Poten@ally search may go into:
serendipitous or exploratory type of behavior • Create meaningful cross-‐sec:ons through archive • Audiovisual archive interlinked on the fragment level • Automated “Storytelling” type of approaches
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 6: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/6.jpg)
Terminology • Video (e.g, 2 hours)
• Interes@ng segment (e.g. 10 min)
• Anchor: segment for which a user requests a link (e.g., 1 min) “I want to know more about this”
• Hyperlink
• Target: relevant segment for given anchor (e.g., 5 min)
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 7: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/7.jpg)
ME 2013 Search & Hyperlinking • ME 2012 S&HL “brave new” task: – Search: retrieve known-‐item video segment given a natural language descrip@on
– Linking: link known-‐item video segment to similar segments in the collec@on (blib.tv)
• ME 2013 S&HL “regular” task – Search: retrieve known-‐item video segment given a natural language descrip@on
– Linking: link user-‐defined anchor within the known-‐item to relevant target video segments in the collec@on (bbc archive)
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 8: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/8.jpg)
ME 2013 Search & Hyperlinking
7/2/13 DGA workshop -‐ July 2013, Paris
Anchor Jump-‐in point
![Page 9: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/9.jpg)
Task set-‐up
1. Organizers: provide anchors to par@cipants
2. Par@cipants: find relevant targets
3. Organizers: evaluate relevance targets
4. Organizers+Par@cipants: discuss at workshop 7/2/13 DGA workshop -‐ July 2013, Paris
![Page 10: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/10.jpg)
1. Provide anchors
• Requirements: – Large video data set – Real ‘users’ selec@ng anchors
• Data: – 270h of BBC archival data
• One complete week • Excluded: material not owned by BBC (e.g., Dr. Who)
• Users: …
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 11: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/11.jpg)
Researchers & Educators AXES Research system
Journalists Research
Academic researchers & students
Inves:gate
Academic educators
Educate
Home users AXES Home system
Home users Entertainment, Edutainment
Main group User Target
Media Professionals AXES Pro system
Broadcast Professionals
Reuse
Media Archivists Annotate
User requirements study
7/2/13 DGA workshop -‐ July 2013, Paris
!
!!
![Page 12: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/12.jpg)
Users
• Recruitment Agency • In an@cipa@on on a rather complex task: public users with computer & online experience
• 30 Public Users – Na@onality: UK – Age group: 16-‐30 – Computer familiarity: high – Online ac@vity: high
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 13: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/13.jpg)
ANCHOR SELECTION BBC User Trials
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 14: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/14.jpg)
Search for an interes@ng video
7/2/13 DGA workshop -‐ July 2013, Paris
potentially interesting videoSEARCH
BBC ARCHIVE
![Page 15: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/15.jpg)
7/2/13 DGA workshop -‐ July 2013, Paris
Formulate Informa@on
need
Text search Visual search
AXES PRO system
![Page 16: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/16.jpg)
Select an interes@ng clip
7/2/13 DGA workshop -‐ July 2013, Paris
BBC ARCHIVE
potentially interesting video
interesting clip
SEARCH
![Page 17: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/17.jpg)
Select an anchor
7/2/13 DGA workshop -‐ July 2013, Paris
BBC ARCHIVE
potentially interesting video
interesting clip
interesting segment for linking
SEARCH
![Page 18: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/18.jpg)
7/2/13 DGA workshop -‐ July 2013, Paris
Describe target
Describe anchor
Select anchor using sliders
![Page 19: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/19.jpg)
Trial evalua@on
• Most par@cipants understood the task • Not difficult • Technology would be a very nice-‐to-‐have men@oning the “serendipitous” type of browsing behavior
• Par@cipants want music as link anchors (which we did not an@cipate on!)
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 20: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/20.jpg)
Classifica@on of selected clips
7/2/13 DGA workshop -‐ July 2013, Paris
0
5
10
15
20
25
![Page 21: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/21.jpg)
Classifica@on of anchors
7/2/13 DGA workshop -‐ July 2013, Paris
0
10
20
30
40
50
60
70
80
90
Whole Scene Speech Other Object, moving
Object, sta:c
First FA cup Final at Wembley
Mayor Bloomberg of New York talking about Ken Livingstone
![Page 22: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/22.jpg)
2. Automa@c target selec@on • ~ 15 Par@cipants
• Data – 50 NL queries, 50 anchors – 270h of BBC archival data (used for anchor selec@on) – + One month before ~1000h
• Metadata – Archival metadata – Sub@tles – Speech transcripts (provided by LIMSI & LIUM) – Shot segmenta@on – (Self-‐derived features)
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 23: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/23.jpg)
7/2/13 DGA workshop -‐ July 2013, Paris
BBC ARCHIVE
potentially interesting video
interesting clip
interesting segment for linking
related video clip
video hyperlink
SEARCH
LINK
![Page 24: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/24.jpg)
3. Evaluate target relevance
7/2/13 DGA workshop -‐ July 2013, Paris
BBC ARCHIVE
potentially interesting video
interesting clip
interesting segment for linking
related video clip
video hyperlink
SEARCH
LINK
RelevanceAssessment
![Page 25: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/25.jpg)
Relevance assessment • Default procedure:
– Par@cipants of 1st user trials – Evaluate for their own anchors the automa@cally generated targets from
ME par@cipants • Risks:
– No shows – Par@cipants can only evaluate top-‐N (N-‐max=5?) targets for a given
anchor • Fall-‐back op@on:
– Use Amazon Mechanical Turk approach – (same as we did in ME-‐2012 task) – Drawback
• relevance assessment not based on personal trail “informa@on need” >> “clip selec@on” >> “anchor selec@on” >> “target evalua@on”
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 26: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/26.jpg)
Assessment interface
7/2/13 DGA workshop -‐ July 2013, Paris
Original anchor & target
descrip@on
ANCHOR SUGGESTED TARGETS
Relevance assessment & comments
![Page 27: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/27.jpg)
Evalua@on metrics • MAP evalua@on for both searching and linking • Issues: – MAP does not provide means to evaluate segmenta@on (e.g., correct video but jump-‐in point is incorrect)
– limit relevant results to a window of N (N=60 seconds) before and awer the actual beginning of the relevant segment
• Addi@onal metric: – Mean Generalized Average Precision (mGAP)
• awards runs that are closer to the jump-‐in point of the relevant content
7/2/13 DGA workshop -‐ July 2013, Paris
![Page 28: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/28.jpg)
4. Discuss at workshop
7/2/13 DGA workshop -‐ July 2013, Paris
h3p://www.mul:mediaeval.org
18-‐19 October 2013 Barcelona, Spain
![Page 29: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/29.jpg)
7/2/13 DGA workshop -‐ July 2013, Paris
Work funded by:
Task Organizers: Robin Aly Maria Eskevitch Gareth Jones Roeland Ordelman
h3p://www.mul:mediaeval.org
![Page 30: Presentation on MediaEval Search & Linking task 2013](https://reader034.vdocument.in/reader034/viewer/2022042714/54b9fbc24a795984188b4569/html5/thumbnails/30.jpg)