multimedia information retrieval · kmi.open.ac.uk since 1995: 117 projects & 67 technologies...
TRANSCRIPT
Multimedia Information Retrieval
Prof Stefan Rüger
Multimedia and Information SystemsKnowledge Media Institute
The Open Universityhttp://kmi.open.ac.uk/mmis
kmi.open.ac.uk
kmi.open.ac.uk
kmi.open.ac.uk
Since 1995: 117 projects & 67 technologies
Current year
17 live projects , typically per year£2.5m (¥300m) ext, £1m (¥120m) internal• 10 EU• 3 UK • 1 US• 3 internal (iTunes U, SocialLearn)
Multimedia Information Retrieval
1. What are multimedia queries?
2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation
8. Browsing, search and geography
Multimedia Information Retrieval
1. What are multimedia queries? - What is multimedia? - Query by image - Current best practice for image search - Snaptell/Google goggles - Shazam - Discussion: Challenges and difficulties
2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation8. Browsing, search and geography
What is Multimedia?
Within this lecture:One or more mediaPossibly interlinkedDigitalFor communication (not only entertainment)
Sensō-ji ( � � � � � � Kinryū-zan Sensō-ji?) is an ancient Buddhist templelocated in Asakusa, Taitō, Tokyo, Japan. It is Tokyo's oldest temple, and one of its most significant. Formerly associated with the Tendai sect, it became independent after World War II. Adjacent to the temple is a Shinto shrine,the Asakusa Shrine [http://en.wikipedia.org/wiki/Sensō-ji]
Multimedia queries
Web-based image searching
“Tokyo temple”
Google ImagesBing ImagesFlickrYahoo ImagesЯндекс
Web-based image searching
Best current practice is a text search:Find text in filename, anchor text, caption, ...
Text search works by creating a large index:
GoogleTokyo temple
BingTokyo temple
FlickrTokyo temple
YahooTokyo temple
YandexTokyo temple
New search types
query doc
conventional text retrieval
hum a tune and get a music piece
you roar and get a wildlife documentarytype “floods” and get BBC radio news
Example
text
video
images
speech
music
sketches
multimedia
loca
tion
sound
hum
min
g
mot
ion
text
imag
e
spee
ch
Exercise
Organise yourself in groupsDiscuss with neighbours - Two Examples for different query/doc modes? - How hard is this? Which techniques are involved? - One example combining different modes
Exercise
query doc
Discuss
- 2 examples
- How hard is it?
- 1 combination
loca
tion
sound
hum
min
g
mot
ion
text
imag
e
spee
ch
loca
tion
sound
hum
min
g
mot
ion
text
imag
e
spee
ch
text
video
images
speech
music
sketches
multimedia
Near-duplictate detection:Cool access mode!
Snaptell: Book, CD and DVD covers
Snaptell: Book, CD and DVD covers
Snaptell: Book, CD and DVD covers
Snaptell: Book, CD and DVD covers
Snaptell: Book, CD and DVD covers
Link from real world to databases
doi: 10.2200/S00244ED1V01Y200912ICR010
The Open Univerity'sSpot & Search
Scott Forrest: E=MC squared
"Between finished surface texture and raw quarried stone. Between hard materials and soft concepts.
Between text and context."
More information
[with Suzanne Little]
Spot & Search
[with Suzanne Little]
Near duplicate detection
Works well in 2d: CD covers, wine labels, signs, ...Less so in near 2d: buildings, vases, …Not so well in 3d: faces, complex objects, ...
Shazam
Rueger, Multimedia IR, 2010explains it all! Buy it now
Near duplicate detectionExercise
Find applications for near-duplicate detection - be imaginative: the more “outragous” the better - can be other media types (audio, smells, haptic, ...) - can be hard to do
Near-duplicate detectionWhere are the challenges?
[Victoria and Albert museum, London, ceramics collection, 2010]
Leaf detectionWhat are the challenges?
[with Natural History Museum, London, and Goldsmiths]