A. Frank

Multimedia

Multimedia/Video Search

A. Frank2

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

A. Frank3

Multimedia Resources

• ,Pictures, Photos, Maps תמונות – Graphics, Animations

• Audio, Sounds – שמע

• Music, MP3 –מוסיקה

• Radio, Podcasts, Speeches – הקלטות

• Video, Flash, Vlogs –חוזי

• Movies, TV – סרטים

A. Frank4

MM Search & Information Retrieval (MMIR)

• Search & information retrieval of multimedia objects is hard!!• Using SEs to find relevant, quality MM objects can be difficult

and frustrating. • Sample query: find a movie that has a large mouse chasing a

Persian cat with twins watching from above?• Common MM search is text-based; object text can be

derived from:– caption of the MM object – text surrounding the MM object– filenames of MM object and the containing page– manual/automatic annotations

• Recently, tag-based MM search is effective (social computing).• However, long due, content-based MM search on the Web

is forthcoming.

A. Frank5

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

A. Frank6

Multimedia: Google

A. Frank7

Multimedia: Google

A. Frank8

Multimedia: Yahoo

A. Frank9

Yahoo Audio Search

A. Frank10

A. Frank11

AOL Video Search

A. Frank12

Multimedia: MSN Live

A. Frank13

MSN Live

A. Frank14

MSN Live Search

A. Frank15

AltaVista Multimedia Search

A. Frank16

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

A. Frank17

Images: Picsearch

A. Frank18

Picsearch Advanced Interface

A. Frank19

Audio: FindSounds

A. Frank20

Types of Sounds

A. Frank21

Video: SearchVideo

A. Frank22

Video: Clipblast!

• Organizes and makes the video Web relevant, fast and simple to navigate.

• Video Search and Navigation technology makes it easy to search, browse and personalize the video that viewers want, when they want it.

• Crawls the entire Web for all video content available, indexing more video content providers than any other video search engine.

• Largest video distribution platform ever.

A. Frank23

Video: Clipblast

A. Frank24

Multimedia: Flurl

A. Frank25

Flurl list of sites

A. Frank26

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

A. Frank27

Flickr

A. Frank28

Flickr

A. Frank29

Flickr

A. Frank30

Image/Maps: Panoramio

A. Frank31

Image/Maps: Panoramio

A. Frank32

Image/Maps: Panoramio

A. Frank33

Video: YouTube

A. Frank34

Keotag Meta-SE

A. Frank35

Keotag Meta-SE

A. Frank36

Contents

• Multimedia (MM) and search/retrieval

• Text-based MM search in General SEs

• Text-based MM search in Vertical SEs

• Tag-based MM search

• Content-based MM search

A. Frank37

“Content-based“ Image Search/Retrieval

• “Content based” image DB or Web site(s) that contain a large number of images.

• Image analysis techniques used to describe images by their extracted visual features such as color, texture, shape and orientation.

• These objects can be represented by sets of image features and then compared with the sets extracted from other images.

• Feature sets are described by their statistics, and the computer searches for statistically likely matches.

• Queries can be formulated in terms of visual features or by similarity options.

A. Frank38

Image: Squid

• Has about 1100 images of marine creatures in its database.

• Each image shows one distinct species on a uniform background.

• Every image is processed to recover the boundary contour, which is then represented by three global shape parameters and the maxima of the curvature zero-crossing contours in its Curvature Scale Space (CSS) image.

A. Frank39

Images: Squid

A. Frank40

Images: Squid

A. Frank41

Images: Squid

A. Frank42

Images: Squid

A. Frank43

Image: retriever

• Experimental service which lets you search and explore in a selection of Flickr images by drawing a rough sketch.

• Matches the most pronounced shapes and slabs of colors.

• The results are usually fairly good, sometimes even stunning,

• But it doesn't do object/face recognition of any kind, so if drawing an outline sketch of a chair, it almost certainly won't get you one back.

A. Frank44

Images: retriever

A. Frank45

Images: retriever

A. Frank46

Images: retriever

A. Frank47

MelodyHound

A. Frank48

MelodyHound

A. Frank49

MelodyHound

A. Frank50

MelodyHound

A. Frank51

Musipedia

A. Frank52

Audio: MusiPedia

A. Frank53

Music: Musipedia

A. Frank54

Musipedia or Web Search?

A. Frank55

Video: Blinkx

• World's largest and most advanced video search engine.

• Use of visual analysis and speech recognition to better understand rich media content.

• Advanced image and audio analysis techniques are used to automatically extract information about a video in real time and to create a rich index about the video content.

• Automatically and implicitly conducts searches based on the content being viewed by each individual user.

• With the advent of blinkx TV, users can now search and access news, movie trailers, popular multimedia segments and other video formats on demand.

A. Frank56

Video: Blinkx

A. Frank57

Video: Blinkx

A. Frank58

Blinkx TV Architecture

A. Frank59

References

• Kate Greene, Better, More Accurate Image Search, Technology Review, April 9, 2007, http://www.technologyreview.com/Infotech/18501/

• Kate Greene, Surfing TV on the Internet, Technology Review, April 18, 2007, http://www.technologyreview.com/Infotech/18545/

• Amit Agarwal, 10 Interesting Things You Can Do with YouTube or Google Videos, Digital Inspiration (Blog), July 27, 2006, http://labnol.blogspot.com/2006/07/10-interesting-things-you-can-do-with.html