multimedia information retrieval on a very large scale · i sensed a scream passing through nature;...

Fabrizio Falchifabrizio.falchi@cnr.it

2016-03-04

Multimedia Information Retrievalon a Very Large Scale

Introduction

Overview

IntroductionFabrizio Falchi

1. Visual FeaturesFabrizio Falchi

2. Indexing for Similarity SearchGiuseppe Amato

3. Large Scale CBIR using Standard Text Retrieval EnginesClaudio Gennaro

Multimedia Information Retrieval

• The process of

o searching for and finding multimedia documents

• The corresponding research field is concerned with

o building the best possible multimedia search engines.

• The intriguing bit here is that

o the query itself can be a multimedia excerpt.

[“Multimedia Information Retrieval”, Stefan Rüger 2009]

Multimedia (adj)

• Of art, education etc.:

using more than one medium of expression or communication

• Of computer applications:

incorporating audio and video, especially interactively

Multimedia documents

• Consist of multimedia data

(text, images, audio, video, etc.)

• Are semistructured, i.e., contain

o structured data (e.g., metadata)

o unstructured data (e.g., text, images, audio, video, etc.)

[“Multimedia Information Retrieval”, Stefan Rüger 2009]

Multimedia retrieval

• There are basically two options

o Metadata based

• Multimedia documents are described by metadata

• Search is performed on metadata

• Metadata can be generated manually or automatically

o Similarity based (often called content based)

• Mathematical descriptions of media content is generated

• Retrieval is performed by searching for similar mathematical

descriptions

• Automatic medatadata generation

o Obtained leveraging on classification techniques

Change of the Search Paradigm

• Traditional YES-NO keyword search will not suffice - sortable

domains of data (numbers, strings) are assumed

• New types of data need gradual comparison and/or ranking

based on:

o similarity,

o dissimilarity,

o proximity,

o distance, closeness, etc.

Similarity Search

Focus on:

• efficient ways

• to locate user-relevant information in collections of objects,

• the similarity of which is quantified using a

pairwise distance measure

Image Similarity Search Problem

image database

Feature-based Approach

image layer

feature layer

Library

Library Catalogue

Catalog card

How did/do we search for content?

• Content was in book stored in libraries

• We use(d) card catalog containing metadata

• Metadata can be:

o Structural: data about the containers of data

o Descriptive: about the data content

• Nowadays we usually search in the (text) content

(e.g., web search engines)

Google Books

How would you search for the name of this Library?

Results

Same images, different sizes

Guessed text

What’s that?

The Scream, Edvard Munch

• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg

• One of the files of the same picture as

• One of the files of the same picture

• Almost the same as

• Almost the same

• A picture of the object at National Gallery, Oslo as

• Almost the same

• A picture of the object at National Gallery, Oslo

• One of “The Scream” by Edvard Munch as

• Almost the same

• One of “The Scream”s by Edvard Munch

• A painting by Edvard Munch as

• Almost the same

• A painting by Edvard Munch

• One of “The Scream”s by various artists as

• Almost the same

• One of “The Scream”s by various artists

• An expressionist painting as

• Almost the same

• An expressionist painting

• A painting as

• Almost the same

• A painting

• An hand made object as

• Almost the same

• A painting

• An hand made object

• An artificial objectbeing the product of intentional human manufacture

Recognition and Semantic

• Represents Valhallveien, above Oslo

Painting meaning?

• “I stopped and looked out over the fjord—the sun was setting, and the clouds

turning blood red. I sensed a scream passing through nature; it seemed to me

that I heard the scream. I painted this picture, painted the clouds as actual

blood. The color shrieked. This became The Scream.” (Edvard Munch)

• Reddish sky in the background is the artist's memory of the effects of the

powerful volcaniceruption of Krakatoa

• The imagery of The Scream has been compared to that which an individual

suffering from depersonalization disorder experiences, a feeling of distortion of

the environment and one's self, and also facial pain.

• "Whistler's Mother, Wood's American Gothic, Leonardo da Vinci's Mona

Lisa and Edvard Munch's The Scream have all achieved something that most

paintings—regardless of their art historical importance, beauty, or monetary

value—have not: they communicate a specific meaning almost immediately

to almost every viewer. (Martha Tedeschi)

Wikipedia

• Almost the same

• A painting

• An artificial object

being the product of intentional human manufacture

Low-level features

High-level semantic

• Almost the same

• A painting

• An artificial object

being the product of intentional human manufacture

Classification

Matching

Recognition

Related Research Fields

• Computer Vision

o To understand what is in a visual content the device have to “see”

• Multimedia Information Retrieval

o To retrieve visual content from a huge datasets

(it is not feasible to “see” everything online, even for the computers)

• Data Mining

o How to extract knowledge from the visual content we have

Matching

• Dictionary:

o to equal; be equal to

• In some terms, the match have to be exact

• Can often be done using signature

• Examples:

o Copy detection

Classification (of the visual content)

Contains:

• Red/orange sky

• 3 Humans

• Road

• Oslo

• 2 coves

• Cathedral

• Edvard Munch

• The Scream

• Norway

• Oslo

• Scream

• Art robbery

• Munchmuseet

• Oil

• Tempera

• Pastel

• Cardboard

• Skrik

• …

An index term assigned to a piece of information; a type of meta-

information that captures knowledge about an information resource

Flickr Automatic Tagging

https://www.flickr.com/photos/fabriziofalchi/2810872125

MIR for Augmented Reality

MIR for Tourism

http://www.visitotuscany.it/

To provide tourists with immediate access to information related to

monuments and artworks

Smart Ticketing

To provide citizens with effective ways to get information related to events

and making the booking process quick and easy

Smart shopping/marketing

To provide effective service in supermarkets by rising the efficiency of total

supply chain through quick billing and promotion of products.

multimedia information retrieval on a very large scale · i sensed a scream passing through nature;...

Documents

scream opening analysis

game design: i-scream zombies

scream magazine volume i

“the scream” by edvard munch expressionist artist...

scream postmodern

scream house

magazine jalouse ice cream i scream

the scream: interactive screaming painting · when edvard...

i)scream,)you)scream,)we)all)scream)for)ice)cream!) · i...

hindesight investor letter april 2013 - i want to scream

i have no reason to scream at this underture

i scream, path to well-being you scream! special events

i scream for public typography

as i scream

little scream

scream - rewrite

foto scream

film opening - scream

media scream

my favourite food. i scream, you scream, we all scream:...