mmi 05 slides

O. Abou Khaled, D. Lalanne, J. BapstAssistants : Tadeusz Senn, Sandro Gerardi, Florian Evéquoz, Bruno Dumas

[5] Multimedia content representation &

Information Visualization

Multimodal Interfaces

The MMI team 2MMI_05

Multimodal Interfaces OverviewOverview

Multimedia Content RepresentationIntroduction, definitions, etc.

Audio, Image, VideoMultimedia content protection (Watermarking)Multimedia storage and transmissionDigital library, content-based multimedia systems.

Multimedia Information RetrievalAudio, image, video

Examples & research Projects

Goal: have a wide idea about Multimedia fundamental and multimedia system construction/access (over the networks)Multimedia Information Retrieval


Multimodal Interfaces OverviewOverview

Information VisualizationIntroductio, DefinitionsThe Power of Information VisualizationVisualization for What ?

Examples of Information Visualization

Goal: have a wide idea about Visualization techniques



WhatWhat’’s Multimedia?s Multimedia?Multi: ManyMedia:

A means to distribute and represent information: Text, graphics, pictures, voice, sound and music..

Perception media (how do humans perceive information?)– Audio/visual media

Representation media (how is information encoded?)– ASCII, JPG, MPEG, PAL.

Presentation media (medium used for output/input)– Input/output media (keyboards, papers)

Storage media (Where is information stored?)– Magnetic disk, optical disk

Multimedia:To distribute and present information coded as

Text, Graphics, animation, audio and video..By Computer, TV, phone, etc.

Multimedia: a working definitionA combination of two or more categories of information having different transport signal characteristics. Typically, one medium is a continuous medium while another is discreteImage, audio, video and graphics are usually the examples of media


Multimodal Interfaces Why we need Media?Why we need Media?

Our words cannot exactly describe the images.

Speaking is faster than writingListing is easier than readingShowing is easier and clearer than describing


Multimodal Interfaces The Types of MediaThe Types of Media

Perception MediaThe nature of information perceived by humans (How do humans perceive information?)Auditory media and Visual media

Representation MediaHow information is represented internally to the computer (How is information encoded in the computer?)Character (ASCII) , image (JPEG), audio (PCM) , video (TV signal, MPEG)

Presentation MediaPhysical means used by systems to reproduce information for humans (Which medium is used to output information from the or input in the computer?)Monitors, keyboard, cameras (Input output devices)


Multimodal Interfaces The Types of MediaThe Types of Media

Storage MediaPhysical means for storing computer data (where is information stored?)Magnetic tapes, magnetic disks, optical disks.

Transmission MediaPhysical means that allow the transmission of signals. (which medium is used to transmit data?)Cables, Radio tower, satellite...

Information Exchange MediaAll data media used to transport information. (which data medium is used to exchange information between different locations?)


Multimodal Interfaces ContextContext

Recent advances in the technologies of communication, computer science and electronics have facilitated production, and distribution of multimedia dataHuge quantity of multimedia data is accessible in different domains (education, entertainment, communication, etc.)Rich content of multimedia data

Integration of different media (audio, video, still images, text)Complex relationships (spatio-temporal, composition, etc.)

Professional/non professional users need to access multimedia data

=> Important need for elaborate multimedia indexing and retrieval techniques.


Multimodal Interfaces Multimedia IR vs. Text IRMultimedia IR vs. Text IR

People are used to express their needs using natural language Natural language queries are frequently used for text information retrievalMatching between text queries and text documents is more or less straightforwardMultimedia documents contain non-textual dataTo provide text queries on top of multimedia documents, the document content should be described (annotated) textually



Two approaches in retrieving multimedia Two approaches in retrieving multimedia documentdocument

Text-based retrievalApproach

Annotating the multimedia content with text descriptions and allowing text queries on top of these descriptions

ProsSome multimedia contain text zones which can be used for description

ConsAutomatic description is very hard to achieveManual description is time consumingSome contents are difficult to describe.



Content-based retrievalApproach

Querying the content based on similarities with a given multimedia example, a sketch, etc.

ProsCan be useful for restricted/specialised content (such as logo databases)

ConsExamples are not always easy to find/createPerformances are very limitedSemantic retrieval is almost impossible.

Two approaches in retrieving multimedia Two approaches in retrieving multimedia documentdocument


Multimodal Interfaces What is a Multimedia System?What is a Multimedia System?

A system that involves:Generation

Representation

Storage

Transmission

search and retrieval

delivery of multimedia information

A system that involves:production/authoring tools

compression and formats

file system design

networking issues

database management

server design, streaming



How to Build Multimedia Database Systems?How to Build Multimedia Database Systems?

How to build text database?

Text document Natural language processing

Tree-based indexingText database

Multimedia data Multimedia analysis

Multimedia Indexing

Multimedia database

Yahoo, Google

TransmissionActions

Transmission

Actions


Multimodal Interfaces AudioAudio

Sound FundamentalsSound is a continuous wave that travels through the air. The wave is made up of pressure differences.Sound is detected by measuring the pressure level at a location.Sound waves have normal wave properties (reflection, refraction,diffraction, etc.).

Sound reflects off walls if small wave length– reflection

Sound bends around walls if large wave lengths– diffraction

Sound changes direction due to temperature shifts– refraction

Not covered subjectsSignal Fundamental, Human Perception, Sound Quality Measures, Sound Codec Standards, etc.


Multimodal Interfaces Audio Information RetrievalAudio Information Retrieval

The Basics of Audio Search and Audio Information Retrieval

Audio Information Retrieval is the process of retrieving audio information by using various available resources:

If the available resources are series of keywords annotated manually -> Text-based retrievalText based searching for audio information is most commonIf the available resource is a piece of audio information (ex: a melody of a song) -> Content-based retrieval.Content based audio research is promising and attractive, but there is a long way to go …



What are some Audio IR Mechanisms?Annotation based Audio RetrievalContent-based Audio Retrieval

Annotation based Audio RetrievalPeer-to-peer file sharing softwareFTP, Streaming audio, WebsitesOnline network drives, Clip Art

ProblemsSpyware, virus, unrelated results returned



Content-based Audio RetrievalAudio feature extractionAudio classification and Retrieval

What’s content-based retrieval? The retrieval is facilitated by the information content in contrast to simple retrieval based on manual index terms or keywords.

What’s the content?The semantic concept meaning of the information.

Why key-words annotation is not good.Subjective and expensiveInsufficient

How to describe the content?Features:Audio: Loudness, bandwidth, pitch..Image: Color, Texture, Objects..Video: Temporal information change, Image+Audio


Multimodal Interfaces General Audio Retrieval FrameworkGeneral Audio Retrieval Framework

Audio Repository Features Extraction

Classification: Male, Laughing,

Indexing:Using feature describe

audio unit

Audio DatabaseRetrievalAudio Example Features

ExtractionUser

InterfaceKeywords

Browsing


Multimodal Interfaces ContentContent--based Audio Retrievalbased Audio Retrieval

General Audio Features for information retrieval.Time-domain FeaturesFrequency-domain Features

Audio classification Goal

Classify audio into speech, music, and other categories and subcategories

Motivation Different audio types require different processing and indexing retrieval techniquesDifferent audio types have diff signification to different application Search space after classification is reduced to a particular audio class


Multimodal Interfaces ContentContent--based Audio Retrievalbased Audio Retrieval

A demo on Speech and Music Classificationhttp://www.musclefish.comhttp://www.soundfisher.com/download/

Content-based Audio database browse and retrievalhttp://www.soundfisher.com/index_flash.htmlhttp://www.musclefish.com/frameset.html

Conclusions Audio Information Retrieval

Annotation Based:– Peer-to-peer system, ftp

Content Based:– Audio Feature extraction– Audio Classification and Retrieval– Music Retrieval.


Multimodal Interfaces Image Image

What’s an Image?An image is a 2D rectilinear array of Pixels

Image Data StructurePixel

Picture elements in digital images, it usually indicate a “point” in an image.Image Resolution

The number of pixels in a digital image.Depth

The number of bit used to characterize each pixel information.– Bit Map: 1 bit/pixel, Gray scale: 2-8bits/pixel, Full color: 24 bits/ pixel, Color

mapped: 8 bits/ pixel

The Quality of the imageResolution (The number of pixel)Image DepthAdopted compression algorithm (if adopted)

Not covered subjectsImage Depth, Monochrome/Bit-Map, Dithering, Gray Scale Images, 8-bit/24-bit Color Images, Image Format, etc.


Multimodal Interfaces Image Retrieval Image Retrieval

Text-based retrieval Using the text surrounding the image

Text close to the image in the containing document – URL: http://www.host.com/animals/dogs/poodle.gif– Alt text: <img src=URL alt="picture of poodle">– Hyperlink text: <a href=URL>Sally the poodle</a>

Using the text inside the imageRequired OCR technique

Some image search engines use this techniqueGoogle, Altavista, www.ditto.com

ProsEasy to implement and useUseful for simple and non-professional image retrieval

ConsIt is incomplete and subjectiveSome features are difficult to define in text such as texture or object shape


Multimodal Interfaces Image retrievalImage retrieval



Almost impossible to describe all the contents.Some contents are difficult to describe.



Content-based image retrieval (CBIR)The commonly acceptable way is

To show a sample image, or draw a sketch of the desired images to computer, and ask the system to retrieve all the images similar to that sample image or sketch.It relies on features such as colour, shape, texture

ExamplesIBM’s Query By Image Content (QBIC)

Retrieves based on visual content, including properties such as color percentage, color layout and texture.Fine Arts Museum of San Francisco uses QBIC.

Virages’s VIR Image Engine Can search based on color, composition, texture and structure.

ChallengesThe term “similarity” has different meaning for different people.Even the same person uses different similarity measures in different situations.


Multimodal Interfaces Images containing similar colorsImages containing similar colors


Multimodal Interfaces Images containing similar shapeImages containing similar shape


Multimodal Interfaces Images containing similar contentImages containing similar content


Multimodal Interfaces Variety of SimilarityVariety of Similarity

• Similar color distribution

• Similar texture pattern

• Similar shape/pattern

• Similar real content

Degree of difficulty

Histogram matching

Texture analysis

Image Segmentation,Pattern recognition

Eternal goal :-)


Multimodal Interfaces IndexingIndexing

Use any text available: Title, Subject, CaptionUse content information: Colour histogram, Shape, Texture



Demonstrationshttp://zomax.wins.uva.nl:5345/ret_user/http://www.ifp.uiuc.edu/~nakazato/CBIR/

Other demoshttp://eidetic.ai.ru.nl/egon/cogw/co440/CBIR_Demo-s.htmlhttp://www.ee.surrey.ac.uk/Research/VSSP/imagedb/demo.htmlhttp://www.fb9-ti.uni-duisburg.de/rotdemo.htmlhttp://mmdb.ece.ucsb.edu/~demo/corelacm/

ConclusionsImage RetrievalContent-based Image Retrieval (CBIR)General Measures:

Gray intensity, Color, Texture, ShapeDistances Measures:

Color similarity, Texture similarity, Shape similarity, Object and relationship similarity.


Multimodal Interfaces VideoVideo

Video consists of imagesWhat’s the interval of images?

Sampling rates must be high enough to avoid motion "aliasing”.1. At least 15 frames/Sec2. 30 frames/ Sec appears smoothly3. At least 50 frames/ sec needed in the ideal case

Not covered subjectsVideo standards, Broadcast System Elements, Analog Video Representations, human perception, Compression Standards, Video Processing Techniques, etc.


Multimodal Interfaces ContentContent--based Video Retrievalbased Video Retrieval

ApplicationsVideo Surveillance

Find where else the person appearsExperience On-Demand

Help to remember previous eventsProvide useful information on traveling

Equipment on cars to retrieve useful multimedia information according to your location/preference

Typical Retrieval FrameworkUser : provide query information that represents his informationneeds Database: store a large collection of video dataGoal: Find the most relevant shots from the database

Shots: “paragraph” in video, typically 20 – 40 seconds, which is the basic unit of video retrieval


Multimodal Interfaces Sample QuerySample Query

Text : Find pictures of George Washington

Image: Video:


Multimodal Interfaces Bridging the Gap Bridging the Gap

Video Database User

Result


Multimodal Interfaces Automatically Structure Video DataAutomatically Structure Video Data

The first step for video retrieval: Video “programmes” are structured into logical scenes, and physical shots If dealing with text, then the structure is obvious:

paragraph, section, topic, page, etc.

All text-based indexing, retrieval, linking, etc. builds upon this structure;Automatic shot boundary detection and selection of representative keyframes is usually the first step;


Multimodal Interfaces Typical automatic structuring of videoTypical automatic structuring of video

A set of shots

a video document

Keyframe browser combined with transcript or object-based search


Multimodal Interfaces Bridging the Gap Bridging the Gap

Video DatabaseUser

Video Structure Information Need

Result


Multimodal Interfaces Ideal solution Ideal solution

Video DatabaseUser

Video Structure Information NeedUnderstanding the semantic meaning and retrieve

Result


Multimodal Interfaces Ideal solution Ideal solution

Video DatabaseUser


Result

However, 1. Hard to represent query in natural

language and for computer to understand2. Computers have no experience3. Other representation restriction like

position, time


Multimodal Interfaces Alternative SolutionAlternative Solution

Video DatabaseUser


Match and combine

Result

Provide evidence of relevant information ( text, image, audio)


Multimodal Interfaces EvidenceEvidence--based Retrieval Systembased Retrieval System

General framework for current video retrieval systemVideo retrieval based on the evidence from both users and database, including

Text information Image informationMotion informationAudio information

Return a relevant score for each evidenceCombination of the scores


Multimodal Interfaces More Evidence in Video RetrievalMore Evidence in Video Retrieval

Video DatabaseUser


Text Information Keyword

Image Information Query

Images

MotionInformation

Audio Information

Motion

Audio


Multimodal Interfaces Combination of multiCombination of multi--modal resultsmodal results

Difference characteristics between multi-modal informationText-based Information: better for middle and high level queries

e.g. Find the video clip of dancing women wearing dresses Image-based Information: better for low and middle level queries

e.g. Find the video clip of green trees

Combination of multi-modal information


Multimodal Interfaces Video Video retrievalretrieval

Primitives of Color Moments Methodhttp://debut.cis.nctu.edu.tw/Demo/ContentBasedVideoRetrieval/CBVR/PrimitivesE/index.html

Dominant Colors Methodhttp://debut.cis.nctu.edu.tw/Demo/ContentBasedVideoRetrieval/CBVR/DominantE/index.html

Combination Methodhttp://debut.cis.nctu.edu.tw/Demo/ContentBasedVideoRetrieval/CBVR/demoE.html


Multimodal Interfaces Ideal solution: TSR Case study Ideal solution: TSR Case study

Video DatabaseUser


Result


Multimodal Interfaces TSR StudyTSR Study

Indexing

Retrieval

TV news

Production


Multimodal Interfaces Problems (I): Problems (I): IndexingIndexing

Non-optimal reuse of information

Production

Indexing

Retrieval

ScriptSubtitleTeletextEdited VideoDescribed rushesJournalist commentaries

Video segments described following the TSR scheme (places, events, persons, dates, etc.)

Ineffective information exchange

Described video relevant to the

query


Multimodal Interfaces MPEGMPEG--7 in Practice7 in Practice

Library of audiovisual descriptions Coverage

Providing comprehensive set of descriptions needed in known audiovisual applications

InteroperabilityData Definition Language based on the W3C XML Schema


Multimodal Interfaces Example of MPEGExample of MPEG--7 Description7 Description

TV news audiovisual data



<Mpeg7>…

<StillRegion id = “news”></StillRegion>

…</Mpeg7>

Title



<Mpeg7><Mpeg7>……

<<StillRegionStillRegion id = id = ““newsnews””> > <<SpatialDecompositionSpatialDecomposition>>

<StillRegion id = <StillRegion id = ““backgroundbackground””> > <<VisualDescriptorVisualDescriptor

xsi:typexsi:type==““DominantColorTypeDominantColorType””>>110 108 140 110 108 140

</</VisualDescriptorVisualDescriptor>><StillRegion id = <StillRegion id = ““speakerspeaker””>>

</</SpatialDecompositionSpatialDecomposition>></</StillRegionStillRegion>>……

</Mpeg7></Mpeg7>

Back ground features



Example of MPEGExample of MPEG--7 Description7 Description

<Mpeg7><Mpeg7>……

<<StillRegionStillRegion id = id = ““speakerspeaker””>><TextAnnotation><TextAnnotation>

<<FreeTextAnnotationFreeTextAnnotation> Journalist > Journalist Anna BlancoAnna Blanco……

</</FreeTextAnnotationFreeTextAnnotation> > </</TextAnnotationTextAnnotation>><<MaskMask xsi:typexsi:type="="SpatialMaskTypeSpatialMaskType">">

<<SubRegionSubRegion>><Poly><Poly>

<<CoordsCoords> 80 288, 100 200, > 80 288, 100 200, ……, , 352 288 352 288

</</CoordsCoords>></Poly></Poly>

</</SubRegionSubRegion>></</MaskMask>>

</</StillRegionStillRegion> > ……

</Mpeg7></Mpeg7>

More features


Multimodal Interfaces Semantic Views Model (I)Semantic Views Model (I)

GoalProvide a common TV news retrieval platform for professional and non-professional usersCover a rich combination of content descriptions and AV structure via a simple model

DesignAnalyzing queries asked by different users in Télévision Suisse Romande(TSR) revealed a set of common description types

Example

A news item in the context of Euro 2000 football games containing a shot of at least 5 seconds

showing a French football supporter saying « que le meilleur gagne »

Find


Multimodal Interfaces Semantic Views Model (II)Semantic Views Model (II)

=> Users describe AV information following a set of “Views”

Its duration isat least 5 seconds

It isin the contextof EURO 2000 football games

It isa news item

It contains a shot showing an French football supporter

I can hear« Que le meilleur

gagne! »

A video segment

PhysicalView

ThematicView

Production View

VisualView

AudioView



TV news indexing and retrieval platform of COALTV news indexing and retrieval platform of COAL

COALA

AudiovisualRepository

system

Descriptionsystem

TV newsMPEG-7 corpus

Retrieval system

Visualizationsystem

COALA

AudiovisualRepository

system

Descriptionsystem

TV newsMPEG-7 corpus

Retrieval system

Visualizationsystem


Multimodal Interfaces Indexing toolIndexing tool

Demo


Multimodal Interfaces Indexing toolIndexing tool



Querying tool based onQuerying tool based onSemantic Views ModelSemantic Views Model

BasicViewEntities ViewDescriptions

IntraViewRelations

InterViewRelations

Five Views


Multimodal Interfaces Hierarchical browsing toolHierarchical browsing tool

Demo


Multimodal Interfaces Semantic views browsing toolSemantic views browsing tool


Multimodal Interfaces Conclusion and discussionConclusion and discussion

Recent approaches to the problem of multimedia IR are mostly based on the extraction of text/audiovisual featuresExtraction/creation of descriptions is hard and expensive

Manual approaches are time consumingAutomatic approaches are not always possible, some are not sufficiently accurate

Multimedia descriptions are very preciousApplications need to exchange themCreated descriptions should be conserved

Important need for a standard multimedia description language


Multimodal Interfaces Information visualizationInformation visualization

What is Information Visualization?Visualize: to form a mental image or vision of …

Visualize: to imagine or remember as if actually seeing.American Heritage dictionary, Concise Oxford dictionary

“Transformation of the symbolic into the geometric” (McCormick et al., 1987)

“... finding the artificial memory that best supports our natural means of perception.'‘ (Bertin, 1983)

The depiction of information using spatial or graphical representations, to facilitate comparison, pattern recognition, change detection, and other cognitive skills by making use of the visual system (Hearst 03).



The Power of VisualizationThe Power of Visualization

Visualization for Problem SolvingVisualization for ElicitingKnowledge from Data

statistics

Visualization for ClarificationMappy, etc.



Two Different Primary Goals:Two Different Primary Goals:Two Different Types of Two Different Types of VizViz

Explore/CalculateAnalyzeReason about Information

CommunicateExplain Make DecisionsReason about Information

In more detail, visualization should:Make large datasets coherent

(Present huge amounts of information compactly) Present information from various viewpoints Present information at several levels of detail

(from overviews to fine structure) Support visual comparisons Tell stories about the data


Multimodal Interfaces Human Perceptual FacilitiesHuman Perceptual Facilities

Use the eye for pattern recognition; people are good atScanning, recognizing, remembering images

Graphical elements facilitate comparisons via Length, shape, orientation,texture

Animation shows changes across time Color helps make distinctionsAesthetics make the process appealing


Multimodal Interfaces Information visualization: contextInformation visualization: context

Large amount of information vs. relatively small computer screen.

locating a given item of information? interpreting an item of information? relating an item with some other items?

Two Category of ApproachesNon-distortion-oriented approaches.

Displaying a portion of the information at a time;Scrolling or paging accessProviding hierarchical accessStructure-specific presentation

Distortion-oriented Approaches:Distort an image of large amount of information so that it can fit in screen.Allow the user to examine a local area in detail;At the same time, present a global view of the information space;Provide navigation mechanism.


Multimodal Interfaces Information visualization: contextInformation visualization: context


Multimodal Interfaces DistortionDistortion--based Techniquesbased Techniques

Idea of Distortion-based TechniquesCo-existence of local details with global context at reduced magnification.A focus region to display detailed information.Demagnified view of the peripheral areas is presented around the focus area.A distorted view is created by applying a transformation function to an undistorted image.A magnification function, provides a profile of the magnification factors for the entire area of image.

Ex. Bifocal DisplayDistortion at one or two dimensions with linear transformation function.Combination of detailed view and two distorted side views.



Ex. Polyfocal DisplayPerspective Wall

A conceptual descendent of the Bifocal display.Smoothly integrated detailed and contextual views.Side panels are demagnified directly proportional to their distance from the viewer.



Fisheye ViewBasic idea: more relevant information presented in great detail; the less relevant information presented as an abstraction.Relevance is computed on basis of the importance of information

elements and their distance to the focus.

Graphical Fisheye ViewAn extension of the fisheye view concept.Could be also considered as a special case of polyfocal display.


Multimodal Interfaces Why Visualize Text?Why Visualize Text?

To help with Information Retrievalgive an overview of a collectionshow user what aspects of their interests are present in a collectionhelp user understand why documents retrieved as a result of a query


Multimodal Interfaces Exploiting Visual PropertiesExploiting Visual Properties

Analyzing retrieval resultsKartOO http://www.kartoo.com/

Grokker http://www.groxis.com/service/grok


Multimodal Interfaces Exploiting Visual PropertiesExploiting Visual Properties

mmi 05 slides

Documents

categories of information

describingthe mmi team

examples of mediathe

presentation media medium

auditory media

multimedia fundamental

data medium

computer data