Event-Based Modeling and Processing of Digital Media

Rahul Singh Zhao Li Pilho Kim Derik Pack Ramesh JainExperiential Systems Group

Georgia Institute of Technology

Ubiquity of Media

Surveillance, biometrics, situation monitoring, …

Genomics, Proteomics, Structural Biology

Biomedical imaging, medical records

Personal Media

Database Research

Digital Information Processing Research

Motivation

Database Research

Digital Information Processing Research

Images

Audio

Text

Media DependentFeatures and SimilarityFunctions

Molecular Index

Problem Formulation

Semantic correlations between media are important!

ImagesText

A General Model To Semantically Unify Multimedia

Goals

• Provide a model to unify multimedia semantically

• Storage scheme for efficient queries of structured and unstructured data

• Semi-Automatic signal processing and tagging of media

Outline

I. Event ModelII. Event StorageIII. Event Tagging SystemIV. Encompassing ArchitecturesV. Conclusions

Event Model

Event-Based Unified Modeling of Multimedia

Media 1

Media 2

Media n

Media 3

Images

Video

Text

Audio

Event Model

Physical Event

Definition: An event is an observed physical reality parameterized by space and time. The observations describing the event are defined by the nature or physics of the observable, the observation model, and the observer.

The Conceptual Model

Event

EventInformation }

TimeLocationParticipants…

EventRelations }

Spatio-Temporal RelationsEvent Aggregations…

MediaSupport } Media Type(s)

Media Locators, Indexes

Modeling Time

• Discrete time-stamping does not work very well• Direct application of interval based models is not possible for many types of media

My Trip to Paris for CVDB

June 10 June 14June 11 June 12

• Assign to events a mixed (point or interval) character

• Point-Point Relations, Point-Interval Relations, Interval-Interval Relations

TimeBefore Simultaneous After

TimeBefore During After

TimeBefore Equal StartMeet Overlap During Finish

Motivating Example--Meetings

• Why?– Significant research into smart meeting rooms– Need to facilitate remote participation– Various media can be found in meetings

Event Storage

Event Storage in Meetings

• A Closer Look at Meeting Data– Static Information such as the name of the attendee etc. – Dynamic Information such as the process of the meeting.– Most of static information is structured and most dynamic

information is unstructured.

• Our Approach– Using relational database approach to handle the static

information.– Using the XML database approach to deal with the dynamic

information.

Meeting System ER&E Diagram-- Entities Relationship and Event Diagram

Time Title

Location

M

N

Event infrastructure

Compound, Sub-events

Event Relationship

(Temporal, causality)

Related Multi-silo Data resources (video, audio, PowerPoint)

Structured Data Unstructured Data

Static Info. Dynamic Info.

Meeting

Attendee

AttendName

Birthday Position

Tables of The Meeting Information System

Meeting

Attendee

MT# Time Location MeetingProcessing (XMLType)

Attend Relationship

E_mail MT#

E_mail FName LName Position Image

XML Schema for Meetings

Queries

Relational/XML Queryselect extract((e.meetingprocessing),'/meeting/Presentation/how/ask_question/video')from meeting ewhere meetinglocation='TSRB1’;

XML Queryselect extract(value(e),'//MEETING[Location=”TSRB1”]/Presentation/how/ask_question/video')from xmldemotable e";

Performance Evaluation

Event Tagging System

Example of Meeting Related Event Flow

Time Line

Setup Meeting

Identify Member

Meeting Capture

Presentation Material

Post MeetingDiscussion

Different Color means

Different types of data

How to combine manual annotation to automatic feature detection in network

DataEvent

Detector

Symbol Candidate

ManualEvent

Tagging

AssignTeaching

ValueSymbol

DomainAnalysis

On-Line Processing

Off-Line Processing

Both

Data N

etwork

Symbol N

etwork

BackgroundNoise Filtering

SpeechDetection

ExtractVocal Feature

Compare withSpeaker Database

SpeakerCandidate

Specific Microphonein Specific Environment

Limited Speakers

Prior Assumptions ofBackground Noise

Prior Assumptions ofSpeech Signal Features

Prior Assumptions ofAcoustic Features

Prior Knowledge onParticipants

Speaker Identification Flow

Encompassing Architectures

Architecture Diagram

EventBase

Sensors EventDetector

Continuous Query Server

Update Server

Data Silo

Domain

Client

Conclusions

Conclusions

• Event model handles complex spatio-temporal reasoning and the semantic unification of heterogeneous multimedia

• Event tagging incorporates humans in the event detection process

• Combining structured and unstructured meta-data improves efficiency in event storage

Future Work

• Non ad-hoc approaches for processing techniques over multiple media types

• Effect of top-down event approach on signal processing and computer vision

• Researching system implementations of unified multimedia models

Questions?