multimedia semantics – from mpeg-7 to web 3.0 jane hunter [email protected] samt 2008
TRANSCRIPT
![Page 2: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/2.jpg)
Agenda• MPEG-7• MPEG-7 Ontologies• Semantic Gap Approaches• Web 2.0• Web 3.0
– New forms of multimedia– Hybrid classification– Weighted classifications
• Conclusions
SAMT 2008
![Page 3: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/3.jpg)
Multimedia Semantics
• How to find user-relevant multimedia
• Using search terms meaningful to the user
• Content-based search– Events – tries, coral bleaching– People - “Barak Obama”– Objects – astrocytoma, cancerous cells– Scenes – church scenes in “In Bruges”
(Focus is not on metadata – format, creator, date
Focus is not on query-by-example)
![Page 4: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/4.jpg)
The Semantic Gap
SAMT 2008
![Page 5: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/5.jpg)
MPEG-7 Standard• Multimedia Content Description Interface• ISO/IEC standard by MPEG • Objectives are to:
– Standardize content-based descriptions for audiovisual information
– Address a wide range of applications– Describe images, audio (speech, music),
video, graphics, 3D models, compositions– Be independent of storage, coding, display,
transmission, medium, or technology
![Page 6: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/6.jpg)
MPEG-7 Components
Descriptors:Descriptors:(Syntax & semantics(Syntax & semanticsof feature representation)of feature representation)
D7
D2
D5
D6D4
D1
D9
D8
D10
Description Definition extension extensionLanguage
DefinitionDefinition
101011 0
Encoding&
Delivery
TagsTags
<scene id=1> <time> .... <camera>.. <annotation</scene>
InstantiationInstantiation
D3
Description SchemesDescription Schemes(Relationships between Ds (Relationships between Ds and DSs)and DSs)
D1
D3D2
D5D4D6
DS2
DS3
DS1
DS4
StructuringStructuring
Video-on-DemandTV-Anytime
![Page 7: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/7.jpg)
Multimedia DSs
Datatype &structures
Link & medialocalization
Basic DSsBasicBasicelementselements
Navigation &Navigation &AccessAccess
Summary
Variation
AnalyticModel
Collection &Classification
Content organizationContent organization
Content descriptionContent description
Content managementContent management
Creation &production
Media ContentUsage
UserUser
User preferences
Conceptualaspects
Structuralaspects
![Page 8: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/8.jpg)
Visual and Audio DescriptorsFeature Descriptors
Color DominantColor ScalableColor ColorLayout ColorStructure GoFGoPColor
Texture Homogeneous Texture
TextureBrowsing Shape EdgeHistogram
RegionShape ContourShape Shape3D
Motion CameraMotion MotionTrajectory ParametricMotion MotionActivity
Feature Descriptors Silence Silence Timbre InstrumentTimbre HarmonicInstrumentTimbre PercussiveInstrumentTimbre Speech Phoneme Articulation Language Musical Structure
MelodicContour, Rhythm
SoundEffects Reverberation, Contour, Noise, Pitch
![Page 9: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/9.jpg)
MPEG-7 Ontology
Necessary in order to:• Bridge the semantic gap • Enhance discovery• Add multimedia to the semantic web• Enable machine reasoning both within and
across multimedia content• Enhance semantic interoperability • Facilitate integration of multimedia content
through common understandings
![Page 10: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/10.jpg)
MPEG-7 Class Hierarchy
![Page 11: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/11.jpg)
Vid e oSe g me n t
StillR e g io n Mo vin g
R e g io n
co lo r
v isu a ld e scr ip to r
domain
range
C o lo r
rd fs :R e so u rce
D o min a n tC o lo r
Sca la b leC o lo r
C o lo rL a yo u t
C o lo rStru ctu re
G o F G o PC o lo r
Colour Descriptors
![Page 12: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/12.jpg)
Semantic Inferencing Architecture
SAMT 2008
Ontologies – MPEG-7 + Domain-specific
Define RuleML, SWRL rules
![Page 13: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/13.jpg)
Rules-By-Example
![Page 14: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/14.jpg)
SAMT 2008
![Page 15: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/15.jpg)
![Page 16: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/16.jpg)
Richer User-Centred Queries
How does the mean catalyst size in electrodes of width < 20 microns effect
electrode conductivity?
Manufacturing
Performance
Microscopy
Image analysis
Annotations
Inferencing
![Page 17: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/17.jpg)
Upper Ontologies
SAMT 2008
ABC Ontology
MPEG-7 Ontology CIDOC CRM – Museum contentFUSION – Fuel Cell imagesOBOE – Environmental images/video
PhysicalEntityAbstractEntityInformationObjectEventPlaceTimeAgent
![Page 18: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/18.jpg)
Related Approaches
• Low level feature extraction – still room for improvement
• Statistical classification methods– Cluster multimedia objects based on similarity
• Machine Learning/Black box approaches– Bayesian, Probabilistic models– Neural networks– Genetic algorithms– Decision tree learning
SAMT 2008
![Page 19: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/19.jpg)
Web 2.0, eScience -> New Content
• 3D – cultural, biomedical, nano-structural• 4D – Scientific Visualizations• Compound Objects (Provenance, Learning Objects)• Access Grid, EVO Sessions• Multi-sensor networks – webcams, sensors, remote
sensing satellite imagery• YouTube, PodCasts• Video/Audio Blogs• FaceBook, MySpace
SAMT 2008
![Page 20: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/20.jpg)
![Page 21: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/21.jpg)
Indexing and Search for 3D Cultural Artefacts
SAMT 2008
![Page 22: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/22.jpg)
AccessGrid Session Recording & Playback
![Page 23: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/23.jpg)
![Page 24: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/24.jpg)
![Page 25: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/25.jpg)
Web 2.0
Social networks/community participation- define information environments- drive the technologies- rank information resources- define own tags (folksonomies)
Distributed Multimedia Collections
![Page 26: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/26.jpg)
Metadata vs TagsTags
– Topical, relevant, adaptive, light-weight– Cheap – generated by masses– Inconsistent, inaccurate, won’t scale, flat structure– Systems don’t interoperate (TagCommons)
Authoritative metadata– Complex, fixed, hierarchical structures– Don’t evolve, irrelevant, anachronistic terms– Very expensive– High quality
![Page 27: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/27.jpg)
Annotations vs Tags
• Tags– Light-weight, unstructured, organic, folksonomies
• Annotations– Structured, schemas, controlled vocabs, ontologies– creator, date, type, content/description, context
• Ontology-based tags in RDF enable:– machine-processing of annotations– resources to become part of semantic web– enhanced discovery and reasoning
![Page 28: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/28.jpg)
Collaborative 3D Object Annotation
Semi-Automatic Indexing/Description (DC/MPEG-7) &
Access/Rights/Traditional Care
Content Viewer(e.g. MPEG-2,
JPEG2000, 3D objects)
Shared Annotation/Discussion (Annotea)
![Page 29: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/29.jpg)
System Architecture
OAIRepository
OAIRepository
OAIRepository
WebSearchInterface
PeriodicallyHarvestedMetadata OAI-PMH
Community generating annotations
AnnotationServer
OAI-PMH
HarvestedMetadata
Store
PeriodicallyHarvestedAnnotations
AugmentedMetadata
Institutional Repositories/Metadata
Authenticated Annotation
Service
Shibboleth
WebSearchinterface
![Page 30: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/30.jpg)
Tag Clouds
![Page 31: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/31.jpg)
![Page 32: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/32.jpg)
Issues
• Quality Control on community-generated data
• Ontology-directed folksonomies
• How to identify tags to add to ontology– Tag convergence over time
• When, where to add them?
SAMT 2008
![Page 33: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/33.jpg)
Annotating Relationships
• Tag/Annotate a set of objects (or segments)
• Annotate one multimedia object with another multimedia object– Audio description of a photo
• Tag/Annotate relationships between objects or segments – label the relationship
• Kickstart greater semantic inferencing
SAMT 2008
![Page 34: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/34.jpg)
Web VideoServer
Secure Annotation
Server
Co-Annotea User Interface- attach, share, view, edit associations
Web Browser + Plug-ins
HTTP
See this association between scenes in the Seven Samurai andMagnificent Seven
Lecturer
Students ofFilm and Media 101
Annotating Relationships
![Page 35: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/35.jpg)
Annotating Relationships
SAMT 2008
![Page 36: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/36.jpg)
![Page 37: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/37.jpg)
![Page 38: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/38.jpg)
OAI-ORE - Complex Compound Objects
PSHTML
MP3
OAI/ORE Named Graphs/Resource Maps:• Define set of components • Typed Relationships between components• Different views of the compound object• Metadata attached to compound object• Publish as RDF Named Graph
cites
is_derived_from
hasRepresentation
View1.html
hasRepresentation
View2.smil
Identifier
URIDOIPURL
http://arXiv.org/astro-ph/061175/
http://openarchives.org/ore
![Page 39: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/39.jpg)
OAI-ORE Compound Objects
SAMT 2008
![Page 40: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/40.jpg)
Future -> Hybrid Approaches• Traditional authoritative metadata – high quality,
expensive, hierarchical, rigid• Combine with:
– automatic low-level feature extraction – colour, texture, shape, pitch
– social tagging – topical, cheap, flat, non-conformist– machine learning/statistical – need corpus
• Assign weightings based on:– reliability of source/method– social network information - FOAF
SAMT 2008
![Page 41: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/41.jpg)
Web 3.0 – Social Web meets Semantic Web
Photo in LOC
VRA Metadata
Library of CongressCurator
Community Tags/Tag Cloud
FlickrCommons
MPEG-7 Ds- Colour histogram- Region shape-> label
Image processing+ rules
Machine-learning, statisticalmethods
AutomaticTags
Aggregated description – that includes sources and weightings
![Page 42: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/42.jpg)
Hybrid Architectures
SAMT 2008
Collection of images
assign tags
Collective Intelligence
TaggedCollection
Corpus
Machine LearningMethod
New photo
Preprocessing
Structured, weighted semantically richmachine-processable metadata
Place (Geotags),DateCreator
Format,Size,Regions,MPEG-7
Semantic tags
e.g., Flickr SciVee
Workflow
Social tags
![Page 43: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/43.jpg)
Weighted Multimedia Metadata
Bibliographic: Title, Creator, Subject, Genre, Date_created
Formatting: Format, encoding, storage, dimensions, hardware/software
Structural: Regions, Segments
Content: Description, Transcript Colours, Textures Shapes, Motion Pitch, Tempo, Volume
Life History/Provenance: Events, Rights mgt.
Semantics: People, place, objects, events, concepts
Machine LearningBayesianNeural NetworkStatistical
Image ProcessingAudio Processing
Weightings
High
High
Low-Medium – depends on Trust rating
Medium
Medium
Source
Manual
Automatic
Manual
Semi-Automatic
Automatic
![Page 44: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/44.jpg)
Hybrid Search and Browse Interfaces
SAMT 2008
• Tag Clouds + Ontology-based querying• Combined keyword plus descriptor (e.g., shape) searches• SPARQL + spatio-temporal search interfaces
• Give me sub-tropical rainforests in S-E Qld above 3000m with reduced rainfall and containing endangered species
![Page 45: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/45.jpg)
Future Research Challenges I
YouTubeFlickrPodCasts
BlogosphereYouTubePodCasts
Challenges – 1. Identify relationships between resources on the Web (is_about)2. Extract structured information/metadata/tags from unstructured sources (RDFa?)3. Assign trust or reliability weighting based on the source
- how to crawl blogs and identify discussions around particular videos- how to extract structured information about the video/image from a blog
Increasing volumes of community-generated content +community generated metadata/annotations in multiple formats
about
![Page 46: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/46.jpg)
Future Research Challenges II
Multimedia search engines that support:– Ontology-based searches
• Ontology is dynamic – incorporates folksonomies over time• Combined with content-based, QBE, spatio-temporal searches
– Dynamic inferencing – rules change over time
– Identification of your social network and trust metrics– Ranking of results based on:
• tags assigned by users from same social networks• weightings applied to metadata that reflect reliability,
precision
SAMT 2008
![Page 47: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/47.jpg)
Acknowledgements
• Australia Research Council• Dept of Innovation Science and Research• NCRIS 5.16• Suzanne Little• Laura Hollink• Ronald Schroeter• Imran Khan• Ron Chernich• Anna Gerber
SAMT 2008
![Page 48: Multimedia Semantics – From MPEG-7 to Web 3.0 Jane Hunter jane@itee.uq.edu.au SAMT 2008](https://reader036.vdocument.in/reader036/viewer/2022062803/56649cb95503460f9497f911/html5/thumbnails/48.jpg)
Questions?
http://www.itee.uq.edu.au/~eresearch
SAMT 2008