![Page 1: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/1.jpg)
CS257 Modelling Multimedia InformationCS257 Modelling Multimedia Information
LECTURE 5LECTURE 5
![Page 2: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/2.jpg)
IntroductionIntroduction
• Let’s review…– What we’re dealing with (video data)– What kinds of moving images might be stored
in a digital video database– What kinds of queries people might want to
make to a video database
Techniques for modelling and querying video content
![Page 3: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/3.jpg)
Digital Video Data is…Digital Video Data is…
• A sequence of frames, where each frame is an image (typically 25 frames per second)
• May include a soundtrack: Speech (commentary / monologue / dialogue); Music; Sound effects
• May include text, i.e. subtitles / closed captions• Video data has temporal aspects as well as spatial
aspects: the temporal organisation of the moving images conveys meaning in its own right – change the order of scenes change the meaning of a film
• Cinematic techniques (pan, zoom, etc.) and editing effects can also convey information
![Page 4: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/4.jpg)
Moving Images include…Moving Images include…
• Television• Cinema• Meteorological images• Surveillance cameras• Medical images• Biomechanical images• Dance
Diverse range of video database applications, users and information needs
![Page 5: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/5.jpg)
What is Video Content?What is Video Content?
• Generally speaking video content can be said to comprise:– Objects (including people) with properties
AND– Activities (actions, events) involving 0 or more objects– Organised in space and time
• (This is an attempt at a generic approach, i.e. one that is applicable to all kinds of moving images)
![Page 6: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/6.jpg)
Example queries to film Example queries to film archivesarchives
“Transport”“Meercats”“Portillo”“Ferries departing at night”“Animal rights protesters outside Hillingdon cat
farm, wearing masks”
From Enser and Sandom (2002).
![Page 7: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/7.jpg)
Queries to Video DatabasesQueries to Video Databases
• User may not want to see the whole of a video file (e.g. a whole film), may just want certain parts or intervals (e.g. shots or scenes)
Need to attach metadata to different parts of the video [More on this in Part 1 of this Lecture]
![Page 8: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/8.jpg)
Queries to Video DatabasesQueries to Video Databases
• Users may want to query for a particular event involving particular people, e.g. “find me video with Bill hitting Tom” – why not use a list of keywords [hit, Bill, Tom] for query and to represent film content?
Need more structured descriptions of what’s happening (both for queries and for video metadata), i.e. who is doing what to whom with what and why. [More on this in Lecture 6]
![Page 9: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/9.jpg)
Queries to Video DatabasesQueries to Video Databases
• User may want to specify a temporal sequence of events, e.g. “find me video where this happens then this happens while that happens”
[More on this in Lecture 6]
![Page 10: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/10.jpg)
Overview of LECTURE 5Overview of LECTURE 5
• PART 1: Need to be able to index (attach metadata to) and retrieve parts of video data files: Video Data Models
• PART 2: How developments in the MPEG standards have enabled new ways to access and create digital video
• LAB – Finish previous exercises; bring coursework questions
![Page 11: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/11.jpg)
PART 1:PART 1: Video Data Models Video Data Models
• Must decide how to model (i.e. structure) video data so that metadata can be attached appropriately – must consider potential user information needs
• Video data can be modelled as:– BLOb’s (Binary Large Objects)– Frames– Intervals (discrete, hierarchical, overlapping); – Object-based schemes (cf. MPEG-4)
• NOTE, this does not entail chopping up the video data file
![Page 12: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/12.jpg)
Modelling Video Data as a BLObModelling Video Data as a BLOb
• Metadata may be associated with the video data as a whole
• The kinds of metadata for visual information discussed in Lecture 4 apply equally well to moving images; but note – ideally only metadata that is true for the whole video data file should be associated with a BLOb
![Page 13: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/13.jpg)
Modelling Video Data as a BLObModelling Video Data as a BLOb
Time
Attribute1:
Attribute2:
Attribute3:
AttributeX…
![Page 14: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/14.jpg)
Modelling Video Data as FramesModelling Video Data as Frames
• An exhaustive metadata description of a video data file would include details for each and every frame (remember each frame is a still image)
• However, with 25-30 fps, the cost of this is usually prohibitive and there are few applications where it would be beneficial
![Page 15: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/15.jpg)
Modelling Video Data as FramesModelling Video Data as Frames
Time
Frame: 1
Attribute A:
Attribute B:
Frame: 2
Attribute A:
Attribute B:
Frame: 3
Attribute A:
Attribute B:
![Page 16: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/16.jpg)
Modelling Video Data as IntervalsModelling Video Data as Intervals
• It is more usual to model video data as ‘meaningful’ intervals – where ‘meaningful’ depends on the particular domain and application
• The intervals may be discrete or overlapping• The intervals may be arranged in a hierarchy so
that metadata descriptions can be inherited
![Page 17: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/17.jpg)
Discrete IntervalsDiscrete Intervals
Time
Interval: 1
Start Time / Frame: 1
End Time / Frame: 103
Attribute A:
Attribute B:
Attribute C:
Interval: 2
Start Time / Frame: 104
End Time / Frame: 155
Attribute A:
Attribute B:
Attribute C:
Interval 1 Interval 2 Interval 3 Interval 4
![Page 18: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/18.jpg)
Overlapping IntervalsOverlapping Intervals
Time
Interval 1
Interval 2
Interval 3
Interval 4
Interval: 2
Start Time / Frame: 008
End Time / Frame: 035
Attribute A:
Attribute B:
Attribute C:
Interval: 3
Start Time / Frame: 045
End Time / Frame: 086
Attribute A:
Attribute B:
Attribute C:
![Page 19: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/19.jpg)
Hierarchical IntervalsHierarchical Intervals
Time
Interval 1 Interval 2 Interval 3 Interval 4 Interval 5 Int 6Interval 7 Interval 8 Interval 9
Interval 10
Interval: 10
Start Time / Frame: 000
End Time / Frame: 210
Attribute A:
Attribute B:
Attribute C:
![Page 20: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/20.jpg)
Hierarchical IntervalsHierarchical Intervals• One example of video data that can modelled as hierarchical intervals are news broadcasts, made up of separate parts (main news, sports news, weather) and each part has separate stories, and each story is made up of shots and scenes
• Then attach metadata to each shot and scene
![Page 21: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/21.jpg)
From: Corridoni et al (1996), ‘Multi-perspective Navigation of Movies’, Journal of Visual Languages and Computing 7, 445-466. [Available through Library eJournals].
![Page 22: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/22.jpg)
Exercise 5-1Exercise 5-1
• Metadata about both audio and video data streams can be organised by modelling the data as a Binary Large Object (BLOb) or in terms of intervals. Intervals may be modelled as single- or multi-levelled, they may be discrete or overlapping, and they may be organised in a hierarchy. Give an example of a video stream and an example of an audio stream which would be best modelled as:
• (i) A Binary Large Objects (BLOb)• (ii) Overlapping multi-level intervals• (iii) A hierarchy of discrete intervals
• For each of (i), (ii) and (iii) you should discuss the information needs of potential users and give examples of the metadata you would attach to the data streams
![Page 23: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/23.jpg)
Part 2: Developments in MPEG Part 2: Developments in MPEG standards for digital videostandards for digital video
Video Data: In the beginning….• Before the mid-1980s, digitising, storing and displaying still images,
never mind moving ones, required dedicated high power and high cost hardware
• Now video data is commonplace on home PCs; and throughout the TV and film industries
• Moreover, the ways in which video data can be accessed have become increasingly intuitive and flexible. In a large part these developments have been driven by successive MPEG standards
The MPEG standards: some hows and whys….
![Page 24: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/24.jpg)
Moving Pictures Expert GroupMoving Pictures Expert Group• A committee comprising 100s of international
experts• MISSION: “Development of international
standards for compression, decompression, processing and coded representation of moving pictures, audio, and their combination, in order to satisfy a wide variety of applications”
Each standard developed with a specific aim
![Page 25: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/25.jpg)
MPEG-1: why?MPEG-1: why?
Compression: video data at 1.5 Mbits/s• e.g. for storage on a CD-ROM and playback on
a low-spec PC • then for distribution of video data over the
Internet (not necessarily in real time) – progressive rendering
Started 1988; finished 1992(MPEG-1 Audio Layer III MP3)
![Page 26: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/26.jpg)
MPEG-1: how?MPEG-1: how?
Combines visual, audio and timing information into one data stream suitable for digital transmission
Removes redundancy• Inter-frame redundancy (cf. still images)• Intra-frame redundancy – i.e. regions that do not
change (significantly) -> store some frames and interpolate others
Quality issues: fidelity of colour map; pixel resolution; frame resolution
![Page 27: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/27.jpg)
MPEG-2: why?MPEG-2: why?
Television requirements• higher quality• “efficient transmission over error-prone delivery
systems”Set top boxes and DVDsTV and film production
Started 1990; finished 1996
![Page 28: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/28.jpg)
MPEG-2: how?MPEG-2: how?
Digital Storage Media Command and Control: protocols for managing bitstreams
Requirements• Higher bandwidth• Dedicated hardware for coding (far from real
time); professional systems still cost many £K
(?MPEG-3?)
![Page 29: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/29.jpg)
MPEG-4: why?MPEG-4: why?
Next generation mobile communication• Further compression• PLUS… Reuse / Personlisation /
Customisation / Interactivity / Creativity
Requirements– Composition of real and synthetic media– Multiplex and synchronise objects over networks– Interact with audiovisual scene that is generated at the receiver’s
end
(Started 1993; parts finished in 2000/1)
![Page 30: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/30.jpg)
MPEG-4: how?MPEG-4: how?
**FROM FRAMES TO AUDIO-VISUAL OBJECTS**
Coding of audio-visual objects - real or synthetic• Still images (e.g. a fixed background)• Video objects (e.g. talking person); need not be
rectangular frames• Audio objects (e.g. person’s speech)• Text and graphics – inc. VR• Synthetic heads, speech and sound
![Page 31: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/31.jpg)
MPEG-4: how?MPEG-4: how?
Standard scene description (cf. VRML):• Place media objects in coordinate system• Apply transforms to change geometry / acoustics• Form compound media objects• Apply streamed data to objects to modify
attributes• Interactively change user’s viewing and listening
points
![Page 32: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/32.jpg)
![Page 33: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/33.jpg)
MPEG-7MPEG-7
Indexing and retrieval
“Multimedia Content Description Interface”
“different from previous MPEG standards… what is represented is not the information itself but information about the information”
i.e. it is a metadata standard.
![Page 34: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/34.jpg)
MPEG-7MPEG-7
Description Definition Language– for low-level visual features; – for low-level audio features;– for ‘semantic content’, e.g. descriptions
of the narrative world depicted by a piece of media
• Started 1997; official standard in 2001• MPEG-7 cameras developed to capture
some features at time of recording
![Page 35: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/35.jpg)
EXERCISE 5-2EXERCISE 5-2• Consider the following people who are each developing a different
multimedia system to handle digital video data. Which of the MPEG standards would be of interest to each person and why?
– Annabel: she has collected a digital library of historical documentaries which she would like people to be able to access via a high-speed intranet by making a range of queries about semantic content
– Brian: he is distributing a video browser with his home-made cartoons on a CD-ROM
– Claire: she is setting up a pay-per-view digital television service– Dave: he is an experimental filmmaker who is developing a system to help him
edit video material in new and interesting ways
• It can be argued that the format in which multimedia data is coded has an impact on the potential functionality of systems that will store, retrieve and manipulate that data. Discuss, with explanations, the impact on potential system functionality that you think could arise from the following choices of coding format:
– Coding a moving image in MPEG-1 or coding it in MPEG-4– Coding a piece of music in ‘.wav’ or coding it in ‘.midi’
![Page 36: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/36.jpg)
LECTURE 5:LECTURE 5:LEARNING OUTCOMESLEARNING OUTCOMES
• After the lecture, you should be able to:– Select an appropriate video data model given
a specific kind of video data and some user information needs, and explain / justify your decision
– Describe and explain how each MPEG standard enabled new ways to access and create digital video
![Page 37: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/37.jpg)
OPTIONAL READINGOPTIONAL READING
• For information about all the MPEG standards, see: www.chiariglione.org/mpeg/index.htm
• For an overview of MPEG-7: www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm
![Page 38: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/38.jpg)
OPTIONAL READINGOPTIONAL READING
• Benitez et al. (2002), “Description of a Single Multimedia Document”, in: Manjunath, Salembier and Sikora (eds.), Introduction to MPEG-7: multimedia content description interface.
There are 4 copies of this article in the Library Article Collection.
• Note, focus on pages 113-4 and 124-138, and particularly on Figs. 8.9, 8.11, 8.14 and 8.16.
![Page 39: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/39.jpg)
OPTIONAL READINGOPTIONAL READING
Multimedia Standards in Television / Media Production• Issues of multimedia standards are discussed by a
number of other professional groups, including those interested in mxf (multimedia exchange format) and aaf (advanced authoring format).
For a short article about mxf and aaf, see: www.snellwilcox.com/knowledgecenter/mxf_aaf.html
Homepages;mxf: www.pro-mpeg.org/aaf: www.aafassociation.org/
![Page 40: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/40.jpg)
OPTIONAL READINGOPTIONAL READING
• The following articles are available for download via the University Library’s e-journals section. Many of the authors are from the MPEG-7 committee but the papers are more concise than the official MPEG technical documents.
Martinez, J.M., Koenen, R., and Pereira, F. (2002), “MPEG-7: the generic multimedia content description standard, part 1.” IEEE Multimedia 9 (2), April-June 2002, 78–87
Martinez, J.M. (2002), “Standards - MPEG-7 overview of MPEG-7 description tools, part 2. ” IEEE Multimedia 9 (3), July-Sept. 2002, 83 -93.
Chang, S.-F., Sikora, T. and Purl (2001), “An Overview of the MPEG-7 Standard”, IEEE Transactions on Circuits and Systems for Video Technology, 11 (6), 688-695.
Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 1.” IEEE Multimedia 6 (3), July-Sept. 1999, 65 –77.
Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 2.” IEEE Multimedia 6 (4), Oct.-Dec. 1999 , 64 –73
![Page 41: CS257 Modelling Multimedia Information LECTURE 5](https://reader030.vdocument.in/reader030/viewer/2022032523/56649d7d5503460f94a6010b/html5/thumbnails/41.jpg)
Today’s LabToday’s Lab
• Continue with image retrieval