1

05MSE E13- Multi Media Computing

By

U.RAHAMATHUNNISA

ASSISTANT PROFESSOR

SITE

2

Introduction

Multimedia Description Why multimedia systems? Classification of Media Multimedia Systems Data Stream Characteristics

CIS 104 3

Multimedia Description Multimedia: is an integration of continuous media

(e.g. audio, video) and discrete media (e.g. text, graphics, images) through which digital information can be conveyed to the user in an appropriate way.

Multimedia technology is the computer-based integration of text, sound, still images, animation and digitized motion video.

Merges capabilities of computers with televisions, VCRs, CD players, DVD players, video and audio recording equipment, music and gaming technologies.

CIS 104 4

Why Multimedia Computing? Application driven

e.g. medicine, sports, entertainment, education Information can often be better represented using

audio/video/animation rather than using text, images and graphics alone.

Information is distributed using computer and telecommunication networks.

Integration of multiple media places demands on computation power

storage requirements networking requirements

CIS 104 5

Multimedia Information Systems

Technical challenges Sheer volume of data

Need to manage huge volumes of data Timing requirements

among components of data computation and communication. Must work internally with given timing constraints - real-time

performance is required. Integration requirements

need to process traditional media (text, images) as well as continuous media (audio/video).

Media are not always independent of each other - synchronization among the media may be required.

CIS 104 6

High Data Volume of Multimedia Information

Speech 8000 samples/s 8Kbytes/s

CD Audio 44,100 samples/s, 2 bytes/sample

176Kbytes/s

Satellite Imagery

180X180 km 2̂ 30m 2̂ resolution

600MB/image (60MB compressed)

NTSC Video 30fps, 640X480 pixels, 3bytes/pixel

30Mbytes/s (2-8 Mbits/s compressed)

CIS 104 7

Technology Incentive

Growth in computational capacity MM workstations with audio/video processing capability Dramatic increase in CPU processing power Dedicated compression engines for audio, video etc.

Rise in storage capacity Large capacity disks (several gigabytes) Increase in storage bandwidth,e.g. disk array technology

Surge in available network bandwidth high speed fiber optic networks - gigabit networks fast packet switching technology

CIS 104 8

Application Areas

Residential Services video-on-demand video phone/conferencing systems multimedia home shopping (MM catalogs,

product demos and presentation) self-paced education

Business Services Corporate training Desktop MM conferencing, MM e-mail

CIS 104 9

Application Areas

Education Distance education - MM repository of class videos Access to digital MM libraries over high speed

networks Science and Technology

computational visualization and prototyping astronomy, environmental science

Medicine Diagnosis and treatment - e.g. MM databases that

provide support for queries on scanned images, X-rays, assessments, response etc.

CIS 104 10

Classification of Media

Perception Medium How do humans perceive information in a computer?

Through seeing - text, images, video Through hearing - music, noise, speech

Representation Medium How is the computer information encoded?

Using formats for representing and information ASCII(text), JPEG(image), MPEG(video)

Presentation Medium Through which medium is information delivered by the

computer or introduced into the computer? Via I/O tools and devices paper, screen, speakers (output media) keyboard, mouse, camera, microphone (input media)

CIS 104 11

Classification of Media (cont.)

Storage Medium Where will the information be stored? Storage media - floppy disk, hard disk, tape, CD-ROM etc.

Transmission Medium Over what medium will the information be transmitted? Using information carriers that enable continuous data

transmission - networks wire, coaxial cable, fiber optics

Information Exchange Medium Which information carrier will be used for information

exchange between different places? Direct transmission using computer networks Combined use of storage and transmission media (e.g.

electronic mail).

CIS 104 12

Media Concepts

Each medium defines Representation values - determine the

information representation of different media Continuous representation values (e.g. electro-

magnetic waves) Discrete representation values(e.g. text characters

in digital form)

Representation space determines the surrounding where the media are presented. Visual representation space (e.g. paper, screen) Acoustic representation space (e.g. stereo)

CIS 104 13

Media Concepts (cont.)

Representation dimensions of a representation space are: Spatial dimensions:

two dimensional (2D graphics) three dimensional (holography)

Temporal dimensions: Time independent (document) - Discrete media

Information consists of a sequence of individual elements without a time component.

Time dependent (movie) - Continuous media Information is expressed not only by its individual value but

also by its time of occurrence.

CIS 104 14

Multimedia Systems

Qualitative and quantitative evaluation of multimedia systems Combination of media

continuous and discrete. Levels of media-independence

some media types (audio/video) may be tightly coupled, others may not.

Computer supported integration timing, spatial and semantic synchronization

Communication capability

CIS 104 15

Data Streams

Distributed multimedia communication systems data of discrete and continuous media are broken

into individual units (packets) and transmitted. Data Stream

sequence of individual packets that are transmitted in a time-dependant fashion.

Transmission of information carrying different media leads to data streams with varying features Asynchronous Synchronous Isochronous

CIS 104 16

Data Stream Characteristics

Asynchronous transmission mode provides for communication with no time restriction Packets reach receiver as quickly as possible, e.g. protocols for

email transmission Synchronous transmission mode

defines a maximum end-to-end delay for each packet of a data stream.

May require intermediate storage E.g. audio connection established over a network.

Isochronous transmission mode defines a maximum and a minimum end-to-end delay for each

packet of a data stream. Delay jitter of individual packets is bounded.

E.g. transmission of video over a network. Intermediate storage requirements reduced.

CIS 104 17

Data Stream Characteristics

Data Stream characteristics for continuous media can be based on Time intervals between complete transmission of

consecutive packets Strongly periodic data streams - constant time interval Weakly periodic data streams - periodic function with finite period. Aperiodic data streams

Data size - amount of consecutive packets Strongly regular data streams - constant amount of data Weakly regular data streams - varies periodically with time Irregular data streams

Continuity Continuous data streams Discrete data streams

CIS 104 18

Classification based on time intervals

Strongly periodic data stream

Weakly periodic data stream

Aperiodic data stream

T

T

T1

T3

T2

T1

T2

T

CIS 104 19

Classification based on packet size

TD1

D1

TD1D2D3D1D2D3

D1D2D3

Dn

Strongly regular data stream

Weakly regular data stream

Irregular data stream

t

t

t

CIS 104 20

Classification based on continuity

Continuous data stream

Discrete data stream

D

D1 D2 D3 D4

D

D1 D2 D3 D4

CIS 104 21

Logical Data Units

Continuous media consist of a time-dependent sequence of individual information units called Logical Data Units (LDU).

a symphony consists of independent sentences a sentence consists of notes notes are sequences of samples

Granularity of LDUs symphony, sentence, individual notes, grouped samples, individual

samples film, clip, frame, raster, pixel

Duration of LDU: open LDU - duration not known in advance closed LDU - predefined duration

CIS 104 22

Granularity of Logical Data Units

Film

Clip

Frame

Blocks

Pixels

Multimedia from a user‘s point of view

Presentation of Information also using Audio and Animation

Information is easier conveyed Integration of Media with the Computer allows for

Interactive Presentations Transmission across computer networks

23CIS 104

Multimedia and the Industry

Sectors affected by the developments in Multimedia

Telecommunications Consumer Electronics TV- and Radiostations Publishing Houses

=> Convergence of Sectors

24CIS 104

Properties of multimedia processing

High data troughput Realtime Synchronisation

25CIS 104

Global Structure/Areas in Multimedia-Systems

Databases Operating Systems Communication System

Computer Technology

Memory Networks

Compression

Audio Video26CIS 104

Areas in Multimedia-Systems

Abstractions

Documents

Applications

User Interface

27CIS 104

Terminology

Multi [lat. much], as a Prefix Medium, mediating element, means to reproduce

Information

28CIS 104

The term „medium“

Means for the distribution and presentation of information Examples: Text, Figure, Image, Voice,

Music Further differentiation of the term in the MHEG-

Standard (1993) On the following two slides

29CIS 104

Different media terms

Perception-Medium To See, to Hear (how?)

Representation-Medium JPEG, MPEG (Structure, Coding)

Presentation-Medium Screen, Speakers, Keyboard

(Input/Output)

30CIS 104

Different media terms

Storage-Medium Paper, Harddisk, CD-ROM (where from?)

Transmission-Medium Network (on what?)

Information-Exchange-Medium Transmitter / Storage of Information: CD,

Network (higher abstraction)

31CIS 104

Presentation values and domains

Presentation domains Screen, Paper (visual presentation

domains) Stereophonetics

Presentation values Character, shock wave

32CIS 104

Presentation dimension

Every presentation domain has presentation dimensions Screen: 2 spatial dimensions

In Addition, Time as a dimension possible Time as additional dimension

fundamental for multimedia systems

33CIS 104

Classes of media

Classification using the time dimension Discrete Media

Independent from time Examples: Text, Figures

Continuous Media Dependent on time Examples: Sound, Motion Picture

34CIS 104

The Term „Multimedia“

Qualitative rather than quantitative understanding What kind of media instead of multiple media

A Combination of text and picture is not sufficient for the term „Multimedia“

„Controversial term“

35CIS 104

Main properties of a multimedia system

Combination of Media Processing of discrete and continuous

media Independency

in the layout of the media in space and time

Computerized Integration Computer as a tool

(Systems able to communicate) Distributed Multimedia-Systems

36CIS 104

Definition of „Multimedia“

A multimedia system is defined through the computerized, integrated creation, manipulation, presentation, storage and communication of independent information coded in at least one continuous and one discrete medium.

37CIS 104

Usage of the term „Multimedia“

In a narrow sense Continuous Media Borders in Time

In a broader sense Shared processing of multiple media

38CIS 104

Transmission of media data

Transmission of data of discrete and continuous media

For information exchange Based on data streams

39CIS 104

Data stream

Units (Packets) Send from source to sink Data stream is a sequence of single packets For discrete and continuous media

40CIS 104

Asynchronous Transmission

No restrictions in time for the packets of the data stream Example: IP

Normally not suitable for multimedia

41CIS 104

Synchronous Transmission

Defined Maximum End-To-End Latency For multimedia this means a guarantee on when

packets will be arriving

42CIS 104

Isochronous Transmission

Defined maximum and minimum end-to-end latency Jitter of single packets is restricted Buffers can be calculated more exact Suitable for „Streaming“, transmission of very large

amounts of data

43CIS 104

Data streams of continuous media

Different Properties Time interval between packets Amount of data / size of packets Correlation of packets

Possibly because of compression Example: Audio- and video transmission

44CIS 104

Strongly periodic data stream

Time interval between neighbouring packets is constant

Example: Voice in a telephone operation systemT

45CIS 104

Weakly periodic data stream

Time interval between packets follows a function of finite period duration

Example: Animation T

T1 T2 T3 T1 T2

46CIS 104

Aperiodic data stream

Neither strongly nor weakly periodic

Example: Transmission of events (e.g. Mouse interactions)

T1 T2 Tn

• • •

47CIS 104

Strongly regular

Constant amount of data of all packets

D1

Dn

t48CIS 104

Weakly regular

Amount of data of packets varies periodicly

D1

t

D2

D3

D1

D2

D3

T

49CIS 104

Not regular

Amount of data is neither constant nor periodic

D1

t

D2

D3

Dn

• • •

50CIS 104

Correlation of successive packets

Coherent data stream Packets are transmitted one after another

without gaps

Incoherent data stream– Gaps between the data packets

t

t51CIS 104

Units of information

Medium consists of a sequence of single units of information

PDU: protocol data unit LDU: logical data unit Example: Logical units of a movie sequence

Movie, Clip, Frame, Raster, Pixel

52CIS 104

Characteristics of multimedia applications

Large quantities of continuous data Timely and smooth delivery is critical

deadlines throughput and response time guarantees

Interactive MM applications require low round-trip delays Need to co-exist with other applications

must not hog resources Reconfiguration is a common occurrence

varying resource requirements Resources required:

Processor cycles in workstations and servers Network bandwidth (+ latency) Dedicated memory Disk bandwidth (for stored media)

At the right timeand in the right quantities

53CIS 104

Application requirements

Network phone and audio conferencing relatively low bandwidth (~ 64 Kbits/sec), but delay times must be

short ( < 250 ms round-trip)

Video on demand services High bandwidth (~ 10 Mbits/s), critical deadlines, latency not critical

Simple video conference Many high-bandwidth streams to each node (~1.5 Mbits/s each),

high bandwidth, low latency ( < 100 ms round-trip), synchronised states.

Music rehearsal and performance facility high bandwidth (~1.4 Mbits/s), very low latency (< 100 ms round

trip), highly synchronised media (sound and video < 50 ms).

54CIS 104

System support issues and requirements

Scheduling and resource allocation in most current OS’s divides the resources equally amongst all comers (processes) no limit on load can’t guarantee throughput or response time

MM and other time-critical applications require resource allocation and scheduling to meet deadlines Quality of Service (QoS) management

Admission control: controls demand QoS negotiation: enables applications to negotiate

admission andreconfigurations

Resource management: guarantees availability of resources for admitted applications

real-time processor and other resource scheduling

55CIS 104

Characteristics of typical multimedia streams

Data rate(approximate)

Sample or frame frequency size

Telephone speech 64 kbps 8 bits 8000/secCD-quality sound 1.4 Mbps 16 bits 44,000/secStandard TV video(uncompressed)

120 Mbps up to 640 x 480pixels x 16 bits

24/sec

Standard TV video (MPEG-1 compressed)

1.5 Mbps variable 24/sec

HDTV video(uncompressed)

1000–3000 Mbps up to 1920 x 1080pixels x 24 bits

24–60/sec

HDTV videoMPEG-2 compressed)

10–30 Mbps variable 24–60/sec

Figure 15.3

56CIS 104

QoS Parameters

Bandwidth rate of flow of multimedia data

Latency time required for the end-to-end transmission

of a single data element

Jitter variation in latency :– dL/dt

Loss rate the proportion of data elements that can be

dropped or delivered late57CIS 104

What is a Multimedia System? A system that involves:

Generation: production/authoring tools Representation: compression and formats Storage: file system design Transmission: networking issues, QoS Search and retrieval: database

management Delivery: service design, QoS of

multimedia information

58