Temporal relationships

.

What is meant by temporal relationship ?

• Temporal (timing) relationships are important in a multimedia presentation.

• Ex: A speaker’s lip movement and the words she utters must match in a video clip

Type of temporal relationship and accuracy depend upon:

• application and the• type of media used. Deviations to the required temporal relationships can

occur• on the transmitting end,• on the communication link, and• on the receiving end.

Topic of study

• Fundamentals of temporal relationships

• Classification and specification of temporal relationships

. Temporal relationships based on object stream interaction

Intramedia Interaction between the objects of a single stream.Ex: Animation without soundLet an animation clip is stored at the rate of 14 frames per secDuring playback the rate will be 14 frames per sec• Intermedia• Interaction between the objects of two parallel streams.• Ex: Video clip with sound• A precise temporal relationship must be maintained between

the image frames and the digitized voice to maintain lip sync

.

• User interaction based temporal relationship

• Interactivity being discussed here relates to the interaction between the user and the system

• The response time of a standalone or networked interactive multimedia system should be small and consistent

.• Temporal relationships based on Media LevityLive presentations• Information captured, transmitted, and presented in real-time.• Ex: Video conference is an example of a live presentation• Requirement: minimum end-to-end delay.Stored Presentations• Captured and stored on secondary storage systems.• Strict real-time constraint during retrieval.• Video on demand is an example of a stored and retrieved non live

presentationMixed presentation• Uses a combination of live and stored information.• Example: Collaborative conference• One window on the screen shows live pictures while other windows are

used to show stored information

Temporal relationship and isochronous multimedia

• A video clip played without sound

• If it was stored at 24 fps,then it must be played back at 24 fps

• It is not acceptable to play the video at 20fps in one sec and then compensate by playing it at 28 fps in the next

Temporal relationship and synchronous multimedia

• Video clip when played along with audio is a synchronous media

• Sound and video streams should play isochronously at correct rate

• Must also be synchronized wrt each other

SYNCHRONIZATION ACCURACY SPECIFICATION

• The factors used to specify synchronization accuracy are:

• Delay,

• Jitter,

• Skew,

• Error rate

.

• Delay• Interactive application: response time.• Streamed application: time take to traverse the

network.• Jitter• shaky picture.• quivering voice.• Skew• slow or fast moving picture.• lower or higher pitch than the normal.

.

• SYNCHRONIZATION MECHANISMS• Introducing intentional delay - removing jitter by• buffering:• All delay jitter can be removed by buffering.• Not desirable for most applications.• Acceptable for• One way communication applications such as:• NVoD (near video on demand).• Not acceptable for• Interactive applications, such as:• FVoD (full video on demand) and other• Needs larger buffer.

Processing delays: Transmitting end

• Stored objects:

Retrieval delays for stored objects:Tquery is the time taken to process and transform the query into the

location of the objects on the storage medium

Tseek is the time taken to locate the objects on the storage medium

Taccess is the time taken to read and transfer the object from the medium

These delays are not applicable to objects captured in real time

Capturing delays for real-time objects

• Apply to objects such as audio and video captured in real-time

• In natural form all real-life quantities are analog in nature

• For digital processing must be converted to digital values

• Tsample is the time taken for converting analog to digital value

• Tencode is the time for encoding like compression or encryption

• On the contrary stored objects are ready for transmission

Networking delays

• Networking delays apply to stored as well as captured objects

• Tpacketize is the time taken to break the object data into packets

• Ttransmit is the time taken to pass the object through the network interface

• Ex:Nbits/Rbps• Tpropagate is the time delay from the instant the

object is into the network on the transmitting end to the instant when it comes out of receiving end

• In ckt switching propagation delay=Distance/speed of transmission

Processing delays: receiving end

• Data must be buffered on the receiving end also

• Tbuffer is the delay added due to buffering at the receiving end

• Buffere delay may also happen in the transmitting end

• The recived data must be depacketized Tdepacketize delay

• Tdecode delay is time for decoding at the reciving end