chapter 2: multimedia basics and representation

Chapter 2:Multimedia Basics and

Representation•Digital multimedia characteristics

•Audio format and MIDI

•Image format and Color models

•Video format and Color models

1chapter1: Introduction to Multimedia

Characteristics of a Multimedia

Systems • A Multimedia systems has four basic

characteristics:

• Multimedia systems must be computercontrolled.

• Multimedia systems are integrated.

• The information they handle must berepresented digitally.

• The interface to the final presentation of mediais usually interactive.

2chapter1: Introduction to Multimedia

Computer Controlled

• Producing the content of the information – e.g. byusing the authoring tools, image editor, sound andvideo editor

• Storing the information – providing large andshared capacity for multimedia information.Transmitting the information – through thenetwork.

• Presenting the information to the end user – makedirect use of computer peripheral such as displaydevice (monitor) or sound generator (speaker).

chapter1: Introduction to Multimedia 3

Characteristics of a Multimedia Systems

Integrated

• All multimedia components (audio, video, text,graphics) used in the system must be somehowintegrated.

• Every device, such as microphone and camera isconnected to and controlled by a singlecomputer.

• A single type of digital storage is used for allmedia type.

• Video sequences are shown on computer screeninstead of TV monitor.



Interactivity

• Level 1: Interactivity strictly on informationdelivery. Users select the time at which thepresentation starts, the order, the speed and theform of the presentation itself.

• Level 2: Users can modify or enrich the content ofthe information, and this modification is recorded.

• Level 3: Actual processing of users input and thecomputer generate genuine result based on theusers input.



• Digitally Represented

• Digitization: process involved in transformingan analog signal to digital signal.



7

Audio Format: Digital AudioApplication of Digital Audio - Selected Examples • Music Production

– Hard Disk Recording – Sound Synthesis – Samplers – Effects Processing

• Video - Audio Important Element: Music and Effects

• Web - Many uses on Web

– Spice up Web Pages – Listen to Cds– Listen to Web Radio

• Many More Uses - try and think of some?

8

Digitization of Sound

• Sound is a continuous wave that travelsthrough the air

• The wave is made up of pressuredifferences. Sound is detected bymeasuring the pressure level at aparticular location.

• Sound waves have normal waveproperties (reflection, refraction,diffraction, etc.).

9

Digitizing Audio

• That is the basic idea of digitizing a soundunfortunately things are (practically speaking)not so simple.

• Questions for producing digital audio(Analogue-to-Digital Conversion):

1. How often do you need to sample the signal?

2. How good is the signal?

3. How is audio data formatted?

10

Computer Manipulation of Sound • Once Digitised processing the digital sound is essentially

straightforward although it depends on the processingyou wish to do (e.g. volume is easier to code thanaccurate reverb)

Sample Rates and Bit SizeHow do we store each sample value (Quantisation)?• 8 Bit Value

– (0-255)

• 16 Bit Value– (Integer) (0-65535)

How many Samples to take?– 11.025 KHz Speech (Telephone 8KHz)– 22.05 KHz Low Grade Audio

(WWW Audio, AM Radio)– 44.1 KHz CD Quality

11

Implications of Sample Rate and

Bit Size

Affects Quality of Audio

• Ears do not respond to sound in a linear fashion

• Decibel (dB) a logarithmic measurement ofsound

• 16-Bit has a signal-to-noise ratio of 98 dB -virtually inaudible

• 8-bit has a signal-to-noise ratio of 50 dB

• Therefore, 8-bit is roughly 8 times as noisy

– 6 dB increment is twice as loud

12

Typical Audio Formats

• Popular audio file formats include .au (Unixworkstations), .aiff (MAC, SGI), .wav (PC,DEC workstations)

• A simple and widely used audiocompression method is Adaptive Delta PulseCode Modulation (ADPCM). Based on pastsamples, it predicts the next sample andencodes the difference between the actualvalue and the predicted value.

13

Computer Music - MIDI

• Sound waves, whether occurred natural or man-made, are often very complex, i.e., they consistof

many frequencies. Digital sound is relativelystraight forward to record complex sound.However, it is quite difficult to generate (orsynthesize) complex sound.

• There is a better way to generate high qualitymusic. This is known as MIDI — MusicalInstrument Digital Interface.

14

MIDI

• It is a communication standard developed in theearly 1980s for electronic instruments andcomputers. It specifies the hardware connectionbetween equipments as well as the format inwhich the data are transferred between theequipments.

• Common MIDI devices include electronic musicsynthesisers, modules, and MIDI devices incommon sound cards.

• General MIDI is a standard specified by MIDIManufacturers Association. To be GM compatible,a sound generating device must meet the GeneralMIDI system level 1 performance requirement.

15

MIDI• Musical Instrument Digital Interface

a protocol that enables computer, synthesizers, keyboards, and other musical devices to communicate with each other.

Typical Sequencer setup

Setup:

MIDI OUT of synthesizer is connected to MIDI IN of sequencer.

MIDI OUT of sequencer is connected to MIDI IN of synthesizer and "through" to each of the additional sound modules.

Working:

During recording, the keyboard-equipped synthesizer is used to send MIDI message to the sequencer, which records them.

During play back, messages are sent out from the sequencer to the sound modules and the synthesizer which will play back the music.

THRU

THRU

THRU

IN OUT IN OUT

IN

IN

MIDI Interface/Sound Card

(Sequencer)

Synthesizer/Keyboard

MIDI Module A

MIDI Module B

Etc.

16

MIDI: Data Format

• Information traveling through the hardware is encodedin MIDI data format.

• The encoding includes note information like beginning ofnote, frequency and sound volume; upto 128 notes

• The MIDI data format is digital

• The data are grouped into MIDI messages

• Each MIDI message communicates one musical eventbetween machines. An event might be pressing keys,moving slider controls, setting switches and adjustingfoot pedals.

• 10 mins of music encoded in MIDI data format is about200 Kbytes of data. (compare against CD-audio!)

17

MIDI files

When using computers to play MIDI music, theMIDI data are often stored in MIDI files. EachMIDI files contains a number of chunks. There aretwo types of chunks:

• Header chunk — contains information about theentire file: the type of MIDI file, number of tracksand the timing.

• Track chunk — the actual data of MIDI track.

There three types of MIDI file:• 0 single multi-channel track• 1 one or more simultaneous track of a sequence• 2 one or more sequentially independent single-

track patterns

18

Tracks, channels and patches

• Multiple tracks can be played at the sametime.

• Each track can be assigned to a differentchannel.

• Each channel can accept more than onetrack.

• Each channel is assigned a patch,therefore generates sound of a particularinstrument.

19

How MIDI Sounds Are Synthesized

A simplistic view is that:

• the MIDI device stores the characteristicsof sounds produced by different soundsources;

• the MIDI messages tell the device whichkind of sound, at which pitch is to begenerated, how long the sound is playedand other attributes the note should have.

20

There are two ways of synthesizing sounds:

• FM Synthesis (Frequency Modulation)—Usingone sine wave to modulate another sine wave,thus generating a new wave which is rich intimbre. It consists of the two original waves, theirsum and difference and harmonics.

The drawbacks of FM synthesis are: the generatedsound is not real; there is no exact formula forgenerating a particular sound.

• Wave-table synthesis— It stores representativedigital sound samples. It manipulates thesesamples, e.g., by changing the pitch, to create thecomplete range of notes.

21

MIDI Sound Attributes

• The shape of the amplitude envelop hasgreat influence on the resulting characterof sound. There are two different types ofenvelop:

• Diminishing sound — gradually die out;

• Continuing sound — sustain until turnedoff.

chapter 2: Image format 22

Image Format and color model:

Analog versus bitmap images• The visual world is analog, which is to say that

real-world images is a continuous spectrum of colors.

• Infinity is uncomfortably large to a computer. A computer-stored image is split into little bits of light.

• A bitmap is a matrix of colored dots.

.


Bitmaps, Pixels & Colors

• The colored dots that make up a bitmap areproperly called "pixels".

• Color is represented on a computer by usingvarying amounts of red, green and blue light.

• In the simplest sort of bitmapped image, eachpixel is represented by three numbers to storethe amounts of red, green and blue light thatdefine the color of the pixel in question.


True Color Bitmap• Each pixel requires one byte for each color index

for a total of three bytes per pixel.

• As a byte represents eight bits, each pixelrequires 24 bits to store all its color information.

• This defines the maximum number of discretecolors this sort of bitmap can represent as 224, or16,777,216.

• Such graphics are referred to as “True Color"images, or just as "24-bit" graphics.


Bit-Depth = Color-Depth• Number of Colors = 2^(Bit-depth)

• Bit-depth is the number of bits.

– It is also called “Color resolution”.

Bit depth Color resolution Calcuation

1-bit 2 colors 2^1 = 2



4-bit 16 colors 2^4 = 16

8-bits 256 colors 2^8 = 256

16-bits 65,536 colors 2^16 = 65536

24-bits 16,777,215 colors 2^24 = 16.7 million


Vector Graphics• Vector graphics define pictures as collections of

lines, ellipses, triangles, polygons and other basicgraphic "primitives."

• Vector graphics are limited to storing mechanicalart

• Vector graphic formats include

– Corel Draw files, which use the extension CDR,

– AutoCAD drawings, which use the extensionDXF.


Bitmap vs. Vector graphics


CompressionGraphic file formats can be regarded as being of three types.

• The first type stores graphics uncompressed.

– Windows BMP (.bmp) files are an example of this sort offormat.

• The second type is called "non-lossy“ or “lossless”compression formats.

– Most graphic formats use lossless compression - the GIFformats are among them.

• The third type of bitmapped graphic file formats is called"lossy" compression.

– the details are what prevent areas from being all the samecolor, and as such from responding well to compression.

– perhaps too subtle to be discernable by your eye

Examples of common interchange formats are:•- GIF - Graphics Interchange Format

•- PNG - Portable Network Graphics

•- JFIF - JPEG (Joint Photographic Experts Group) File Interchange Format

•- TIFF - Tagged Image File Format

•- PGM - Portable Gray Map

• -PBM - Portable Bit Map

• -PPM - Portable Pix Map

• - FITS - Flexible Image Transport System 29chapter 2: Image format

Image File Formats

Test Image in Various Formats

• JPG 67 kb

• GIF 185 kb (256 colors, optimized)

• BMP 899 kb (640*479*3/1024 = 898.125)

• TIF 902 kb

• PNG 418 kb

• RAW 900 kb

30chapter 2: Image format

GIF (Graphics Interchange Format )

• GIF creates a table of up to 256 colors from a pool of 16 million.

• If the image has fewer than 256 colors, GIF can render the image exactly.

• GIF is "lossless" only for images with 256 colors or less. For a rich, true color image, GIF may "lose" 99.998% of the colors.

• The GIF format can contain animations. Its internal structure issuch that it can store multiple images and the controls to makethem appear as real time animation– animated GIF.

• The GIF format also allows a special color as to be specified as"using the background." This results in the image looks liketransparent– transparent GIF.



Animated GIF

• Advantages:– No need other software or plugins

– GIF is the standard format on the Web.

– GIF animated tools are available to use.

• The file contains layered frames ontop of each other.

• Tips:– Avoid more than one animated GIF on

a page.

– Avoid animated GIF on text-rich pages.

– Examine the pause between repetitions.


Transparency GIF

• The image can be shapes other than rectangles.

• One position of the color palette is designated as“Transparent”.

• All pixels of the image that have this particular colorindex will be painted as transparent when viewing.

a transparent GIF


JPG/JPEG Format• JPEG – Joint Photographic Experts Group

• The JPEG format uses "lossy" compression to get moregraphics into a smaller file than would otherwise bepossible.

• An image written to the JPEG format will be degraded.

• The amount of degradation, the "quality factor," can usuallybe set in a graphics software.– If the value is set to 100, almost no degradation will occured.

– If it is set to a value close to zero, the resulting image will be a verysmall file but unrecognizable.

– The default value of 75 is usually a good compromise.

chapter 2: Image format35

Progressive JPEG• There are two types of JPEG files extant as of this writing,

called "sequential" and "progressive".

– A sequential JPEG file stores its image as a simplebitmap.

– A progressive JPEG files stores its image such that it canappear initially out of focus when it begins to downloadto a web page, and resolve itself as more of the image isreceived by your web browser.

• Advantage: Provide indication of the whole image to the viewerbefore the entired image is loaded.

• Disadvantage: Require more computational power to display.

GIF Versus JPG• For photographic images JPG is better both for quality

and file size

• For line art GIF is better both for quality and file size

• The future of line art is probably in vector file formats like SVG (Scalable Vector Graphics)

• VML (Vector Markup Language) is Microsoft backed


GIF JPEG

Best application Line Art, Image with few color text

Photographs, Image with many colors

How to reduce display time?

Interlace Interlace (Progressive)

Display speed Fast Slower, more computation

Benefits Transparency,

Animation

Greatest compress for photographs, more color

Max. color 256 16.7 million

TIFF (Tagged Image File Format )• TIFF can be lossless or lossy.

• The details of the image storage algorithm areincluded as part of the file.

• TIFF is used almost exclusively as a lossless imagestorage format that uses no compression at all.

• Most graphics programs that use TIFF do not usecompression .

• File sizes are quite big.– (Sometimes LZW is used, but it is not universally

supported.)


PNG ( Portable Network Graphics )• PNG is a lossless storage format.• It looks for patterns in the image that it can use to compress file size.• The compression is exactly reversible, so the image is recovered exactly.• Support three image type: true color, grayscale, palette-based (8-bit).

• JPEG supports the first 2.• GIF supports the 3rd one.

Advantages– Better Compression

• Deflate is an improved version of the Lempel-Ziv compression algorithm.– Improve Interlacing

• Display image quicker than Interlaced GIF.– True Color and Transparency

• Support 16-bit (Grey scale) or 48-bit (True Color)• 16-bit for alpha channel (Transparency).

– Gamma storage• Store the gamma setting of the platform of the creator.

Disadvantages– Not support by old browsers (Netscape 2,3,4 and IE 2,3,4)


BMP bitmap (DIB) file format /bitmap• BMP is an uncompressed proprietary format invented

by Microsoft.

• There is really no reason to ever use this format.


PSD•PSD is a proprietary format used by Photoshop.

•Working format to edit images in the software

•This package uses layers to build complex images, and layerinformation may be lost in the nonproprietary formats such asTIFF and JPG.

•However, it is best to save the product as a TIFF or JPG, so itcan be viewed in the future when software changes.

Number of colors

• Images have differing numbers ofcolor within them.

• Black and white images take upthe least amount of space with 1bit of information.

• Current video cards are able todisplay 24 bits of color per pixel.

• This makes it possible for 16million colors to be displayed (224

colors).

• This requires a significant amountof disk space.



8-bit and 24-bit photos• Can you see the difference?


Color PalettesThe store images having a maximum of 16,777,216 colours, isone which can store images having a maximum of 256colours.

• Such files are referred to as using "palette-colour“. Thecolours in a palette-colour file are derived from a potentialpalette of 16,777,216 colours, but no more than 256 of themcan be used in any one image.

The Web Palette• Color in Browser – 24-bit color displays on an 8-bit monitor

display.

• Need a common palette (Web Palette) for browser, guarantees theimage will look the same on different platforms.

• If the browser is in 24-bit display, the palette does not affect theimages.

• Web Palette consists of 216 colors

• Also called: Web-safe Palette, Non-dithering Palette, 6x6x6 cube.


Which Color Palette?

Blue (0,0,255)Black (0,0,0)

Red (255,0,0)

White(FF,FF,FF)

(255,255,255)

Green (0,255,0)


6x6x6 Color Palette


Define RGB Values

• Decimal– Ranging from 0 to 255.

– 51-51-255 means red value is 51, green value is 51, blue value is 255.

• Hexadecimal– Base-16 number system

– Only six characters to describe RGB color.

– 51-51-255 is 3333FF.

• Percentage– 51-51-51 is 20%-20%-100%

– 00,33,66,99,CC,FF – 0%,20%,40%,60%,80%,100%

– Some old Macintosh programs use it.


Gamma control• the overall brightness of the display of a computer monitor.

• The relationship of voltage and the light intensity.

• Default gamma setting varies from platform to platform– Macintosh – 1.8 Gamma

– PC – 2.2 Gamma

– UNIX – 2.4 Gamma

• The higher the gamma value, the darker the display.

• Image created on PC looks wash out on Macintosh.

Different Gamma Value

Hue - Ink Mixing

• Subtractive mix of transparent inks

• Primary: CMY(Cyan, Magenta, yellow)

• Secondary: RGB

• ~Black: C + M + Y

• Actually use CMYKto get true black


Assumption: ink printed on pure white paper

CMY = White – RGB:

C = 1 – R, M = 1 – G, Y = 1 – B

CMYK from CMY (K is black ink): K = min(C, M, Y)

C = C – K, M = M – K, Y = Y - K

Hue - Light Mixing

• Additive mix of coloredlights

• Primary: RGB

• Secondary: CMY

• White = R + G + B

• Show demonstration of optical mixing


Video Format

• Video formats involve two distinct, and verydifferent technology

• concepts: containers (sometimes called wrappers)& codecs (short for coder/decoder).

• Codecs are used inside of a container and becauseof this video formats can be confusing.


• Container

The container describes the structure of the file: where thevarious pieces are stored, how they are interleaved, andwhich codecs are used by which pieces. It may specify anaudio codec as well as video. It is used to package thevideo & its components (audio/metadata) and is identified(usually) by a file extension such as .AVI, .MP4 or .MOV.

• Codec

A codec (short for "coder/decoder") is a way of encodingaudio or video into a stream of bytes. It is the method usedto encode the video and is the chief determiner of quality.


How do they work together?

Think of the container as the file itself.

Think of the codec as its contents.

The important thing to realize is that most good containerformats can hold many codecs.

For example a .MOV container can hold almost any kindof codec data. The same goes for .MP4 and even .AVI filescan hold a wide variety of codecs as their contents. In noway does the container decide the quality or features ofthe video itself, that is up to the codec.

The proper way to describe video is to indicate both: A.MOV file containing H.264 data. An .AVI file containingDivX data. Professionals use shorthand’s for this, sayingthings like "Give me an H.264 Quicktime file (.mov)".


Why is it so complicated?

At first this seems straightforward, but it's not. Theproblem lies in the confusion and imprecision in thecommon use of these terms. To make matters worse,software companies try to simplify their documentationand instructions by ignoring the difference altogether.The result is that people believe that phrases like "I'llgive you a MOV file," or "An MP4 file will be fine" arelegitimate ways to talk about video.

To make life even more confusing, some names, such as"mpeg-‐4", describe both a codec and a container, so it'snot always clear from context, which is being used. Youcould have a movie encoded with an mpeg-‐4 codecinside an avi container, for example, or a movie encodedwith the Sorenson codec inside an mpeg-‐4 container.


What Video Format Should I Use?

File size & quality

Digital Storage Space -‐ To calculate the amount of storage space you willneed for a project, digital video requires approximately 200 MB perminute of footage, or roughly 12 GB per hour. Of course this variesaccording to your recording device and the quality it is set to record at.

Frames per Second – The standard for FPS is 29.97, increasing the FPS allows

for more images per second thus a smoother image. Decreasing FPS will make the video a bit choppy and not nearly as smooth.

Video Bitrate -‐ Bitrate is a measurement of the number of bits that aretransmitted over a set length of time. Your overall bitrate is a combination of your video stream, audio stream & metadata in your file with the majority coming from your video stream. The higher the bit rate the better the quality the bigger it will be.

Resolution – this is the number of pixels present in the images of the video. This determines whether your video is standard definition or high definition. The higher the resolution the clearer the image the bigger the file.


List of Most Common Codecs• MPEG (Moving Pictures Expert Group): three video formats, MPEG 1, 2,

and 4.

• MPEG-‐1: Old, supported by everything (at least up to 352x240), reasonably efficient. A good format for the web.

• MPEG-‐2: A version of MPEG-‐1, with better compression. 720x480. Used in HDTV, DVD, and SVCD.

• MPEG-‐4: A family of codecs, some of which are open, others Microsoft proprietary.

• H.264: Most commonly used codecs for videos uploaded to the web. Part of the MPEG-‐4 codec.

• MPEG spinoffs: mp3 (for music) and VideoCD.

• MJPEG (Motion JPEG): A codec consisting of a stream of JPEG images. Common in video from digital cameras, and a reasonable format for editing videos, but it doesn't compress well, so it's not good for web distribution.

• DV (Digital Video): Usually used for video grabbed via firewire off a video camera. Fixed at 720x480 @ 29.97FPS, or 720x576 @ 25 FPS. Not very highly compressed.


• WMV (Windows Media Video): A collection of Microsoft proprietaryvideo codecs. Since version 7, it has used a special version of MPEG4.

• RM (Real Media): a closed codec developed by Real Networks forstreaming video and audio.

• DivX: in early versions, essentially an ASF (incomplete early MPEG-‐4)codec inside an AVI container; DivX 4 and later are a more full MPEG-‐4 codec…no resolution limit. Requires more horsepower to play thanmpeg1, but less than mpeg2. Hard to find mac and windows players.

• Sorenson 3: Apple's proprietary codec, commonly used for distributingmovie trailers (inside a Quicktime container).

• Quicktime 6: Apple's implementation of an MPEG4 codec.• RP9: a very efficient streaming proprietary codec from Real (not

MPEG4).• WMV9: a proprietary, non-‐MPEG4 codec from Microsoft.• Ogg Theora: A relatively new open format from Xiph.org.• Dirac: A very new open format under development by the BBC.


List of Most Common Codecs

List of Most Common Containers

• AVI (Audio Video Interleave): a Windows' standard multimedia container.

• MPEG-‐4 Part 14 (known as .mp4): is the standardized container for MPEG-‐4.

• FLV (Flash Video): the format used to deliver MPEG video through Flash Player.

• MOV: Apple's QuickTime container format.

• OGG, OGM & OGV: open-‐standard containers.

• MKV (Mastroska): another open-‐specification container that you've seen if you've ever downloaded anime.

• VOB (DVD Video Object): It's DVD's standard container.

• ASF: a Microsoft format designed for WMV and WMA—files can end in .wmv or .asf


Sample Bit Rates

• 16 kbit/s – videophone quality (minimum necessary for a consumer-‐acceptable “talking head” picture using various video compression schemes)

• 128 – 384 kbit/s – business-‐oriented video conferencing quality using video compression

• 1.5 Mbit/s max – VCD quality (using MPEG1 compression)

• 3.5 Mbit/s typ — Standard-‐definition television quality (with bit-‐rate reduction from MPEG-‐2 compression)

• 9.8 Mbit/s max – DVD (using MPEG2 compression)

• 8 to 15 Mbit/s typ – HDTV quality (with bit-‐rate reduction from MPEG-‐4 AVC compression)

• 19 Mbit/s approximate — HDV 720p (using MPEG2 compression)

• 24 Mbit/s max — AVCHD (using MPEG4 AVC compression)

• 25 Mbit/s approximate — HDV 1080i (using MPEG2 compression)

• 29.4 Mbit/s max – HD DVD

• 40 Mbit/s max – Blu-‐ray Disc (using MPEG2, AVC or VC-‐1 compression)


chapter 2: multimedia basics and representation

Documents