1JJIIJI

BackClose

Multimedia SystemsModule No: CMP632

c David Marshall 19992004

2JJIIJI

BackClose

About This Course Single Module Tutorials Labs Multimedia LabAssessment:

Exam 70% Coursework 30%

3JJIIJI

BackClose

Course Web Sitehttp://www.cs.cf.ac.uk/Dave/ISE Multimedia/

PDFs of Slides (Colour) PDF Additional Notes. HTML based notes Lots of Links to other material Under Development More to be added

See also related BSC Course:http://www.cs.cf.ac.uk/Dave/Multimedia/

4JJIIJI

BackClose

Aims of Module Broad grounding in issue surrounding multimedia,

role of multimedia Systems design of multimedia Systems

Multimedia data digital audio, graphics and video, etc Underlying concepts and representations of sound, pictures

and video, Data compression and transmission, Integration of media, Multimedia authoring, Delivery of multimedia. Applications

5JJIIJI

BackClose

Objectives of Course

Understand the relevance and underlying infrastructure ofthe multimedia systems.

Understand core multimedia technologies and standards(Digital Audio, Graphics, Video,data transmission/compression)

Be aware of factors involved in multimedia systemsperformance, integration and evaluation

6JJIIJI

BackClose

Syllabus Outline

Topics in the module include the following: Introduction: Multimedia applications and requirements (e.g.,

overview of multimedia systems, video-on-demand, interactivetelevision, video conferencing, hypermedia courseware,groupware, World Wide Web, and digital libraries).

Audio/Video fundamentals including analog and digitalrepresentations, human perception, andaudio/video equipment, applications.

Audio and video compression perceptual transform coders for images/video

(e.g., JPEG, MPEG, H.263, etc.), scalable coders (e.g., pyramid coders), and perceptual audio encoders. image and video processing applications and algorithms.

7JJIIJI

BackClose

Recommended Course BooksSupplied Text: Managing Multimedia:

Project Management for Interactive Media(2nd Edition)Elaine England and Andy Finney,Addison Wesley, 1998(ISBN 0-201-36058-6)

8JJIIJI

BackClose

Recommended Course Book

Fundamentals of MultimediaMark S. Drew, Li Ze-NianPrentice Hall, 2003(ISBN: 0130618721)

9JJIIJI

BackClose

Other Good General Texts

Multimedia Communications:Applications, Networks, Protocols andStandards,Fred Halsall,Addison Wesley, 2000(ISBN 0-201-39818-4)OR

Networked Multimedia Systems,Raghavan and Tripathi,Prentice Hall,(ISBN 0-13-210642)

10

JJIIJI

BackClose

The following books are highly recommended reading: Hypermedia and the Web: An Engineering Approach, D. Lowe and W. Hall, J. Wiley

and Sons, 1999 (ISBN 0-471-98312-8). Multimedia Systems, J.F.K, Buford, ACM Press, 1994 (ISBN 0-201-53258-1). Understanding Networked Multimedia, Fluckiger, Prentice Hall, (ISBN 0-13-190992-4) Design for Multimedia Learning, Boyle, Prentice Hall, (ISBN 0-13-242215-8) Distributed Multimedia:Technologies, Applications, and Opportunities in the Digital

Information Industry (1st Edition) P.W. Agnew and A.S. Kellerman , Addison Wesley,1996 (ISBN 0-201-76536-5)

Multimedia Communication, Sloane, McGraw Hill, (ISBN 0-077092228) Virtual Reality Systems, J. Vince, Addison Wesley, 1995 (ISBN 0-201-87687-6) Encyclopedia of Graphics File Formats, Second Edition by James D. Murray and William

vanRyper, OReilly & Associates, 1996 (ISBN: 1-56592-161-5)

11

JJIIJI

BackClose

Multimedia Authoring Useful for Assessed Coursework

Macromedia Director MX Demystified,Phil Gross, Macromedia Press (ISBN:0321180976)

Macromedia Director MX and Lingo:Training from the Source Phil Gross,Macromedia Press (ISBN: 0321180968)

Director 8 and Lingo (InsideMacromedia), Scott Wilson, Delmar(ISBN: 0766820084)

Director/Lingo Manuals ApplicationHelp and in Library

SMIL: Adding Multimedia to the WebTim Kennedy and Mary Slowinski,Sams.net (ISBN: 067232167X)

12

JJIIJI

BackClose

The following provide good reference material for parts of themodule:Multimedia Systems

Hyperwave:The Next Generation Web Solution, H. Maurer,Addison Wesley, 1996 (ISBn 0-201-40346).

13

JJIIJI

BackClose

Digital Audio

A programmers Guide to Sound, T. Kientzle, Addison Wesley,1997 (ISBN 0-201-41972-6)

Audio on the Web The official IUMA Guide, Patterson andMelcher, Peachpit Press.

The Art of Digital Audio, Watkinson,Focal/Butterworth-Heinmann.

Synthesiser Basics, GPI Publications. Signal Processing: Principles and Applications, Brook and

Wynne, Hodder and Stoughton. Digital Signal Processing, Oppenheim and Schafer, Prentice

Hall.

14

JJIIJI

BackClose

Digital Imaging/Graphics/Video

Digital video processing, A.M. Tekalp, Prentice Hall PTR,1995.

Encyclopedia of Graphics File Formats, Second Edition byJames D. Murray and William vanRyper, 1996, OReilly &Associates.

15

JJIIJI

BackClose

Data Compression

The Data Compression Book, Mark Nelson,M&T Books, 1995. Introduction to Data Compression, Khalid Sayood, Morgan

Kaufmann, 1996. G.K. Wallace, The JPEG Still Picture Compression Standard CCITT, Recommendation H.261 D. Le Gall, MPEG: AVideo Compression Standard forMultimediaApplications

K. Patel, et. al., Performance of a Software MPEGVideo Decoder P. Cosman, et. al., Using Vector Quantization for Image Processing

16

JJIIJI

BackClose

Introduction to Multimedia

What is Multimedia?

17

JJIIJI

BackClose

Introduction to MultimediaWhat is Multimedia?

Multimedia can have a many definitions these include:

Multimedia means that computer information can berepresented through audio, video, and animation in additionto traditional media (i.e., text, graphics/drawings, images).

18

JJIIJI

BackClose

General Definition

A good general definition is:

Multimedia is the field concerned with the computer-controlledintegration of text, graphics, drawings, still and moving images(Video), animation, audio, and any other media where everytype of information can be represented, stored, transmitted andprocessed digitally.

19

JJIIJI

BackClose

Multimedia Application Definition

A Multimedia Application is an Application which uses acollection of multiple media sources e.g. text, graphics, images,

sound/audio, animation and/or video.

20

JJIIJI

BackClose

What is HyperText and HyperMedia?

Hypertext is a text which contains links to other texts.The term was invented by Ted Nelson around 1965.

Figure 1: Definition of Hypertext

21

JJIIJI

BackClose

Hypertext is therefore usually non-linear (as indicated below).

Figure 2: Illustration of Hypertext Links

22

JJIIJI

BackClose

Hypermedia

HyperMedia is not constrained to be text-based. It can includeother media, e.g., graphics, images, and especially thecontinuous media sound and video.

Figure 3: Definition of HyperMedia

23

JJIIJI

BackClose

Example Hypermedia Applications?

24

JJIIJI

BackClose

Example Hypermedia Applications?

The World Wide Web (WWW) is the best example of ahypermedia application.

Powerpoint Adobe Acrobat Macromedia Director Many Others?

25

JJIIJI

BackClose

Multimedia Systems

A Multimedia System is a system capable of processingmultimedia data and applications.

A Multimedia System is characterised by the processing,storage, generation, manipulation and rendition of Multimediainformation.

26

JJIIJI

BackClose

Characteristics of a Multimedia System

A Multimedia system has four basic characteristics:

Multimedia systems must be computer controlled. Multimedia systems are integrated. The information they handle must be represented digitally. The interface to the final presentation of media is usuallyinteractive.

27

JJIIJI

BackClose

Challenges for Multimedia Systems

Distributed Networks Temporal relationship between data

Render different data at same time continuously. Sequencing within the mediaplaying frames in correct order/time frame in video

Synchronisation inter-media scheduling

E.g. Video and Audio Lip synchronisation is clearlyimportant for humans to watch playback of video and audioand even animation and audio.

Ever tried watching an out of (lip) sync film for a long time?

28

JJIIJI

BackClose

Key Issues for Multimedia Systems

The key issues multimedia systems need to deal with here are:

How to represent and store temporal information. How to strictly maintain the temporal relationships on play

back/retrieval What process are involved in the above. Data has to represented digitally AnalogDigital

Conversion, Sampling etc. Large Data Requirements bandwidth, storage, compression

29

JJIIJI

BackClose

Desirable Features for a Multimedia System

Given the above challenges the following feature a desirable (ifnot a prerequisite) for a Multimedia System:Very High Processing Power needed to deal with large data

processing and real time delivery of media. Special hardwarecommonplace.

Multimedia Capable File System needed to deliver real-timemedia e.g. Video/Audio Streaming.

Special Hardware/Software needed e.g RAID technology.Data Representations File Formats that support multimedia

should be easy to handle yet allow forcompression/decompression in real-time.

30

JJIIJI

BackClose

Efficient and High I/O input and output to the file subsystemneeds to be efficient and fast. Needs to allow for real-timerecording as well as playback of data. e.g. Direct to Diskrecording systems.

Special Operating System to allow access to file system andprocess data efficiently and quickly. Needs to support directtransfers to disk, real-time scheduling, fast interrupt processing,I/O streaming etc.

Storage and Memory large storage units (of the order of 50-100 Gb or more) and large memory (50 -100 Mb or more).Large Caches also required and frequently of Level 2 and 3hierarchy for efficient management.

Network Support Client-server systems common asdistributed systems common.

Software Tools user friendly tools needed to handle media,design and develop applications, deliver media.

31

JJIIJI

BackClose

Components of a Multimedia System

Now let us consider the Components (Hardware and Software)required for a multimedia system:Capture devices Video Camera, Video Recorder, Audio

Microphone, Keyboards, mice, graphics tablets, 3D inputdevices, tactile sensors, VR devices. Digitising/SamplingHardware

Storage Devices Hard disks, CD-ROMs, Jaz/Zip drives, DVD,etc

Communication Networks Ethernet, Token Ring, FDDI, ATM,Intranets, Internets.

Computer Systems Multimedia Desktop machines,Workstations, MPEG/VIDEO/DSP Hardware

Display Devices CD-quality speakers, HDTV,SVGA, Hi-Resmonitors, Colour printers etc.

32

JJIIJI

BackClose

Applications

Examples of Multimedia Applications include: World Wide Web Hypermedia courseware Video conferencing Video-on-demand Interactive TV Groupware Home shopping Games Virtual reality Digital video editing and production systems Multimedia Database systems

33

JJIIJI

BackClose

Trends in Multimedia

Current big applications areas in Multimedia include:

World Wide Web Hypermedia systems embrace nearlyall multimedia technologies and application areas.

MBone Multicast Backbone: Equivalent of conventional TVand Radio on the Internet.

Enabling Technologies developing at a rapid rate to supportever increasing need for Multimedia. Carrier, Switching,Protocols, Applications, Coding/Compression, Database,Processing, and System Integration Technologies at theforefront of this.

34

JJIIJI

BackClose

Multimedia Data: Input and format

Text and Static Data

Source: keyboard, floppies, disks and tapes. Stored and input character by character:

Storage of text is 1 byte per character (text or format character). For other forms of data e.g. Spreadsheet files some formats

may store format as text (with formatting) others may use binaryencoding.

Format: Raw text or formatted text e.g HTML, Rich Text Format(RTF), Word or a program language source (C, Pascal, etc..

Not temporal BUT may have natural implied sequence e.g.HTML format sequence, Sequence of C program statements.

Size Not significant w.r.t. other Multimedia.

35

JJIIJI

BackClose

Graphics

Format: constructed by the composition of primitive objectssuch as lines, polygons, circles, curves and arcs.

Input: Graphics are usually generated by a graphics editorprogram (e.g. Freehand) or automatically by a program (e.g.Postscript).

Graphics are usually editable or revisable (unlike Images). Graphics input devices: keyboard (for text and cursor control),

mouse, trackball or graphics tablet. graphics standards : OpenGL, PHIGS, GKS Graphics files usually store the primitive assembly Do not take up a very high storage overhead.

36

JJIIJI

BackClose

Images

Still pictures which (uncompressed) are represented as abitmap (a grid of pixels).

Input: Generated by programs similar to graphics or animationprograms.

Input: scanned for photographs or pictures using a digitalscanner or from a digital camera.

Analog sources will require digitising. Stored at 1 bit per pixel (Black and White), 8 Bits per pixel

(Grey Scale, Colour Map) or 24 Bits per pixel (True Colour) Size: a 512x512 Grey scale image takes up 1/4 Mb, a 512x512

24 bit image takes 3/4 Mb with no compression. This overhead soon increases with image size Compression is commonly applied.

37

JJIIJI

BackClose

Audio

Audio signals are continuous analog signals. Input: microphones and then digitised and stored usually compressed. CD Quality Audio requires 16-bit sampling at 44.1 KHz 1 Minute of Mono CD quality audio requires 5 Mb.

38

JJIIJI

BackClose

Video

Input: Analog Video is usually captured by a video cameraand then digitised.

There are a variety of video (analog and digital) formats Raw video can be regarded as being a series of single images.

There are typically 25, 30 or 50 frames per second. a 512x512 size monochrome video images take 25*0.25 =

6.25Mb for a minute to store uncompressed. Digital video clearly needs to be compressed.

39

JJIIJI

BackClose

Output Devices

The output devices for a basic multimedia system include

A High Resolution Colour Monitor CD Quality Audio Output Colour Printer Video Output to save Multimedia presentations to (Analog)

Video Tape, CD-ROM DVD. Audio Recorder (DAT, DVD, CD-ROM, (Analog) Cassette) Storage Medium (Hard Disk, Removable Drives, CD-ROM)

40

JJIIJI

BackClose

Multimedia Authoring:Systems and ApplicationsWhat is an Authoring System?

An Authoring System is a program which has pre-programmedelements for the development of interactive multimedia softwaretitles.

Authoring systems vary widely

orientation, capabilities, and learning curve.

41

JJIIJI

BackClose

Why should you use an authoring system?

can speed up programming possibly content developmentand delivery

about 1/8th However, the content creation (graphics, text, video, audio,

animation, etc.) not affected by choice of authoring system; time gains accelerated prototyping

42

JJIIJI

BackClose

Authoring Vs Programming

Big distinction between Programming and Authoring. Authoring

assembly of Multimedia possibly high level graphical interface design some high level scripting.

Programming involves low level assembly of Multimedia construction and control of Multimedia involves real languages like C and Java.

43

JJIIJI

BackClose

Multimedia Authoring Paradigms

The authoring paradigm, or authoring metaphor, is themethodology by which the authoring system accomplishes itstask.

There are various paradigms

Scripting Language

Iconic/Flow Control

Frame

Card/Scripting

Cast/Score/Scripting Macromedia DirectorHypermedia Linkage

Tagging SMIL

44

JJIIJI

BackClose

Scripting Language

closest in form to traditional programming. The paradigmis that of a programming language, which specifies (byfilename) multimedia elements, sequencing, hotspots, synchronization, etc.

Usually a powerful, object-oriented scripting language in-program editing of elements (still graphics, video, audio,

etc.) tends to be minimal or non-existent. media handling can vary widely

45

JJIIJI

BackClose

Examples

The Apples HyperTalk for HyperCard, Assymetrixs OpenScript for ToolBook and Lingo scripting language of Macromedia DirectorHere is an example lingo script to jump to a frame

global gNavSprite

on exitFramego the frameplay sprite gNavSprite

end

46

JJIIJI

BackClose

Iconic/Flow Control

tends to be the speediest in development time best suited for rapid prototyping and short-development

time projects. The core of the paradigm is the Icon Palette, contains: possible functions/interactions of a program, and the Flow Line shows the actual links between the

icons. slowest runtimes programs , high interaction overheads

Examples: Authorware IconAuthor

47

JJIIJI

BackClose

Frame

similar to the Iconic/Flow Control paradigm usually incorporates an icon palette the links drawn between icons are conceptual do not always represent the actual flow of the program.

Examples Quest (whose scripting language is C) Apple Media Kit.

48

JJIIJI

BackCloseFigure 4: Macromedia Authorware Iconic/Flow Control Examples

49

JJIIJI

BackClose

Card/Scripting

paradigm provides a great deal of power(via the incorporated scripting language)

suffers from the index-card structure. Well suited for Hypertext applications, and especially

suited for navigation intensive (e.g. Cyans MYST game)applications.

extensible via XCMDs and DLLs; all objects (including individual graphic elements) to be

scripted; many entertainment applications are prototyped in a

card/scripting system prior to compiled-language coding.

50

JJIIJI

BackClose

Cast/Score/Scripting

uses a music score as its primary authoring metaphor synchronous elements are shown in various horizontal

tracks simultaneity shown via the vertical columns. power of this metaphor lies in the ability to script the

behavior of each of the cast members. easily extensible to handle other functions (such as

hypertext) via XOBJs, XCMDs, and DLLs. best suited for animation-intensive or synchronized media

applications;Examples Macromedia Director Macromedia Flash cut Down director Interface

51

JJIIJI

BackClose

Hierarchical Object

paradigm uses a object metaphor (like OOP) visually represented by embedded objects and iconic

properties. learning curve is non-trivial, visual representation of objects can make very

complicated constructions possible.

52

JJIIJI

BackClose

Figure 5: Macromedia Director Score Window

53

JJIIJI

BackClose

Figure 6: Macromedia Director Cast Window

54

JJIIJI

BackClose

Figure 7: Macromedia Director Script Window

55

JJIIJI

BackClose

Hypermedia Linkage

similar to the Frame paradigm shows conceptual links between elements lacks the Frame paradigms visual linkage metaphor.

56

JJIIJI

BackClose

Tagging

tags in text files to link pages, provide interactivity and integrate multimedia elements.

Examples: SGML/HTML, SMIL (Synchronised Media Integration Language), VRML, 3DML and WinHelp

57

JJIIJI

BackClose

Issues in Multimedia Applications Design

There are various issues in Multimedia authoring.

Issues involved:

Content Design Technical Design

58

JJIIJI

BackClose

Content Design

Content design deals with:

What to say, what vehicle to use.In multimedia, there are five ways to format and deliver yourmessage.

You can

write it, illustrate it, wiggle it, hear it, and interact with it.

59

JJIIJI

BackClose

Scripting (writing)Rules for good writing:1. Understand your audience and correctly address them.2. Keep your writing as simple as possible. (e.g., write out the

full message(s) first, then shorten it.)3. Make sure technologies used complement each other.

60

JJIIJI

BackClose

Graphics (illustrating) Make use of pictures to effectively deliver your messages. Create your own (draw, (color) scanner, PhotoCD, ...), or

keep copy files of art works. Cavemen did it first.

Graphics Styles fonts colors

pastels earth-colors metallic primary color neon color

61

JJIIJI

BackClose

Animation (wiggling) Types of Animation

Character Animation humanise an object Highlights and Sparkles Moving Text Video live video or digitized video

62

JJIIJI

BackClose

2. When to Animate

Enhance emotional impact Make a point (instructional) Improve information delivery Indicate passage of time Provide a transition to next subsection

63

JJIIJI

BackClose

Audio (hearing)Types of Audio in Multimedia Applications:1. Music set the mood of the presentation, enhance the emotion,

illustrate points2. Sound effects to make specific points, e.g., squeaky doors,

explosions, wind, ...3. Narration most direct message, often effective

64

JJIIJI

BackClose

Interactivity (interacting) interactive multimedia systems! people remember 70% of what they interact with (according

to late 1980s study)

65

JJIIJI

BackClose

Types of Interactive Multimedia Applications:1. Menu driven programs/presentations

often a hierarchical structure (main menu, sub-menus, ...)2. Hypermedia

+: less structured, cross-links between subsections of thesame subject > non-linear, quick access to information+: easier for introducing more multimedia features, e.g., moreinteresting buttons-: could sometimes get lost in navigating the hypermedia

3. Simulations / Performance-dependent Simulations e.g., Games SimCity, Flight Simulators

66

JJIIJI

BackClose

Technical Design

Technological factors may limit the ambition of your multimediapresentation.

Studied Later in detail.

67

JJIIJI

BackClose

Storyboarding

The concept of storyboarding has been by animators and theirlike for many years.

68

JJIIJI

BackClose

Storyboarding

used to help plan the generalorganisation

used to help plan the contentof a presentation by recording

organizing ideas on indexcards,

placed on board/wall. Storyboard evolves as the

media are collected andorganised: new ideas and refinements

to the presentation aremade.

69

JJIIJI

BackClose

Storyboard Examples

DVD Example Storyboarding Explained Acting With a Pencil The Storyboard Artist Star Wars Quicktime Storyboard

70

JJIIJI

BackClose

Overview of Multimedia Software Tools

Digital Audio

Macromedia Soundedit - Edits a variety of different formataudio files, apply a variety of effects (Fig 8)

Figure 8: Macromedia Soundedit Main and Control Windows andEffects Menu

71

JJIIJI

BackClose

CoolEdit/Adobe Audtion Edits a variety of different formataudio files

Many Public domain audio editing tools also exist.

72

JJIIJI

BackClose

Music Sequencing and Notation

Cakewalk

Supports General MIDI Provides several editing views (staff, piano roll, event list)

and Virtual Piano Can insert WAV files and Windows MCI commands (animation

and video) into tracks

73

JJIIJI

BackClose

Cubase

A better software than Cakewalk Express Intuitive Interface to arrange and play Music (Figs 9 and 10) Wide Variety of editing tools including Audio (Figs 11 and 12 Score Editing

74

JJIIJI

BackClose

Figure 9: Cubase Arrange Window (Main)

75

JJIIJI

BackClose

Figure 10: Cubase Transport Bar Window Emulates a TapeRecorder Interface

76

JJIIJI

BackClose

Figure 11: Cubase Audio Window

77

JJIIJI

BackClose

Figure 12: Cubase Audio Editing Window with Editing Functions

78

JJIIJI

BackClose

Logic Audio

Cubase Competitor, similar functionalityMarc of the Unicorn Performer

Cubase/Logic Audio Competitor, similar functionality

79

JJIIJI

BackCloseFigure 13: Cubase Score Editing Window

80

JJIIJI

BackClose

Image/Graphics Editing

Adobe Photoshop

Allows layers of images, graphics and text Includes many graphics drawing and painting tools Sophisticate lighting effects filter A good graphics, image processing and manipulation tool

Adobe Premiere

Provides large number (up to 99) of video and audio tracks,superimpositions and virtual clips

Supports various transitions, filters and motions for clips A reasonable desktop video editing toolMacromedia Freehand

Graphics drawing editing packageMany other editors in public domain and commercially

81

JJIIJI

BackClose

Image/Video Editing

Many commercial packages available

Adobe Premier Videoshop Avid Cinema SGI MovieMaker

82

JJIIJI

BackClose

Animation

Many packages available including:

Avid SoftImage Animated Gif building packages e.g. GifBuilder

83

JJIIJI

BackClose

Multimedia Authoring

Tools for making a complete multimedia presentation whereusers usually have a lot of interactive controls.Macromedia Director

Movie metaphor (the cast includes bitmapped sprites, scripts,music, sounds, and palettes, etc.)

Can accept almost any bitmapped file formats Lingo script language with own debugger allows more control

including external devices, e.g., VCRs and video disk players Ready for building more interactivities (buttons, etc.) follows the cast/score/scripting paradigm, tool of choice for animation content (Well FLASH for Web).

84

JJIIJI

BackClose

Authorware

Professional multimedia authoring tool Supports interactive applications with hyperlinks,

drag-and-drop controls, and integrated animation Compatibility between files produced from PC version and

MAC versionOther Authoring Tools mentioned in notes later

85

JJIIJI

BackClose

Multimedia Authoring:Scripting (Lingo)

Cast/Score/Scripting paradigm.

This section is a very brief introduction to Director.

For further Information, You should consult: Macromedia Director Using Director Manual In Library Macromedia Director : Lingo Dictionary Manual In Library Macromedia Director: Application Help Select Help from within the Director

application. This is very thorough resource of information. Macromedia Director Guided tours see Help menu option. A variety of web sites contain director tutorials, hints and information including

http://www.macromedia.com

86

JJIIJI

BackClose

More Director References

Macromedia Director MXDemystified,Phil Gross,Macromedia Press (ISBN:0321180976)

Macromedia Director MX andLingo: Training from the SourcePhil Gross,Macromedia Press (ISBN:0321180968)

Director 8 and Lingo(Inside Macromedia),Scott Wilson,Delmar (ISBN: 0766820084)

87

JJIIJI

BackClose

Related Additional Material and Coursework

Tutorials with additional Director Instructional Material

See Lab Worksheets 1 + 2

Also Assessed Exercise 2

88

JJIIJI

BackClose

Director Overview/Definitions

movies Basic Director Commodity:interactive multimedia pieces that can include animation, sound, text, digital video, and many other types of media. link to external media

A movie can be as small and simple as an animated logo oras complex as an online chat room or game.

Frames Director divides lengths of time into a series of frames,cf. celluloid movie.

89

JJIIJI

BackClose

Creating and editing movies

4 Key Windows:

the Stage Rectangular area where the movie plays

90

JJIIJI

BackClose

the Score : Where the movie is assembled;

91

JJIIJI

BackClose

one or more Cast windows Where the movies media elementsare assembled;

92

JJIIJI

BackClose

andthe Control Panel Controls how the movie plays back.

93

JJIIJI

BackClose

To create a new movie:

Choose File > New > Movie

94

JJIIJI

BackClose

Some other key Director Components (1)

Channels the rows in the Score that contain sprites forcontrolling media numbered contain the sprites that control all the visible media Special effects channels at the top contain behaviors as

well as controls for the tempo, palettes, transitions, andsounds.

Sprites Sprites are objects that control when, where, and how mediaappears in a movie.

95

JJIIJI

BackClose

Some other key Director Components (2)

Cast members

The media assigned to sprites. media that make up a movie. includes bitmap images, text, vector shapes, sounds, Flash

movies, digital videos, and more.

Lingo Directors scripting language, adds interactivity to amovie.

Behaviors pre-existing sets of Lingo instructions.Markers identify fixed locations at a particular frame in a

movie.

96

JJIIJI

BackClose

Lingo Scripting (1)

Commands terms that instruct a movie to do something while themovie is playing. For example, go to sends the playback head toa specific frame, marker, or another movie.

Properties attributes that define an object. For examplecolorDepth is a property of a bitmap cast member,

Functions terms that return a value. For example, the date functionreturns the current date set in the computer. The key functionreturns the key that was pressed last. Parentheses occur at theend of a function,

Keywords reserved words that have a special meaning.For example, end indicates the end of a handler,

97

JJIIJI

BackClose

Lingo Scripting (2)

Events actions that scripts respond to.Constants elements that dont change. For example, the constants

TAB, EMPTY, and RETURN always have the same meaning, andOperators terms that calculate a new value from one or more

values. For example, the add operator (+) adds two or morevalues together to produce a new value.

98

JJIIJI

BackClose

Lingo Data Types

Lingo supports a variety of data types:

references to sprites and cast members, (Boolean) values: TRUE and FALSE , strings, constants, integers, and floating-point numbers.Standard Program structure syntax

99

JJIIJI

BackClose

Lingo Script Types (1)

Director uses four types of scripts.

Behaviors Behaviors are attached to sprites or frames in theScore.

Figure 14: Behavior Icon

Movie scripts available to the entire movie

Figure 15: Movie script icon

100

JJIIJI

BackClose

Lingo Script Types (2)

Parent scripts special scripts that contain Lingo used to createchild objects.

Figure 16: Parent script icon

Scripts attached to cast members independent of the Score.dont appear in the Cast window.

Figure 17: Script button

101

JJIIJI

BackClose

Director Example 1: Simple AnimationA Bouncing Ball Graphic

Run Example in Browser (Shockwave)Run Example in Browser (Lecture ONLY) No Lingo scripting. basic animation where a cast member

102

JJIIJI

BackClose

Creating the Bouncing Ball Graphic

The following steps achieve a simple bouncing ball animationalong a path:

1. Let us begin by creatinga new movie and setting theStage size: Start a New movie: File> New > Movie(Shortcut =Command+N)

Choose Modify > Movie> Properties.In stage size, choose 640x 480.

103

JJIIJI

BackClose

2. Now let us create a ball, using a the vector shape tool:

Choose Window > Vector Shape(Shortcut = Command+Shift+V)

Click the filled ellipse button. Draw an ellipse (circle) about the size of

the Vector Shape Window Click on the Gradient fill button. To change the colours, click the colour

box on the left side of the Gradientcolour control

Change the colour on the right side ofthe Gradient Colours to a dark blue.

Change the Gradient type pull-downmenu from Linear to Radial.

Change the Stroke Colour to white .

104

JJIIJI

BackClose

3. Now let us change a few other properties of this ellipse

Close the Vector Shape window. In the Cast Window, select the ellipse. Choose Edit > Duplicate (Shortcut = Command+D). Double click the new cast, which opens it in the Vector Shape

Tool. Change the Cycles to 3 and the Spread to 200. Name the latest ellipse to bouncing ball

105

JJIIJI

BackClose

4. Now we are going to animate the ball.

Drag bouncing ball from the castmember window to the stage.

You will notice the sprite (theobject that appears in the score)is extended over 20 frames.

Drag the right end of the sprite toframe 40.

Click anywhere in the middle ofthe sprite to select it.

resize the ellipse.

106

JJIIJI

BackClose

4. Ball Animation (Key Frames)

Click on frame 40 in channel 1(the end of the sprite), hold downOption and shift and drag theellipse to the right end of thestage.

To curve the path, we are going toinsert keyframes within the sprite.

Click on frame 10 of the sprite andchoose Insert > Keyframe(Shortcut =Command+Option+K)

Create keyframes at frame 20and 30.

at each keyframe, a circleappears on the path shown onthe stage.

Click on the keyframe 10 circleand drag it up.

Change other Keyframes. Rewind and play the movie.

107

JJIIJI

BackClose

Further Animation: 1.1 Shrinking the ball

Run Example Shrinking the ball (Shockwave)Run Shrinking the ball (Lecture Only) (Optional) Click on keyframe 40 in

the score and drag it to frame 60,notice how all the keyframes spread outproportionally.

(Optional) Click on the keyframes in thescore and adjust the path if you feel likeit.

While moving the keyframes, resize theballs so they slowly get smaller. Noticewhile you resize the balls, the pathchanges and you will need to edit thepath again.

Rewind and play the movie. Save your movie as example2.dir.

108

JJIIJI

BackClose

1.2. Animating sprite colour

Run Example: Animating sprite colour (Shockwave)Run Example: Animating sprite colour (Lecture Only) Working still with example1.dir. Open Property Inspector for Sprite

Right Mouse (or Ctrl) on Sprite(Score or Stage)

Select Properties... Click on the keyframes in the score,

and change the Foreground colour chip,Forecolor, to different colours.

Changing the foreground colour is likeputting a coloured film over your object.The resulting colour is a mixture of theobjects original colour and the film. Forthis reason, light colours work betterthan dark colours for this effect..

Rewind and play the movie. Save as example3.dir

109

JJIIJI

BackClose

1.3. Animating sprite transparency Making the BallDisappear

Run Example: Making the Ball Disappear (Shockwave)Run Example: Making the Ball Disappear (Lecture Only) Open example1.dir Open Property Inspector for Sprite Click on the keyframes in the score, and Change the Blend Transparency to 100, 75, 50, 25, 0 for the

consecutive keyframes. Rewind and play the movie. Save as example4.dir

110

JJIIJI

BackClose

1.4. Animating sprite shape Deforming The Ball

Run Example: Deforming The Ball (Shockwave)Run Example: Deforming The Ball (Lecture Only) Open example1.dir Open Property Inspector for Sprite Click on the keyframes in the score, and Change the Skew Angle to 0, 20, 40, 60 and 80 for the

consecutive keyframes. Rewind and play the movie Save as example5.dir

111

JJIIJI

BackClose

Director Example 2: Importing mediaTo import multimedia data thereare two basic ways:

Choose File > Import ...Useful for importing batchesof data (e.g. Several imagesequences.

Drag and drop source mediainto a cast member locationQuite Intuitive

112

JJIIJI

BackClose

Examples: Simple Image import and Manipulation

Drag an image into a spare cast member. Drag this cast member to the Score Set suitable Properties for Sprite

Manipulate as for a vector item above. Examples:

ex dave roll.dir sets up some keyframes and alters therotation of the image (Shockwave)

ex dave roll.dir sets up some keyframes and alters therotation of the image (Lecture Only)

ex dave sq.dir alters the skew angle (Shockwave) ex dave sq.dir alters the skew angle (Lecture Only)

113

JJIIJI

BackClose

Example: Falling Over Movie, ex dave movie.dir

Example: Falling Over Movie, ex dave movie.dir (Shockwave)Run Example: Falling Over Movie, ex dave movie.dir (Lecture

Only) Several Gif images depicting sequence

exist on disk. Choose File > Import Select items you wish to import by

double-clicking or pressing the Addbutton

Click on the Import Button Several new cast members should be

added Set looping on and play

114

JJIIJI

BackClose

Example: Pinching Movie Movie, ex dave pinch.dir

Example: Pinching Movie Movie, ex dave pinch.dir (Shockwave)Example: Pinching Movie Movie, ex dave pinch.dir (Lecture Only)

Photoshop has been used to set a pincheffect of varying degree for an image.

Import images as before To reverse the image set to obtain a smooth

back and forth animation:

Select the sprite sequence in the score Copy Sequence Press Command+C

(Copy), Click on the frame just after the sprite

sequence Paste Sequence press

Command+V (Paste). Click on this second sprite sequence

and choose Modify > ReverseSequence.

Select the 2 sprites by pressing Shiftand clicking on both. Choose Modify >Join Sprites.

115

JJIIJI

BackClose

Simple Lingo ScriptingDirector Example 3: Very Simple Action

Here we illustrate the basic mechanism of scripting in Directorby developing and extending a very basic example:

Making a button beep and attaching a message to a button

Making the a button beep (Shockwave)Making the a button beep (Lecture Only)

116

JJIIJI

BackClose

Making the Button Beep Movie

Open a new movie. Turn the looping on in the

control panel.

Open the tool palette. Click the push button icon. Draw a button on the stage,

and type in a label:button here

117

JJIIJI

BackClose

Our First Lingo

Now lets write a simple script for the button:

Press Ctrl+click the button in the castwindow and choose Cast MemberScript.

Director writes the first and last line forus, add a beep command so the scriptlook like this:

on mouseUpbeep

end

Close the window. Rewind and play the movie. Click the button a few times.

118

JJIIJI

BackClose

To pop up a message box on button press (and still beep)

Reopen the cast memberscript.

Change the text so it nowreads.

on mouseUpbeepalert "Button Pressed"

end

Close the window. Play the movie and click the

button.

119

JJIIJI

BackClose

Director Example 4: Controlling Navigation with Lingo

A slightly more complex Lingo Example

This examples illustrates how we may use Lingo Scripts as:

Cast Member Scripts Sprite Scripts Behaviour Scripts

120

JJIIJI

BackClose

Director Example 4: Ready Made Example

To save time, we begin we a preassembledDirector movie:

Run Lingo Navigation Example (Shockwave)Run Lingo Navigation Example(Lecture Only) Open lingo ex.3.2.dir Play the movie

press some of the buttons The Numbered buttons record

moves throughScenes/Frames

The Next/Back buttons replaywhat has been recorded.

121

JJIIJI

BackClose

The Loop the Frame script

We are first going to create a loop the frame script: Cast Member 11 controls the Frame

Looping Note we have created a special framemarking channel in the Score.

To create the associated script either Double click on the script icon in the

Score Ctrl-click on the Cast member and

select Member Script The scripting window appears. You can

edit the script text, it now reads:

on exitFramego the frame

end

This frame script tells Director to keepplaying the same frame.

The Loop lasts to frame 24

Pressing down Alt anddragging the frame script inthe Score can change thislength.

122

JJIIJI

BackClose

Scene Markers (1)

Now we will create some markers

To Create a Marker You Click in the marking channel forthe Frame and label some the marker with some typed text

In this example: Markers are at frame 1,10 and 20, naming themscene1, scene2 andscene3 respectively.

Note: You can deletea marker by clicking thetriangle and dragging itbelow the marker channel.

A cast member (9) scriptfor the next button has alsobeen created:

on mouseUpgo to next

end

The go to nextcommand tells Director togo to the next consecutivemarker in the score.

123

JJIIJI

BackClose

Scene Markers (2)

A cast member (10) script for the back button has also beencreated:

on mouseUpgo to previous

end

The go to previous command tells Director to go to theprevious marker in the score.

Once again, Play the movie, click on these buttons to seehow they work.

124

JJIIJI

BackClose

Sprite Scripts

Now We will create some sprite scripts:

Sometimes a button will behave one way in one part of the movie and behave another way in a different part of the movie. A typical example use of sprite scripts.

125

JJIIJI

BackClose

The Next Button Sprite Scripts (1)

Desired Action of Next Button: Jump to next scene

126

JJIIJI

BackClose

The Next Button Sprite Scripts (2)

Here we have split actions to map to ourScene Markers. To achieve this: Click on frame 10 of channel 6 (the

next button) this sprite and chooseModify > Split Sprite.

Do the same at frame 20. To attach a script to each split action:

Select the each sprite sequence(here in channel 6).

Ctrl-click on sequence and selectScript... from the pull-down inthe score to give a script window.

We add a suitable jump to nextscene

In example shown we have go to"scene2" :This command tells Director to sendthe movie to marker "scene2".

Could do other sequences similar But alternatives exist.

127

JJIIJI

BackClose

Behaviour Scripts

Example Here: Another way to write the sprite script on lastslide - using the Behaviour Inspector.

Behaviour Scripts can do A LOTMORE.

128

JJIIJI

BackClose

A Behaviour Script for Next Button (Scene 2) (1)

We now work with the second spritesequence (channel 6 in the score).

We will create (or have created) anassociated behaviour script: Ctrl-click on the second sequence Open the Behaviour Inspector

window Click on the Script Window icon next

to the Behaviour Inspector Tab. To Create/name a new Behaviour:

Click the + icon at the top left of thewindow and select new behaviourfrom the pull-down.

Give the behaviour a name, here itis called next2.

129

JJIIJI

BackClose

A Behaviour Script for Next Button (Scene 2) (2)

To add events/actions to the script youcan:

Under Events click the + iconIn this example we have added amouseUp from the menu.

Under Actions click the + iconIn this example we have chosenNavigation > Go to marker then findscene3 on the list

You can add/edit Lingo textmanually in the Script Editorwindow for the particular behaviour

130

JJIIJI

BackClose

Summary: Sprite Script Order

We now 2 scripts attached to a single object (achieving muchthe same task): a Cast Member script a Sprite script. Sprite scripts take priority over cast member scripts So Here cast member script will be ignored.

131

JJIIJI

BackClose

Some more Lingo to add to our example

Another part of our Application:The jump buttons 1-3 (Buttons 4-6 currently inactive).

We will be using Lingo play/ play do to record actions

We have created a Vector Graphic Image (Cast Member 2)for the main Recorder Interface

132

JJIIJI

BackClose

A problem in Director?

In Director: a script can only be associated with a completeObject

For the way we have created the Recorder Interface we require(and this is clearly a common requirement in many other cases): Only part an image to be linked instead of the whole object. One part for each of the jump buttons 1-3.There is a solution:

Use invisible buttons. These are shape cast members with an invisible border.

133

JJIIJI

BackClose

Creating our Invisible Buttons

We have added our invisible as CastMember 14. To create this component: Open the Tool palette window. Click on the no line button. Click on the rectangle button and

draw a rectangle on the stagearound the 1 button.

We have added this sprite to 8 and wehave attached a sprite script: Ctrl-click on frame 1 of channel 8

and select script. Attach a sprite script to this shape

with the command play scene1. Extend the sprite sequence so it

covers frame 1 to 24. Repeat the steps placing the sprite over

the 2 and 3 button

134

JJIIJI

BackClose

Final Part: the Back Button (1)

Director provides the ability to record actions for future useThe Lingo Play commandThe play command is similar to the go to command but

allows: Director records every time a play is initiated, Keeping track of the users path through the movie. You can move back on along this path by using the playdone command.

135

JJIIJI

BackClose

Final Part: the Back Button (1)

So in this Example Select the sprite sequence in channel 5 and Cast member

10. Attach a Sprite script reading

on mouseUpplay done

end

Rewind, play the movie, click all the 1, 2, 3 buttons in variousorders, click the back button also and observe the effect ofBack button

Complete example: lingo ex3.2.dir (Web Based) Complete example: lingo ex3.2.dir (Local Version)

136

JJIIJI

BackClose

Multimedia Authoring:Tagging (SMIL) Last lecture Lingo scripting This lecture Tagging SMIL an extension of XML for synchronised media integration.

137

JJIIJI

BackClose

What it is SMIL?

SMIL is to synchronized multimedia what HTML is to hyperlinked text. Pronounced smile

138

JJIIJI

BackClose

SMIL :

A simple, Vendor-neutral Markup language

Designed to: For all skill levels of WWW authors Schedule audio, video, text, and graphics files across a

timeline No need to master development tools or complex

programming languages. HTML-like need a text editor only Links to media medie not embedded in SMIL file

139

JJIIJI

BackClose

Drawbacks of SMIL?

Good Points:

A powerful tool for creating synchronized multimediapresentations on the web

Deals with low bandwidth connections.Bad Points:

Meant to work with linear presentations Several types of media can be synchronized to one timeline. Does not work well with non-linear presentations Ability to skip around in the timeline is buggy.For slideshow style mixed media presentations it the best the

web has to offer.

140

JJIIJI

BackClose

SMIL support

The W3C recommended SMIL in June 1998 Quicktime 4.0 supports SMIL (1999) Not universally supported across the Web. NoWeb browser directly support SMIL RealPlayer G2 supports SMIL Many other SMIL-compliant players, authoring tools, and

servers available.

141

JJIIJI

BackClose

Running SMIL Applications

For this course there are basically three ways to run SMILapplications (two use the a Java Applet) so there are basicallytwo SMIL supported mediums:

Quicktime supported since Quicktime Version 4.0.RealPlayer G2 integrated SMIL supportWeb Browser use the SOJA SMIL applet viewer with html

wrapperApplet Viewer use the SOJA SMIL applet viewer with html

wrapper

142

JJIIJI

BackClose

Quicktime media support is richer (see later sections onQuicktime).

You will need to use both as RealPlayer and SOJA supportdifferent media

Media Tag RealPlayer GRiNS SojaGIF img OK OK OK

JPEG img OK OK OKWav audio OK OK -

.au Audio audio OK OK OK.auz Audio Zipped audio - - OK

MP3 audio OK - -Plain text text OK OK OKReal text textstream OK - -

Real movie video OK - -AVI video OK OK -

MPEG video OK OK -MOV video OK - -

143

JJIIJI

BackClose

Using Quicktime Load the SMIL file into a Quicktime plug-in (configure Browser

helper app or mime type) or the Quicktime movie player.

144

JJIIJI

BackClose

Using RealPlayer G2The RealPlayer G2 is installed on the applications HD in the

RealPlayer folder.Real player supports lots of file format and can use plugins.

The main supported formats are:

Real formats: RealText, RealAudio, etc... Images: GIF, JPEG Audio: AU, WAV, MIDI, etc...

145

JJIIJI

BackClose

To run SMIL filesReal Player uses streaming to render presentations.

works better when calling a SMIL file given by a Real Server, rather than from an HTTP one.Locally RUN SMIL files

drag a SMIL file onto the RealPlayer G2 Application Open a local SMIL file inside RealPlayer G2 Application

146

JJIIJI

BackClose

Using the SOJA applet SOJA stands for SMIL Output in JavaApplet.

SOJA is an applet that render SMIL in a web page or in aseparate window. It supports the following formats:

Images: GIF, JPEG Audio: AU and AUZ (AU zipped) SUN Audio files Text: plain text

147

JJIIJI

BackClose

Running SOJATo run SMIL through an applet you have to

call the applet from an HTML file:

the SOJA (soja.jar) archive is located in the SMIL folder onthe Macintoshes.

You may need to alter the CODEBASE attribute for your ownapplications

The PARAM NAME="source" VALUE="MY SMILFILE.smil" ishow the file is called.

148

JJIIJI

BackClose

RUNNING APPLETSThis should be easy to do

Run the html file through a java enabled browser Use Apple Applet Runner

uses MAC OS Runtime Java (Java 1.2) less fat for SMIL applications (we do really need Web

connection for our examples) Efficient JAVA and MAC OS run. Located in Apple Extras:Mac OS Runtime For Java folder TO RUN: Drag files on to application, OR TO RUN: Open file from within application

149

JJIIJI

BackClose

Let us begin to SMIL SMIL Authoring

SMIL Syntax Overview

SMIL files are usually named with .smi or .smil extensions XML based syntax

150

JJIIJI

BackClose

Basic Layout The basic Layout of a SMIL Documents is asfollows:

151

JJIIJI

BackClose

A source begins with and ends with .Note that SMIL is case sensitive

....

152

JJIIJI

BackClose

SMIL documents have two parts: head and body. Each ofthem must have as a parent.

....

....

153

JJIIJI

BackClose

Some tags, such as meta can have a slash at their end:

....

....

This is because SMIL is XML-based.Some tags are written:

...

154

JJIIJI

BackClose

SMIL Layout Everything concerning layout (including windowsettings) is stored between the and the tags in the header as shown in the above subsection.

A variety of Layout Tags define the presentation layout:

......

155

JJIIJI

BackClose

Window settingsYou can set width and height for the window in which your

presentation will be rendered with .The following source will create a window with a 300x200

pixels dimension and also sets the background to be white.

156

JJIIJI

BackClose

Positioning Media It is really easy to position media with SMIL.You can position media in 2 ways:

Absolute Positioning Media are located with offsets fromthe origin the upper left corner of the window.

Relative Positioning Media are located relative to the windowsdimensions.

We define position with a tag

157

JJIIJI

BackClose

The Region tag To insert a media within our presentation we use the

tag.

must specify the region (the place) where it will be displayed. must also assign an id that identifies the region.

158

JJIIJI

BackClose

Lets say we want to insert the Cardiff icon (533x250 pixels) at 30 pixels from the left border and at 25 pixels from the top border.The header becomes:

......

159

JJIIJI

BackClose

The img tagTo insert the Cardiff icon in the region called cardiff icon, we

use the tag as shown in the source below.Note that the region attribute is a pointer to the

tag.

160

JJIIJI

BackClose

This produces the following output:

Figure 18: Simple Cardiff Image Placement in SMIL

161

JJIIJI

BackClose

Relative Position Exampleif you wish to display the Cardiff icon at 10% from the left border and at 5% from the top border, modify the previous source and

replace the left and top attributes.

162

JJIIJI

BackClose

Overlaying RegionsWe have just seen how to position a media along x and y axes

(left and top).What if two regions overlap ?

Which one should be displayed on top ?

163

JJIIJI

BackClose

The following code points out the problem:

164

JJIIJI

BackClose

To ensure that one region is over the other, add z-indexattribute to .

When two region overlay:

the one with the greater z-index is on top. If both regions have the same z-index, the first rendered one

is below the other.

165

JJIIJI

BackClose

In the following code, we add z-index to region 1 andregion 2:

166

JJIIJI

BackClose

fitting media to regions

You can set the fit attribute of the tag to forcemedia to be resized etc.

The following values are valid for fit: fill make media grow and fill the area. meet make media grow (without any distortion) until it

meets the region frontier. slice media grows (without distortion) and fill entirely its

region. scroll if media is bigger than its region area gets scrolled. hidden dont show mediaObviously you set the value like this:

167

JJIIJI

BackClose

Synchronisation There are two basic ways in which we maywant to play media:

play several media one after the other, how to play several media in parallel.In order to do this we need to add synchronisation:

we will need to add time parameter to media elements,

168

JJIIJI

BackClose

Adding a duration of time to media durTo add a duration of time to a media element simply specify a

dur attribute parameter in an appropriate media tag:

.....

.....

169

JJIIJI

BackClose

Delaying Media the begin tagTo specify a delay i.e when to begin set the begin attribute

parameter in an appropriate media tag:If you add begin=2s in the cardiff image tag, you will see

that the Cardiff icon will appear 2 seconds after the documentbegan and will remain during 6 other seconds. Have a look atthe source:

.....

.....

170

JJIIJI

BackClose

Sequencing Media the seq tagScheduling media:The tag is used to define a sequence of media.

The media are executed one after each other:.....

.....

So the setting 1s makes the img2.gif icon appear 1 secondafter img1.gif.

171

JJIIJI

BackClose

Parallel Media the par tagWe use the to play media at the same time:

This will display an image and play some music along with it.

172

JJIIJI

BackClose

Synchronisation Example 1: Planets SoundtrackThe following SMIL code plays on long soundtrack along with

as series of images.Essentially:

The audio file and image sequences are played in parallel The Images are run in sequence with no break (begin =0s)

173

JJIIJI

BackClose

The files are stored on the MACINTOSHES in the MultimediaLab (in the SMIL folder) as follows: planets.html call SMIL source (below) with the SOJA

applet. This demo uses zipped (SUN) audio files (.auz)which are not supported by RealPlayer.

planets.smil sMIL source (listed below),

174

JJIIJI

BackClose

SMIL HEAD DATA

...........

175

JJIIJI

BackClose

SMIL BODY DATA

........

..........

176

JJIIJI

BackClose

177

JJIIJI

BackClose

Synchronisation Example 2: Slides N SoundDr John Rosbottom of Plymouth University has come up with

a novel way of giving lectures.This has one long sequence of parallel pairs of images and audio filesThe files are stored on the MACINTOSHES in the Multimedia

Lab (in the SMIL folder) as follows: slides n sound.smil sMIL source (listed below), play

with RealPlayer G2. NOTE: This demo uses real audio fileswhich are not supported by SOJA:

178

JJIIJI

BackClose

............

179

JJIIJI

BackClose

copyright="Everything is so copyright protected (c)1999"/>

180

JJIIJI

BackClose

SMIL Events Smiles supports event based synchronisation:begin events

When a media begins, it sends a begin event. If another media waits for this event, it catches it.

181

JJIIJI

BackClose

To make a media wait to an event,

one of its synchronisation attributes (begin or end) should be written as follows:

182

JJIIJI

BackClose

For example:

will make the next.gif image begin 2s after cardiff.gifbegins.

183

JJIIJI

BackClose

The switch TagThe syntax for the switch tag is:

The rule is:

The first of the tag children whose test attributesare all evaluated to TRUE is executed.

A tag with no test attributes is evaluated to TRUE. See SMIL reference for list of valid test attributes

184

JJIIJI

BackClose

For example you may wish to provide presentations in englishor welsh:

< audio src ="english.au" />

somewhere in code you will set (or it will be set) thesystem-language

185

JJIIJI

BackClose

Multimedia Systems Technology

Multimedia systems have to deal with the

generation, manipulation, storage, presentation, and communication of information

Lets consider some broad implications of the above

186

JJIIJI

BackClose

Discrete v Continuous Media

RECALL: Our Definition of Multimedia

All data must be in the form of digital information. The data may be in a variety of formats:

text, graphics, images, audio, video.

187

JJIIJI

BackClose

Synchronisation

A majority of this data is large and the different media mayneed synchronisation:

The data will usually have temporal relationships as anintegral property.

188

JJIIJI

BackClose

Static and Continuous Media

Static or Discrete Media Some media is time independent:Normal data, text, single images, graphics are examples.

Continuous media Time dependent Media:Video, animation and audio are examples.

189

JJIIJI

BackClose

Analog and Digital Signals

Some basic definitions Studied HERE Overviewing of technology Studied HERE In depth study later.

190

JJIIJI

BackClose

Analog and Digital Signal Converters

The world we sense is full of analog signals:

Electrical sensors convert the medium they sense intoelectrical signals E.g. transducers, thermocouples, microphones. (usually)continuous signals

Analog signals must be converted ordigitised Digital: discrete digital signals that computer can readily deal

with. Special hardware devices : Analog-to-Digital converters Playback a converse operation: Digital-to-Analog .

191

JJIIJI

BackClose

Multimedia Data: Input and format

How to capture and store each Media format?

Note that Text, Graphics and some images are generateddirectly by computer and do not require digitising:

they are generated directly in some binary format.

Handwritten text would have to digitised either by electronicpen sensing of scanning of paper based form.

192

JJIIJI

BackClose

Text and Static Data

Source: keyboard, floppies, disks and tapes. Stored and input character by character:

Storage of text is 1 byte per character (text or formatcharacter).

For other forms of data e.g. Spreadsheet files someformats may store format as text (with formatting) othersmay use binary encoding.

Format: Raw text or formatted text e.g HTML, Rich TextFormat (RTF), Word or a program language source (C, Java,etc.

Not temporal BUT may have natural implied sequencee.g. HTML format sequence, Sequence of C programstatements.

Size Not significant w.r.t. other Multimedia.

193

JJIIJI

BackClose

Graphics

Format: constructed by the composition of primitive objectssuch as lines, polygons, circles, curves and arcs.

Input: Graphics are usually generated by a graphics editorprogram (e.g. Freehand) or automatically by a program (e.g.Postscript).

Graphics are usually editable or revisable (unlike Images). Graphics input devices: keyboard (for text and cursor

control), mouse, trackball or graphics tablet. graphics standards : OpenGL, PHIGS, GKS Graphics files usually store the primitive assembly Do not take up a very high storage overhead.

194

JJIIJI

BackClose

Images

Still pictures which (uncompressed) are represented as abitmap (a grid of pixels).

Input: Generated by programs similar to graphics oranimation programs.

Input: scanned for photographs or pictures using a digitalscanner or from a digital camera.

Analog sources will require digitising. Stored at 1 bit per pixel (Black and White), 8 Bits per pixel

(Grey Scale, Colour Map) or 24 Bits per pixel (True Colour) Size: a 512x512 Grey scale image takes up 1/4 Mb, a 512x512

24 bit image takes 3/4 Mb with no compression. This overhead soon increases with image size Compression is commonly applied.

195

JJIIJI

BackClose

Audio

Audio signals are continuous analog signals. Input: microphones and then digitised and stored usually compressed. CD Quality Audio requires 16-bit sampling at 44.1 KHz 1 Minute of Mono CD quality audio requires 5 Mb.

196

JJIIJI

BackClose

Video

Input: Analog Video is usually captured by a video cameraand then digitised.

There are a variety of video (analog and digital) formats Raw video can be regarded as being a series of single images.

There are typically 25, 30 or 50 frames per second. a 512x512 size monochrome video images take

25*0.25 = 6.25Mb

for a minute to store uncompressed. Digital video clearly needs to be compressed.

197

JJIIJI

BackClose

Output Devices

The output devices for a basic multimedia system include

A High Resolution Colour Monitor CD Quality Audio Output Colour Printer Video Output to save Multimedia presentations to (Analog)

Video Tape, CD-ROM DVD. Audio Recorder (DAT, DVD, CD-ROM, (Analog) Cassette) Storage Medium (Hard Disk, Removable Drives, CD-ROM)

198

JJIIJI

BackClose

Storage Media

The major problems that affect storage media are: Large volume of date Real time delivery Data format Storage Medium Retrieval mechanisms

199

JJIIJI

BackClose

High performance I/O

There are four factors that influence I/O performance:

Data

high volume, continuous, contiguous vs distributed storage. Direct relationship between size of data and how long it

takes to handle. Compression

200

JJIIJI

BackClose

Data Storage

Depends of the storage hardware and The nature of the data. The following storage parameters affect how data is stored: Storage Capacity Read and Write Operations of hardware Unit of transfer of Read and Write Physical organisation of storage units Read/Write heads, Cylinders per disk,

Tracks per cylinder, Sectors per Track Read time Seek time

201

JJIIJI

BackClose

Data Transfer

Depend how data generated and written to disk, and in what sequence it needs to retrieved. Writing/Generation of Multimedia data is usually

sequential e.g. streaming digital audio/video direct to disk. Individual data (e.g. audio/video file) usually streamed. RAID architecture can be employed to accomplish high

I/O rates (parallel disk access)

202

JJIIJI

BackClose

Operating System Support

Scheduling of processes when I/O is initiated. Time critical operations can adopt special procedures. Direct disk transfer operations free up CPU/Operating

system space.

203

JJIIJI

BackClose

Basic Storage

Basic storage units have problems dealing with large multimediadata

Single Hard Drives SCSI/IDE Drives. AV (Audio-Visual) drives

avoid thermal recalibration between read/writes, suitable for desktop multimedia.

New drives are fast enough for direct to disk audio and videocapture.

not adequate for commercial/professional Multimedia.

204

JJIIJI

BackClose

Removable Media Floppies not adequate Jaz/Zip Drives, CD-ROM, DVD-ROM.

205

JJIIJI

BackClose

RAID Redundant Array of Inexpensive DisksNeeded:

To fulfill the needs of current multimedia and other datahungry application programs,

Fault tolerance built into the storage device. Parallel processing exploits arrangement of hard disks.

Raid technology offers some significant advantages as astorage medium of multimedia data: Affordable alternative to mass storage High throughput and reliability

206

JJIIJI

BackClose

The key components of a RAID System are: Set of disk drives, disk arrays, viewed by user as one or more

logical drives. Data may be distributed across drives Redundancy added in order to allow for disk failure Disk arrays:

store large amounts of data, have high I/O rates and take less power per megabyte (cf. high end disks) but they have very poor reliability

As more devices are added, reliability deteriorates N devices generally have 1N the reliability of a single

device

207

JJIIJI

BackClose

Overcoming Reliability Problems

Redundancy Files stored on arrays may be striped acrossmultiple disks.

Four ways do to this.

208

JJIIJI

BackClose

Four ways of Overcoming Reliability Problems

Mirroring or shadowing of the contents of disk, which canbe a capacity kill approach to the problem. write on two disks - a 100% capacity overhead. Reads to disks may however be optimised.

Horizontal Hamming Codes: A special means toreconstruct information using an error correction encodingtechnique.

Parity and Reed-Soloman Codes: Also an error correctioncoding mechanism. Parity may be computed in a number ofways.

Failure Prediction: There is no capacity overhead in thistechnique.

209

JJIIJI

BackClose

RAID Architecture

Each disk within the array needs to have its own I/O controller,but interaction with a host computer may be mediated throughan array controller

DiskController

DiskController

DiskController

DiskController

ArrayController

Host Processor

HostAdaptor

Manages the controllogic and parity

210

JJIIJI

BackClose

Orthogonal RAID

Possible to combine thedisks together to produce acollection of devices, where

Each vertical array is nowthe unit of dataredundancy.

Such an arrangementis called an orthogonalRAID

Other arrangements ofdisks are also possible.

211

JJIIJI

BackClose

The Eight levels of RAIDThere are 8 levels of RAID technology: each level providing a greater amount of resilience then the

lower levels:

Level 0: Disk Striping

Level 1: Disk Mirroring

Level 2: Bit Interleaving and HEC Parity

Level 3: Bit Interleaving with XOR Parity

Level 4: Block Interleaving with XOR Parity

Level 5: Block Interleaving with Parity Distribution

Level 6: Fault Tolerant System

Level 7: Heterogeneous System

212

JJIIJI

BackClose

First Six RAID levels

SimultaneousWrites onevery drive

Simultaneous readson every drive

RAID 0 Simultaneous readson every drive

Data

drive pairs

Eachwritespansalldrives

Eachreadspansall drives

RAID 1RAID 2

ParallelAccess

Parity

Read andWriteSpan

Parity

EveryWrite must updatea dedicated parity drive

Simultaneous reads

Each drive now also handles parity

Simultaneous reads

RAID 3 RAID 4 RAID 5

Duplication on

Alldrives

on every drive

on every drive

indicated by the filled circle

213

JJIIJI

BackClose

Optical Storage

The most popular storage medium in the multimedia context compact size, High density recording, Easy handling and Low cost per MB. CD and recently DVD (ROM) the most common Laser disc older format.

214

JJIIJI

BackClose

CD StorageThere a now various formats of CD:

CD-DA (Compact Disc-Digital Audio) CD-I (Compact Disc-Interactive) CD-ROM/XA (eXtended Architecture) Photo CDThe capacity of a CD-ROM is

620-700 Mbs depending on CD material, 650/700 Mb (74/80 Mins) is a typical write once CD-ROM

size. Drives that read and write CD-ROMS (CD-RW) also similar.

215

JJIIJI

BackClose

CD Standards

There are several CD standard for different types of media:

Red Book Digital Audio: Most Music CDs.Yellow Book CD-ROM:

Model 1 computer data,Model 2 compressed audio/video data.

Green Book CD-IOrange Book Write once CDsBlue Book Laser disc

216

JJIIJI

BackClose

DVD

The current best generation of optical disc storage technologyfor Multimedia:

DVD Digital Versatile Disc (formal),Digital Video Disc (mistaken).

Larger storage and faster than CD over 2 Hours of Video / Single sided DVD-ROM 2.4 Gb

Formats: DVD-Video and DVD-ROM (DVD-R and DVD-RAM)

217

JJIIJI

BackClose

What are the features of DVD-Video?

The main features of DVD include:

Over 2 hours of high-quality digital video (over 8 on adouble-sided, dual-layer disc).

Support for widescreen movies on standard or widescreenTVs (4:3 and 16:9 aspect ratios).

Up to 8 tracks of digital audio (for multiple languages), eachwith as many as 8 channels.

Up to 32 subtitle/karaoke tracks. Automatic seamless branching of video (for multiple story

lines or ratings on one disc). Up to 9 camera angles (different viewpoints can be selected

during playback).

218

JJIIJI

BackClose

Main features of DVD (Cont)

Menus and simple interactive features (for games, quizzes,etc.).

Multilingual identifying text for title name, album name, songname, cast, crew, etc.

Instant rewind and fast forward, including search to title,chapter, track, and timecode.

Durability (no wear from playing, only from physical damage). Not susceptible to magnetic fields. Resistant to heat. Compact size (easy to handle and store, players can be

portable, replication is cheaper).

219

JJIIJI

BackClose

What are the disadvantages of DVD?

Despite several positive attributes mentioned above there aresome potential disadvantages of DVD:

It has built-in copy protection and regional lockout. It uses digital compression. Poorly compressed audio or

video may be blocky, fuzzy, harsh, or vague. The audio downmix process for stereo/Dolby Surround can

reduce dynamic range. It doesnt fully support HDTV. Some DVD players and drives may not be able to read CD-Rs. Dispute of some DVD-R formats

220

JJIIJI

BackClose

Comparison of DVD and CD-ROM

The increase in capacity inDVD-ROM (from CD-ROM)is due to:

smaller pit length(2.08x),

tighter tracks (2.16x), slightly larger data area

(1.02x), discs single or double

sided

221

JJIIJI

BackClose

Comparison of DVD and CD-ROM (Cont.)

another data layer added to eachside creating a potential for fourlayers of data per disc

more efficient channel bitmodulation (1.06x),

more efficient error correction(1.32x),

less sector overhead ( 1.06x). capacity of a dual-layer disc is

slightly less than double that of asingle-layer disc.

222

JJIIJI

BackClose

Multimedia Data Representation

Issues to be covered:

Digital Audio Graphics/Image Formats Digital Video (Next Lecture) Sampling/Digitisation Compression

223

JJIIJI

BackClose

Digital Audio

Application of Digital Audio Selected Examples

Music Production

Hard Disk Recording Sound Synthesis Samplers Effects Processing

Video Audio Important Element: Music and EffectsWeb Many uses on Web

Spice up Web Pages Listen to Cds Listen to Web Radio

224

JJIIJI

BackClose

What is Sound?

Source Generates Sound Air Pressure changes Electrical Loud Speaker Acoustic Direct Pressure Variations

Destination Receives Sound Electrical Microphone produces electric signal Ears Responds to pressure hear sound (more later(MPEG Audio))

225

JJIIJI

BackClose

Digitising Sound

Microphone produces analog signal Computer like discrete entitiesNeed to convert Analog-to-Digital Specialised Hardware

Also known as Sampling

226

JJIIJI

BackClose

Digital Sampling

Sampling basically involves: measuring the analog signal at regular discrete intervals recording the value at these points

227

JJIIJI

BackClose

Computer Manipulation of Sound

Writing Digital Signal Processing routines range from beingtrivial to highly complex:

Volume Cross-Fading Looping Echo/Reverb/Delay Filtering Signal Analysis

228

JJIIJI

BackClose

Sound Demos

Volume Cross-Fading Looping Echo/Reverb/Delay Filtering

229

JJIIJI

BackClose

Sample Rates and Bit Size

How do we store each sample value (Quantisation)?8 Bit Value (0-255)16 Bit Value (Integer) (0-65535)

How many Samples to take?11.025 KHz Speech (Telephone 8 KHz)22.05 KHz Low Grade Audio

(WWW Audio, AM Radio)44.1 KHz CD Quality

230

JJIIJI

BackClose

Nyquists Sampling Theorem

Sampling Frequency is Very Important in order to accuratelyreproduce a digital version of an Analog Waveform

Nyquists Theorem:

The Sampling frequency for a signal must be at least twicethe highest frequency component in the signal.

231

JJIIJI

BackClose

Figure 19: Sampling at Signal Frequency

232

JJIIJI

BackClose

Figure 20: Sampling at Twice Nyquist Frequency

233

JJIIJI

BackClose

Figure 21: Sampling at above Nyquist Frequency

234

JJIIJI

BackClose

Implications of Sample Rate and Bit Size

Affects Quality of Audio

Ears do not respond to sound in a linear fashion ((more later(MPEG Audio))

Decibel (dB) a logarithmic measurement of sound 16-Bit has a signal-to-noise ratio of 98 dB virtually

inaudible 8-bit has a signal-to-noise ratio of 50 dB Therefore, 8-bit is roughly 8 times as noisy

6 dB increment is twice as loud

Click Here to Hear Sound Examples

235

JJIIJI

BackClose

Implications of Sample Rate and Bit Size (cont)

Affects Size of Data

File Type 44.1 KHz 22.05 KHz 11.025 KHz16 Bit Stereo 10.1 Mb 5.05 Mb 2.52 Mb16 Bit Mono 5.05 Mb 2.52 Mb 1.26 Mb8 Bit Mono 2.52 Mb 1.26 Mb 630 Kb

Figure 22: Memory Required for 1 Minute of Digital Audio

236

JJIIJI

BackClose

Practical Implications of Nyquist Sampling Theory

Must (low pass) filter signal before sampling:

Otherwise strange artifacts from high frequency signals appear.

237

JJIIJI

BackClose

Why are CD Sample Rates 44.1 KHz?

238

JJIIJI

BackClose

Why are CD Sample Rates 44.1 KHz?

Upper range of human hearing is around20-22 KHz Apply Nyquist Theorem

239

JJIIJI

BackClose

Common Audio Formats

Popular audio file formats include .au (Origin: Unix workstations), .aiff (MAC, SGI), .wav (PC, DEC workstations)

Compression can be utilised in some of the above but is notMandatory

A simple and widely used (by above) audio compressionmethod is Adaptive Delta Pulse Code Modulation (ADPCM). Based on past samples, it

predicts the next sample andencodes the difference between the actual value and thepredicted value.

More on this later (Audio Compression)

240

JJIIJI

BackClose

Common Audio Formats (Cont.)

Many formats linked to audio applications Can use some compression

Sounblaster .voc (Can use Silence Deletion (More onthis later (Audio Compression))

Protools/Sound Designer .sd2 Realaudio .ra. Ogg Vorbis .ogg

MPEG AUDIOMore Later (MPEG-3 and MPEG-4)

241

JJIIJI

BackClose

Delivering Audio across a network

Trade off between desired fidelity and file size Bandwidth Considerations for Web and other media. Compress Files:

Could affect live transmission on Web

242

JJIIJI

BackClose

Streaming Audio

Buffered Data: Trick get data to destination before its needed Temporarily store in memory (Buffer) Server keeps feeding the buffer Client Application reads buffer

Needs Reliable Connection, moderately fast too. Specialised client, Steaming Audio Protocol (PNM for real

audio).

243

JJIIJI

BackClose

Synthetic Sounds reducing bandwidth?

Synthesise sounds hardware or software Client produces sound only send parameters to control sound (MIDI next) Many synthesis techniques could be used, For example:

FM (Frequency Modulation) Synthesis used in low-end Sound Blastercards, OPL-4 chip, Yamaha DX Synthesiser range popular in Early 1980s.

Wavetable synthesis wavetable generated from sound waves of real instruments Additive synthesis make up signal from smaller simpler waveforms Subtractive synthesis modify a (complex) waveform but taking out elements Physical Modelling model how acoustic sound in generated in software

Modern Synthesisers use a mixture of sample and synthesis.

244

JJIIJI

BackClose

MIDIWhat is MIDI?

No Longer Exclusively the Domain of Musicians. Midi provides a very low bandwidth alternative on the Web:

transmit musical and certain sound effects data

also now used as a compression control language (modified) See MPEG-4 Section soon

245

JJIIJI

BackClose

MIDI on the Web

Very Low Bandwidth (few 100K bytes)

The responsibility of producing sound is moved to the client: Synthesiser Module Sample Soundcard Software Generated

Most Web browsers can deal with MIDI.

246

JJIIJI

BackClose

Definition of MIDI:A protocol that enables computer, synthesizers, keyboards,

and other musical device to communicate with each other.

247

JJIIJI

BackClose

Components of a MIDI System

Synthesizer:

It is a sound generator (various pitch, loudness, tone colour). A good (musicians) synthesizer often has a microprocessor,

keyboard, control panels, memory, etc.

Sequencer: It can be a stand-alone unit or a software program for a

personal computer. (It used to be a storage server for MIDIdata. Nowadays it is more a software music editor on thecomputer.

It has one or more MIDI INs and MIDI OUTs.

248

JJIIJI

BackClose

Basic MIDI Concepts

Track: Track in sequencer is used to organize the recordings. Tracks can be turned on or off on recording or playing back.Channel: MIDI channels are used to separate information in a MIDI

system. There are 16 MIDI channels in one cable. Channel numbers are coded into each MIDI message.Timbre: The quality of the sound, e.g., flute sound, cello sound, etc. Multitimbral capable of playing many different sounds at

the same time (e.g., piano, brass, drums, etc.)

249

JJIIJI

BackClose

Basic MIDI Concepts (Cont.)

Pitch: The Musical note that the instrument playsVoice: Voice is the portion of the synthesizer that produces sound. Synthesizers can have many (12, 20, 24, 36, etc.) voices. Each voice works independently and simultaneously to produce

sounds of different timbre and pitch.Patch: The control settings that define a particular timbre.

250

JJIIJI

BackClose

Hardware Aspects of MIDI

MIDI connectors: Three 5-pin ports found on the backof every MIDI unit MIDI IN: the connector via

which the device receives all MIDIdata.

MIDI OUT: the connectorthrough which the devicetransmits all the MIDI data itgenerates itself.

MIDI THROUGH: theconnector by which the deviceechoes the data receives fromMIDI IN.

251

JJIIJI

BackClose

MIDI Messages

MIDI messages are used by MIDI devices to communicatewith each other.

MIDI messages are very low bandwidth:

Note On Command Which Key is pressed Which MIDI Channel (what sound to play) 3 Hexadecimal Numbers

Note Off Command Similar Other command (program change) configure sounds to be

played.

252

JJIIJI

BackClose

Structure of MIDI messages:

MIDI message includes a status byte and up to two databytes.

Status byte The most significant bit of status byte is set to 1. The 4 low-order bits identify which channel it belongs to

(four bits produce 16 possible channels). The 3 remaining bits identify the message.

The most significant bit of data byte is set to 0.

253

JJIIJI

BackClose

Classification of MIDI messages:

-- voice messages--- channel messages ---|

| -- mode messages|

MIDI messages ----|| -- common messages--- system messages ---|-- real-time messages

-- exclusive messages

254

JJIIJI

BackClose

Midi Channel messages:

messages that are transmitted on individual channels ratherthat globally to all devices in the MIDI network.

Channel voice messages:

Instruct the receiving instrument to assign particular soundsto its voice

Turn notes on and off Alter the sound of the currently active note or notes

255

JJIIJI

BackClose

Midi Channel Control Messages

Voice Message Status Byte Data Byte1 Data Byte2------------- ----------- ----------------- -----------------

Note off 8x Key number Note Off velocityNote on 9x Key number Note on velocityPolyphonic Key Pressure Ax Key number Amount of pressureControl Change Bx Controller number Controller valueProgram Change Cx Program number NoneChannel Pressure Dx Pressure value NonePitch Bend Ex MSB LSB

Notes: x in status byte hex value stands for a channelnumber.

256

JJIIJI

BackClose

Midi Command Example

A Note On message is followed by two bytes, one to identifythe note, and on to specify the velocity.

To play: Note number 80 (HEX 50) With maximum velocity (127 (Hex 7F) On channel 13 (Hex C),

The MIDI device would send these three hexadecimal bytevalues:

9C 50 7F

257

JJIIJI

BackClose

Midi Channel mode messages:

Channel mode messages are a special case of the ControlChange message (Bx (Hex) or 1011nnnn (Binary)).

The diffe