larry bouthillier csci e12 - april 3, 2007 video & multimedia on the web csci e-12 april 3, 2007

Larry BouthillierCSCI E12 - April 3, 2007

Video & Multimedia on the Web

CSCI E-12April 3, 2007


Larry Bouthillier

• 1st career in audio/multimedia production• Harvard Extension School C.A.S in Software

Engineering, 1996• Director of Software Development/Director of

Educational Technologies, Harvard Business School, 2000-2006

• Multimedia Technology Architect, Harvard University, 2006-

• Author, Speaker, Consultant, Instructor– streamingmedia.com– http://www.learningapi.com/blog


Outline

• Why is video different from other media on the Web?

• Formats and codecs and platforms• Tools for content creation and encoding• Delivery options


Then & Now

• RealVideo - RealPlayer 5.0, 28.8kbps• http://www.educause.edu/ir/library/html/cem/cem97/cem9744.html

• Flash VP6 - Broadband• http://www.brightcove.com/title.jsp?title=627011240&channel=1842533

09

• What’s different?– Better codecs– Faster CPUs– More bandwidth– Standards


Linking an image

• <img src=“sample.gif”>

• Image downloads to browser• Images can be cached• Browser natively supports file formats


What makes video different?

• Large volume of data• Plugin player renders video• Time-sensitive realtime delivery


Delivery - Downloading vs. Streaming

• Http download– Also called progressive download or progressive

streaming– Can be done with a standard Web server

• Streaming– Requires specialized streaming server


HTTP downloading

• Web server accepts request for file• Web server dumps data onto network • Whole file sent as fast as the network can

carry it• User can usually begin to watch video as soon

as enough of it has downloaded


What happens when you stream

• Realtime streaming protocols– rtsp:// rtmp:// mms://

• Two-way conversation between server and player

• Measured delivery of bits at the precise rate required by the bitstream

• Dropped packets are discarded (no resend requests) – (has codec implications)

• Server can request content from the middle of a file - seeking ahead in long content, or begin play from the middle of a video


Streaming vs. http download

HTTP• Short clips• Constant quality, variable

time• Can deliver high-quality

content for low-bandwidth users, if they are willing to wait

Streaming• Longer-form content• Constant time, variable

quality• Scales bitrate to

available bandwidth in realtime

• Random access into content

• Managed bandwidth (benefits infrastructure and costs)

• Viewing statistics• Live video


Streaming metafiles

• Small files linked on your Web page and served by your Web server

• Contain a set of instructions that tell the streaming media player how to connect to the streaming server to play the requested file

• Disentangles the URL/mimetype of the media file from the type and protocol that loads the media player


What happens when you stream - metafiles

(.asx, .ram, .qtl, .smi)l


Metafiles

• Playlists• Alternate media sources

– .asx (WindowsMedia)– .ram (Real)– .qtl (Quicktime)

• Synchronized media• Bitrate and language detection

– .smil (Quicktime & Real)

Metafiles - asx (basic)

• Microsoft Windows Media format

<asx version="3.0"> <entry> <ref href="mms://myserver.com/path_to_movie/myfile.wma" /> </entry></asx>

Metafiles - asx (time offsets)

• Microsoft Windows Media format

 <asx version="3.0"> <title>My Video Title</title> <entry> <ref href="mms://myserver.com/path_to_movie/myfile.wma" />

<starttime value="00:05:00" /> <duration value="00:03:00" />

</entry> </asx>

Metafiles - asx (playlist)<asx version = "3.0"> <title>Backyard outtakes <abstract>Outtakes of a lazy backyard day <banner href="banner.gif"> <abstract>Click here to see this asx source code <moreinfo href="http://www.emediacommunications.biz/blog" /> </banner> <entry> <title>Swingin' on a hammock <abstract>Watching the wildlife float by <ref href = "hammock_150k.wmv" /> <duration value="00:00:10.0" /> </entry> <entry> <title>Ducks and geese <abstract>Goin' for a swim  <banner href="greybanner.gif"> <abstract>Click on this grayscale banner to see the source code <moreinfo href="http://www.emediacommunications.biz/blog" /> </banner> <ref href = "ducksgeese_150k.wmv" /> </entry> <entry>   <ref href = "ducksgeese2_150k.wmv" /> </entry>

</asx>


Metafiles - ram

Basic: rtsp://myserver.com/myvideofile.rm

Time offsets:rtsp://myserver.com/myvideofile.rm?

start=00:05:00&end=00:08:00&title=My video title&screensize=double


Metafiles - smil

• Synchronized Multimedia Integration Language– Offers full multimedia authoring

• Full support in RealPlayer • Partial support in Quicktime• Many mobile devices

<smil> <body> <video src="board.rm"> </body></smil>


Metafiles - smil

<smil> <body> <video src="board.rm"> </body></smil>

Source: Cognitive Development Laboratory; http://www.psych.uiuc.edu/~kmiller/smil/download.html


Metafiles - smil (time offsets)

<smil> <body> <video src="board.rm” clip-begin="16s" clip-end="27s"> </body></smil>



Metafiles - smil (multiple videos)

<smil> <head> <layout> <root-layout width="1080" height="240"/> <region id="video_left" width="360" height="240" left="0" top="0"/> <region id="video_center" width="360" height="240" left="360" top="0"/> <region id="video_right" width="360" height="240" left="720" top="0"/> </layout> </head> <body> <par dur="30s"> <video src="black.rm" clip-begin="1.09s" region="video_left"/> <video src="board.rm" clip-begin="0s" region="video_center"/> <video src="white.rm" clip-begin="1.10s" region="video_right"/> </par> </body></smil>

[play]



Metafiles - smil (voiceover & caption)

<smil> <head> <layout> <root-layout width="1080" height="350"/> <region id="video_left" width="360" height="240" left="0" top="0"/> <region id="video_center" width="360" height="240" left="360" top="0"/> <region id="video_right" width="360" height="240" left="720" top="0"/> <region id="text_subtitle" width="560" height="100" left="260" top="250"/> </layout> </head> <body> <par dur="55s"> <video src="black.rm" begin="5s" clip-begin="1.09s" region="video_left"/> <video src="board.rm" begin="5s" clip-begin="0s" region="video_center"/> <video src="white.rm" begin="5s" clip-begin="1.10s" region="video_right"/> <textstream src="text.rt" region="text_subtitle"/> <audio src="voicetrack.rm"/> </par></body></smil>

[play]



Metafiles - smil (voiceover & caption)<smil> <head> <layout> <root-layout width="360" height="240"/> <region id="video_main" width="360" height="240" left="0" top="0"/> </layout> </head> <body> <par> <seq> <video src="black.rm" begin="2s" clip-begin="0s" dur="2.2s" region="video_main"/> <video src="board.rm" clip-begin="0s" dur="4.2s" region="video_main"/> <video src="black.rm" clip-begin="5s" dur="3.2s" region="video_main"/> <video src="board.rm" clip-begin="8s" dur="4.5s" region="video_main"/> <video src="white.rm" clip-begin="13.5s" dur="2s" region="video_main"/> <video src="board.rm" clip-begin="14s" dur="3.7s" region="video_main"/> <video src="black.rm" clip-begin="18s" dur="6s" region="video_main"/> <video src="board.rm" clip-begin="23s" dur="3.5s" region="video_main"/> <video src="white.rm" clip-begin="27s" dur="4.5s" region="video_main"/> <video src="board.rm" clip-begin="29s" dur="0.5s" region="video_main" fill="freeze"/> </seq> <audio src="voicetrack.rm"/> </par></body></smil>

[play]



Metafiles - Reference

• Articles linked from http://www.learningapi.com/articles.html

• Streamingmedia.com– Streaming Media Metafiles: When And How To Use

Them– Streaming Media Metafiles Part Two: ASX Files– Streaming Media Metafiles Part Three: Quicktime


Codecs and file formats - What do these terms really mean?

• File formats: containers for encoded bitstreams– FLV– MP4– AVI– WMV

• Codecs: specific bitstream encoding schemes– Sorenson Spark– MPEG4 SP– H.264– WM9– MP3/AAC


Codecs & Formats - Some popular options

• Audio– MP3– AAC– WMA

• Video– RealVideo– WindowsMedia– MPEG4 (…MPEG1/MPEG2)– Flash


Audio

• MP3– MP3 is not MPEG3 (MPEG1 Layer 3)– “Universal” format

• AAC– MPEG4 Part 3– iPod, iTunes (w/ DRM from iTunes Music Store)

• WMA– Microsoft platforms and devices– DRM available (from multiple vendors)


RealVideo

• Top quality video streaming codec on the market

• Player supports most file formats and codecs• Server serves MPEG1/2/4, MP3,

WindowsMedia, Quicktime• DRM available• Player penetration is a problem


WindowsMedia

• Family of codecs that includes internet-friendly, lossless, and HD

• Works mostly on Windows and mostly with Internet Explorer

• Mac/Linux, Firefox users on any platform face difficulties

• Use it if you need its DRM• DRM-support for some portable devices


MPEG4

• MPEG4 is a file format and a collection of codecs• MPEG4 SP

– Original MPEG4 codec– Does not scale well to internet bandwidths– Supported on iPod, Sony PSP, Creative Zen Vision

• MPEG4 H.264– Modern MPEG4 codec– Scales well to all bandwidths– Implementations vary

• Plays on iPod, different version plays on PSP• Realplayer, WindowsMedia, Quicktime all profess support• Flash VP6 based on H.264


Flash

• Flash MX 2004 – Sorenson Spark (H.263)– Poor codec, wide support– YouTube

• Flash 8+– On2 VP6– Variant of H.264– High quality codec at all bandwidths


Choosing a codec/format/platform

Player Desktop Penetration

Installer Size

Popular Formats/ Codecs

Flash Player 97 % 1.31 MB FLV (Sorenson Spark or VP6)

Windows Media Player

84 % 6.99 MB WMV

Quicktime Player 66 % 32.30 MB MOV, MP4 (H.264)

Real Player 56 % 8.08 MB RM

Source: http://www.jeroenwijering.com/?item=FLV_Video_Compression


Bitrate vs. Bandwidth

• Bitrate determines video quality, while bandwidth determines the user’s experience

– If (bitrate > bandwidth) { user waits }– If (bitrate < bandwidth) { video plays right away }


Three dimensions of video quality

• Frame size– Aspect ratios

• 4:3 (320x240, 240x180, 400x300)• Widescreen

– Multiple-of-16 rule

• Frame rate– Whole factor of original framerate– 24fps/30fps - (Telecine)

• Image quality– Codec and bitrate dependent


Video Artifacts - Blockiness

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Source: streamingmedia.com Proprietary Codecs Report, 2006


Video Artifacts - “Mosquitoes”





Video Articfacts - Deinterlacing





Video Artifacts - Mottled Colors





Tools matter

• Video codecs define the mathematical scheme for encode/decode of video

• Video codecs do not define the algoritms or processes used to create the encoded bitstream

• Different tools perform very differently, even when encoding with the same codec

• Some tools better at some kinds of content than others


Tools matter - 4 tools encode WindowsMedia

QuickTime™ and aTIFF (Uncompressed) decompressor




Creation tools

• First-party vendor products– RealProducer– WindowsMedia Encoder– Quicktime Pro– Flash Video Encoder– On2 Flix Pro (VP6)

• Third-party products– Sorenson Squeeze– Canopus Procoder– Autodesk Cleaner– Anystream Agility– Telestream FlipFactory


Synchronizing Images (slides) with video

• Event triggers embedded in bitstream (WM, RM, Flash)

• SMIL• Custom Flash• HTML+TIME


More info on encoding

• http://www.akamai.com/html/perspectives/whitepapers_content.html


Hosting

• Destination sites– YouTube, Metacafe, Google

• High-quality syndication– Brightcove– VideoEgg

• CDNs– Akamai

• NetStorage (On your domain, mirrors local storage or can be primary storage)

• Delivery

– VitalStream– mirrorImage– Limelight


CSCI E12 - Thank you!

Larry Bouthillier

[email protected]://www.learningapi.com/blog

larry bouthillier csci e12 - april 3, 2007 video & multimedia on the web csci e-12 april 3, 2007

Documents

media file

streaming media player

video different

media playerwhat

video multimedia

highquality content

content creation

web page