html5 multimedia: where we are, where we're going
DESCRIPTION
A much-hyped feature of HTML5 is native multimedia. In this session we’ll look at embedding and into your pages, and how to make it work cross-browser and degrade gracefully in older browsers. Sound too good to be true? It’s not!We’ll look at the pros and the cons of HTML5 multimedia and see how to write simple controls with JavaScript. Most excitingly, we’ll also look at how HTML5 builds in support for subtitles and captions for multimedia accessibility. And you might pick up a Turkish dancing tip on the way.---Edited version of my Web Directions London talk on 26 May 2011. Slides that don't make sense out of context are removed.TRANSCRIPT
![Page 1: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/1.jpg)
HTML5 multimediaWhere we are, where we're goingBruce Lawson
![Page 2: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/2.jpg)
Anne van Kesteren annevk at opera.comWed Feb 28 05:47:56 PST 2007
Hi,Opera has some internal expiremental builds with an implementation of a <video> element. The element exposes a simple API (for the moment) much like the Audio() object: play() pause() Stop()
The idea is that it works like <object> except that it has special <video> semantics much like <img> has image semantics. In markup you could prolly use it as follows:
<figure> <video src=news-snippet.ogg> ... </video> <legend>HTML5 in BBC News</legend> </figure>
I attached a proposal for the element and as you can see there are still some open issues. The element and its API are of course open for debate. We're not enforcing this upon the world ;-)
Cheers,http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2007-February/009702.html
![Page 3: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/3.jpg)
<object width="425" height="344"><param name="movie"
value="http://www.example.com/v/9sEI1AUFJKw&hl=en&fs=1&"></param>
<param name="allowFullScreen" value="true"></param>
<param name="allowscriptaccess" value="always"></param>
<embed src="http://www.example.com/v/9sEI1AUFJKw&hl=en&fs=1&" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"></embed></object>
![Page 4: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/4.jpg)
<video src=pudding.webm controls autoplay poster=poster.jpg width=320 height=240> <a href=video.webm>Download movie</a></video>
![Page 5: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/5.jpg)
<audio src=bieber.ogg controls autoplay> <a href=bieber.ogg>Download horrid pap</a></audio>
![Page 6: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/6.jpg)
<video src=pudding.webm loop>
![Page 7: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/7.jpg)
<audio src=bieber.ogg preload><audio src=bieber.ogg preload=auto><audio src=bieber.ogg preload=none><audio src=bieber.ogg preload=metadata>
![Page 8: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/8.jpg)
video as native object...why is it important?
● keyboard accessibility built-in● “play nice” with rest of the page● Simple API for controls
![Page 9: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/9.jpg)
"In addition to giving video an HTML element, we must also agree on a baseline video format that will be universally supported, just like the GIF, JPEG and PNG image format are universally supported. It's important that the video format we choose can be supported by a wide range of devices and that it's royalty-free (RF). RF is a well-established principle for W3C standards."
http://people.opera.com/howcome/2007/video/
![Page 10: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/10.jpg)
Ogg Theora
“free and open”, no licensing/royaltiesnot many tools for it, not web optimised
![Page 11: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/11.jpg)
MP4 / H.264
ubiquitous, patent encumbered, licensing/royalties, hardware accelerated
![Page 12: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/12.jpg)
WebM
"open and royalty-free", web optimised, hardware acceleration on its way
![Page 13: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/13.jpg)
video formats
webM Ogg/ Theora mp4/ h264
Opera yes yes
Chrome yes yes Nope (Chrome.soon)
Firefox Yes (FF4) yes
Safari yes
IE9 Yes (if installed) yes
![Page 14: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/14.jpg)
The politics of codecs
![Page 15: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/15.jpg)
audio formats
mp3 Ogg/ Vobis wav
Opera yes yes
Chrome yes yes
Firefox yes yes
Safari yes yes
IE9 yes yes
![Page 16: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/16.jpg)
Giving everybody video
<video controls autoplay poster=… width=… height=…><source src=pudding.mp4 type=video/mp4><source src=pudding.webm type=video/webm><!-- fallback content -->
</video>
![Page 17: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/17.jpg)
<video controls poster="…" width="…" height="…"><source src="movie.mp4" type="video/mp4" /><source src="movie.webm" type="video/webm" /><source src="movie.ogv" type="video/ogg" /><object width="…" height="…" type="application/x-
shockwave-flash" data="player.swf"><param name="movie" value="player.swf" /><param name="flashvars" value=" … file=movie.mp4" /><!-- fallback content -->
</object></video>
still include fallback for old browsershttp://camendesign.com/code/video_for_everybody
![Page 18: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/18.jpg)
Help
● archive.org converts and hosts creative-commons licensed media
● vid.ly has a free conversion/ hosting service (max 1GB source file) see vid.ly/5u4h3e
● Miro video converter is a drag and drop GUI skin on FFMPEG
![Page 19: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/19.jpg)
powerful (simple) API
![Page 21: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/21.jpg)
controlling <video> with JavaScript
var v = document.getElementByTagName('video')[0];
v.play();v.pause();v.volume = … ;v.currentTime = … ;…
![Page 22: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/22.jpg)
events fired by <video>var v = document.getElementById('player');v.addEventListener('loadeddata', function() { … }, true)v.addEventListener('play', function() { … }, true)v.addEventListener('pause', function() { … }, true)v.addEventListener('timeupdate', function() { … }, true)v.addEventListener('ended', function() { … }, true)…
![Page 23: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/23.jpg)
Race condition:html5demos.com/video
![Page 24: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/24.jpg)
video media queries
<video controls><source src=hi-res.ogv media="(min-device-width:800px)"><source src=lo-res.ogv></video>
http://dev.w3.org/html5/spec/video.html#the-source-element
![Page 25: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/25.jpg)
Full-screen video● Currently, WebkitEnterFullscreen();● May 11, WebKit announced it's implementing Gecko API https://wiki.mozilla.org/Gecko:FullScreenAPI
● Opera likes this approach, too
![Page 26: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/26.jpg)
<video> accessibility
![Page 27: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/27.jpg)
WebM release does not support subtitles
http://code.google.com/p/webm/issues/detail?id=11Egg image Kacper "Kangel" Aniołek http://commons.wikimedia.org/wiki/File:Egg.jpg
WHATWG / W3C RFC will release guidance on subtitles and other overlays in HTML5 <video> in the near future. WebM intends to follow that guidance.
![Page 28: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/28.jpg)
<track> element
<video controls poster=… width=… height=…><source src=movie.webm type=video/webm><track src=subtitles-en.vtt kind=subtitles srclang=en><track src=subtitles-de.vtt kind=subtitles srclang=de><!-- fallback content -->
</video>
http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#the-track-element
![Page 29: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/29.jpg)
webVTTWEBVTT FILE
101:23:45,678 --> 01:23:46,789Hello world!
201:23:48,910 --> 01:23:49,101HelloWorld!www.iandevlin.com/blog/2011/05/html5/webvtt-and-video-subtitlesblog.gingertech.net/2011/03/29/webvtt-explained/
![Page 30: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/30.jpg)
WebVTT formatting
Supports positioning of text
Supports <b> and <i>
Colouring individual speakers
Support vertical text
Supports RTL
Supports ruby annotations
http://www.whatwg.org/specs/web-apps/current-work/webvtt.html
![Page 31: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/31.jpg)
<track> polyfillhttp://www.delphiki.com/html5/playr/
![Page 32: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/32.jpg)
Synchronising media elementsEach media element can have a MediaController. A MediaController is an object that coordinates the playback of multiple media elements, for instance so that a sign-language interpreter track can be overlaid on a video track, with the two being kept in sync....
Media elements with a MediaController are said to be slaved to their controller. The MediaController modifies the playback rate and the playback volume of each of the media elements slaved to it, and ensures that when any of its slaved media elements unexpectedly stall, the others are stopped at the same time.
When a media element is slaved to a MediaController, its playback rate is fixed to that of the other tracks in the same MediaController, and any looping is disabled.
http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#synchronising-multiple-media-elements
http://www.w3.org/WAI/PF/HTML/wiki/Media_Multitrack_Media_API
![Page 33: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/33.jpg)
How mature is all this?Looping latency
people.opera.com/patrickl/experiments/audio/hacky-looper/
HTML5 video issues on the iPad and how to solve themblog.millermedeiros.com/2011/03/html5-video-issues-on-the-ipad-and-how-to-solve-them/
Unsolved HTML5 video issues on iOShttp://blog.millermedeiros.com/2011/04/unsolved-html5-video-issues-on-ios/
Audio Sprites (and fixes for iOS) remysharp.com/2010/12/23/audio-sprites/
![Page 34: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/34.jpg)
“...extending the language to better support Web applications … This puts HTML in direct competition with other technologies intended for applications deployed over the Web, in particular Flash and Silverlight.”
Ian Hickson, Editor of HTML5http://lists.w3.org/Archives/Public/public-html/2009Jan/0215.html
![Page 35: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/35.jpg)
DRM on HTML5 without changes
Henri Sivonen
lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2010-July/027051.html
![Page 36: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/36.jpg)
getUserMedia API(previously known as "HTML5 <device>")
![Page 37: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/37.jpg)
<video autoplay></video><script>var video = document.querySelector(video);
If (navigator.getUserMedia) {navigator.getUserMedia('video',successCallback, errorCallback);
function successCallback(stream) { video.src = stream; }...</script>
![Page 38: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/38.jpg)
my.opera.com/core/blog/2011/03/23/webcam-orientation-preview
![Page 39: HTML5 Multimedia: where we are, where we're going](https://reader033.vdocument.in/reader033/viewer/2022051412/5495c06db47959514d8b4e31/html5/thumbnails/39.jpg)
Thanks Mike Taylor @miketaylr for JS help, and Patrick Lauke
@patrick_h_lauke for the Hixie voice. (What does that "H" stand for?)
Ian Devlin @iandevlin and Silvia Pfeiffer @silviapfeiffer for webVTT advice
Philip Jägenstedt @foolip and Simon Pieters @zcorpan for implementing/ QA-ing multimedia in Opera and explaining it
to me slowly and patiently