A gentle introduction to video encoding

A gentle(*) introduction to video encoding

(*)for some definition of gentle

Mark Pilgrim

January 8, 2009

Video is everywhere

HandheldsiPod, cell phone, PSP, ArchosInternetYouTube, Daily Motion, MetacafeTelevisionBroadcast, cable, DVD, Blu-RayIts a birdIts a planeItsBeauregard!PCsQuickTime, Windows Media Player, VLC

Video is everywhere

But how does it get there?

Encoding process

Source material(video + audio)Raw videoCompressed videoDemuxing separates streamsEncoding compresses streamsSingle video fileMuxing combines streams again

Raw audio

Compressed audio

Encoding process

Source material(video + audio)Raw videoCompressed videoSingle video file

Raw audio

Compressed audio

Time-coded captions

Hey, dont forget about accessibility!Transcription

The Zen of video encoding

Elements of a video

The video container holds it all together4The caption format defines when captions are displayed3The audio codec defines how waveforms are compressed2The video codec defines how the pixels are compressed1

How to choose the right formats

Pick a video container that works on your target platform4Pick a caption format that works on your target platform3Pick a audio codec that works on your target platform2Pick a video codec that works on your target platform1

Constraints

CPU capacity for encoding

CPU capacity for decoding and playing in real time

Acceptable delay between recording and delivery

Acceptable quality degradation

Screen dimensions

Your bandwidth

My bandwidth

Hard limits on storage size (for physical media)

Video codec compatibility

Audio codec compatibility

Patent licensing costs

And on and on

There is no right or wrong; there is only what works and what doesnt.The Zen of video encoding

Im looking for a Citigroup quote here to use, if possible

Video codecs

The wool pulled over your eyes to blind you from the truth

A video codec defines the compression algorithm used to store the array of pixels that are eventually displayed on the screen.and they cheat. A lot. And hope you wont notice.What is a video codec?

Im looking for a Citigroup quote here to use, if possible

Acronym overload: video codecs

DiracTheoraVC-1WMVH.264MPEG-4ASPMPEG-2MPEG-1WTF?

I could tell by the pixels

Common video codecs

NoYouTube, Google VideoMPEG WGH.263HDTV broadcastFree Software hippiesBlu-RayMicrosoft world dominationYouTube HD, iTunes, Blu-RayMovie piratesDVDVCDPopular usageYes*On2, Xiph.orgTheoraNoMicrosoftVC-1Yes*BBCDiracNoMicrosoftWMVNoMPEG WGMPEG-2NoMPEG WGMPEG-1NoMPEG WGH.264NoMPEG WGMPEG-4 ASPPatent-free?InventorCodec name

(*) As far as we knowSource: Wikipedia

Video encoders

Adobe PremiereffmpegH.263nonenoneWindows Media Encoder, Compressor, SqueezeWindows Media Encoder, Expression EncoderAdobe Media Encoder, Apple Compressor, On2 Flix Pro, Sorenson Squeeze, TelestreamDivX, 3ivXCompressor, Sonic, TMPGEncQuickTimeNon-OSS toolsoggenc, ThusneldaTheoranoneVC-1dirac-research, SchroedingerDiracnoneWMVffmpeg, mpeg2encMPEG-2ffmpeg, MP1EMPEG-1x264H.264ffmpeg, XvidMPEG-4 ASPOSS toolsCodec name

Source: DigitalContentProducer.com, doom9.org, Wikipedia

Audio codecs

The hand is quicker than the ear

An audio codec defines the compression algorithm used to store the waveforms that are eventually played through your speakers.and they cheat, too.What is an audio codec?

Im looking for a Citigroup quote here to use, if possible

Acronym overload: audio codecs

DTSAC3VorbisWMAAACMP3MP2AMRSeriously,WTF?

Can you hear me now?

Common audio codecs

NoiPod, iTunes Store videosMPEG WGAACBlu-RayDVDFree Software hippiesMicrosoft world dominationMusicVCD, DVD, digital TVRingtones, VoIPPopular usageNoDTS Inc.DTSNoDolbyAC3Yes*Xiph.orgVorbisNoMPEG WGMP2No3GPPAMRNoMicrosoftWMANoMPEG WGMP3Patent-free?OwnerCodec name

(*) As far as we knowSource: Wikipedia

Audio encoders

iTunes, NeroFAACAACDTS, Inc.DolbynoneWindows Media EncoderiTunesAudio Transcoder, Easy CD-DA Extractor3GPP reference encoderNon-OSS toolsnoneDTSffmpeg, AftenAC3ffmpeg, aoTuVVorbisTwoLAMEMP2ffmpeg, RetroCodeAMRffmpegWMALAMEMP3OSS toolsCodec name

Sources: HydrogenAudio.org, Wikipedia

Captioning formats

Just shoot me

A captioning format defines when, where, and how captions for the hearing impaired are displayed on screen while a video is playing.Some formats are reused for subtitles, translations, & commentary.What is a captioning format?

Im looking for a Citigroup quote here to use, if possible

Acronym overload: caption formats

DVDMP4TTSMILSAMIASS(really)SRTTeletextLine 21BabyJesuscries

No, I cant hear you now. What else ya got?

Overview of caption formats

DVDStored as imagesDVD-ForumDVDAnalog TV (Europe)customBBC et. al.TeletextAnalog TV (US + Canada)pseudo-ASCIIElectronic Industries AllianceLine 21UTF-8(multiple)(multiple)UTF-8windows-1252EncodingWindows Media PlayerMSDN pageSAMIMP4 files, iTunes StoreQuickTime (SMIL 1.0 only)AVI files, fansubbingAVI files, fansubbingPopular usageISO 14966-17MPEG-4 Timed TextA doom9.org forum post from 2004SubRipW3C RecommendationSMILSome Word document floating around the netAdvanced SubStation AlphaSpecCaption format

Source: Wikipedia

Container formats

those should be simple, right?

A container format defines how video tracks, audio tracks, captions, and metadata are stored together in a single file.like a ZIP archiveWhat is a video container?

Im looking for a Citigroup quote here to use, if possible

Video metadata

Technical metadata

How many video tracks?

How many audio tracks?

How many caption tracks?

Where is everything within the file? (track IDs, byte ranges)

Human-readable metadata

Video title

Genre

Release date

MPAA rating

Cover art

Metadata matters

Without human-readable metadata, the iTunes Store would look like this:$9.99The.Chronicles.Of.Narnia.Prince.Caspian.2008.SciFi.Fantasy.720x480.02h29m.m4v$9.99Smart.People.2008.Comedy.720x480.01h35m.m4v$9.99Iron.Man.2008.Action.Adventure.720x480.02h42m.m4v$9.99Kung.Fu.Panda.2008.Animation.Comedy.720x480.01h50m.m4v$9.99WALL-E.2008.Animation.Action.Adventure.720x480.01h38m.m4v$9.99Step.Brothers.2008.Comedy.720x480.01h45m.m4v$9.99Wanted.2008.Action.Adventure.720x480.01h50m.m4v$9.99Hancock.2008.Action.Adventure.720x480.01h42m.m4v

Source: list of top movies from RottenTomatoes.com

Acronym overload: container formats

EVOVOBASFOGGFLVMP4MKVAVILOL

One ring to rule them all, and in the darkness bind them.

Overview of container formats

ffmpeg, MP4Box, mp4creatorMPEG-LA.MP4, .M4VMPEG-4ffmpegApple.MOVQuickTimeffmpeg, oggmuxXiph.org.OGGOggffmpeg (in progress)Microsoft.ASF, .WMVAdvanced Systems Formatffmpeg, AVIMux-GUIMicrosoft.AVIAudio/Video Interchangeffmpeg, mkvtoolnixSome guys in Russia*.MKVMatroskaffmpegAdobe.FLVFlash VideoOSS toolsOwnerExtensionContainer Name

(*) Seriously, some guys in Russia. They contributed it to the public domain, so technically you own it.Source: Wikipedia

Putting it all together

Wheres the kaboom? There was supposed to be an Earth-shattering kaboom!

Common combinations

SAMIWMAWMV, VC-1ASFWindowsKateVorbisTheoraOGGLinuxQTTextTrackAACH.264MOVMac/QuickTimeSRT, ASSMP3MPEG-4 ASPAVIDVD piratesBlu-Ray subtitles (SUP)AC3, DTSH.264, VC-1EVOBBlu-RaySRT, ASSAC3, DTSH.264MKVBlu-Ray piratesMP4MP4MP4FLVVOBContainerCustom, if anyMP3H.263Flash 8Line 21, DVD subtitlesAC3MPEG-2DVDAAC+AC3AACAACAudioMPEG-4 Timed TextH.264iPod/iPhoneMPEG-4 Timed TextFlash Timed TextCaptionsH.264Flash 9+H.264AppleTVVideoPlatform/Audience

Source: Wikipedia

Thank You!

Q&A

Google Confidential and Proprietary

Google Confidential and Proprietary

Google Confidential and Proprietary

Google Confidential and Proprietary

Google Confidential and Proprietary

Google Confidential and Proprietary

Google Confidential and Proprietary