advance video distribution
DESCRIPTION
Advance Video Distribution lectureTRANSCRIPT
Advance Video Distribution
www.dsp-ip.comFast Forward Your Development
Day I - Agenda
• Current Delivery technologies (RTP/RTSP)
• Components in depth
▫ File Formats
▫ Codecs
• Video Streaming Evolution
▫ Requirements
www.dsp-ip.comFast Forward Your Development
▫ Requirements
▫ Solutions
• CDN
• Basic Adobe Flash
Day II - Agenda
• Microsoft “Smooth Streaming”
• Adobe “HTTP Streaming”
• Apple “HTTP Streaming”
• Internet Gorillas positioning
• Adobe OSMF
www.dsp-ip.comFast Forward Your Development
Current Delivery Methods
www.dsp-ip.comFast Forward Your Development
Progressive Download
• Simple• Utilizing existing protocols & servers (HTTP)
• Media file is prepared: metadata up front
• Playback - after metadata is received
• Cache-ability - supported
www.dsp-ip.comFast Forward Your Development
• Cache-ability - supported
• Seek-ability – very limited support
• Poor user experience - seek, multi-rate
• Waste of bandwidth when not watched fully
• Low cost
Pseudo Streaming• Media is sent as a regular file like Progressive DW
• Server must understand how the media is
structured
• Playback: after metadata is received
• Existing protocols
▫ Non standard server
www.dsp-ip.comFast Forward Your Development
▫ Non standard server
▫ Non standard client component
• Cache-ability – Limited !
• Seek-ability – supported
• User experience – better than PD, support seek.
• Waste of bandwidth when not watched full
Streaming
• Seek-ability – supported
• Server side – proprietary technology (FMS)
• Cache-ability – requires special servers for streaming
• User experience – very good
• Cost – high
www.dsp-ip.comFast Forward Your Development
• Cost – high
HTTP Streaming Intro
• HTTP Streaming offers the advantages of:▫ Progressive download in terms of � Cost
� Standard Server
� Scalability
� Standard client components (OSMF)
www.dsp-ip.comFast Forward Your Development
� Standard client components (OSMF)
▫ Streaming in terms of � User experience
� Seek-ability of streaming
RTSP Streaming
www.dsp-ip.comFast Forward Your Development
RTSP Protocol
• Real Time Streaming Protocol
• Used for controlling streaming data over the web.
• Designed to efficiently broadcast audio/video-on-demand to large groups.
www.dsp-ip.comFast Forward Your Development
on-demand to large groups.
• Using Directives to control the stream
▫ Options, Describe, Setup, Play, Pause, Record, Teardown.
SDP Protocol• Describes the metadata of the stream.
• Mainly used in: SIP, RTSP and other Multicast sessions.
• Sample SDP description:▫ v=0
▫ o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5
▫ s=SDP Seminar
▫ i=A Seminar on the session description protocol
Session ID
Protocol Version
Session Name
Session Info.
Description URI
www.dsp-ip.comFast Forward Your Development
▫ i=A Seminar on the session description protocol u=http://www.example.com/seminars/[email protected] (Jane Doe)
▫ c=IN IP4 224.2.17.12/127
▫ t=2873397496 2873404696
▫ a=recvonly
▫ m=audio 49170 RTP/AVP 0
▫ m=video 51372 RTP/AVP 99
▫ a=rtpmap:99 h263-1998/90000
Connection Info.
Active session time
Session Attribute lines
Media Name and
Transport address
Media Attribute lines
Client-Server flow
Web
Browser
Web
Server
Client Server
HTTP GET
Stream URI
DESCRIBE
OPTIONS
SDP Information
www.dsp-ip.comFast Forward Your Development
Media
Player
Media
Server
SETUP
PLAY
RTP Media Stream
PAUSE
TEARDOWN
RTP Media Stream
SDP Information
RTSP Protocol Parameters
• version
▫ The version of rtsp. (RTSP/1.0)
• URL
[rtsp/rtspu]://host:port/path
www.dsp-ip.comFast Forward Your Development
*port – the actual stream will be delivered in other port
Reliable
protocol
(TCP)
unreliable
protocol
(UDP)
legal domain name or IP
address
port used to control the
stream
the server stream path
RTSP Protocol Parameters (Ctnd.)• Session ID
▫ Generated by the server
▫ Stays constant for the entire session
• SMPTE – Relative timestamp
▫ A relative time from the beginning of the stream.
▫ Nested types: smpte-range, smpte-type, smpte-time.
▫ smpte-25=(starttime)-(endtime)
www.dsp-ip.comFast Forward Your Development
• UTC – Absolute time
▫ Absolute time using GMT.
▫ Nested types: utc-range, utc-time. utc-date
▫ utc-time = (utcdate)T(utctime).(fraction)Z
• NPT - Normal Play Time
▫ Absolute position from the beginning of the presentation.
▫ npt=123.45-125
RTSP Session Details
Initiation
www.dsp-ip.comFast Forward Your Development
Handling
Termination
RTSP - OPTIONS request
Media URL
www.dsp-ip.comFast Forward Your Development
OPTIONS – Request for information about the communication options available by
the Request-URI.
• CSeq – the request id, a response with the same id will be sent from the server.
•Media URL – the URL of the video.
•Client Player – the user agent of the client.
Media URL
Client Player
Request ID
RTSP – OPTIONS response
Response Code
www.dsp-ip.comFast Forward Your Development
• All RTSP response codes are divided into 5 ranges (RFC 2326 7.1.1) :
1xx – Informational, 2xx – Success 3xx – Redirection, 4xx – Client Error, 5xx – Server Error.
CSeq has the same value as the request CSeq field.
• The server response will return the available methods that it supports.
It May contain any arbitrary data the server want to expose.
Available
Options
RTSP – DESCRIBE request
www.dsp-ip.comFast Forward Your Development
DESCRIBE is used to retrieve the description of the media URL and the session.
The description response MUST contain all media and streaming data needed in order to initialize the session.
Fields: Accept - Used to inform the server which description methods the client supports.
Session Description Protocol (SDP) is highly used.
Notice that CSeq field is increased by one.
Description readers
RTSP – DESCRIBE response
The media URL the response is referring to
The description method used
The length of the SDP message
www.dsp-ip.comFast Forward Your Development
Description readers
SDP
The response will always return the details of the media.
SDP details will be next>
RTSP – GET_PARAMETER request
www.dsp-ip.comFast Forward Your Development
GET_PARAMETER is used to retrieve information about the stream.
The request can be initiated from the Client or from the Server.
The request/response message body is left to server/client implementation.
The parameters can be: packets received, jitter, bps or any other relevant information about the
stream.
RTSP – SETUP request
www.dsp-ip.comFast Forward Your Development
SETUP is used to specify the transport details used to stream the media.
The request/response message body is left to server/client implementation.
The parameters can be: packets received, jitter, bps or any other relevant information about the
stream.
Transport protocol Unicast/Multicast RTP/RTSP client
media portTrack ID
www.dsp-ip.comFast Forward Your Development
SETUP response will contain the session ID.
For each track ( audio/video ) a different SETUP request will be made
After the response is received, a PLAY request can be made to start receiving the media stream.
Transport
protocol
Unicast/Multicast
server option
Unicast
destination ip
Last gateway
source ip
The client port
to receive
media data
The server
port to receive
media data
RTSP – PLAY request
www.dsp-ip.comFast Forward Your Development
PLAY request tells the server to start send data through the streaming details defined in the
SETUP process.
PLAY request maybe queued so that a PLAY request arriving while a previous PLAY request is
still active is delayed until the first has been completed.
Normal Play
TIme
RTSP – PAUSE request
www.dsp-ip.comFast Forward Your Development
Stream URL
PAUSE request tells the server to pause the streaming.
When the user will want to start the stream again he’ll send a PLAY request to the same URL.
The request may contain time information to handle when the pause will take effect.
RTSP – TEARDOWN
www.dsp-ip.comFast Forward Your Development
TEARDOWN stops the stream delivery for the URL specified.
Informs the server that the client is disconnecting from it.
The response will include only the response code.
Description readers
RTSP – More Request types• RECORD:
▫ Initiates recording operation given a time information and stream URL.
• REDIRECT:
▫ Server to Client request that informs the client he needs to switch the server he connected to.The request will contain the new server URL.
www.dsp-ip.comFast Forward Your Development
The request will contain the new server URL.
• SET_PARAMETER:
▫ sends a request to change a value of the presentation stream.The response code will contain the answer.
• ANNOUNCE:
▫ Can be initiated both by client/server. Informs the recipient that the SDP table of the object has changed.
Components
•Containers•CODECs
www.dsp-ip.comFast Forward Your Development
•CODECs
File Format
www.dsp-ip.comFast Forward Your Developmentframe
samplesample
frame
sample
mdat
trak
moovAudio track
trak
Video track
Movie (meta-data)
Media Data
sample
Agenda
• Intro to file formats
• Second Generation formats
▫ RIFF: AVI, WAV
• Third Generation Containers
▫ MPEG4 FF
▫ MKV
www.dsp-ip.comFast Forward Your Development
▫ MKV
File Format Segmentation
File Formats
3rd Generation
2nd Generation
1st Generation
www.dsp-ip.comFast Forward Your Development
Generation
XML BasedObject Based
Generation
Media Muxer
Generation
Raw / Proprietary
2nd Generation File Formats
www.dsp-ip.comFast Forward Your Development
2ND Generation Files features
• Multiple media track in the same file
• Identification of codec
▫ Usually by FourCC
• Interleaving
www.dsp-ip.comFast Forward Your Development
2nd Generation File Formats
2nd Generation FF
RIFF ASF MPEG2 FLV
www.dsp-ip.comFast Forward Your Development
RIFF
WAV AVI
ASF
WMA WMV
MPEG2
MP2PS VOB
MP2TS
FLV
AVI FILE FORMAT
www.dsp-ip.comFast Forward Your Development
AVI Overview
• AVI files use the AVI RIFF format (like WAV)
• Introduced by Microsoft on 1992
• File is divided into:
▫ Streams – Audio, Video, Subtitles
▫ Blocks “Chunks” -
www.dsp-ip.comFast Forward Your Development
Blocks / Chunks
• A RIFF File logical unit
• Chunks are identified by four letters (FOUR-CC)
• RIFF file has two mandatory sub-chunks and one optional sub-chunk
• Mandatory Chunks:
▫ hdrl – File headerRIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI '
LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘
www.dsp-ip.comFast Forward Your Development
▫ hdrl – File header
▫ movi - Media Data
• Optional Chunk
▫ idx1 - Index
RIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI ' RIFF ('AVI ' LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘LIST ('hdrl‘
'avih'(<Main AVI Header>)'avih'(<Main AVI Header>)'avih'(<Main AVI Header>)'avih'(<Main AVI Header>)LIST ('strl’ ... ) . . . ) LIST ('strl’ ... ) . . . ) LIST ('strl’ ... ) . . . ) LIST ('strl’ ... ) . . . )
LIST ('movi‘ . . . ) LIST ('movi‘ . . . ) LIST ('movi‘ . . . ) LIST ('movi‘ . . . ) ['idx['idx['idx['idx1111'<AVI Index>]'<AVI Index>]'<AVI Index>]'<AVI Index>]
)
*This order is fixed
AVI main headerRIFF 'AVI ' - Identifies the file as RIFF file.
LIST 'hdrl' - Identifies a chunk containing sub-chunks that define the format of the data.
'avih' - Identifies a chunk containing general information about the file. Includes:▫ dwMicrosecPerFrame - Time between frames
www.dsp-ip.comFast Forward Your Development
▫ dwMicrosecPerFrame - Time between frames
▫ dwMaxBytesPerSec – number of bytes per second the player should handle
▫ dwReserved1 - Reserved
▫ dwFlags - Contains any flags for the file.
Example - headers
chunk ID chunk IDformat
Time between
frames
Data rate
chunk size
flages
Total no. of
Initial frame
streams
Avi file header
www.dsp-ip.comFast Forward Your Development
Total no. of
frames
Frame
width 320
Frame
height
Stream header
Junk chunk
identifier
Size of padding
reserved
Example – data chunks
Audio data chunk
(stream 01)
www.dsp-ip.comFast Forward Your Development
(stream 01)
video data chunk
(stream 00)
AVI Summary
• Advantages
▫ Includes both audio and video
▫ Index-able
• Disadvantage
▫ Not suited for progressive DW
▫ Very rigid format
www.dsp-ip.comFast Forward Your Development
▫ Very rigid format
▫ Insufficient support for: seeking, metadata multi-reference frames
3RD GENERATION FILE FORMATS
www.dsp-ip.comFast Forward Your Development
Why “Fix it”?
• 2nd Generation Formats are missing:
• Metadata
▫ Separate from Media
▫ Info on angle, language, Synchronization
▫ Versioning
• Better Streaming Support
www.dsp-ip.comFast Forward Your Development
• Better Streaming Support
▫ Reduce CPU per stream
▫ Better seeking support
• Better parsing
▫ XML
▫ Atom Based
Main Attributes
• File format is not just a Video / Audio multiplexer
• Separation between
▫ Media – Audio, Video, Images, Subtitles
▫ Metadata – Indexing, frame length, Tags
www.dsp-ip.comFast Forward Your Development
3rd Generation File Formats
3rd Generation
XML Based Object Based
www.dsp-ip.comFast Forward Your Development
Matruska (MKV) MOV MPEG4 FF
3GPP FFFragmented MPEG4 FF
MPEG4 FILE FORMAT
www.dsp-ip.comFast Forward Your Development
MP4 File Format
• File Structuring Concepts
▫ Separate the media data from descriptive (meta) data.
▫ Support the use of multiple files.
▫ Support for hint tracks:
� support of real time streaming over any protocol
www.dsp-ip.comFast Forward Your Development
� support of real time streaming over any protocol
Separate Metadata and Media
• Key meta-information is compact▫ The type of media present▫ Time-scales▫ Timing▫ Synchronization points etc.
• Enables
www.dsp-ip.comFast Forward Your Development
• Enables▫ Random access▫ Inspection, composition, editing etc.▫ Simplified update
Multiple file support
• Use URLs to ‘point to’ media
▫ Distinct from URLs in MPEG-4 Systems
• URLs use file-access service
▫ e.g. file://, http://, ftp:// etc.
• Permits assembly of composition without requiring data-copy
www.dsp-ip.comFast Forward Your Development
requiring data-copy
• Referenced files contain only media
▫ Meta-data all in ‘main’ file
Logical File Structure
• Presentation (‘movie’) contains…
• Tracks which contain…
• Samples
www.dsp-ip.comFast Forward Your Development
Physical Structure—File
• Succession of objects (atoms, boxes)
• Exactly one Meta-data object
• Zero or more media data object(s)
• Free space etc.
www.dsp-ip.comFast Forward Your Development
Example Layout
moovAudio track
trak
Video track
Movie (meta-data)
www.dsp-ip.comFast Forward Your Development
frame
samplesample
frame
sample
mdat
trak
Media Data
sample
Meta-data tables
• Sample Timing• Sample Size and position• Synchronization (random access) points, priority
etc.• Temporal/physical order de-coupled
▫ May be aligned for optimization
www.dsp-ip.comFast Forward Your Development
▫ May be aligned for optimization▫ Permits composition, editing, re-use etc. without
re-write
• Tables are compacted
Multi-protocol Streaming support
• Two kinds of track
• Media (Elementary Stream) Tracks
▫ Sample is Access Unit
• Protocol ‘hint’ tracks
▫ Sample tells server how to build protocol transmission unit (packet, protocol data unit etc.)
www.dsp-ip.comFast Forward Your Development
transmission unit (packet, protocol data unit etc.)
Track types
• Visual—’description’ formats
▫ MPEG4
▫ JPEG2000
• Audio—’description’ formats
▫ MPEG4 compressed tracks
▫ ‘Raw’ (DV) audio
www.dsp-ip.comFast Forward Your Development
▫ ‘Raw’ (DV) audio
• Other MPEG-4 tracks
• Hint Tracks (streaming)
Track Structure
• Sample pointers (time, position)
• Sample description(s)
• Track references
▫ Dependencies, hint-media links
• Edit lists
▫ Re-use, time-shifting, ‘silent’ intervals etc.
www.dsp-ip.comFast Forward Your Development
▫ Re-use, time-shifting, ‘silent’ intervals etc.
Hint Tracks
• May include media (ES) data by ref.
• Only ‘extra’ protocol headers etc. added to hint tracks — compact
▫ Make SL, RTP headers as needed
• May multiplex data from several tracks
• Packetization/fragmentation/multiplex through
www.dsp-ip.comFast Forward Your Development
• Packetization/fragmentation/multiplex through hint structures
• Timing is derived from media timing
Hint track structure
moov
Hint track
trak
Video track
Movie (meta-data)
www.dsp-ip.comFast Forward Your Development
frame
sample
header
pointer
hint
sample
frame
sample
header
pointer
hint
sample
mdat
trak
Hint track
Sample Data
Extensibility
• Other media types.
▫ Non-sc29 sample descriptions (e.G. Other video).
▫ Non-sc29 track types (e.G. Laboratory instrument trace).
• Copyright notice (file or track level) etc.
• General object extensions (GUIDs).
www.dsp-ip.comFast Forward Your Development
• General object extensions (GUIDs).
Advantages
• Compatibility
▫ files can be played by other companies players.
� Real Player with envivo plug-in.
� Windows media player etc.
▫ Files can be streamed by other companies streaming server
www.dsp-ip.comFast Forward Your Development
streaming server
� Darwin Streaming Server.
� Quick Time Streaming Server.
Single File-Multiple data types
• No need to do an export process for files, one file type is used for storage of video, audio, events, continues telemetry data from sensors and JPEG images in one file.
www.dsp-ip.comFast Forward Your Development
AudioAudio
VideoVideo
JPEGJPEG11
Sensor Continues dataSensor Continues data
eventsevents
Métadonnées
Métadonnées
Single file playback
• All video track of a site could be stored in one file. In order to view many cameras in a synchronized manner the MPEG-4 file format can hold all the views of multiple cameras in one file.
www.dsp-ip.comFast Forward Your Development
AudioAudio
Video cam 1Video cam 1
Video cam 2Video cam 2
Video cam …….Video cam …….
Video cam NVideo cam N
Métadonnées
Métadonnées
Skimming
• Skimming – shortening a long movie to its interesting points, much like creating a “promo”. For example skimming a surveillance movie of two hours to 2 minutes where there is movement and people are entering the building.
• MPEG-4 FF enables the creation of skims within the file through the use of edit-list (part of the
www.dsp-ip.comFast Forward Your Development
the file through the use of edit-list (part of the standard) without overhead.
MKV FILE FORMAT
www.dsp-ip.comFast Forward Your Development
XML Based File-Format
MKV - File Format
• Container file format for videos, audio tracks, pictures and subtitles all in one file.
• Announced on Dec. 2002 by Steve Lhomme.
• Based on Binary XML format called EBML (Extensible Binary Meta Language)
www.dsp-ip.comFast Forward Your Development
(Extensible Binary Meta Language)
• Complete Open-Standard format. (Free for personal use).
• Source is licensed under GNU L-GPL.
MKV - Specifications
• Can contain chapter entries of video streams
• Allows fast in-file seeking.
• Metadata tags are fully supported.
• Multiple streams container in a single file.
www.dsp-ip.comFast Forward Your Development
• Multiple streams container in a single file.
• Modular – Can be expanded to company special needs.
• Can be streamed over HTTP, FTP, etc.
MKV Support software & hardware
• Players:▫ All Player, BS.Player, DivX Player, Gstreamer-Based
players, VLC media, xine, Zoom Player, Mplayer, Media Player Classic, ShowTime, Media Player Classic and many more…
• Media Centers:
www.dsp-ip.comFast Forward Your Development
• Media Centers:▫ Boxee, DivX connected, Media Portal, PS3 Media
Server, Moovida, XBMC etc.
• Blu-Ray Players:▫ Samsung, LG and Oppo.
• Mobile Players:
▫ Archos 5 android device, Cowon A3 and O2.
MKV - EBML in details
• A binary format for representing data in XML-like format.
• Using specific XML tags to define stream properties and data.
• MKV conforms to the rules of EBML by defining a set of tags.
www.dsp-ip.comFast Forward Your Development
a set of tags.
▫ Segment , Info, Seek, Block, Slices etc.
• Uses 3 Lacing mechanisms for shortening small data block (usually frames).
▫ Uses: Xiph, EBML or fixed-sized lacing.
MKV – Simple representationDescriptionType
Version info, EBML type ( matroska in our case ).Header
Optional, Allows fast seeking of other level 1 elements in file.Meta Seek Information
File information - title, unique file ID, part number, next file ID.
Segment Information
Basic information about the track – resolution, sample rate, codec info.
Track
www.dsp-ip.comFast Forward Your Development
codec info.
Predefines seek point in media.Chapters
Video and audio frames for each trackClusters
Stores cue points for each track. Allows fast in track seeking. Cueing Data
Any other file relates to this. ( subtitles, Album covers, etc… )Attachment
Tags that relates to the file and for each track (similar to MP3 ID3 tags).
Tagging
MKV – Streaming
• Matroska supports two types of streaming.
• File Access
▫ Used for reading file locally or from remote web server.
▫ Prone to reading and seeking errors.
▫ Causes buffering issues on slow servers.
www.dsp-ip.comFast Forward Your Development
▫ Causes buffering issues on slow servers.
• Live Streaming
▫ Usually over HTTP or other TCP based protocol.
▫ Special streaming structure – no Meta seek, Cues, Chapters or attachments are allowed.
File Format Summary - Trends
• Metadata is important
▫ Simple metadata or XML
▫ Separated from media
• Forward compatibility
▫ Not crash if don’t understand a data entry
• Progressive download oriented
www.dsp-ip.comFast Forward Your Development
• Progressive download oriented
• Multi-bitrate oriented
• Fragmentation -> Lower granularity
▫ Self contained File fragments
• CDN-ability
Video Codecs
www.dsp-ip.comFast Forward Your Developmentframe
samplesample
frame
sample
mdat
trak
moovAudio track
trak
Video track
Movie (meta-data)
Media Data
sample
Why Advance ? MPEG2 Works….• Coding efficiency
• Packetization
• Robustness
• Scalable profiles
• Internet requires Interaction
▫ Scalable & On demand
www.dsp-ip.comFast Forward Your Development
▫ Scalable & On demand
▫ Fast-Forward / Fast Rewind / Random Access
▫ Stream switching
• Multi
▫ Bitrate
▫ resolution /screen
Coding efficiency Motivation
www.dsp-ip.comFast Forward Your Development
Sorenson Spark video Codec
• H263 variant
• Low footprint (code size) ~100K
• Good performance for 2002
• Quality SPARK vs Optimal MPEG (H263+)
▫ 20-30% less efficient
www.dsp-ip.comFast Forward Your Development
• SPARK Quality RT vs Offline
▫ RT has Considerably lower quality due to processing power and RT (delay) constraints
Sorenson Spark - 2
• Does Not support:▫ Arithmetic coding▫ Advance prediction▫ B-frames
• Features▫ De-blocking filter mode
www.dsp-ip.comFast Forward Your Development
▫ De-blocking filter mode▫ UMV - Unrestricted Motion Vector mode ▫ Arbitrary frame dimensions▫ Supported by FFMPEG ▫ D – Frames
D-Frames
• D (Disposable) frames
▫ One way prediction
▫ Provides flexible bit-rate: I-D-P-D-P-D-P
▫ D-frames used only when feeding a flash communication server
www.dsp-ip.comFast Forward Your Development
On2 TrueMotion VP6
• Features
▫ Compressed I-frames (Intra-compression makes use of spatial predictors)
▫ unidirectional predicted frames (P-frames)
▫ Multiple reference P-frames
▫ 8x8 iDCT-class transform (4x4 in VP7)
www.dsp-ip.comFast Forward Your Development
▫ 8x8 iDCT-class transform (4x4 in VP7)
▫ improved quantization strategy (preserves image details)
▫ Advance Entropy Coding
VP6 Features
• Entropy Coding ▫ various techniques are used based on complexity
and frame size including:� VLC � Context modeled binary coding (like H264 CABAC)
• Bit Rate Control
www.dsp-ip.comFast Forward Your Development
▫ To reach the requested data rate, VP6 adjusts� Quantization levels� Encoded frame dimensions� Entropy Coding� Drop frames
VP6 motion prediction
• Motion Vectors▫ One vector per MacroBlock (16x16)
or ▫ 4 vectors for each block (8x8)
• Quarter pel motion compensation support• Unrestricted motion compensation support
www.dsp-ip.comFast Forward Your Development
• Unrestricted motion compensation support• Two reference frames:
▫ The previous frameor
▫ Previously bookmarked frame
VP6 vs H264
• VP6 is much simpler than H.264
▫ Requires less CPU resourced for decoding & encoding
▫ Code size is considerably smaller.
• Simpler means less efficient? NO! Techniques
www.dsp-ip.comFast Forward Your Development
• Simpler means less efficient? NO! Techniques used:
▫ Mix of adaptive sub-pixel motion estimation
▫ Better prediction of low-order frequency coefficients
▫ Improved quantization strategy
▫ de-blocking and de-ringing filters
▫ Enhanced context based entropy coding,
720p High Profile H.264 vs VP7
45
45.5
46
46.5
47
Alexander Trailer
Tips for reading this kind of a graph (a PSNR graph):
Draw a line straight
Draw a line straight across until you intersect the lower line ( in this case x264. i.e. keep the quality/ psnr constant )
Draw a line straight
What this means: On this clip VP7 at 2750 kbps has the same quality / PSNR as x264 high profile at 3620 kbps. i.e. you’d need 30% higher
PSNR Graphs are used for comparative analysis of compression quality. Each line represents the encode quality on a given clip at multiple datarates. The highest line represents the codec with the best quality. In this case VP7 clearly is better than x264.
Th
is a
xis
re
pre
sen
ts q
ua
lity
. H
igh
er
is b
ett
er
Pick any point on the top line, in this case it’s VP7.
www.dsp-ip.comFast Forward Your Development
42.5
43
43.5
44
44.5
45
1400 1900 2400 2900 3400 3900 4400
PSNR
Kbps
Vp7
x264
2750 kbps 3620 kbps
Draw a line straight down from that point to the datarate axis. The crossing point tells you the datarate at that point.
Draw a line straight down from that point to the datarate axis. The crossing point tells you the datarate at that point.
at 3620 kbps. i.e. you’d need 30% higher datarate to get the same quality out of x264 that you got from vP7.
Th
is a
xis
re
pre
sen
ts q
ua
lity
. H
igh
er
is b
ett
er
This axis represents datarate in kilobits per second.
VP6 vs. H264• There is a difference between the codec
technology and a codec implementation.
www.dsp-ip.comFast Forward Your Development
On2 VP7
• Not open source
• Non-standard royalties model
• Better video quality than H264
• Used by:
▫ Part of EVD – China standard for HD-DVD
▫ Skype Beta (V 2.0)
www.dsp-ip.comFast Forward Your Development
▫ Skype Beta (V 2.0)
▫ Flash Player
Windows Media
• Windows media is a format used by Microsoft for encoding and distributing Audio and Video.
• Windows Media has two types of media:▫ Windows Media Audio (WMA) ▫ Windows Media video (WMV)
• Windows Media Video
www.dsp-ip.comFast Forward Your Development
• Windows Media Video ▫ A modified version of MPEG 4▫ Codec version has initially started from version 7
for windows media player 7 and then evolved to version 8-10
Windows Media 9 - VC1 Format
• Microsoft has submitted Version 9 codec to the Society of Motion Picture and Television Engineers (SMPTE), for approval as an international standard. SMPTE is reviewing the submission under the draft-name "VC-1")
• This codec is also used to distribute high definition video on standard DVDs in a format Microsoft has branded as
www.dsp-ip.comFast Forward Your Development
on standard DVDs in a format Microsoft has branded as WMV HD. This WMV HD content can be played back on computers or compatible DVD players.
• The Trial version of standards were published by SMPTE in September 2005
• WMV9 was approved by SMPTE, April 2006
H.264
www.dsp-ip.comFast Forward Your Development
H.264 Terminology
• The following terms are used interchangeably:▫ H.26L
▫ “JVT CODEC”
▫ The “AVC” or Advanced Video CODE
• Proper Terminology going forward:
www.dsp-ip.comFast Forward Your Development
▫ MPEG-4 Part 10 (Official MPEG Term)
� ISO/IEC 14496-10 AVC
▫ H.264 (Official ITU Term)
H264 Standard ideas
• “Blocks” size fixed ->Variable
▫ Slice
▫ Block
• Block Size order/scanning –> different orders
▫ Zig-zag, Flexible Macroblock Order
• Additional spatial prediction - >Intra prediction
www.dsp-ip.comFast Forward Your Development
• Additional spatial prediction - >Intra prediction
• Inter prediction 1 frame only ->Multiple frames
▫ P and B picture
▫ Multiple reference frame
H264 Standard Ideas
▫ Pixel interpolation
▫ Motion vectors
• In-loop Deblocking filter
• Improved Entropy coding
www.dsp-ip.comFast Forward Your Development
• Improved Entropy coding
New Features of H.264 - summarized
• SP, SI - Additional picture types
• NAL (Network Abstraction Layer)
• CABAC - Additional entropy coding mode
• ¼ & 1/8-pixel motion vector precision
• In-loop de-blocking filter
• B-frame prediction weighting
www.dsp-ip.comFast Forward Your Development
• B-frame prediction weighting
• 4×4 integer transform
• Multi-mode intra-prediction
• NAL - Coding and transport layers separation
• FMO - Flexible MacroBlock ordering.
Block diagram
www.dsp-ip.comFast Forward Your Development
Profiles and Levels
• Profiles: Baseline, Main, and X▫ Baseline: Progressive, Videoconferencing & Wireless
▫ Main: esp. Broadcast
▫ Extended: Mobile network
• Wireless <> Mobile
www.dsp-ip.comFast Forward Your Development
www.dsp-ip.comFast Forward Your Development
Baseline Profile
• Baseline profile is the minimum
implementation▫ No CABAC, 1/8 MC, B-frame, SP-slices
• 15 levels▫ Resolution, capability, bit rate, buffer, reference #
▫ Built to match popular international production and
www.dsp-ip.comFast Forward Your Development
▫ Built to match popular international production and
emission formats
▫ From QCIF to D-Cinema
• Progressive (not interlaced)• I and P slices types
Baseline Profile
• 1/4-sample Inter prediction • Deblocking filter, Redundant slices • VLC-based entropy coding (no CABAC)• 4:2:0 chroma format • Flexible Macroblock Ordering (FMO) • Arbitrary Slice Order (ASO)
www.dsp-ip.comFast Forward Your Development
• Arbitrary Slice Order (ASO)▫ Decoder process slices in an arbitrary order as they
arrive to the decoder.▫ The decoder dose not have a wait for all slices to
be properly arranged before it starts processing them.▫ Reduces the processing delay at the decoder.
Baseline Profile
• FMO: Flexible Macroblock Ordering ▫ With FMO, macroblocks are coded according to a
macroblock allocation map that groups, within a given slice.
▫ Macroblocks from spatially different locationsin the frame.
www.dsp-ip.comFast Forward Your Development
in the frame.▫ Enhances error resilience
• Redundant slices:▫ allow the transmission of duplicate slices.
H.264 Profiles & Levels - Main
• All Baseline features Plus� Interlace � B slice types (bi directional reference )� CABAC� Weighted prediction
• All features included in the Baseline profile
www.dsp-ip.comFast Forward Your Development
• All features included in the Baseline profile except:� Arbitrary Slice Order (ASO)� Flexible Macroblock Order (FMO)� Redundant Slices
Main Profile
• CABAC
• Good performance (bit rate reduction) by
▫ Selecting models by context
▫ Adapting estimates by local statistics
▫ Arithmetic coding reduces computational complexity
• Improve computational complexity more than
www.dsp-ip.comFast Forward Your Development
• Improve computational complexity more than10%~20% of the total decoder execution time at medium bitrate
• Average bit-rate saving over CAVLC 10-15%
Extended Profile
• All Baseline features plus
▫ Interlace
▫ B slice types
▫ Weighted prediction
www.dsp-ip.comFast Forward Your Development
Variable block size
Slices�A picture split into 1 or several slices
�Slices are a sequence of macroblocks
Macroblock�Contains 16x16 luminance samples and
www.dsp-ip.comFast Forward Your Development
Slice 0
Slice 1
Slice 2
�Contains 16x16 luminance samples and
two 8x8 chrominance samples
�Macroblocks within a slices depend on
each others
�Macroblocks can be further partitioned
Basic Marcoblock Coding Structure
Entropy
Coding
Scaling & Inv.
Transform
Control
Data
Quant.
Transf. coeffs
Coder
Control
Decoder
Transform/
Scal./Quant.-
Input
Video
Signal
Split into
Macroblocks
16x16 pixels
www.dsp-ip.comFast Forward Your Development
Motion-
Compensation
Motion
Data
Intra/Inter
Motion
Estimation
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
Motion Compensation
Entropy
Scaling & Inv.
Transform
Control
Data
Quant.
Transf. coeffs
Coder
Control
Decoder
Transform/
Scal./Quant.-
Input
Video
Signal
Split into
Macroblocks
16x16 pixels
www.dsp-ip.comFast Forward Your Development
Entropy
Coding
Motion-
Compensation
Motion
Data
Intra/Inter
Motion
Estimation
16x16 pixels
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
Various block sizes and shapes
8x8
0
4x8
0 10 1
2 3
4x48x4
1
08x8
Types
0
16x16
0 1
8x16
MBTypes
8x8
0 1
2 3
16x8
1
0
Variable block size
0
0
1
0 1
2 30 1
Mode 1 Mode 2 Mode 3 Mode 4
�Block sizes of
16x8, 8x16, 8x8,
8x4 , 4X8 and
4X4 are
available.
www.dsp-ip.comFast Forward Your DevelopmentYossi Cohen
DSP-IP
0 1 2 3
4 5 6 7
0 1 2 3
4 5 6 7
8 9 1
0
1
1
1
2
1
3
1
4
1
5
0 1
2 3
4 5
6 7
Mode 11 16x16 block
Mode 22 16x8 blocks
Mode 32 8x16 blocks
Mode 44 8x8 blocks
Mode 58 8x4 blocks
Mode 68 4x8 blocks
Mode 716 4x4 blocks
�Using seven different
block sizes can translate
into bit rate savings of
more than 15% as
compared to using only a
16x16 block size.
How to select the partition size?
www.dsp-ip.comFast Forward Your Development
The partition size that minimizes the coded
residual and motion vectors
Inter prediction modesmotion vectors
• MVs for neighboring partitions are often highly correlated.
• So we encode MVDs instead of MVs
• MVD = predicted MV – MVp
• ¼ pixel accurate motion compensation
www.dsp-ip.comFast Forward Your Development
Multiple Reference Frames
Entropy
Coding
Scaling & Inv.
Transform
Control
Data
Quant.
Transf. coeffs
Coder
Control
Decoder
Transform/
Scal./Quant.-
Input
Video
Signal
Split into
Macroblocks
16x16 pixels
www.dsp-ip.comFast Forward Your Development
Coding
Motion-
Compensation
Motion
Data
Intra/Inter
Motion
Estimation
Intra-frame
Prediction
De-blocking
Filter
Output
Video
Signal
Multiple Reference Frames for
Motion Compensation
Multiple Reference Frames
www.dsp-ip.comFast Forward Your Development
Intra prediction modes4x4 luminance prediction modes
0(vertical) 1(Horizontal) 2(DC) 3(Diagonal 4(Diagonal
Down/left) Down/right)
5(Vertical-right) 6(Horizontal-down) 7(Vertical-left) 8(Horizontal-top)
www.dsp-ip.comFast Forward Your Development
Mode 2 (DC)
Predict all pixels from
(A+B+C+D+I+J+K+L+4)/8 or
(A+B+C+D+2)/4 or (I+J+K+L+2)/4
Intra prediction modes4x4 luminance prediction modes
www.dsp-ip.comFast Forward Your Development
Intra prediction modesIntra 16x16 luminance and 8x8 chrominance prediction modes
www.dsp-ip.comFast Forward Your Development
Inter prediction modeschrominance Pixel interpolation
dx
dy
S-dx
Quarter chrominance Pixels are
interpolated by tacking weighted
averages of distance from the new
pixel to four surrounding original
A B
www.dsp-ip.comFast Forward Your Development
S-dx
S-dy
V=(s-dx)(s-dy)A+dx(s-dy)B+(s-dx)dyC+dxdyD+s2/2
S2
pixel to four surrounding original
pixels.C D
In-loop De-blocking Filter
• Highly compressed decoded inter picture
• Significantly reduces prediction residuals
www.dsp-ip.comFast Forward Your Development
Without filter with H.264/AVC De-blocking
Entropy coding
www.dsp-ip.comFast Forward Your Development
Entropy coding
• Entropy coding methods:• CABAC - Discussed• UVLC
▫ H.264 offers a single Universal VLC (UVLC) table for all symbol
• CAVLC
www.dsp-ip.comFast Forward Your Development
• CAVLC▫ CAVLC (Context-based variable Length Coding
)
▫ Probability distribution is static▫ Code words must have integer number of bits (Low
coding efficiency for highly peaked pdfs)
CABAC - Binarization
www.dsp-ip.comFast Forward Your Development
Context Probability Coding
update probability estimation
CABAC: Technical OverviewCABAC: Technical Overview
www.dsp-ip.comFast Forward Your Development
Context
modelingBinarization Probability
estimation
Coding
engine
Adaptive binary arithmetic coder
Chooses a model
conditioned on
past observations
Maps non-binary
symbols to a
binary sequence
Uses the provided model
for the actual encoding
and updates the model
H.264 NAL & RTP
www.dsp-ip.comFast Forward Your Development
H264 Layer Structure
Video Coding Layer
Data PartitioningContr
ol D
ata
Macroblock
www.dsp-ip.comFast Forward Your Development
Data Partitioning
Network Abstraction Layer
H.320 H.324 H.323/IP MPEG2
Slice/Partition
H264 & NAL
• Motivation
▫ Many delivery methods are based on packet based networks
▫ Its better to do the packetization inside the encoder where all coding information than in other separate modules
• Architecture: NAL units as the transport entity
www.dsp-ip.comFast Forward Your Development
• Architecture: NAL units as the transport entity
▫ NAL units may be mapped into a bit stream▫ NAL units are self-contained-independently
decodable▫ The decoding process assumes NAL units are in
decoding order
Network Abstraction Layer (NAL)
• H.264 encoder is composed of two layers:
• VCL - Video Coding Layer – unit which translates the video information into a stream of bits
• NAL - Network Abstraction Layer (NAL). Which maps and packetize the VCL biststream into unitsprior to transmission or storage
• Each NAL unit contains:
www.dsp-ip.comFast Forward Your Development
• Each NAL unit contains:
▫ Payload – RBSP (Raw Byte Sequence Payload), which contains set of data corresponding to coded video data or header information
▫ NAL Unit Header – which contains the NAL header
NAL
• The coded video sequence is represented by a sequence of NAL units that can be transmitted over a packet-based network or a bitstream transmission link or stored in a file
• There are two NAL Types
▫ VCL Units – NAL units which represents encoded
www.dsp-ip.comFast Forward Your Development
▫ VCL Units – NAL units which represents encoded video data
▫ Non-VCL Units – Parameters sets
NAL Unit Header
NAL unit header- 1 byte consisting of:• Forbidden_bit(1 bit) may be used to signal that a NAL unit is
corrupt
• nal_storage_idc(2 bit): signal relative importance, and if the picture is stored in the reference picture buffer.
NAL unit header NAL unit payload
www.dsp-ip.comFast Forward Your Development
picture is stored in the reference picture buffer.
• nal_unit_type(5 bit): signal 1 of 10 different NAL unit types:▫ Coded slice (regular VCL data)
▫ Coded data partition. (DPA,DPB,DPC)
▫ Instantaneous decoder refresh (IDR)
▫ Supplemental enhancement information (SEI)
▫ Sequence and picture parameter sets (SPS,PPS)
▫ Picture delimiter (PD) and filler data (FD)
RBSP (NAL Payload) Types• Parameter Set – global parameters for a sequence
includes: Resolution, video format, macroblock, allocation map
• Supplemental Enhancement Information
• Picture Delimiter – boundary between video pictures
• Coded slice – header and data for a slice, this unit contains actual coded video data
www.dsp-ip.comFast Forward Your Development
contains actual coded video data
• Data Partition A,B or C – Data Partitioned slice layer data (A – header data for all MBs in the slice, B – intra coded data, C – inter coded data)
• End of sequence
• End of stream
• Filler data
RTP payload format for H.264
• Based on IETF RFC 3984, February 2005
• Described how to use H.264 NAL inside RTP with proper packetization Employs the native NAL (Network Abstraction Layer) interface, based on NAL units (NALUs)
• NALU – byte string of variable length that contains syntax elements of a certain class
• NALU header – defines the information within the NAL
www.dsp-ip.comFast Forward Your Development
• NALU header – defines the information within the NAL Unit (Corrupted, Type etc)
• There are two basic methods for RTP packetization of NAL units:
▫ Non fragmented NAL units
▫ Fragmented NAL units
RTP Payload for H.264 NAL
• The most common method is to configure the encoder to output one NAL unit for each RTP packet. Each NAL unit is ~1.4KB
www.dsp-ip.comFast Forward Your Development
• Fragment a large NAL unit (Frame) into many RTP Packets. The difference is in the RTP Header information
RTP and H.264
• RTP Packetization of NAL allows both aggregation of many NAL units into one RTP Packet and fragmentation of one NAL units into many RTP packets
www.dsp-ip.comFast Forward Your Development
Comparison
www.dsp-ip.comFast Forward Your Development
Summary• New key features are:
▫ Enhanced motion compensation
▫ Small blocks for transform coding
▫ Improved de-blocking filter
▫ Enhanced entropy coding
• Substantial bit-rate savings (up to 50%) relative to
other standards for the same quality
www.dsp-ip.comFast Forward Your Development
other standards for the same quality
• The complexity of the encoder triples that of the
prior ones
• The complexity of the decoder doubles that of the
prior ones
Google VP8
www.dsp-ip.comFast Forward Your Development
Before we start
• VP8 goal is NOT to delivery the best video quality in any given bitrate
• VP8 was designed as a mobile video decoder and should be examined in this context:
▫ VP8 vs H.264 base profile
www.dsp-ip.comFast Forward Your Development
Google VP8
• Last month, in Google IO (its developer confrence), Google released VP8 as open source
• VP8 is a light weight video codec developed by On2.
• VP8 provide quality which is the same/higher than H.264 base profile
www.dsp-ip.comFast Forward Your Development
than H.264 base profile
• VP8 memory requirements are lower than H.264 base profile
• After optimization, VP8 might have better MIPS performance than H.264 base profile
Genealogy • VP8 is part of a well know codec family
• VP3 was released to open source to become XIPH Theora
• VP6 is used in Flash video
• VP7 is used in Skype
Sorenson
Spark
www.dsp-ip.comFast Forward Your Development
• Motivation:
▫ “No Royalties” CODEC
VP8
VP7 VP6
VP3Theora
ADAPTATION – WHO USE IT?
Software
Hardware
www.dsp-ip.comFast Forward Your Development
Hardware
Platform & Publishers
Software Adaptation
• Android, Anystream, Collabora
• Corecodec, Firefox, Adobe Flash
• Google Chrome, iLinc,
• Inlet, Opera, ooVoo
• Skype, Sorenson Media
• Theora.org, Telestream, Wildform.
www.dsp-ip.comFast Forward Your Development
• Theora.org, Telestream, Wildform.
Hardware adaptation
• AMD, ARM, Broadcom
• Digital Rapids, Freescale
• Harmonic ,Logitech, ViewCast
• Imagination Technologies, Marvell
• NVIDIA, Qualcomm, Texas Instruments
• VeriSilicon, MIPS
www.dsp-ip.comFast Forward Your Development
• VeriSilicon, MIPS
Platforms and Publishers
• Brightcove
• Encoding.com
• HD Cloud
• Kaltura
• Ooyala
• YouTube
www.dsp-ip.comFast Forward Your Development
• YouTube
• Zencoder
VP8 MAIN FEATURES
www.dsp-ip.comFast Forward Your Development
(According to On2/Google)
Adaptive Loop Filter
• Improved Loop filter provides better quality & performance in comparison to H.264
• ALF is a feature of H.265
www.dsp-ip.comFast Forward Your Development
Source: On2
Golden Frames
• Golden frames enables better decoding of background which is used for prediction in later frames
• Could be used as resync-point:
▫ Golden frame can reference an I frame
• Could be hidden (not for display)
www.dsp-ip.comFast Forward Your Development
• Could be hidden (not for display)
Source: On2
Decoding efficiency
• CABAC is an H.264 feature which improves coding efficiency but consumes many CPU cycles
• VP8 has better entropy coding than H.264, this leads to relatively lower CPU consumption under the same conditions
• Decoding efficiency is
www.dsp-ip.comFast Forward Your Development
• Decoding efficiency is important for smooth operation and long battery life in netbooks and mobile devices
Source: On2
Resolution up-scaling & downscaling
• Supported by the decoder
• Encoder could decide dynamically (RT applications) to lower resolution in case of low bit rate and let the decoder scale.
• Remove decision from the application
• No need for an I frame
www.dsp-ip.comFast Forward Your Development
• No need for an I frame
VP8 BASICS
Definitions
Bitstream structure
www.dsp-ip.comFast Forward Your Development
Bitstream structure
Frame structure
Definitions
• Frame – same as H.264
• Segment – Parallel to slice in H.264. MB in the same segment will use the settings such as:
▫ Entropy encoder/decoder context
▫ De-blocking filter settings
• Partition – block of byte aligned compressed
www.dsp-ip.comFast Forward Your Development
• Partition – block of byte aligned compressed video bits.
Definitions
• Block – 8x8 matrix of pixels
• Macro-block –processing unit, contains a 16x16 Y pixels, and 2 8x8 matrix of U and V:
▫ 4* 8x8Y block
▫ 1* 8x8U block
▫ 1* 8x8V block
www.dsp-ip.comFast Forward Your Development
▫ 1* 8x8V block
• Sub-block – 4x4 matrix of pixels. All DCT / WHT operations are done on sub-blocks.
Frame Types
• I Frame
• P Frame
• No B Frames due to patents / delays but we have “Future Alt-Ref” frame. What is the diff?
• Prediction
▫ Previous frame
www.dsp-ip.comFast Forward Your Development
▫ Previous frame
▫ Golden Frame
▫ Alt-ref frame
Frame Structure
• Include three sections:
• Frame Header
• Partition I
• Partition II
www.dsp-ip.comFast Forward Your Development
Frame
Header Partition I Partition II
partitions
Frame Header
• Byte aligned uncompressed information
• Frame type - 1-bit frame type
▫ 0 for key frames, 1 for inter-frame.
• Level - A 3-bit version number
▫ 0 - 3 are defined as four different profiles with different decoding complexity; other values for future use
www.dsp-ip.comFast Forward Your Development
• show_frame - A 1-bit show_frame flag
▫ 0 – current frame not for display
▫ 1 - current frame is for display
• Length - A 19-bit field containing the size of the first data partition in bytes.
Partition I
• Header information for the entire frame
• Per-macroblock information specifying how each macroblock is predicted.
• This information is presented in raster-scan order
www.dsp-ip.comFast Forward Your Development
Partition II
• Texture information - DCT/WHT quantized coefficients
• Optionally each macroblock row could be mapped to a separate partition.
• Partition II might be divided to several partitions for parallel processing
www.dsp-ip.comFast Forward Your Development
partitions for parallel processing
Frame
Header Partition I
Texture Data
Partition IIA Partition IIB Partition IIn
Decoder
• Holds 4 frames:
▫ Current remonstrated frame
▫ Previous frame
▫ Previous “Golden Frame”
▫ Previous Alt-ref frame
• Frame dimension can change in every frame
www.dsp-ip.comFast Forward Your Development
• Frame dimension can change in every frame
VP8 block diagram
Entropy
Coding
Scaling & Inv.
Transform
Control
Data
Quant.
Transf. coeffs
Coder
Control
Decoder
Transform/
Scal./Quant.-
Input
Video
Signal
Split into
Macroblocks
www.dsp-ip.comFast Forward Your Development
Motion-
Compensation
Motion
Data
Intra/Inter
Motion
Estimation
Intra-frame
Prediction
Dynamic
De-blocking
Output
Video
VP8 BLOCK CODING
www.dsp-ip.comFast Forward Your Development
VP8 Macroblock coding
Divide to
16x16
Macroblock
Divide to
8x8
blocks
Process as
4x4
sub blocks 4x4
WHT
4x4
DCT
Each Macroblock is divided into 25 sub-blocks
DC/AC Coeff
www.dsp-ip.comFast Forward Your Development
Each Macroblock is divided into 25 sub-blocks•16Y sub-blocks•4 U sub-blocks, •4 V sub-blocks•1 Y2 DC values sub-block (WHT)aka Hierarchical transform
VP8 Transforms
• Very inefficient raw implementation of the transform – uses 16bit multipliers
• Uses exact values of pixels
▫ +Memory
▫ +Accuracy and no drift
www.dsp-ip.comFast Forward Your Development
static const int cospi8sqrt2minus1 = 20091; //sqrt(2) * cos(pi/8)
static const int sinpi8sqrt2 = 35468; //sqrt(2) * sin (pi/8)
temp1 = (ip[4] * sinpi8sqrt2 + rounding) >> 16;
iDCT
• Like in H.264 or VP6 the 2D transform is done by two separable 1D transforms (LLM method) there are two methods of implementation:
1. Transpose, vertical T, Transpose, vertical T.
2. Vertical T., transpose, vertical T, transpose.
• Duo to SIMD command its better to do it as 1 and
www.dsp-ip.comFast Forward Your Development
• Duo to SIMD command its better to do it as 1 and eliminate the first transpose like H.264 does
• VP8 does it as 2 (Why?!) wasted 1-2% CPU
Quantization
• There are 6 quantizer types each has its own levels
• The quantizer depends on (multiplication of)
▫ Plane: Y,U, V
▫ Coefficient AC, DC
• Quantizer level is indicated by a 7 digit number which is an entry into one of the 6 quantization
www.dsp-ip.comFast Forward Your Development
which is an entry into one of the 6 quantization levels
VP8 PREDICTION
Inter-prediction
www.dsp-ip.comFast Forward Your Development
Inter-prediction
Intra prediction
Macroblock Intra Prediction
• Intra-prediction exploits the spatial coherence between Macro-blocks without referring to other frames.
• Modes
▫ Same as H.264 in i16x16 and i4x4
▫ Missing modes like i8x8 which exists in H.264
www.dsp-ip.comFast Forward Your Development
▫ Missing modes like i8x8 which exists in H.264
Intra prediction - blocks used
Not Relevant
MNot Available Not Available
www.dsp-ip.comFast Forward Your Development
M
Not Available Not Available Not Available Not Available
Luma i4x4 Intra Prediction
• 4x4 block are predicated by
▫ Orig four 16x16 prediction methods
▫ six “diagonal” prediction methodsDiagonal Down/left Diagonal Down/right
www.dsp-ip.comFast Forward Your Development
Diagonal Down/left
Vertical-right Horizontal-down Vertical-left Horizontal-top
Diagonal Down/right
8x8 Chroma prediction modes• U,V, Y prediction are done separately and one
channel prediction does not affect the other channels.
www.dsp-ip.comFast Forward Your Development
Inter-frame prediction
• Definition - Inter-prediction exploits the temporal coherence between frames to save bitrate.
• Luma sub-block prediction
▫ Method - each Y 4x4 sub-blocks is related to a 4x4 sub-block of the prediction frame.
www.dsp-ip.comFast Forward Your Development
sub-block of the prediction frame.
▫ Precision – motion vectors precision is q-pel.
▫ interpolation pixel is calculated by applying a kernel filter three pixels horizontally and vertically.
▫ Based on deltas from previous MVs “NEARST” and NEAR modes
Inter-frame prediction
• SPLITMV mode
▫ Supports up to 16 MV inside a single MB
▫ MV reuse (delat) is done not only on MB level but also on sub-block level
• Example: Partition of Block into 3 MV areas
www.dsp-ip.comFast Forward Your Development
16
3
Inter-frame prediction - Chroma• Chroma prediction - motion vector for each
8X8 chroma block is calculated separately by one of four prediction methods listed below:
1. Vertical - Copying the row from above throughout the prediction buffer.
2. Horizontal - Copying the column from left
www.dsp-ip.comFast Forward Your Development
2. Horizontal - Copying the column from left throughout the prediction buffer.
3. DC - Copying the average value of the row and column throughout the prediction buffer.
4. Extrapolation from the row and column using the (fixed) second difference (horizontal and vertical) from the upper left corner.
Inter-frame Prediction - Chroma
• Chroma precision - the calculated chroma motion vectors have 1/8 pixel resolution
• averaging the vectors of the four Y sub-blocks that occupy the same area of the frame.
www.dsp-ip.comFast Forward Your Development
Entropy Coding
• Entropy coding is based on binary tree like CABAC but unlike H.264 its not context adaptive like in H.264 and does not adapt on every operation
• Tables does not change all the time
• Unlike H.264 which makes all Symbol->Bit
www.dsp-ip.comFast Forward Your Development
• Unlike H.264 which makes all Symbol->Bit decision as text, VP8 has a tree which represent the transform of each symbol -> faster implementation (not faster processing time)
16
6
Entropy Coding
• According to X264Developer “probability values are constant over the course of the frame”
• According to Google document, there could be up to 4 “Entropy coding context” in a frame
• We can view this as “switched context BAC” and not as CABAC
www.dsp-ip.comFast Forward Your Development
not as CABAC
• Obviously “switched context BAC”<<CABAC on a CPU level
• However setting the context for each fragment is hard to implement in hardware
16
7
PARALLEL PROCESSING
Segment
www.dsp-ip.comFast Forward Your Development
Segment
Partition
Segment Processing
• Segmentation enables creation of MB groups within one logical unit.
• MB are associated with a segment by the MB Segment ID
• All MBs in a segment has the same adaptive adjustments which includes:
www.dsp-ip.comFast Forward Your Development
adjustments which includes:
▫ Same Quantization level
▫ Loop filter strength (0-2)
Frame Processing Architecture
• Frame Header and Partition I are processed first to initialize probabilistic decoder and prediction scheme for each MB. A Serial operation
• Each sub-partition might be processed in parallel to other partitions. probabilistic model of one sub-partition does not interact with another sub-
www.dsp-ip.comFast Forward Your Development
sub-partition does not interact with another sub-partition
Frame
HeaderPartition I
Sub-partition
Partition
IIALength
IIA-IIn-1
Partition
IIB
Partition
IIn
VP VS H.264 DIFFRENCES
www.dsp-ip.comFast Forward Your Development
H.264 Loop Filter
• H.264 Loop filter strength depends on boundary strength, MB type
www.dsp-ip.comFast Forward Your Development
17
2
VP8 Loop Filter
• Lop filter is adjusted by 6 bit variable on two levels:
▫ Global Frame Loop filter level
▫ Per MB level
• VP8 supports two filter types.
• Per MB level is set as “delta” from frame filter level
• L1 – enable/Disable Loop filter in code
www.dsp-ip.comFast Forward Your Development
• L1 – enable/Disable Loop filter in code mode_ref_lf_delta_update
17
3
VP8 Loop filter complexity
• Before final optimization 70% of CPU time is
deblocking filter!!! According to
http://x264dev.multimedia.cx/?p=486
• a 4×4 transform requires a total of 8 length-16 and 8 length-8 loopfilter calls per macroblock while theora 8x8 filter requires half the amount
www.dsp-ip.comFast Forward Your Development
while theora 8x8 filter requires half the amount of calls
17
4
AltRef Noise Reduction
• One of the usages of Alt-Ref frame is for noise reduction
• For noisy sources using a filtered image as reference improves compression (so is pre-processing)
• Improvement 0.25DB for noisy clip
www.dsp-ip.comFast Forward Your Development
• Improvement 0.25DB for noisy clip
• Example with noise reduction set to 5
17
5
AltRef Frame Future Frame
• Just don’t call it a B-Frame and it will give you 1db
www.dsp-ip.comFast Forward Your Development
17
6
H.264 vs VP8 Transforms differences
• No 8x8 transforms
• H.264 simplifies the DCT transform and implement it as a series of:
1. Add
2. Subtract
3. Right shift by 1
www.dsp-ip.comFast Forward Your Development
3. Right shift by 1
It sacrifice accuracy (~1%) for speed/CPU
VP8 uses large,16bit multipliers for accuracy (20091, 35468), this is redundant unlike VC which uses small multiplier
17
7
H.264 vs VP8 Transforms differences
• Unlike H.264 which de-corellates DC values (Hadamard transform) ONLY in Intra i116x16 MB, VP8 uses this method also on some p16x16 MBs
www.dsp-ip.comFast Forward Your Development
17
8
VP8 / H.264 summary
• “Golden Frames” – exist at some level in H.264 slice group map type 2
• Slice granularity is better in H.264
▫ support MB instead of MB line
• Interlacing – not supported
www.dsp-ip.comFast Forward Your Development
• AltRef Frame – a reference frame which is not displayed. Could aggregate all sorts of useful MBs
• Filter – VP has an adaptive complex and slow filter
17
9
Install
• Visual Studio 2010 version – I couldn’t make it work
• Download Visual Studio 2005/2008 code version
• Download and Install YASM
• Follow instructions on
www.dsp-ip.comFast Forward Your Development
• Follow instructions on http://www.tortall.net/projects/yasm/wiki/VisualStudio2005 to integrate with Visual Studio 2005/2008 (copy files and set paths)
18
0
Basic Decoder work
• Compile IVFDEC Project
• Set parameters
▫ --codec=vp8 -o a.i420 qcif1.ivf
• Compile IVFEnc Project
▫ --codec=vp8 --i420 -v a.i420 qcif1.ivf
• Check that the decoder unders
www.dsp-ip.comFast Forward Your Development
• Check that the decoder unders
18
1
COMPARISON (FINALLY)
www.dsp-ip.comFast Forward Your Development
Talking heads, Low motion
• Low motion videos like talking heads are easy to compress, so you'll see no real difference
www.dsp-ip.comFast Forward Your Development
Low motion
In another low motion video with a terrible background for encoding (finely detailed wallpaper), the VP8 video retains much more detail than H.264. Interesting result.
www.dsp-ip.comFast Forward Your Development
Medium motion
VP8 holds up fairly well
www.dsp-ip.comFast Forward Your Development
High motion• In high motion videos, H.264 seems superior. In this
sample, blocks are visible in the pita where the H.264 video is smooth. The pin-striped shirt in the right background is also sharper in the H.264 video, as is the striped shirt on the left.
www.dsp-ip.comFast Forward Your Development
Very High motion
In this very high motion skateboard video, H.264 also looks clearer, particularly in the highlighted areas in the fence, where the VP8 video has artifacts.
www.dsp-ip.comFast Forward Your Development
Final
In the final comparison, I'd give a slight edge to VP8, which was clearer and showed fewer artifacts.
www.dsp-ip.comFast Forward Your Development
Quality Comparison
www.dsp-ip.comFast Forward Your Development
Test yourself
1. Why VP8 is less effective in high motion?
2. Is it patent free?
3. Will you use it?
www.dsp-ip.comFast Forward Your Development
Adobe Flash Video
www.dsp-ip.comFast Forward Your Development www.dsp-ip.com
Flash File Format
Flash Movie / SWF
� File type: .swf
� Includes:
� Graphics, text, video
Web Page
� File type: .htm, .html, .asp, etc>
Flash video / FLV
� File type: .flv
� Includes:
�Video
� Plays in:
.swf
www.dsp-ip.comFast Forward Your Development
� Graphics, text, video
� Video controls
� Client logic
� Plays in:
� Flash Player
� Served from:
� Web server
� Synonyms:
� Application
� Video player
� .swf
� Served from:
�Web server
�Flash Communication Server
FLV (Flash Video) File Format
• Headers in the beginning of the file. Why?
• FLV and MP4 File Formats
• Video support:
▫ SPARK
▫ TrueMotion VP6
www.dsp-ip.comFast Forward Your Development
▫ H264 and VP8 (future support)
• Audio
▫ Nelly Moser codec by ASAO
▫ MP3 Codec
▫ ADPCM (not compressed)
• Alpha Channel
SWF File
• Vector Graphic Format
• Container file
• Includes FLV files
• Action Scripts
• Players
www.dsp-ip.comFast Forward Your Development
• Players
SWF and Personalized Ads
• Encoding a special video version for each user is:
▫ Expensive
▫ Degrades the video
• SWF Enables:
▫ Using one video version stored in FLV
▫ Changing the text or image stored in the outer SWF
www.dsp-ip.comFast Forward Your Development
▫ Changing the text or image stored in the outer SWF per user
Video & Audio
FLV File
Images, Scripts, special Effects
Personalized Video
HTTP Live Streamingby
Apple
www.dsp-ip.comFast Forward Your Development
Agenda
• System Overview
• Components
• Session
www.dsp-ip.comFast Forward Your Development
Apple’s Note
Note: Many existing streaming services require specialized servers to distribute content to end users. It requires specialized skills to set up and maintain these servers, and in a large-scale deployment these servers can be costly. Apple
www.dsp-ip.comFast Forward Your Development
deployment these servers can be costly. Apple has designed a system that avoids this by using standard HTTP to deliver the streams.
System Overview
www.dsp-ip.comFast Forward Your Development
Components Review
• Server
▫ Encoder
▫ Segmenter
• Distributer
▫ Basic HTTP Server
• Client
www.dsp-ip.comFast Forward Your Development
• Client
Server
• Receives Digital / Analog input stream
• Encodes / Transcode video/audio
▫ H.264 Video
▫ AAC audio (HE-AAC or AAC-LC)
• Encodes / Transcode audio only:
▫ MPEG-2 elementary streams, HE-AAC or AAC-LC
www.dsp-ip.comFast Forward Your Development
▫ MPEG-2 elementary streams, HE-AAC or AAC-LC files, or MP3 files
• Encapsulate in MPEG2
▫ Transport Stream
▫ Program Stream
Segmenter
• All segments should be with the same duration
• All segments are placed in a separate file
• Creates Index file with references to segment files
• For protection, the Segmenter might encrypt each media segment and create a key file
www.dsp-ip.comFast Forward Your Development
each media segment and create a key file
Distribution
• Distribution system is a regular HTTP Server
• Could be Apache or small embedded Server
www.dsp-ip.comFast Forward Your Development
Files
• Segments – stored as *.ts files
• Index files – stored as *.m3u8
• Index file format example:
#EXTM3U#EXT-X-TARGETDURATION:10#EXTINF:10,
www.dsp-ip.comFast Forward Your Development
#EXTINF:10,http://media.example.com/segment1.ts#EXTINF:10,http://media.example.com/segment2.ts#EXTINF:10,http://media.example.com/segment3.ts#EXT-X-ENDLIST
Session types
• Live Stream Broadcast
▫ Index file is continues updated
▫ Include a moving window of segments around “live” part of the session
▫ Client should continuously refresh the Index file
• VoD Session
www.dsp-ip.comFast Forward Your Development
• VoD Session
▫ Index file static
▫ Includes ALL the segments of the file
▫ Enables “Seek” operation
Multi-bitrate multi–device support
• Multi-bitrate is enabled via multiple index files
• Index files are pointed by a global index files
• Client can select a stream according to:
▫ Device properties
▫ Available bit rate
• This method is less efficient
www.dsp-ip.comFast Forward Your Development
• This method is less efficientthan Silverlight Global File
Test yourself
1. What are the two Live streaming file types?
2. What is the role of the Segmenter?
3. On which delivery protocol is the live streaming based?
www.dsp-ip.comFast Forward Your Development
Silverlight Smooth Streaming
www.dsp-ip.comFast Forward Your Development
SILVERLIGHT INTRODUCTION
www.dsp-ip.comFast Forward Your Development
Smooth Streaming
• Microsoft’s implementation of HTTP-based adaptive streaming
• A hybrid media delivery method that acts like streaming but is in fact a series of short progressive downloads
www.dsp-ip.comFast Forward Your Development
• Leverages existing HTTP caches• Client can seamlessly switch video quality and bit
rate based on perceived network bandwidth and CPU resources
Streaming or Progressive Download?
Traditional Streaming
• Responsive User Experience
• Bandwidth Use• User Tracking
Progressive Download
• Works from a Web Server
• World-wide scale w/HTTP
www.dsp-ip.comFast Forward Your Development
Challenges
• No cache-ability• Separate,
smaller streaming networks
Challenges
• Limited User Experience
• User tracking• Bandwidth Use
(20% watched)
Smooth Streaming Design
• Smooth Streaming File Format based on MP4 (ISO Base Media File Format)
• Video is encoded and stored on disk as one contiguous MP4 file▫ Separate file for each bit rate
• Each video Group of Pictures (GOP) is stored in a
www.dsp-ip.comFast Forward Your Development
• Each video Group of Pictures (GOP) is stored in a Movie Fragment box▫ This allows easy fragmentation at key frames
• Contiguous file is virtually split up into chunks when responding to a client request
Content Provider Benefits
• Cheaper to deploy
▫ Can utilize any generic HTTP caches/proxies
▫ Doesn’t require specialized servers at every node
• Better scalability and reach
▫ Reduces “last mile” issues because it can
www.dsp-ip.comFast Forward Your Development
▫ Reduces “last mile” issues because it can dynamically adapt to inferior network conditions
• Audience can adapt to the content, rather than requiring the content providers to guess which bit rates are most likely to be accessible to their audience
End User Benefits• Fast start-up and seek times
▫ Start-up/seeking can be initiated on the lowest bit rate before moving up to a higher bit rate
• No buffering, no disconnects, no playback stutter
▫ As long as the user meets the minimum
www.dsp-ip.comFast Forward Your Development
▫ As long as the user meets the minimum bit rate requirement
• Seamless bit rate switching based on network conditions and CPU capabilities.
• A generally consistent, smoothplayback experience
Evolution
• Previous versions of MS streaming divide the file into many chunkc 0001.vid 0002.vid etc
• Problematic in caching, CDNs, CMS etc
• Today all fragments of a file are contained in a single bitstream container. Typically 1 fragment = 1 video GOP.
www.dsp-ip.comFast Forward Your Development
= 1 video GOP.
SILVERLIGHT FILES
www.dsp-ip.comFast Forward Your Development
Containers & Configuration files
Format options
• ASF/WMV – native Microsoft Format
• MPEG4 File-Format
• AVI
• OGG
www.dsp-ip.comFast Forward Your Development
MP4 over ASF file format• MP4 is a lightweight container format with less
overhead than ASF
• MP4 is easier to parse in managed (.NET) code
• MP4 is based on a widely used standard, making 3rd party adoption and support easier
• MP4 has native H.264 video support
www.dsp-ip.comFast Forward Your Development
• MP4 has native H.264 video support
• MP4 was designed to natively support payload fragmentation within the file
MP4 File format
• MP4 has two format types
▫ Disk Format - for file storage
▫ Wire format - for transport
• Wire format enables easy CDN support and integration
www.dsp-ip.comFast Forward Your Development
Smooth Streaming File Format
www.dsp-ip.comFast Forward Your Development
Smooth Streaming Wire Format
www.dsp-ip.comFast Forward Your Development
File extensions
• Media Files
▫ *.ismv - Audio & Video
▫ *.isma – Audio only
• Manifest Files
▫ *.ism – Server manifest. Describes to the server Relation between tracks, bitrates & files on disk.
www.dsp-ip.comFast Forward Your Development
Relation between tracks, bitrates & files on disk. Based on SMIL 2.0 XML format specification
▫ *.ismc – Describes to the client the available streams, CODECS used, bitrates encoded, video resolutions, markers, captions. First file delivered to client. It’s the first file delivered to client (“SDP” like).
Directory Structure
Manifest Files
Media file in different bitrates
www.dsp-ip.comFast Forward Your Development
Manifest files• VC-1, WMA, H.264 and AAC codecs
• Text streams
• Multi-language audio tracks
• Alternate video & audio tracks (i.e. multiple camera angles, director’s commentary, etc.)
• Multiple hardware profiles (i.e. same bitrates
www.dsp-ip.comFast Forward Your Development
• Multiple hardware profiles (i.e. same bitrates targeted at different playback devices)
• Script commands, markers/chapters, captions
• Client manifest Gzip compression
• URL obfuscation
• Live encoding and streaming
ISM file sample<?xml version="1.0" encoding="utf-16" ?>
- <!-- Created with Expression Encoder version 2.1.1206.0 -->
- <smil xmlns="http://www.w3.org/2001/SMIL20/Language">
- <head>
<meta name="clientManifestRelativePath" content="NBA.ismc" />
</head>
- <body>
- <switch>
- <video src="NBA_3000000.ismv" systemBitrate="3000000">
www.dsp-ip.comFast Forward Your Development
<param name="trackID" value="2" valuetype="data" />
</video>
- <video src="NBA_2400000.ismv" systemBitrate="2400000">
<param name="trackID" value="2" valuetype="data" />
</video>
- <video src="NBA_1800000.ismv" systemBitrate="1800000">
<param name="trackID" value="2" valuetype="data" />
</video>
ISM file sample- <video src="NBA_1300000.ismv" systemBitrate="1300000">
<param name="trackID" value="2" valuetype="data" />
</video>
- <video src="NBA_800000.ismv" systemBitrate="800000">
<param name="trackID" value="2" valuetype="data" />
</video>
- <video src="NBA_500000.ismv" systemBitrate="500000">
<param name="trackID" value="2" valuetype="data" />
</video>
www.dsp-ip.comFast Forward Your Development
- <audio src="NBA_3000000.ismv" systemBitrate="64000">
<param name="trackID" value="1" valuetype="data" />
</audio>
</switch>
</body>
</smil>
*.ISMC sample<?xml version="1.0" encoding="utf-16" ?>
- <!-- Created with Expression Encoder version 2.1.1206.0 -->
- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="4084405506">
- <StreamIndex Type="video" Subtype="WVC1" Chunks="208" Url="QualityLevels({bitrate})/Fragments(video={start time})">
<QualityLevel Bitrate="3000000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3FE27F1678A27F859E80C9082DB8D44A9C00000010E5A67F840" />
<QualityLevel Bitrate="2400000" FourCC="WVC1" Width="1056" Height="592"
www.dsp-ip.comFast Forward Your Development
<QualityLevel Bitrate="2400000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3FE20F1278A20F849E80C9082493DEDDCC00000010E5A67F840" />
<QualityLevel Bitrate="1800000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBF81A70EF8A1A783BE80C908236EE5265400000010E5A67F840" />
<QualityLevel Bitrate="1300000" FourCC="WVC1" Width="640" Height="352" CodecPrivateData="250000010FCBE813F0AF8A13F82BE80C9081A7ABF704400000010E5A67F840" />
ISMC File - 2- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">
- <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})">
<QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" /> …..
<c n="0" d="20000000" />
<c n="1" d="20000000" />
.....
<c n="298" d="5000001" />
</StreamIndex>
www.dsp-ip.comFast Forward Your Development
</StreamIndex>
- <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})">
<QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" />
<c n="0" d="20433560" /> ....
<c n="297" d="20433560" />
<c n="298" d="4393197" />
</StreamIndex>
</SmoothStreamingMedia>
SILVERLIGHT SESSION
www.dsp-ip.comFast Forward Your Development
Initiation and Flow
Smooth Streaming Protocol
• Smooth Streaming Protocol uses HTTP [RFC2616] as its underlying transport .
• The Server role in the protocol is stateless
▫ Enabling (potentially) different instance of the server to handle client requests
www.dsp-ip.comFast Forward Your Development
server to handle client requests
▫ Request can utilize any generic HTTP caches/proxies - > Lowering CDN costs
Messages
• Smooth Streaming Protocol uses 4 different messages:
▫ Manifest Request
▫ Manifest Response
▫ Fragment Request
▫ Fragment Response
www.dsp-ip.comFast Forward Your Development
▫ Fragment Response
• All messages follow the HTTP/1.1 specification
Messages Flow
Server Client
Manifest Request
Manifest Response
Fragment Request
www.dsp-ip.comFast Forward Your Development
Fragment Response
Fragment Request(s)
Messages
• Manifest Request and Fragment Request message MUST use the HTTP "GET" method, generated by the client.
• Manifest Request and Fragment Request message use the HTTP Response messages.
www.dsp-ip.comFast Forward Your Development
message use the HTTP Response messages.
Status-Code SHOULD be 200.
Smooth Streaming Transport Protocol Session Details
Manifest Request
Manifest ResponseVideo Fragment Request
Fragment ResponseAudio Fragment Request
www.dsp-ip.comFast Forward Your Development
Session Details - Manifest Request
• In order to initiate a presentation the Client MUST send the server a Manifest Request using the HTTP GET method.
www.dsp-ip.comFast Forward Your Development
the HTTP GET method.
Session Details - Manifest Response
• The Response is a ISMC Manifest file describing the session. - <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">
- <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})">
<QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" />
…..
<c n="0" d="20000000" />
<c n="1" d="20000000" />
www.dsp-ip.comFast Forward Your Development
<c n="1" d="20000000" />
.....
<c n="297" d="20000000" />
<c n="298" d="5000001" />
</StreamIndex>
- <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})">
<QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" />
<c n="0" d="20433560" />
....
<c n="297" d="20433560" />
<c n="298" d="4393197" />
</StreamIndex>
</SmoothStreamingMedia>
Manifest Response reviewed• We can see in the ISMC file that the server can support 8 different levels
of quality (bitrate) for the client can chose from between 2.75Mbit to 0.35 Mbit.
- <SmoothStreamingMedia MajorVersion="1" MinorVersion="0" Duration="5965419999">
- <StreamIndex Type="video" Subtype="WVC1" Chunks="299" Url="QualityLevels({bitrate})/Fragments(video={start time})">
<QualityLevel Bitrate="2750000" FourCC="WVC1" Width="1280" Height="720" CodecPrivateData="250000010FD3BE27F1678A27F859E804508253EBE8E6C00000010E5AE7F840" />
<QualityLevel Bitrate="2040000" FourCC="WVC1" Width="1056" Height="592" CodecPrivateData="250000010FD3BE20F1278A20F849E80450823E414DD1400000010E5AE7F840" />
<QualityLevel Bitrate="1520000" FourCC="WVC1" Width="848" Height="480" CodecPrivateData="250000010FCBAE1A70EF8A1A783BE8045081AE62F3F7400000010E5AE7F840" />
www.dsp-ip.comFast Forward Your Development
CodecPrivateData="250000010FCBAE1A70EF8A1A783BE8045081AE62F3F7400000010E5AE7F840" />
<QualityLevel Bitrate="1130000" FourCC="WVC1" Width="704" Height="400" CodecPrivateData="250000010FCBA215F0C78A15F831E8045081A27BD635C00000010E5AE7F840" />
<QualityLevel Bitrate="845000" FourCC="WVC1" Width="576" Height="320" CodecPrivateData="250000010FCB9A11F09F8A11F827E804508199C94077400000010E5AE7F840" />
<QualityLevel Bitrate="630000" FourCC="WVC1" Width="448" Height="256" CodecPrivateData="250000010FCB920DF07F8A0DF81FE804508113396020C00000010E5AE7F840" />
<QualityLevel Bitrate="470000" FourCC="WVC1" Width="368" Height="208" CodecPrivateData="250000010FC38E0B70678A0B7819E80450810E5747B6C00000010E5AE7F840" />
<QualityLevel Bitrate="350000" FourCC="WVC1" Width="320" Height="176" CodecPrivateData="250000010FC38A09F0578A09F815E80450808AADEACF400000010E5AE7F840" />
Manifest Response – reviewed• The client also receives the number of chunks for audio and video tracks
and the duration of each chunk so it can request the chunk which fits the desired position in the file
<c n="0" d="20000000" />
<c n="1" d="20000000" />
<c n="2" d="20000000" />
<c n="3" d="20000000" />
....
<c n="297" d="20000000" />
<c n="298" d="5000001" />
</StreamIndex>
www.dsp-ip.comFast Forward Your Development
</StreamIndex>
- <StreamIndex Type="audio" Subtype="WmaPro" Chunks="299" Url="QualityLevels({bitrate})/Fragments(audio={start time})">
<QualityLevel Bitrate="64000" WaveFormatEx="6201020044AC0000451F0000CF05100012001000030000000000000000000000E00042C0" />
<c n="0" d="20433560" />
<c n="1" d="19969161" />
<c n="2" d="19969161" />
<c n="3" d="20433560" />
<c n="4" d="20433560" />
<c n="297" d="20433560" />
<c n="298" d="4393197" />
</StreamIndex>
</SmoothStreamingMedia>
Session Details – Fragment Request
• Client-Server requests are based on RESTFull URLs:GET /mediadl/iisnet/smoothmedia/Experience/BigBuckBunny_720p.ism/QualityLevels(350000)/Fragments(video=0)
• The URL includes reference to:
www.dsp-ip.comFast Forward Your Development
• The URL includes reference to:
▫ Bitrate as QualityLevels which maps to a media file
▫ Fragment number
Session Details – Fragment Response
• The Server:
▫ checks “BigBuckBunny_720p.ism” server manifest file to find the media file associated with the quality level(350000)
▫ Opens and parses the associated media file to get the chunk with requested time offset (0).
www.dsp-ip.comFast Forward Your Development
requested time offset (0).
▫ Sends the requested media fragment to the client as HTTP response with status code set to 200
Refrences
• Most valuable refrence: http://alexzambelli.com/blog/2009/02/10/smooth-streaming-architecture/
www.dsp-ip.comFast Forward Your Development
SILVERLIGHT PROGRAMMING
www.dsp-ip.comFast Forward Your Development
MediaElement Code Walkthrough
Creating New Silverlight Project
• Choose new “Silverlight Application” in the new project dialog. C# or VB.Net is available.
• Choose the default settings at the dialog opened
www.dsp-ip.comFast Forward Your Development
New Project configuration
• A new project has been created.
• The starting point of the application is “App.xaml”.
▫ The starting page of the application is called by default: “MainPage.xaml” ( can be changed at “App.xaml.cs”)
www.dsp-ip.comFast Forward Your Development
• Media element Represents a control that contains audio and/or video.
• Media files cannot be used as project resource.
www.dsp-ip.comFast Forward Your Development
▫ Must be delivered separately.
• To preserve video original aspect ratio DON’T change both Width and Height.
Media Element – Main Events
• Media Events
▫ MediaOpened Occurs when media loading has finished.� ‘ActualWidth’ and ‘ActualHeight’ are set after this event is called.
▫ MediaEnded Occurs when the media has ended.
▫ MediaFailed Occurs when an error is encountered.� Use ‘ExceptionRoutedEventArgs’ to get failure details.
www.dsp-ip.comFast Forward Your Development
• Buffering Events:
▫ BufferingStarted Occurs when media buffering has begun.
▫ BufferingEnded Occurs when media buffering has ended.
Controlling Media Element
• Setting Audio/Video input.
• Playing file from current position.
▫ Call this only after MediaOpened event is called.
www.dsp-ip.comFast Forward Your Development
• Stop Playing current
▫ Stops the media playing and sets current to start
• Pause command
Full Source - MainPage.xaml
www.dsp-ip.comFast Forward Your Development
Full Source - MainPage.xaml.cs
www.dsp-ip.comFast Forward Your Development
Adobe HTTP Streaming
www.dsp-ip.comFast Forward Your Development
Agenda
• Overview
• Components
• Files
• Session
www.dsp-ip.comFast Forward Your Development
HTTP Streaming Intro
• HTTP Streaming offers the advantages of:▫ Progressive download in terms of � Cost
� Standard Server
� Scalability
� Standard client components (OSMF)
www.dsp-ip.comFast Forward Your Development
� Standard client components (OSMF)
▫ Streaming in terms of � User experience
� Seek-ability of streaming
Overview
Adobe HTTP Dynamic Streaming is a solution that allows you to stream live and on-demand content over HTTP to Adobe Flash Player. When content streams over HTTP, clients can seek quickly to any
www.dsp-ip.comFast Forward Your Development
HTTP, clients can seek quickly to any location.
HTTP Dynamic Streaming supports adaptive streaming, DVR functionality, and Adobe Flash Access protection (DRM).
COMPONENTS
• Content Ingest
• Server
www.dsp-ip.comFast Forward Your Development
• Server
• Client
Main components
• Preparation
▫ File Packager
▫ Live Packager for HTTP Dynamic Streaming
• Server
▫ Apache module (HTTP Origin Module)
▫ Flash Access
www.dsp-ip.comFast Forward Your Development
▫ Flash Access
• Client
▫ Player with OSMF classes
▫ Flash Player version 10.1+
▫ Air 2.0+
Adobe Solution main components
www.dsp-ip.comFast Forward Your Development
Ingest- File Packager
• A command-line tool
• Used for converting offline content to formats required for Adobe HTTP streaming
• Translates on-demand media files into fragments and writes the fragments to F4F files.
www.dsp-ip.comFast Forward Your Development
Ingest - Live Packager
• The Live Packager for HTTP Dynamic Streaming is part of Adobe Flash Media Server.
• The server ingests a live stream over RTMP and translates it into F4F files in real-time.
• The built-in Apache HTTP Server uses the HTTP Origin Module to deliver the live content over
www.dsp-ip.comFast Forward Your Development
Origin Module to deliver the live content over HTTP.
Server - Apache module (HTTP Origin)
• Extension to Apache HTTP Server version 2.2.
• Enables processes of Adobe Files:
▫ F4F, F4M,F4X
▫ .bootstrap and
▫ .drmmeta
• Flash Media Interactive Server
www.dsp-ip.comFast Forward Your Development
• Flash Media Interactive Server 3.8 includes Apache HTTP Server.
Server - Flash Access
• DRM Server
• Flash Access delivers protected media to Flash Player
• For content protection, both File Packager and Flash Media Server are required to package and encrypt the
www.dsp-ip.comFast Forward Your Development
required to package and encrypt the content
Client - OSMF classes
• The OSMF Player uses the ActionScript 3.0 NetStream.appendBytes() API to deliver data to Flash Player.
• OSMF is a robust framework designed to deliver high-quality video.
• Adobe strongly recommends using OSMF
www.dsp-ip.comFast Forward Your Development
• Adobe strongly recommends using OSMF to build HTTP Dynamic Streaming players.
ADOBE HTTP STREAMING FILES
F4F, F4M,F4X
.bootstrap and
www.dsp-ip.comFast Forward Your Development
.bootstrap and
.drmmeta
Files
• The files required for HTTP streaming are:▫ F4F - MPEG4 media format. Holds the media
▫ F4M – Media description file(codec, resolution)
▫ F4X - Fragments location file
▫ .bootstrap – bootstrap information for each segment
www.dsp-ip.comFast Forward Your Development
segment
▫ .drmmeta – DRM encryption information
*.F4F File
• Standard MP4 format with open file specification
• Each file contains a segment of the source file.
• Each segment contains one or more fragments of content.
• The file formats stores any flash supported codec
• A player can use a URL to address each fragment.
www.dsp-ip.comFast Forward Your Development
• A player can use a URL to address each fragment.
HTTP Streaming file types
• *.F4X File▫ Flash Index file.
▫ Contains the location of specific fragments within a stream.
• *.F4M File▫ Flash Media Manifest file.
www.dsp-ip.comFast Forward Your Development
▫ Flash Media Manifest file.
▫ Contains information about the media codecs, resolution, and the availability of multi-bitratefiles.
F4M File Sample<?xml version="1.0" encoding="utf-8" ?>
- <manifest xmlns="http://ns.adobe.com/f4m/1.0">
<id>Dynamic Streaming</id>
<duration>253</duration>
<mimeType>video/mp4</mimeType>
<baseURL>rtmp://cp67126.edgefcs.net/ondemand</baseURL>
<media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_dream_768x428_24.0fps_408kbps" bitrate="408" width="768" height="428" />
www.dsp-ip.comFast Forward Your Development
ream_768x428_24.0fps_408kbps" bitrate="408" width="768" height="428" />
<media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_dream_768x428_24.0fps_608kbps" bitrate="608" width="768" height="428" />
<media url="mp4:mediapm/ovp/content/demo/video/elephants_dream/elephants_dream_1024x522_24.0fps_908kbps" bitrate="908" width="1024" height="522" />
</manifest>
HTTP Streaming file types
• .bootstrap
▫ Bootstrap file.
▫ Contains the bootstrap information for each segment of the file
• .drmmeta
▫ DRM Header file.
www.dsp-ip.comFast Forward Your Development
▫ DRM Header file.
▫ Contains additional header information about the encryption.
Folder example
• Example for File “foo” Folder will include :
▫ fooSeg#.f4f (many fragments)
▫ foo.f4x
▫ foo.meta
▫ foo.bootstrap
▫ foo.drmmeta (if the stream is configured for
www.dsp-ip.comFast Forward Your Development
▫ foo.drmmeta (if the stream is configured for encryption)
Messages Flow
Server Client
Manifest Request
Manifest Response
Fragment Request
www.dsp-ip.comFast Forward Your Development
Fragment Response
Fragment Request(s)
Session
• The player fetches foo.f4m manifest.
• The player request fragment#
• HTTP Origin Module locates the required f4f files by looking at F4X index file and send to client.
• If the client seeks further, the Origin module searches at the f4m and again locates the f4f file that fits, starts streaming again.
www.dsp-ip.comFast Forward Your Development
streaming again.
• If the player constantly monitor bandwidth and performance and ask the Origin Module for the proper bit rate associated f4f.
HTML5 Video
www.dsp-ip.comFast Forward Your Development
HTML5
• Drafts by WHAT WG
▫ Web Hypertext Application Technologies
• Merging into W3C specifications• “One of HTML5’s goals is to move the Web away from
proprietary technologies such as Flash, Silverlight, and JavaFX, says Ian Hickson, co-editor of the HTML5
www.dsp-ip.comFast Forward Your Development
JavaFX, says Ian Hickson, co-editor of the HTML5 specification.”—Paul Krill, reporting for InfoWorld, June 16, 2009
• Browser support
Fragmented Web - Description
• Multimedia coding on the web is fragmented
• Many video codecs:
▫ DIVX, XVID, H.264
▫ WMV, VC-1, VP6
• Many containers (File Format)
▫ AVI, MKV
www.dsp-ip.comFast Forward Your Development
▫ AVI, MKV
▫ MPEG4 FF, 3GPP
• Many delivery methods
▫ RTSP/RTP Streaming, Progressive download
▫ Live HTTP, Smooth Streaming
Fragmented Web - Challenges
• Proprietary Plug-ins - like Flash
• Vertical market control on media distribution –like Apple
• Media Distributers need to support many:
▫ Codecs
▫ Containers
www.dsp-ip.comFast Forward Your Development
▫ Containers
▫ Delivery Formats
• in order to support all device and audiences
XIPH
• XIPH.org is a non profit organization which aims to create free multimedia coding standards
• XIPH defined
▫ Vorbis – Audio codec
▫ Ogg – a free file format media container
▫ Speex – voice codec
www.dsp-ip.comFast Forward Your Development
▫ Speex – voice codec
▫ Theora – Video Codec
• HTML5 Video first based its video codec and container standard on XIPH Standards
HTML5 Video
• HTML5 video first defined XIPH formats as the base HTML5 video: “User agents should support Theora video and Vorbis audio, as well as the Ogg container format.” December 10, 2007, the HTML5 specification
• This was later replaced by a statement which
www.dsp-ip.comFast Forward Your Development
• This was later replaced by a statement which basically stated: we cant make up our mind, use whatever you like.
HTML5 Video - Fragmented• Support Theora (version of VP3)
▫ Old codec
▫ Poor performance (BR/Quality ratio)
▫ Free no royalties
▫ Hardware support?
• Also H.264
www.dsp-ip.comFast Forward Your Development
▫ Much better quality per bitrate
▫ But it requires royalties….
• Google opens VP8
▫ Good Quality
▫ No Royalties (?)
HTML5 Video Code
<videosrc="movie.ogg" controls="controls">If you can see this text, your browser does not support the HTML5 video tag.</video>
www.dsp-ip.comFast Forward Your Development
Source W3C School
Browser CODEC Support
Browser Ogg Theora H.264/MPEG-4 AVC
Internet Explorer NO 9.0
Mozilla Firefox 3.5 No
www.dsp-ip.comFast Forward Your Development
Google Chrome 3.0 3.0
Safari No 3.1
Opera 10.50
What is missing
• Standard Multi-bitrate support
• HTTP Streaming (not PD)
• Option for live streams
• Transmit your camera (ChatRoulette Style)
• P2P Interaction
www.dsp-ip.comFast Forward Your Development
Is that the Flash Killer?
WebM Project
www.dsp-ip.comFast Forward Your Development
WebM Overview
• Google Sponsored Project
• Aims to create: Open, Royalty free media coding formats for the open web
• Defines
▫ File Format / Container
▫ Audio CODEC
www.dsp-ip.comFast Forward Your Development
▫ Audio CODEC
▫ Video CODEC
H.264 vs Theora
• Theora is a royalty free code but has lower quality than H.264
• H.264 requires royalties for IP owners but has better quality
• Some browser used H.264 video some stayed with Theora.
www.dsp-ip.comFast Forward Your Development
with Theora.
• HTML5 video fragments the web instead of unit it
Browser Ogg Theora H.264
Internet Explorer
NO 9.0
Mozilla Firefox
3.5 No
Google Chrome
3.0 3.0
Safari No 3.1
Opera 10.50
Source: Wikipedia
WebM
• WebM fills the gap left by HTML5 standardization.
• Defines: video, audio and container formats
• Solves the royalty free Theora vs the superior quality H.264 by providing a royalty free video codec with the same (or better) video quality as H.264
www.dsp-ip.comFast Forward Your Development
H.264
Source: On2
CDNs
www.dsp-ip.comFast Forward Your DevelopmentSource: Limelight
Previous CDN classification
• Server based – Incumbents, Akamai
• Traditional P2P
▫ “Bittorrent”-s
▫ Pando Networks
Classify CDN based on delivery technology
www.dsp-ip.comFast Forward Your Development
▫ Pando Networks
• Grid Based
• Hybrid Architecture
▫ “Peer Assisted Streaming”
▫ Joost
Classic CDN Segmentation
CDN
PoPBased
P2PMulti-stream
Grid
www.dsp-ip.comFast Forward Your Development
Distributed Centralized Pure P2PPeer
Assisted
Modern CDN Classification
Classify CDN based on
▫ Who pays for the service?
▫ What is the service goal?
not on the delivery technology
• Content Provider CDN
www.dsp-ip.comFast Forward Your Development
• Content Provider CDN
• External Service provider CDN
• Internal Service Provider CDN
• Application CDN
Modern CDN Classification
CDN
Service Provider -
Service Provider -
Content Based
www.dsp-ip.comFast Forward Your Development
Provider -Internal
Premium Content
OTT
Provider -External
Based
Content Provider CDN
• Goal – Assure delivery, Reduce load and delay from content providers’ sites.
• Examples
▫ Akamai
▫ Limelight
• Located – in main service provider’s POP
www.dsp-ip.comFast Forward Your Development
• Located – in main service provider’s POP deployed globally
• Client - Content distributers and owners such as CNN, Hulu and “Youtube”-like sites.
External Service providers CDN
• Goal - Reduce EXTERNAL bandwidth cost for ISP by caching content within the ISP network.
• Examples
▫ Oversi
▫ PeerApp
• Located – in main ISP POPs
www.dsp-ip.comFast Forward Your Development
• Located – in main ISP POPs
• Equipment is PC server based or embedded
• Clients - service providers like 012 or 014 (Israeli examples)
Internal Service Provider CDN• Goal - reduce INTERNAL bandwidth load inside
the service provider network by caching.
• Used mainly in IPTV to distribute premium content (due to cost)
• Examples: Mostly VoD service providers
▫ Bitband (Motorola)
www.dsp-ip.comFast Forward Your Development
▫ Bitband (Motorola)
▫ Velolcix (Alcatel-Lucent)
• Located – Distributed within service provider’s network
• Equipment is PC server based or embedded
• Clients – IPTV service providers
Peer based CDNs
• Bittorrent
• Giraffic
• RayV
www.dsp-ip.comFast Forward Your Development
Video Technology PositioningMicrosoft
Apple
www.dsp-ip.comFast Forward Your Development
Apple
Adobe
Microsoft
• Large Install base 60%+ for Silverlight
• Good connection to media and content companies (DRM – Play Ready)
• Innovative in video technologies
▫ HTTP Streaming
▫ CDN-oriented
www.dsp-ip.comFast Forward Your Development
▫ CDN-oriented
▫ Moving from proprietary to standard technology
Apple
• User base
▫ Relatively small and fanatic
• Video innovation – minor
▫ HTTP Streaming FF is based on MPEG2
▫ Many fragments - hard to manage and CDN
▫ First use of front video cam on iPhone 4.0
www.dsp-ip.comFast Forward Your Development
▫ First use of front video cam on iPhone 4.0
• Closed platform
• Vertical market control
• Relaying too much on HTML5
• Annoying clients and business partners
Adobe
• Install base
▫ Biggest 99% of connected computers
▫ Company image suffers from bad PR lately
• Innovation
▫ Very innovative in the past
▫ First massive use of progressive download
www.dsp-ip.comFast Forward Your Development
▫ First massive use of progressive download
▫ Good on Integration Solution – OSMF
▫ Losing its edge:
� Last to implement HTTP Streaming
Adobe
• Acquisition strategy - Poor
▫ Can’t monetize past acquisitions: Macromedia, Amicima
▫ Paid too much for last acquisition: Omniture
• Technology
▫ Got good technologies
www.dsp-ip.comFast Forward Your Development
▫ Got good technologies
▫ Focused on developing technologies and less on monetizing them
• Install base
▫ Medium
▫ Good growth potential on mobile / browser
• Innovation
▫ Good Innovation Internally
▫ Aggressive acquisition strategy (Android, On2)
www.dsp-ip.comFast Forward Your Development
▫ Aggressive acquisition strategy (Android, On2)
▫ Open system as innovation
• Open platform strategy
▫ Not technology oriented
• Partnerships
▫ Good relations with partners
• Acquisition strategy – Great!
▫ Does not pay too much on companies
▫ Monetize and build acquired companies IP: Android
▫ Great move with On2 IP
• Technology
www.dsp-ip.comFast Forward Your Development
• Technology
▫ Good mix of internal and external technologies
▫ Use standard API & Protocols
▫ Standardize proprietary Technologies (VP8)
▫ Does not depend on a single delivery technology: HTML5 or Flash
How do we define success?
• Who is first in the video race:
▫ Install base – Speed
▫ Innovation - acceleration Install
Base
www.dsp-ip.comFast Forward Your Development
Innovation
Positioning
• This is my understanding.
• What do you think?Install
Base
www.dsp-ip.comFast Forward Your Development
Innovation
Summary
• Scaling Video delivery:
▫ CDNs
▫ HTTP Streaming
▫ Standards
▫ Interoperability
• Lowering costs
www.dsp-ip.comFast Forward Your Development
• Lowering costs
▫ HTTP Streaming
• User experience
▫ PD -> Streaming
▫ Multi-bit-rate
DSP-IP Contact information
Download slides at: Download slides at: www.dspwww.dsp--ip.comip.com
Course materials & lecture requestCourse materials & lecture request
Adi YakovAdi YakovTraining ManagerTraining Manager
++972972--99--86519338651933
www.dsp-ip.comFast Forward Your Development
Projects development services:Projects development services:
++972972--99--86519338651933
wwwwww..dspdsp--ipip..comcomMailMail :: info@[email protected]:: ++972972--99--88509568850956,,FaxFax :: ++972972--5050-- 89629108962910
Alona AshkenaziAlona Ashkenazi
Development Development ServicesServices
alona@[email protected]
++972972--99--88509568850956
APPENDIX OPTIONAL SLIDES
www.dsp-ip.comFast Forward Your Development
OSMF
www.dsp-ip.comFast Forward Your Development
Goal
• Accelerate Development
• Standardize API between modules
For Video Players
www.dsp-ip.comFast Forward Your Development
Support Media Types
• Streaming audio
▫ Mp3 , AAC , Speex and Nellymoser
• Streaming video
▫ FLV, F4V, MP4, MPEG:MP4, M4V, 3GPP
www.dsp-ip.comFast Forward Your Development
3GPP
• Progressive audio & video
▫ Mp3, FLV, F4V, MP4, MP4V-ES, M4V,3GPP, QuickTime
Basic Structure an OSMF player
Media player Control playback of a media
element
Media containerDisplay media element (s)
On the stage
Play, Stop, Pause etc Add to stage, show element(s)
www.dsp-ip.comFast Forward Your Development
Media element Loads any support media type
Video, images, sound, etc or
Composite element
load stuff
Imports
Import org. osmf. containers. Media container;
Import org. osmf. Elements. Video element;
Import org. osmf. Media .media player ;
Import org. osmf. Media. URL Resource;
www.dsp-ip.comFast Forward Your Development
The media container
Import org. osmf. containers. Media container;
Import org. osmf. Elements. Video element;
Import org. osmf. Media .media player ;
Import org. osmf. Media. URL Resource;
www.dsp-ip.comFast Forward Your Development
// create the container class that displays the media .
var container: media container = new media container ( ) ;
The media container
Import org. osmf. containers. Media container;
Import org. osmf. Elements. Video element;
Import org. osmf. Media .media player ;
Import org. osmf. Media. URL Resource; // create the container class that displays the media .var container: media container = new media container ( ) ;
www.dsp-ip.comFast Forward Your Development
//add the Media container instance to the stage
Add child (container) ;
The media container
Import org. osmf. containers. Media container;
Import org. osmf. Elements. Video element;
Import org. osmf. Media .media player ;
Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ;//add the Media container instance to the stage
www.dsp-ip.comFast Forward Your Development
Var video Element =
new video Elements ( new URL resource ) ;
//add the Media container instance to the stage Add child (container) ;
The media container
Import org. osmf. containers. Media container;
Import org. osmf. Elements. Video element;
Import org. osmf. Media .media player ;
Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ;//add the Media container instance to the stage
www.dsp-ip.comFast Forward Your Development
// cerate the media player instance
Var media player : media player = mew media player ( );
//add the Media container instance to the stage Add child (container) ;Var video Element =
new video Elements ( new URL resource ) ;
The media containerImport org. osmf. containers. Media container;
Import org. osmf. Elements. Video element;
Import org. osmf. Media .media player ;
Import org. osmf. Media. URL Resource; // create the container class that displays the media var container: media container = new media container ( ) ;//add the Media container instance to the stage Add child (container) ;
www.dsp-ip.comFast Forward Your Development
Add child (container) ;Var video Element = new video Elements ( new URL resource ) ;// cerate the media player instance Var media player : media player = mew media player ( );//Set the media Elements on the media player// because auto play defaults to true , play back , begin immediately media player . media = Video Elements ;
ADDING CAPABILITIES
Playback control
www.dsp-ip.comFast Forward Your Development
Media Factory
Layouts
Default Playback
// set auto play to false
Media player .auto play=false;
www.dsp-ip.comFast Forward Your Development
// listen for the click event on the stage
Stage . Add Event Listener ( mouse Event . CLICK,_ on click );
Function _on click (evt ): void
{
// if the media player is playing
If (media player .playing )
{
// pause it
Default Playback
www.dsp-ip.comFast Forward Your Development
// pause it
media player. pause ( );
}
else {
// resume it
media player . Play ( );
}
}
Media Factory
Create
Media Elements, with a URL
Media Factory
www.dsp-ip.comFast Forward Your Development
Media Elements:
video ElementImage Element sound Elements
The Media Factory
Import org .osmf .media .default media factory ;
// create the default media factory instance
Var media factory = new default media factory ();
// have the media factory create the video element
www.dsp-ip.comFast Forward Your Development
Var
Media Element = media factory create media element ( new URL resource )
LayoutsMedia container
Layout metadata:
www.dsp-ip.comFast Forward Your Development
Controls how a media element is
Displayed in the media container
Media element
Layout metadata
Width
Height
X
Y
>
Layouts
www.dsp-ip.comFast Forward Your Development
>
Can be absolute or relative
Layout metadata
// Create a Layout metadata object
Var layout = new Layout metadata( ) ;
// set the width and height of the Layout metadata object
Layout . width = 640
Layout height =480
// tie the Layout metadata instance to the
www.dsp-ip.comFast Forward Your Development
// tie the Layout metadata instance to the
Media element instance
Media element . add metadata (Layout metadata. LAYOUT_MANESPACE, layout ) ;
Media element
Netloader
www.dsp-ip.comFast Forward Your Development
Netloader
HTTP Streaming NetLoader
RTMPDynamic Streaming NetLoaderNet Loader
Netloader
www.dsp-ip.comFast Forward Your Development
DVRCastNetLoader
Change
var video element : video element
= new video element ( resource ) ;
var video element : video element = new video element (
To
www.dsp-ip.comFast Forward Your Development
var video element : video element = new video element ( resource , new RTMPDynamicstreamingNetLoader ( ) )
Media Element
Vector.<DynamicstreamingReousorcecletem>
www.dsp-ip.comFast Forward Your Development
Dynamic Streaming
// create a new Dynamic streaming Resource pointing to FMS application
Var resource : Dynamic streaming Resource = new
Dynamic streaming Resource ( RMTP_ URL ) ;
// create a Vector to store the
Dynamic Streaming Item object
www.dsp-ip.comFast Forward Your Development
Var vector : vector . < Dynamic Streaming Item > =
New vector . < Dynamic Streaming Item > ( 3 ) ;
// create 3 entries in the vector, each a DynamicStreamingItem , with a file name and a minimum bit rate
Vector ( 0 ) = new DynamicStreamingItem
( “ 2012 _ high “ , 1500 ) ;
Dynamic Streaming
www.dsp-ip.comFast Forward Your Development
( “ 2012 _ high “ , 1500 ) ;
Vector ( 1 ) = new DynamicStreamingItem
( “ 2012 _ low “ , 400 ) ;
Vector ( 2 ) = new DynamicStreamingItem
( “ 2012 _ medium “ , 600 ) ;
// set the StreamingItems property on the
DynamicStreamingResource to our vector
Resource. StreamingItems = vector ;
// create a new VideoElement , based on the resource
Var videoElement = new Video Element ( resource ) ;
Dynamic Streaming
www.dsp-ip.comFast Forward Your Development
Var videoElement = new Video Element ( resource ) ;
Dynamic Streaming in a few linesVar resource = new Dynamic Streaming resource ( RMTP_ URL ) ;
Var vector = new vector . < DynamicStreamingItem > ( 3 ) ;
Vector ( 0 ) = new DynamicStreamingItem
( “ 2012 _ high “ , 1500 ) ;
Vector ( 1 ) = new DynamicStreamingItem
( “ 2012 _ low “ , 400 ) ;
Vector ( 2 ) = new DynamicStreamingItem
www.dsp-ip.comFast Forward Your Development
( “ 2012 _ medium “ , 600 ) ;
Resource . StreamItems = vector ;
Var VideoElement = new VideoElement (resource ) ;