GStreamer Multimedia Framework

Part 2

2

Contents

Description

Part 1 Introduction to GStreamer

Part 2 GStreamer Plugin Internals

Part 3 Advanced GStreamer Concepts

Part 4 Concepts of GStreamer Application

Part 5 An Example Application Code

Important GStreamer Plugin Components

Overview

4

The GStreamer Element

• An Element in GStreamer is the most important class of objects. GStreamer plugins that have been discussed so far are all Elements.

• Element performs one defined specific function. Example Decode MP3 Parse AVI file Render PCM audio Read from file

• A sequence or combination of elements is used by GStreamer to achieve a defined multimedia scenario. Such a sequence of elements is called a graph or pipeline. Examples of multimedia scenarios could be: Playback an audio video stream Record a video stream using camera

• The element is represented by GstElement abstract data type. All plugins are derived from this data type.

5

Element Object Hierarchy

GObject

GstObject

GstElement

The fundamental GObject data type provided by GLib

The GStreamer’s base object type. Adds the following functionalities on top of Gobject:

1. Reference counting2. Name3. Locking mechanism for synchronisation

The GStreamer’s Element object type. Defines the plugin interface which every module has to implement to qualify as a GStreamer plugin

6

sink pad

Pads

• Pads belong to Elements and they are the ports through which data flows in and out of elements.

• GStreamer constructs graph of multimedia components by connecting the elements at their pads.

• Apart from read/write properties of elements, pads are the only interface through which data is given to and taken out from elements

• In any given pad, data can flow in only one defined direction. Pads are thus classified in to two types based on the direction of data flow Sink pad: receives data Source pad: generates data

element src padReceives Data Outputs Data

7

Element & Pad configurations

sink pad

sink element source element src pad

sink pad filter element

src pad

sink pad

demuxerssrc pad

src pad

sink pad

sink pad

multiplexerssrc pad

8

Pad classification based on availability

Pad Classification

Description Examples

Always Pads Static pads that are present always. They are created along with the element itself.

SRC pad of an MP3 decoderSINK pad of a 3GP parser

Sometimes Pads Dynamic pads that could be added to an element at run time. Application can come to know of a pad being added through the “pad-added signal.”

SRC pad(s) of any file format parser elements such as 3GP parser or AVI parser. Audio SRC pad may be added only if the audio track is detected in the media file

Request Pads Dynamic pads that are added to an element based on request from application.

SRC pad(s) of a tee element

9

Element chaining through pads

alsasrc src sink

mp3 encoder

src sink

mp3 file format writer

src sink filesink

Direction of Data Flow

10

Bin

• A bin is a container which can hold other elements• The advantage of using a bin is that, it allows for a set of related elements to

be grouped together and treated as a single logical element• From a programmer’s perspective Bin is also a Element, i.e. it derives from the

GstElement class. This allows for GStreamer core to treat a bin and a normal element as the same. This also allows for a bin to be included in to another bin.

GObject

GstObject

GstElement

GstBin

11

video bin

Bin as a container

• Bin does not have a pad of it’s own. • A pad belonging to an element within the bin can be made to appear as a pad

for the bin. This is known as a ghost pad

sinkdivx

decodersrc sink

colour converte

r

src sink

xvimage sinksink

Ghost Pad

12

Pipeline

• In GStreamer, a pipeline is a predefined and special type of bin.• Pipeline is the top most level of bin. In other words, a pipeline cannot be added

in to another bin.

GObject

GstObject

GstElement

GstBin

GstPipeline

13

Features of Pipeline

• A generic bin allows for logical grouping of elements and enables the core to treat a set of elements as a single element

• In addition to this feature, a pipeline adds the following functionalities Takes care of scheduling Selects and distributes a clock which will be used by all elements Provides a bus which can be used by applications to get message

notifications• A GStreamer application needs to have at least one pipeline element.

The top level bin in a graph has to be a pipeline.

14

Video/Audio (Media) Plane

Pipeline

Audio Sink

Audio Decode

rQueue

Graphics Plane (Menus and Labels)

X Window

GDK

Component Architecture Example

Media Player Application

GTK GStreamer Framework Core

File Src Parser

QueueVideo

Decoder

Video Sink

Concept of CapabilitiesAll about media representation and

negotiation

16

GstCapsHolds an array of objects of type GstStructure

“Caps” : GStreamer’s Solution for Media Representation • GStreamer is introducing the concept of capabilities or caps in short as an efficient

mechanism for media representation and negotiation.• GStreamer uses the data type GstCaps to represent capabilities• GstCaps is implemented as an array of objects of type GstStructure • Each GstStructure object has a collection of key/value pairs• Each Key/Value pair can be of different type (either fundamental or derived)

Collection of key/value pairs

GstStructure GstStructure GstStructure GstStructure

Each GstStructure represents one media

type

17

Caps …contd.

• The key in each key/value pair is generally a string representing a field or property (e.g. samplingrate, channels, etc).

• The value can be of any data type. Example: INT, FRACTION, BOOLEAN, FLOAT, STRING

• GStreamer adds new types to represent values in the form of Ranges (Int, float and fraction) Lists (Arrays) FOURCC codes

• Looking at the hierarchy of objects used in Caps data type, it may seem to be difficult to construct, manage and retrieve objects of GstCaps data type. However, GStreamer provides utility functions to convert from GstCaps structure to string representation and from string representation to GstCaps structure. Some sample APIs include gst_caps_new_simple gst_caps_from_string gst_caps_to_string

18

Capabilities of an Element

• In a GStreamer element, as the data passes only through pads of the element, the media handling capability of the element is entirely described by the caps associated with it’s pads.

• Element capabilities: The type of media that the element can receive as input is identified by the caps

associated with the sink pad(s) of the element The type of media that the element can generate as output is identified by the caps

associated with the source pad(s) of the element• GStreamer caps can be classified in to following based on functionality

Template caps Allowed caps Negotiated caps

• GStreamer caps can be classified in to following based on content Full caps Simple caps Fixed caps

19

Caps classification based on its content

• Full caps is the most generic. It… Contains more than one GstStructure object within, i.e. contains more than

one media type The values for the various properties (keys) in each GstStructure object can

be either a range or list of values or a single/distinct value• Simple caps is a specialized case of full caps, where in

There is only one GstStructure object within, i.e. contains the description of only one media type

The values for the various properties (keys) for that GstStructure object can be either a range or list of values or a single/distinct value

• Fixed caps is a specialized case of simple caps, where in There is only one GstStructure object within, i.e. contains the description of

only one media type The values for the various properties (keys) for that GstStructure object

need to have only single/distinct values (no ranges or lists)

20

Caps classification based on functionality

• Template caps: Applicable to a single pad Represents the complete set of media types (universal set) that a pad is designed to

handle. Template caps would be of the form “full caps.” Examples:

sinkaudio sink

audio/x-raw-int, channels={1, 2}, rate=[16000, 96000], width=16, depth=16,audio/mpeg, mpegversion=1, layer=3, channels={1, 2}, rate=[16000, 96000]

Template caps of sink pad

21

Caps classification based on functionality

• Allowed Caps: Applicable to a connection between pads of two plugins Represents the set of media that can flow between two plugins. This will typically be an

intersection of the template caps of the pads at each end of the connection. Allowed caps would also be of the form “full caps.”

sinkaudio sink

mp3 decoder src

audio/mpeg, mpegversion=1, layer=3, channels=2, rate=[8000, 48000]

audio/x-raw-int, channels={1, 2}, rate=[8000, 96000], width=16, depth=16,audio/mpeg, mpegversion=1, layer=3, channels={1, 2}, rate=[8000, 96000]

Template Caps Template Caps

audio/mpeg, mpegversion=1, layer=3, channels=2, rate=[16000, 48000]

Allowed Caps on this Link

22

sinksink

Caps classification based on functionality

• Negotiated Caps: Applicable to pads Represents the set of media that the pad is currently configured to handle based on the stream

being processed. Pads are said to be in negotiated state when an agreement has been reached on what would be the final media type that would flow through the pipeline.

Negotiated caps would be of the form “fixed caps.”

sinkaudio sink

mp3 decoder srcfilesrc src

mp3 parser src

audio/mpeg, mpegversion=1, layer=3, channels=2, rate=44100

Negotiated Caps

23

What flow through pads?

• There are two kinds of information that flows between plugin elements through the pads Events Buffers

• Buffers contain the actual multimedia data that the pipeline is created to handle in the first place

• Events are light weight structures that carry control information Data start indicator event (new segment event) End of stream event Flush events Seek request events

24

Terminology: Upstream and Downstream

Plugin 1 Plugin 2 Plugin 3 Plugin 4

• Downstream: left to right Example downstream flow

‐ Data (buffer) flow‐ Events such as flushes, end of stream indicator, beginning of data flow indicator

• Upstream: right to left Example upstream flow

‐ Seek request events

Downstream

Upstream

States in GStreamer ElementsOverview

26

GStreamer Element States

STATE DESCRIPTION

VOID STATE Invalid state type

NULL STATE First state when element is just created. Initialised from GObject perspective, but not yet initialised from GStreamer perspective.

READY STATERun time memory requirements have been allocated. Capabilities are not yet negotiated and the plugin is not yet aware of the kind of media that would be flow.

PAUSED STATEPlugin is aware of the final media type that is to be streamed. Plugin is ready to process any data that flows through it. Pipeline clock is not running.

PLAYING STATEPlugin is processing data. Pipeline clock is running.

27

State Transitions

1. NULL STATE

2. READY STATE

3. PAUSED STATE

4.PLAYING STATEUpward State Transitions

Downward State Transitions

NULL TO READY

READY TO PAUSED

PAUSED TO PLAYING

PLAYING TO PAUSED

PAUSED TO READY

READY TO NULL

28

Difference between PAUSED State and PLAYING State

• For SINK elements In paused state

‐ No data is being rendered to peripheral. Internal rendering threads, if any, will be in paused/blocked state.

‐ Clock is not running In playing state

‐ Data rendering to peripheral is in continuous flow‐ Clock is running

• For NON-SINK elements There is absolutely no difference between paused and playing state for

non-sink elements. Non-sink elements are required to process data even in paused state.

Scheduling ModesPlugin operation choices

30

Push mode

• The most generic case is where one element pushes data to the next down stream element connected on its source pad

MP3 Decoder Audio Sink

GStreamer Framework Core

src sink

1. Audio Sink is designed to passively receive data on its sink pad, i.e. implement chain function

Data Flow Management

Logic

2. MP3 decoder pushes data buffer on its source pad

3. Core forwards data to sink plugin through its sink pad

4. Audio sink’s chain function is invoked with buffer as argument

31

Pull Mode

• A plugin operating in pull mode will request data from upstream plugin. Upstream plugin will not send data by itself and responds only when requested.

File Source 3GP Demuxer

GStreamer Framework Core

src sink

Data Flow Management

Logic

1. filesrc plugin is designed to support requests for data to be retrieved on its source pad, i.e. implement get_range function

2. 3GP demuxer asks for data of given size at given offset to be pulled on its sink pad

3. Core forwards request to filesrc plugin

through its src pad

4. filesrc’s _get_range() function gets called

5. When the sequence of synch apis return, the demuxer will have data

32

Scheduling mode requirements

• For push mode, The downstream plugin should support _chain() function, i.e. it should

passively receive data The upstream plugin will call the api gst_pad_push() to push data on its src

pad• For pull mode,

The upstream plugin should support _get_range() function, i.e. it should support requests for data retrieval

The downstream plugin will call the api gst_pad_pull_range() to pull data through its sink pad

Thank You