embedded streaming media with gstreamer and...
TRANSCRIPT
![Page 1: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/1.jpg)
1
Embedded Streaming Media with
GStreamer
and BeagleBoard
Presented by Todd Fischer
todd.fischer (at) ridgerun.com
![Page 2: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/2.jpg)
Agenda
BeagleBoard-XM multimedia features
GStreamer concepts
GStreamer hands on exercises
DMAI and GStreamer
Questions
![Page 3: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/3.jpg)
What's New
Performance
◦ Tuning not as critical
Streaming media not central to product
HD – more power for higher resolution
![Page 4: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/4.jpg)
AM3730 Architecture
Multimedia features:
◦ Cortex A8 with Neon
◦ C64x+ DSP
◦ HD video accelerators
◦ How to utilize the
hardware features ?
![Page 5: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/5.jpg)
AM3730 Architecture
Cortex A8
◦ Neon
◦ Super-scaler
◦ Ghz clock
![Page 6: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/6.jpg)
AM3730 Architecture
C64+ DSP
◦ HD video encode /
decode
![Page 7: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/7.jpg)
AM3730 Architecture
Graphics Accelerator
◦ Dedicated hardware
◦ IVA – image, video
audio accelerator
◦ SGX accelerator
Supports OpenGLES
![Page 8: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/8.jpg)
GStreamer
Streaming media framework – audio and video
Close to 200 plug-ins available
Higher level than just input / filters / output
Networking, audio/video mixed streams, auto data
handling
Various options utilizing hardware accelerators
![Page 9: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/9.jpg)
GStreamer Overview
Elements
◦ Source, filters,
sinks
Bins and Pipelines
◦ Containers,
pipeline is the
overall bin
Pads
◦ Element source /
sink connection
points
Caps
Capabilities
organized by
stream type with a
set of properties
Plugin
Collection of
elements
![Page 10: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/10.jpg)
Hands On Exercise 0
Double click on
GStreamer Class icon
In terminal window, type
source ./s
Need the period
![Page 11: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/11.jpg)
Hands On Exercise 0
Run video pipeline
v1 Actual command
gst-launch videotestsrc ! ffmpegcolorspace ! xvimagesink
See script contents
cat v1 There are lots of scripts
ls
![Page 12: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/12.jpg)
Simple MP3 Player
Create dynamically using
gst-launch
Source element reads from a file
Filter element converts MP3 to PWM
Sink element passes to ALSA output
See script a0
![Page 13: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/13.jpg)
Simple Audio Player
Source Code
Create pipeline, source, filter, sink
◦ Set element properties
Build into pipeline
◦ Connect src and sink pads
Setup pipeline event handler
◦ End of stream
Set pipeline state to play
Run
See source a_gst.c
![Page 14: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/14.jpg)
Keeping Plug-ins Organized
Each known plug-in is added to registry
Most aspects of plug-in are tracked in the
registry
Registry support run-in pipeline creation
and dynamic filter selection
Use gst-inspect to list plug-ins
![Page 15: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/15.jpg)
Hands On Exercise 1
Using gst-inspect, list
◦ All plug-ins
◦ All video plug-ins
◦ Element properties for filesrc plug-in
![Page 16: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/16.jpg)
Hands On Exercise 2
GStreamer demultplexing pipelines
d5 – flash video
◦ First demultiplex into audio and videogst-launch filesrc location=sprc720.flv ! flvdemux name=demux
◦ Second, process audiodemux.audio ! queue ! mad ! alsasink
◦ Third, process video demux.video ! queue ! ffdec_vp6f ! omapdmaifbsink
Idea is the same
◦ source data, filter data, send data to sink
![Page 17: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/17.jpg)
GStreamer Daemon
Separates audio / video streaming from
controlling application
Uses D-Bus messages to control pipeline
Simplifies application development
– No interaction with Gstreamer API
Simplifies testing
– Test app just sends D-Bus messages
![Page 18: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/18.jpg)
Performance
Data Passing
Minimize data copies
Stream held in buffers with data,
timestamp, other info
When possible, buffer memory allocated
by sink pad
Use hardware when data copy is
necessary
![Page 19: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/19.jpg)
Performance
Data Transformation
Cortex A8 compiler optimization
NEON
◦ Single Instruction Multiple Data
C64+
◦ Video accelerator
DMA and other data movers
![Page 20: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/20.jpg)
Performance
Scheduling GStreamer elements may not be tuned
for embedded use model
Decoder may starve output device
◦ Noticeable audio clicks
Adjust buffering to pace entire pipeline
Adjust thread priority
![Page 21: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/21.jpg)
Davinci Multimedia
Application Interface
![Page 22: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/22.jpg)
DMAI and GStreamer
Davinci Multimedia Application Interface
◦ Exposes OMAP/Davinci hardware using high
level of abstraction
Stream audio / video
Graphics display
Hardware optimized frame/data copy
![Page 23: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/23.jpg)
Sitara Codec Engine
Isolates users for audio/video codecs
from those implementing the codecs
Codec can run in several places without
the calling application being aware
◦ Cortex A8, NEON, C64, hardware
accelerator
◦ Uses DSPLink and DSPBios conventions to
support DSP based algorithms dynamically
![Page 24: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/24.jpg)
Convenience Video
The power of the AM3730
◦ Streaming audio / video can be added to most any
product
Example: stream from DM365 Leopard Board
365v4l2src ! dmaienc_mpeg4 ! rtpmp4vpay ! udpsink
BeagleBoard XMudpsrc ! rtpmp4vdepay ! ffdec_mpeg4 ! omapdmaifbsink
![Page 25: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/25.jpg)
DSPLink
ARM9 C64
![Page 26: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/26.jpg)
GStreamer in
3 Layers
![Page 27: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/27.jpg)
Backup Slides
• DSPLink presentation
![Page 28: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/28.jpg)
DSPLink
ARM / DSP Communication
GPP
App
DSP
Task
Shared
MemoryDSPARM
Data
App
Control
DSPLINK Control
See diagrams in
DSPLINK
Programmers
Guide
![Page 29: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/29.jpg)
DSPLink
Communication Modules• Notify
– Low frequency
communication
– Small messages
MSGQ
single reader
multiple writers
Variable size
messages
Fixed buffer size
MPLIST
multiple readers
multiple writers
priority order
processing
![Page 30: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/30.jpg)
DSPLink
Communication Modules
• CHNL
– Single reader
– Single writer
– Fixed size buffers
– Legacy SIO
– Simplified buffer
handling
RingIO
Single reader
Single writer
Low
reader/writer
coupling
variable data
creation/
consumption
Independent
execution
![Page 31: Embedded Streaming Media with GStreamer and …beagle.s3.amazonaws.com/esc/GStreamer-esc-chicago-2010.pdfHands On Exercise 0 Run video pipeline v1 Actual command gst-launch videotestsrc](https://reader036.vdocument.in/reader036/viewer/2022070616/5d18834b88c99313688dcc73/html5/thumbnails/31.jpg)
DSPLINK
Support Modules• PROC
– hardware setup
– DSP code load and
boot
– ARM/DSP
communication
– DSP shutdown
POOL
Manage shared
memory
Allocate / free
Address
translation
Cache alignment