The move from a hardware centric product design

towards a software centric product design:

Gstreamer as a driving forceMarc Leeman

@den_erpel

Introduction

● The Early Days● First Paradigm shift● A new technology● At the start of the new millennium● A typical 2000 design● What changed?● A new world● Gstreamer, the early days● Gstreamer, 2nd try● Gstreamer, wider use● Gstreamer, from here on out● Change does not come easy● A Changed way of working● New challenges

The Early Days

● Barco: Belgian American Radio Company● Established in 1934● Focus: Radios

● Vacuum tubes● Long Lasting ● In those days, buying a radio was costly● A status symbol

First Paradigm Shift

● 2nd World War● Refocus of the industry towards the War Machine

● Radios were restricted● Control the information● Were hidden and penalties were severe● Jamming signals were (partly) aimed for detection

A new technology

● The advent of television● Started back in the 30s, halted for 5 years● Multi standard (PAL, SECAM, NTSC)

● Content driven● Window on the world● Not unlike 16:9 or HD● UHD?

● Real breakthrough after exhibition of 1958● From the mid-60s onwards, start of to focus on industrial projects

At the start of a the new millennium

● Technologies last long● Non disruptive changes● Improving on existing, slow move to new

● Competence centre in optics● RGB projectors

● Full chain development with exception of CRT● Components out

A typical 2000 design

● Core of the design is hardware accelerated● High density solutions● FPGA at the core of the processing

● “The rest are peripherals”● The rise of DSP solutions● Uc as a controller

● Software as network controller● Management software provides logical access point● Hides 100s of smaller devices

● Boxes each containing 10-20 PCBs● Each processing up to 4 network streams

What Changed?

● Long lead times● Full chain development, not flexible during design● Maintenance is much higher once product is introduced

● Custom stack; all development is integrated, very much tailored to “exceptions”

● Component life time● Industrial range● Expensive● Underperforming wrt cutting edge

● Industry is not driving technology, but consumer● Mobile... mobile... mobile...

Mobile?

● Before: research was mainly on:● Focus on power efficiency● Focus on performance within constrained device

● Android...● Java and AMM is not efficient● Contradicts previous focus

● Solution was not to make the software better but to beef the specs of the devices● Same story as compression research● More memory, deeper memory hierarchy● Faster cores, more cores

A new world

● Suddenly, a lot of new processor cores are flooding the market● > 1GHz is the norm● > 2 cores is the norm

● Sounds awfully familiar to GPP● Difference between GPP software and 'embedded' software is

decreasing faster than ever before

● Gstreamer as a typical example

GStreamer, the early days

● Initial attempts on MPC8349 core● uClibc based

● uClibc proved very bad thread behaviour (meant for controller tasks)● unfair comparison: compared with MPC8245 based 'limited' protocol support● uC core was not as fast as advertised

● Discontinued● YACS: Yet Another Custom Stack

● Modular design was retained● 2nd time to write a streaming system from scratch

● 2006

GStreamer, 2nd try

● Something stuck● modular design● open source (experience with Linux, uclibc, busybox, ffmpeg, X11, …)● YACS just felt wrong

● MGS: Media Gateway Server● x86-64 server based solution● removes much of the previous mentioned limitations

● Small team can support a wide range of systems● key was modularity● cooperation● peer review

● 2008

GStreamer, wider use

● Hard to not notice the advantages● Stability● Protocol support● Time-to-market● Flexibility

● Replacement of custom stack with GStreamer● Integration of custom codecs in GStreamer

● the power of an external API

GStreamer, from here on out...

● Established on x86● Once in userland, why the focus on one architecture

● NSLU2, SheevaPlug, Raspberry Pi, Odroid, ...

● Platform approach● re-use of experience and code● encoder and platform support

● For devices, added value is the same● Multi protocol support● Standard protocol support● Not so standard protocol support

Streaming,...or not?

● RAW video handling (loss-less)– Low latency

– High quality high bandwidth

● 1 GHz Device handling streams +10Gbps?– At first glance, it cannot be done

● Not for the pure data handling– But for the configuration of the blocks that do the handling

– Configuration scaler parameters● Caps negotiation..

V4L2V4L2 V4L2V4L2 V4L2V4L2

FPGAFPGA

LinuxLinux

V4l2src V4l2src V4l2sinkV4l2sink

GStreamer applicationCaps negotiation

Abstraction, Obstruction

● Raw, compressed: Oh, wibbly-wobbly, timey-wimey... ● Keep hardware contained

– As close as possible to the kernel

● Standardised interfaces– e.g. V4L2 for RAW handling

– Isolate frameworks behind standard interface● e.g. OMX

– No new Gstreamer modules● Extend and improve● e.g. uri handing

– e.g. Piggy back on V4L2 interface

Change does not come easy

● "GStreamer is slow"● "GStreamer adds overhead"● Typically from the 'established'

● Optimised code from day 1?

● Prove the opposite● GStreamer _can_ be slow● BUGS, not by design● Fix the problems

● Alsa● H.264 parsing

● fixing a bug can never be an excuse for re-writing a complete software stack

A Changed Way of Working

● The Linux kernel has been used for some time● Kernel is often very specific to the SoC/board● Kernel is linked to FPGA logic

● In contrast: Gstreamer allows more re-use● Userspace● 3rd party camera stacks can easily be transferred● Easy integration of codecs and codec implementations throughout

the divisions

Cooperation invites tooling

● Tooling is no longer an arbitrary choice– Loads of commercial “good and less good” tools

● In order to participate in a community, you need to follow the rules of that community– A similar way of working makes life easier

● Svn → git; deployment, ...

● Commercial solution is limited to sugar coating– e.g. layer on top of git

New Challenges

● The future is software● Training, training● Migrate from hardware to software

● Software is cheap● Anyone can create a network system

● Typical reflex● Protectionism

● Do not use GPL● Do not use LGPL● Do not use Open Source Software

New Challenges

● Luckily we past that phase before software became the dominating factor

● Cooperation is key● Focus on standards

● Step away from “our use-case is special”● Added value is in solutions, not components

● Is our added value really in supporting e.g. camera X support?● Keep moving...

● The return of the Garage start-ups?

In Practice

● Software design is now the core of product design– Used to be hardware driven– Board created, software was added

● Now we see inter-divisional architecture meetings– Cross checking and review of design

● Gstreamer provides a stable environment to develop in– Stable APIs

– Peer review (when submitted)– Common reference within the company– Easy to get external experts for temporary bottlenecks

●

In Practice

● Training is key– Excess in hardware designers

– Convert to Software designers

– Again, external training

● Focus is on adding value, not re-inventing the wheel

Questions?

www.barco.comwww.twitter.com/Barcowww.facebook.com/Barcowww.youtube.com/BarcoTV