erik noreke st ericsson | opensl es chair

© Copyright Khronos Group, 2009 -

Erik Noreke ST Ericsson | OpenSL ES Chair

© Copyright Khronos Group, 2005-2009 -

Over 3.5 Billion phones built based on our technologies

Offers the full range of access technologies from 2G to 3G to LTE

LET’S CREATE IT

Leading global supplier of platforms and semiconductors for wireless

Joint Venture owned equally by STMicroelectronics and Ericsson

A workforce of about 8000


•  Lots of fragmentation!

• Many proprietary audio APIs of varying functionality -  Even playing a simple sound on different platform requires different code.

• No standard way to access audio hardware acceleration -  Lots of work for developers to re-write code for every platform – no application/source code

level portability

• Newer multimedia devices incorporating more advanced audio functionality -  Increases in audio quality & functionality further complicate the efforts of content developers

aggravating the portability problem.

Solution - Open standard application level API for embedded audio !


High performance & low latency access to audio

features

Support a diversified market space of

embedded devices

• Diversified collection of embedded devices -  Target devices include “Mobile phones, personal media players & handheld gaming consoles”

• High performance, low latency access to audio • Application developer friendly -  Source-level portability of native code from platform to platform -  Same interface for both hardware and software solutions

• Royalty-free open standard

Open standard for embedded audio

Open Standard API for application developers enabling application

portability


• What about ALSA, OSS, GStreamer, OpenAL? -  OpenAL is targeted for desktop PCs -  OSS is obsolete, replaced by ALSA -  ALSA is Linux specific -  GStreamer is not designed to be hardware accelerated -  Are all released under variations of GNU Public License

-  They can provide the foundation for an OpenSL ES implementation

• OpenSL ES -  Designed specifically with embedded software and hardware in mind -  Designed to be hardware accelerated -  Pure API, independent of underlying implementation and architecture -  Royalty-free open standard – No copy-left implications


4Q07 1H08 2H08 1Q09

OpenSL ES 1.0 Provisional Release

Conformance Tests and OpenSL ES 1.0 Final

2Q09 3Q09

• OpenSL ES 1.0 released in provisional form in September 2007 -  Encouraged developer feedback before specification finalization

•  Final specification released March 2009 -  Full set of conformance tests available


•  Target devices include: -  Basic, Feature & Gaming mobile phones -  Personal Media Players & PDA’s -  Portable gaming consoles -  Just about any portable device

•  Target applications include -  User Interface sounds -  Music and Ring-tone playback -  Simple 2D and advanced 3D games -  Media recording and sequencers


• Playback of audio files -  Playback PCM and encoded content -  Good for sound effects; device UI sounds

•  SP-MIDI, Mobile DLS, Mobile XMF -  For interactive music and ring-tones

•  Effects & Controls -  Music and media player effects -  Advanced environmental effects for gaming

•  3D Audio -  Provided for gaming as companion to OpenGL ES

Doppler

MIDI messages

Preset Reverb

EQ Pitch

Volume

Buffer Queues Rate Metadata extraction

Environmental Reverb

Virtualization

Stereo widening LED & Vibra

3D position


• OpenSL ES 1.0 consists of: •  The specification -  58-page user guide providing a comprehensive overview of the API -  497-page API reference that goes into details of the objects and interfaces

• Header files -  Available to the public for download

• Conformance Tests -  Ensures application portability across implementations -  Passing required for implementations before adoption is accepted -  Adopters that pass the conformance tests are able to use the OpenSL ES logo

• Adopters Package -  Defines the process and requirements for a conformant implementation


Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record

functionality, simple 2D audio games

Game-centric mobile devices Advanced MIDI functionality, sophisticated audio

capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc.

Music-centric mobile devices High quality audio, ability to support

multiple music audio codecs, audio streaming support


•  Phone + Music = Phone combined with portable music player -  Advanced stereo ring tones -  Digital music playback

• Game + Music = Advanced music player / Musical games -  High quality music playback -  3D audio games with advanced audio effects

•  Phone + Game = Gaming phone -  3D ring signals -  Games utilizing advanced audio techniques

•  Phone + Game + Music = Ultimate Audio Rendering Device -  Multiple audio sources perceived -  Encapsulates the entire functionality of OpenSL ES


•  Same API for H/W and S/W solutions

Middleware, JSR-135, JSR-234, High-level Audio Libraries

Applications

Native Audio Solution

Media silicon (audio HW, CPUs, DSPs)

OpenMAX IL


• OpenSL ES has an object-oriented programming model -  Simplifies common use cases – but also extensible

•  Engine Objects are central to any OpenSL ES session -  Objects created using methods on the Engine Object interfaces

• OpenSL ES Objects enable PLAY and RECORD of audio -  Perform some operation on an input and emit the result as output -  Can handle almost any audio use case

• Objects have control interfaces -  Play, Seek, Rate, Audio, Metadata Extraction -  Record, Audio Encoder, MIDI

• Audio Objects input and output to Devices

OpenSL ES Audio Object URI

Memory

Audio Input

URI

Memory

Audio Mix

DSrc DSnk


•  Create Engine object -  To drive this session

•  Create Audio Output Mix object -  Method on Engine interface -  Mix object drives audio output devices

•  Create Media Player object -  Method on Engine interface -  Input is URI pointing to a local mp3 file -  Output drives audio output mix

•  Register event callback -  Method on Media Player interface

•  Set PlayState to Playing -  Method on Media Player interface

•  Wait for end of file event -  Via registered callback

Engine Object

Media Player Object

Output Mix Object

EngineItf

PlayItf

Play Event Callback

Application


Audio Player

SLPlayItf

SLPlayItf

SL3DLocationItf

URI

Memory

Buffer Queue

URI

Memory

Buffer Queue

Output Mix

Audio Player SLVolumeItf

SLVolumeItf


Overlap streaming media functionality uses consistent APIs

3D Audio

Audio Effects

Advanced MIDI

Audio Playback

Audio Recording

MIDI

Video playback

Camera

Video recording

Image capture & display Buffer

queues

Analog Radio & RDS

(Enhanced audio API) (Streaming Media APIs)


•  Independent -  No dependency between the APIs – either one can exist by itself -  A device may support any combination of the APIs that most suits the device: -  OpenSL ES only (Phone, Game, Music) -  OpenMAX AL only (Multimedia Playback/Recording) -  OpenMAX AL + OpenSL ES (Music + Multimedia Playback/Recording) -  OpenMAX AL + OpenSL ES (Game + Multimedia Playback/Recording) -  ….

• Compatible -  Working groups collaborated to make sure the APIs work together well

• Consistent -  Identical API architecture -  Identical APIs for same functionality

• Distinct -  OpenSL ES provides advanced audio functionality -  3D, audio effects and advanced MIDI

-  OpenMAX AL provides wide-range, basic multimedia functionality -  audio, video and image


Audio Player URI Output Mix

File Reader

Parser

Codec Audio Effect

Audio Effect Mixer

Sink


•  Same API regardless of underlying solution -  hardware accelerated -  software implementations

•  Supports JSR-135 and JSR-234 audio functionality -  Simplifies integration of third party JSR implementations -  Provides a stable foundation for Java support

•  Simplifies porting between platforms -  Spend application development time on application development, not porting

• Does for audio what OpenGL ES does for graphics -  Brings your applications and devices to life!


The OpenSL ES Workgroup

erik noreke st ericsson | opensl es chair

Documents