20090114_unipv_nomadik
TRANSCRIPT
Nomadik Application Processor
Andrea GalloGiancarlo Asnaghi
January 14, 2009
MP3 PortablePlayer
Set Top Box DVD Player
ST is #1 world-wide leader inDigital TV and Consumer Audio
Digital Satellite RadioDigital Car Radio
MMDSP+ inside more than 200 million produced chips
January 14, 2009
ST leader in mobile phone chips
January 14, 2009
Nomadik
Nomadik is based on this heritage providing:
– Unrivalled multimedia performances– Very low power consumption– Scalable performances
January 14, 2009
Best Application Processor 2004Best Application Processor 2004
Nominees:
Intel XScale PXA260, NeoMagicMiMagic 6, Nvidia MQ-9000, STMicroelectronics Nomadik STn8800, Texas Instruments OMAP 1611
Lowest power consumptionScalable performanceVideo/Audio qualityCost-effective
January 14, 2009
Nomadik
Nomadik is a family of Application Processors
– Distributed processing architecture ARM9 + multiple Smart Accelerators
– Support of a wide range of OS and applications
– Seamless integration in the OS through standard API drivers and MM framework
January 14, 2009
roadmap
...
January 14, 2009
Some Nomadik products on the market...
January 14, 2009
STn8815 block diagram
January 14, 2009
Nomadik : a true real time multiprocessor platform
SDRAM
•Unlimited Space•Limited Bandwidth
NAND Flash
•Unlimited Space•“No” Bandwidth•Mass storage
Mem
ory Controller
GeneralPurposeSystemDMA
ARM926(L1 + L2)
Master OS
MMDSP+ Audio133 MHz, 24-bit
RTOSMulti-thread
MMDSP+ Video66 MHz, 16-bit
RTOS(Scheduler FSM)
Peripherals
multi-layer AHB bus
SRAM
(Level 2Cache
for Video)
HW Acceleration
Audio Peripherals Video Peripherals
January 14, 2009
Context & task relation: Logical view
User side
System side (kernel)System calls
Drivers &
Kernel components
GUI, User Applications
Application engines
& serversMultiple
Address Space
HW/IO process
ARM boundary (WinCE & Linux OS type)
SAA boundarySVA boundary
Single
Address Space
Flat
Address Space
Virtual world
Physical world
IO process
IO
IO
January 14, 2009
Picture Post Processing
January 14, 2009
Resized: zoom out (preview)
Examples of Video Resize filters targeting Mobile TV
Regular: display 1:1 scale
Wide: zoom to full height
Panoramic crop and zoom to full width
Stretch zoom to full width and height
January 14, 2009
Rotate
January 14, 2009
Hardware/Software partitioning
Handled by Hardware.No flexibility, high computation
Handled by Software on the host CPU.High flexibility, low computationOS friendly
Block
Macro-Block
FirmwareGroup of Macro-Block
Picture
Group of picture
Sequence
MPEG4 software stack
AR
M S
WH
W/F
W
January 14, 2009
MM integration in an open OS
Application (Player,…)
SourceFilter
Codec
Audio
Sink
MM Framework(plug-ins based)
Audio
Filter/Codec
Video
Filter/CodecVideo
Sink Virtual Media pipeline
Integration/Adaptation layer
SAA HW Driver (control path)
SAA
SVA HW Driver (control path)
SVA
User side
Kernel side
Video
Codec
Post-
Processing
Audio
Codec
Display FrameBuffer
PCM if
January 14, 2009
t1 t2 t3 t4 t5
Concurrent execution
ARM
SAA
SVA
Read & Parse
Audio Decode
Video Decode
Audio Playback
Video Display
Read & Parse
Audio Decode
Video Decode
Audio Playback
Video Display
Read & Parse
Audio Decode
Video Decode
Audio Playback
Video Display
Read & Parse
Audio Decode
Video Decode
Read & Parse
January 14, 2009
Power Management support
~200mA
~mA
~µA
Standby mode: ARM in WFI mode (Wait For Interrupt)
• PLL1 & 2 are running
• All peripheral clocks are gated
Power Save mode: • ARM in WFI mode
• PLL1 OFF, PLL2 Running
• CPU & Bus clocked at XTAL frequency
• All peripheral clocks unused are gated
RUN mode:
Full clock speed Doze mode: • ARM in WFI mode
• PLL1 & 2 OFF, clocks = 32 kHz
• All peripheral clocks are gated
Sleep mode: • ARM in WFI mode
• PLL1 & 2 OFF, clocks are stopped except the non-switchable part @ 32 kHz
• All peripheral clocks are gated
Deep-Sleep mode:- Main part of the logic is no more powered- The non switchable part is clocked @32 kHz
Wake up:3ms
Standby mode: ARM in WFI mode (Wait For Interrupt)
• PLL1 & 2 are running 64-393MHz
• All peripheral clocks are gated
Power Save mode• ARM in WFI mode
• PLL1 OFF, PLL2 Running 19MHz
• CPU & Bus clocked at XTAL frequency
• All peripheral clocks unused are gated
RUN mode:Full clock speed
64 – 393MHz Doze mode: • ARM in WFI mode
• PLL1 & 2 OFF, clocks 32 kHz
• All peripheral clocks are gated
Sleep mode: • ARM in WFI mode
• PLL1 & 2 OFF, clocks are stopped except the non-switchable part @ 32 kHz
• All peripheral clocks are gated
Deep-Sleep mode:- Main part of the logic is no more powered- The non switchable part is clocked @32 kHz
Wake up:3ms
SW architectureLinux
January 14, 2009
Screenshots and pictures
WVGA 800x480 @ 24bpp Philips TPO display
20%CPU
3%CPU
January 14, 2009
Average CPU load
VGA 30fps 2MbpsXine FFMPEG 10-20%GSTLaunch MPEG4 27%GSTLaunch H264 37%
Playbin +20pp
OpenGL-ESpanel carousel demoARM load 3%
Khronos compliance ongoingLauterbachETM traceH264 VGA
30fps 2Mbps AAC
LauterbachETM trace
MPEG4 VGA 30fps 2Mbps MP3
January 14, 2009
System overview for STn8815
Full open source Linux 2.6.20 kernel
OpenedHandPokyLinux for system test
In line with GMAE, Limo and Maemo
Power Management
cmp K+ architecture
Kernel and driver level
Middleware
Multimedia Interfaces
+ ALSA+ V4L2+ X.11+ xvideo+ EXA+ DirectFB
Multimedia Driv ers
+ SAA Driver+ SGA Lib and Driver+ SVA Driver
GStreamer
+ GStreamer+ OpenMAX-IL+ FFMPEG+ SAA Block Library
OpenGL-ES 1.1
+ Clutter+ EGL+ OpenGL Driver+ OpenGL-ES 1.1+ GLES Library
Graphic User Interface
+ Matchbox+ GTK+
Communication
+ Connection Manager+ Bluetooth+ Wi-Fi+ Ethernet+ Telephony
MEMS
+ Gesture Library+ MEMS Driver
Security
+ Cryptoengine driver+ IPSec+ OpenSSL+ Security
Camera
+ Camera ISP+ ISP Driver
Application Framework
+ Application Framework+ Azingo+ Google Android+ Access ALP+ Trol ltech Qtopia+ OpenMoko+ Poky
Kernel and Driv ers
+ DMA+ I2C+ Linux kernel 2.6.20+ MSP+ SPI+ UART
Multimedia HCL
+ SAA HCL+ SGA HCL+ SVA HCL
Power Management
+ PM framework+ Governor+ Battery Manager+ CPUFreq+ Linux DPM+ APM+ LiMo PM
Connectiv ity
+ SD/MMC+ USB Hi-Speed
File systems
+ JFFS2+ YAFFS+ CRAMFS+ RAMFS+ MTD
January 14, 2009
A/V features at a glanceVideo
– MPEG4 / H.264 decoder up to VGA 30fps 2Mbps– MPEG4 encoder up to VGA 30fps– H.263 codec– JPEG codec
Audio– MP3, AAC / eAAC+, WMA– G.711, G.723, G.726, G.729– AMRnb / AMRwb– Full duplex noise and acoustic echo cancelling– Effects, e.g. equaliser, reverberation, spatial sound
January 14, 2009
Multimedia interfaces
cmp Multimedia Interfaces
V4L2
xvideo
ALSA X.11
EXA
DirectFB
Multimedia Drivers::SGA Lib and Driver
Multimedia Drivers::SVA Driver
Kernel and Drivers::MSP
«use»«flow»
«use» «use»
«use»«use»
Standard Linux interfaces
Nomadik drivers
January 14, 2009
Graphic and video subsystem2D Graphics
– X.11 EXA for copy, solid fill, compositing
– Good acceleration up to 3x on big pixmaps
– Penalties for smaller primitives ARM + L2C is much faster
3D Graphics– ARM for EGL and
parsing– SVA+FPU for TnL– SGA for rendering– Good fit for smart GUI,
not for 3D gamesVideo rendering
– Xvideo extension with hw acceleration in SVA
– Resize, rotate, colourconversion and blitting
January 14, 2009
Multimedia with FFMPEGcmp Karakum
Matchbox
SVA driverSAA Driver
SGA Driver
BlockLibSGA user lib
PMP application Xine
ALSA
Xvideo
EXA
Xserver
GTK+
Web Browser -Minimo
FFMPEG video codecs
FFMPEG audio codecs
AudioCodec
Linphone
V4L2 API
frame buffer
«flow»
«use» «use»«use»
«use»
«use»
«use»
«flow»
«use»«flow»
«flow»
«use»
«flow»
«flow»
«flow»
«flow»
«flow»«flow»
«flow»
«flow»
«flow»
«flow» «flow» «flow» «flow»
«use»
January 14, 2009
cmp Static View
INTEGRATION
FRAMEWORK
APPLICATION
NOMADIK HW
KERNEL
XVideo
«Gstreamer Based»Player App
OMX::OMX_Core
OMX::OMX_AudioDecComponent
OMX::OMX_VideoDecComponent
GSTREAMER_OMX::GstOmxMp3Dec
GSTREAMER_OMX::GstOmxMpeg4Dec
GSTREAMER::GstXVImageSink
GSTREAMER::GstAlsaSink
SAA User Lib::Blocklib
Linux DDrivers::SAA driverLinux DDrivers::ALSA driver Linux DDrivers::SVA driver
ST_FFmpeg::ffmpeg libavcodec
Nomadik HW::SAA
Nomadik HW::SVA
Nomadik HW::Audio Codec (ex. 5095)
Nomadik HW::Frame Buffer
«instantiate» «instantiate» «instantiate» «instantiate»
PCM_Data_Buffer
«flow»«flow»
GStreamer and OpenMAX layers
January 14, 2009
Google Android on Nomadik
LIBRARIES
Media Framework
FreeType
SQLite
WebKit
SSL libc
APPLICATION FRAMEWORKActivity
Manager
Package Manager
WindowManager
Telephony Manager
Content Providers
Resource Manager
View System
Location Manager
Notification Manager
XMPP Service
APPLICATIONS
Home Contacts Phone Browser . . .
LINUX KERNEL
Display Driver
USB Driver
Camera Driver
Keypad Driver
BluetoothDriver
WiFi Driver
Flash Memory Driver
A/V Driver
Binder (IPC) Driver
Power Management
ANDROID RUNTIME
Dalvik Virtual Machine
Core libraries
OpenGL|ES
SGL
Surface Manager
ModemDriver
Crypto Engine Driver
January 14, 2009
Multiple clipping video rendering
Minimise bandwidth usage when not needed!!!
R1
R2 R3
January 14, 2009
Thread Model
omx mpeg4 msg thread omx mp3 msg threadOMX mp3 dec threadOMX mpeg4 dec thread
Play bin thread
gst player
gst media parser
gst-omx mpeg4 dec plugin
gst-omx mp3 decplugin
ffmpeg mpeg4 dec ffmpeg mp3 dec
SAA block lib
Linux SAA driverLinux SVA driver
ALSA libXVLib
gst XVImageSink gst ALSA sink
Input file
omx mpeg4 deccomponent
omx mp3 deccomponent
omx mpeg4 decmsg handler
omx mpeg4 decbuf manager
omx mp3 dec msghandler
omx mp3 dec bufmanager
January 14, 2009
Power ManagementCPUfreq to scale voltage and frequencyCPUIdle to switch amongst running, idle, sleep and deep sleepCustom API to control clock gating to all internal IP’sFeasibility on-going with DPM
Kernel Space
Constraint Manager
/sysfs
PowerTop
Drivers and Bus Drivers
Driver Interface
CpuIdle
/sysfs
Dynamic Tick (Wakeup event)
cpufreq
/sysfs
Governor
«flow»
«use»
January 14, 2009
DVFS – Dynamic Voltage Frequency Scaling
stm Operating Points PH1
489.6/163.2
Initial
393.6/131.2
326.4/163.2
264.0/132.0
100.8/100.8201.6/100.8
Boot
(lab only)
1.0V
1.2V
1.2V
1.2V
1.26V
1.45V
MHzMHz
MHz
MHz
MHz
MHz
January 14, 2009
PM Dump Tool
PM monitor to optimise operating modes and ULP, estimate consumption
– (Under development)
UARTSDMMCMCDETVOUTAUDIOSAASVABTWIFILANCPU 0CPU 1
I (mA) 100 95 20 50 10 100 95 20 50 10 100 95 20V 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9Clock 393 393 64 166 19 393 393 64 166 19 393 393 64
35510ms 35530ms 35550ms 35570ms 35590ms 35610ms 35630ms
Thank you!