qtdd11_qtmultimediakitonmobile
DESCRIPTION
Slides from my talk at Qt Developer Days 2011, "Getting the most out of QtMultimediaKit on mobile"TRANSCRIPT
![Page 1: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/1.jpg)
Getting the most out of QtMultimediaKit on mobile
Gareth Stockwell
![Page 2: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/2.jpg)
Accenture
• It’s not just a management consultancy ...• I work for a division called Accenture Mobility
– Provides Mobility aspects of end-to-end solutions– Work on many mobile/embedded platforms
• Android, iOS, MeeGo, Symbian, Windows Phone ...
– In a wide range of industry sectors• Healthcare, automotive, retail, consumer electronics ...
– Some of us even get to hack on Qt
![Page 3: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/3.jpg)
About me
• Based in London, UK• 6 years in embedded software
– Mostly on Symbian– Mostly developing multimedia adaptations, lately some
graphics
• 2 years developing Symbian port of Qt– Phonon MMF backend– QAudio backend– QtMultimediaKit, mostly around video rendering
![Page 4: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/4.jpg)
Conventions
• Source code for the demos shown is available at
$DD11/examples/foo
• Logos show which platform(s) a topic or example applies to
https://gitorious.org/garethstockwell-qtdevdays-2011/mainline
• Path within the repository is shown as
![Page 5: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/5.jpg)
Overview
• QtMultimediaKit provides cross-platform abstractions– So why do you need to care whether you are targeting a
mobile device?– Where are the limitations and leaks in those
abstractions?
• How can you tune your code so that it works well in the constrained mobile environment?
![Page 6: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/6.jpg)
Mobile considerations
Latency
Less processing power
Reliance on co-processors
Multiple connectivity options
Limited memory
Cool demos
Low latency processing
Access point control
Memory management and
monitoring
Portable – always in your pocket
![Page 7: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/7.jpg)
Achieving low latency
![Page 8: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/8.jpg)
Sources of latency
Resourceallocation
Buffering Playback
time
Media playback use case
Real-time use cases
![Page 9: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/9.jpg)
Media playback: startup latency
Resourceallocation
Buffering Playback
NoMedia Loading
Loaded Buffering Buffered
audio.source = “foo.mp3” audio.play()
MediaStatus
m_audio->setMedia(...);
• Startup latency can be minimised by pre-loading media sources where possible
• Buffering latency is not controllable via QtMultimediaKit high-level APIs
QML
Native m_audio->play()
![Page 10: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/10.jpg)
Media playback: startup latency
Demo $DD11/audioplayer
Pure QML app
![Page 11: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/11.jpg)
Low level audio APIs
QIODevice
QAudioInput
QIODevice
QAudioOutput
QIODevice *QAudioOutput::start();
void QAudioOutput::start(QIODevice *device);
Push
Pull
Easy to use – no need to implement QIODevice
Requires an extra buffer copy from client data into the audio stack
+
-
Best choice for low-latency processing
![Page 12: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/12.jpg)
Real time audio latency
Buffering Playback
time
Impulsee.g. User input
Real-time data generation
void QAudioOutput::setBufferSize(int value);
• Keep buffer size to a minimum– But beware of underflow
• This is controllable via an API
– But this isn’t supported on Symbian– Here, you can control the buffering instead via
the source QIODevice
qint64 QIODevice::bytesAvailable() const;
![Page 13: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/13.jpg)
Real time audio latency
Demo $DD11/qpiano
Hybrid QML / C++ app
Guess what it does ...
![Page 14: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/14.jpg)
Network streaming
![Page 15: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/15.jpg)
Access point selection
• By default, the access point will be chosen by the platform– May involve popping up a dialog for the user
• Apps may want programmatic control– For application-specific reasons (e.g. video chat protocol
won’t work well over laggy cellular networks)– To simplify the UX
QList<QNetworkConfiguration> QNetworkConfigurationManager::allConfigurations(...) const;
void QMediaPlayer::setNetworkConfigurations( const QList<QNetworkConfiguration> &configurations);
QNetworkConfiguration QMediaPlayer::currentNetworkConfiguration() const;
![Page 16: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/16.jpg)
Access point selection
Demo $DD11/networkplayer
Hybrid QML / C++ app
No access point selection QML
bindings provided in QtMultimediaKit
Network configuration stateUndefined
Defined
Discovered
Active
![Page 17: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/17.jpg)
Memory managementand monitoring
![Page 18: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/18.jpg)
GPU memory constraints
• Current S^3 devices have only 32MB graphics memory– This is shared by the graphics subsystem (GLES and VG
engines) and the multimedia subsystem (video and camera)
– Combination of heavy graphics use cases (e.g. storing large QPixmaps or textures on the GPU) with MM (e.g. playback of high-resolution video clips; starting camera viewfinder) can quickly exhaust the memory
• It can be difficult to understand whether a given bug is due to GOOM– There is an API via which GPU memory usage can be
queried– There are some strategies which can be used to
minimise GPU memory usage
• The good news: the next generation of S^3 devices, starting with the C7-01, have 128MB
![Page 19: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/19.jpg)
Memory usage monitoring
• EGL_NOK_resource_profiling– Provides total memory, total used memory, and per-
process usage– API is not very friendly (particularly when compared to
Qt APIs)– So I wrote a Qt wrapper
class GraphicsMemoryMonitor : public QObject{ Q_PROPERTY(qint64 totalMemory READ totalMemory NOTIFY totalMemoryChanged) Q_PROPERTY(qint64 usedMemory READ usedMemory NOTIFY usedMemoryChanged) Q_PROPERTY(qint64 currentProcessUsage READ currentProcessUsage NOTIFY currentProcessUsageChanged) ...};
$DD11/snippets/graphicsmemorymonitor
![Page 20: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/20.jpg)
Memory usage monitoring
Total usage
Current process usage *
Total memory
• Not all memory allocations are correctly tagged– Here, the 3.84MB is the EGL window surface– Memory consumed by the video decoder is not
represented– So the only reliable measurement is the total usage
Demo $DD11/videoplayer
![Page 21: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/21.jpg)
m_camera->setCaptureMode(QCamera::CaptureModeVideo);
Mimimising camera memory usage
Demo $DD11/viewfinder
• The image capture ISP firmware module consumes several MB– So if you only need a viewfinder, switch
to video capture mode
– You may also need to lower the viewfinder resolution
QMediaRecorder *mediaRecorder = new QMediaRecorder(m_camera);QVideoEncoderSettings videoSettings = mediaRecorder->videoSettings();videoSettings->setResolution(QSize(320, 240));mediaRecorder->setEncodingSettings(audioSettings, videoSettings);
![Page 22: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/22.jpg)
Advanced video rendering
Cool stuff alert
![Page 23: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/23.jpg)
Video post-processing with GLSL
• If QAbstractVideoSurface returns texture handles, we can use QGLShaderProgram to apply effects
• From Qt 4.7.4, this becomes much easier, as we can embed GLSL directly in QML via ShaderEffectItem
ShaderEffectItem { property variant source: ShaderEffectSource { sourceItem: anItem; hideSource: true } property real granularity: 10 property int targetSize: 256
fragmentShader: " uniform highp float granularity; uniform sampler2D source; uniform highp float targetSize; uniform lowp float qt_Opacity; varying highp vec2 qt_TexCoord0; void main() { vec2 uv = qt_TexCoord0.xy; float dx = granularity / targetSize; float dy = granularity / targetSize; vec2 tc = vec2(dx*(floor(uv.x/dx) + 0.5), dy*(floor(uv.y/dy) + 0.5)); gl_FragColor = qt_Opacity * texture2D(source, tc); }"}
![Page 24: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/24.jpg)
Video post-processing with GLSL
• Possible use cases– Video playback transition effects (e.g. pixellated fade-in)– Camera ‘ageing’, applied both to viewfinder and captured
images
Demo $DD11/qmlvideofx
Pure QML app (!)
![Page 25: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/25.jpg)
Video and Qt3D
• Qt3D provides the Material abstraction, which can be a wrapper around a texture– But at present, it doesn’t abstract
texture streams
• With a bit of C++, we can provide the glue which pipes QAbstractVideoSurface output into a Qt3D material– Requires definition of some new
QML elements
import QtQuick 1.0import QtQuick3D 1.0import VideoTexture 1.0import TextureStream 1.0
Rectangle { width: 600 height: 600
VideoTexture { id: video source: "test.mp4” }
Viewport { Cube { id: cube } }
TextureStream { source: video target: cube }}
![Page 26: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/26.jpg)
Video and Qt3D
Demo Source not available yet, sorry...
Hybrid QML / C++ app
C++ provides the glue which sticks
QtMultimediaKit to Qt3D
![Page 27: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/27.jpg)
Summary
• Low latency– Startup– Steady state
• Network access point control
• Graphics memory usage– Monitoring– Reduction
• Advanced video rendering– Shader effects– 3D
![Page 28: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/28.jpg)
Feedback
Remember to send your session feedback via the Qt Developer Days app
Get the app by• Tapping one of the NFC tags on the event
floor• Downloading the “Qt Developer Days” app
from Nokia Store• Downloading from
qt.nokia.com/qtdevdays2011• Visiting m.qtdevdays2011.qt.nokia.com to
use web version
![Page 29: qtdd11_qtmultimediakitonmobile](https://reader035.vdocument.in/reader035/viewer/2022081602/54b6875f4a79598c308b4597/html5/thumbnails/29.jpg)
Questions?Thanks for listening