ID 710C: Developing Smart Screens for Industrial Applications

Patrick Shelly

Field Application Engineer

Version: 1.1

QNX Software Systems

13 October 2010

2

QNX Software Systems Overview

3

Agenda

� QNX Overview

� HMI Technology Architecture

� Adobe Flash as a next-gen embedded HMI tool

� Challenges with HMI frameworks in embedded

� Reliability

� Performance

� Integrating the HMI to real-time components

� ActionScript extensions

� Persistent Publish-Subscribe

� Blending multiple apps into a seamless interface

4

QNX Core Product Offerings

ServicesRuntime Product Portfolio Aviage Middleware Portfolio

Development and productivity tools

QNX Neutrino RTOS (Micro

Kernel)

QNX Photon microGUI

Instrumented kernel

File systems

Multicore

Adaptive partitioning

High availability

Connectivity and

networking

Integrated Java

QNX Momentics IDE

Multicore visualization

Performance optimization

Memory and system

analysis

Performance and footprint

optimization

Photon application builder

Target information &

remote debug

Libraries and GNU tools

Support

Custom engineering

Professional Services

Training

QNX Aviage multimedia

solution

QNX Aviage HMI suite

(including Flash Lite 3)

QNX Aviage acoustic

processing

Voyager web browser

QNX has served the embedded market for over 30 years. Our products include the award winning QNX

Neutrino RTOS, QNX Momentics Eclipse based IDE, QNX Aviage middleware and professional services.

The Aviage Middleware portfolio includes a complete multi-media engine, a runtime environment for

Adobe Flash that allows Flash to be used in embedded environments and acoustic echo

cancellation/noise suppression technology.

4

5

Advanced Runtime Module

QNX Runtime Product Portfolio

Multimedia

Suite

QNX Neutrino

RTOS

QNX Neutrino

Extensions

QNX Aviage

Middleware

Instant

Device

Activation

Extended

Networking

(security)

Multi-coreAdaptive

Partitioning

Web

Browser

Java

Acoustic

Processing

HMI

Suite

QNX Neutrino RTOS Runtime

Core

File Systems

Peripheral

Hardware

Core

Networking

Core

OS

TDP/QNETHigh

Availability

2D/3D

GraphicsPhoton

The QNX Runtime portfolio includes the RTOS, graphics, networking, filesystems and a high availability

framework. Extensions include full symmetric multi-processing capability, CPU partitioning, instant on

(fast boot) and networking security protocols such as IPSec.

5

6

QNX Standards and Certifications

QNX continues to invest in standards and certifications. Most recently we added SIL3 and EAL 4+ to

respond to customer requirements. Neutrino is a POSIX operating system so Linux application code can

often be recompiled and reused without additional coding.

While an operating system cannot achieve FDA certification on its own, QNX has worked with many

medical customers to help them achieve FDA certification of their products.

6

77

QNX HMI Architecture

7

8 8

QNX HMI TechnologyQNX HMI Technology

Flash

Adobe FlashAdobe Flash

OS interfacesOS interfaces

QNX HMI Technology ArchitectureVertical Stacks

& Demos

Vertical Stacks

& Demos

Core Applications

WebKit engine

WebKit engine

HTML

App store

App store BrowserBrowserMedia

player

Media player

Widget

Examples

Widget

Examples

Graphics EngineGraphics Engine

OpenVGOpenVG

Screen ManagementScreen Management

Native and EGL interfaceNative and EGL interface

HW Accelerated Screen TransitionsHW Accelerated Screen TransitionsHW & Virtual Layer ManagementHW & Virtual Layer Management

Operating SystemOperating SystemQNX RTOSQNX RTOS

Video Render

Video Render

Video

OpenGL

Application

OpenGL

Application

3D native

OpenVG

Application

OpenVG

Application

2D native

Composition Manager

HW AccelerationHW Acceleration

Application framework (ActionScript 3 based)

Flash Services and widgetsFlash Services and widgets

Scroll ListScroll List ButtonsButtons SlidersSliders Soft keyboardSoft keyboard NavBarNavBar ......TransitionsTransitions ThemesThemes

Native OS ServicesNative OS Services

MediaMedia BrowserBrowser SQLSQL RemoteRemote ......Graphics LayersGraphics LayersPPSPPSApplication launcherApplication launcher

LoadingLoading SettingsSettings StateState ......ResizeResize

InputInput

Touch, keyboard, mouse Touch, keyboard, mouse

QNX CARQNX CARIndustrial

SMA Demo

Industrial

SMA DemoIndustrial

White Goods Demo

Industrial

White Goods DemoMedical DemoMedical Demo

OpenGL ES 1.1 & 2.0OpenGL ES 1.1 & 2.0

DocsDocs

Core Documentation

Generic DemoGeneric Demo

The QNX HMI architecture builds on the QNX Neutrino RTOS. We provide Open VG and OpenGL ES

and do take advantage of hardware acceleration wherever possible. The QNX composition manager

provides a framework for managing screens as well as input devices. The ActionScript 3 based

application framework allows developers to access native OS services.

8

9 9

QNX HMI Technology Components: Flash

� Flash Lite 4

� Hardware accelerated OpenVG (on supported platforms)

� ActionScript3 support

� Debugging capabilities

– X-Ray

� AS3 compatible extensions

– Webkit ASE – Browser

– mm-render ASE – video, audio playback

– PPS ASE –data distribution services

– QDB ASE – database access

– etc

� Native video decode/render for supported formats and platforms

� Complete HMI framework provided by QNX

� Apps, widgets, services, lifecycle management

The QNX HMI makes use of Flash Lite 4. Flash Lite 4 can be hardware accelerated using OpenVG when

the SoC supports this. ActionScript3 extensions include a webkit browser, multi-media, the QNX

Persistent Publish Subscribe services and database to name a few. In addition, a complete framework

provides developers with a broad selection of applications, widgets and OS services.

9

10

QNX HMI Technology Components: Webkit and Multimedia

� Webkit

� HTML5 support

– Includes new additions to HTML language for video, audio, geolocation etc

� Uses Skia standard for 2D drawing

� Current Webkit version 525; updating to version 533

� Improved components for user interaction

� High performance panning and zooming

� HTML rendering engine; browser app provided in Flash

� Multimedia

� Based on Aviage 1.2.1

– Providing audio/video playback without needing full framework

� New hardware encode/decode support for new targets

� Streaming or file-based video playback

In addition to the Flash HMI capabilities, the QNX environment integrates the Webkit browser. Currently

Webkit v525 is supported and v533 will be available soon. QNX Aviage Multimedia is also integrated. The

multimedia engine provides the ability to encode/decode audio and video and supports steaming and file-

based video playback.

10

Combine and overlay graphics from different render engines

Flash acts as a “Window Manager”

Based on standards

Action Script, OpenGL, OpenVG, OpenKode, HTML, JavaScript, POSIX

QNX HMI Technology Components: HMI Composition

CompositionComposition

FlashFlash

HTMLHTML

OpenVGOpenVG

OpenGLOpenGLVideoVideo

The framework includes the QNX Composition Manager. The Composition Manager allows developers to

combine and overlay graphics from a variety of different rendering engines and supports Action Script,

OpenGL, OpenVG and other technologies.

11

12 12

QNX HMI Technology Components: Tools and Documentation

� Tools� QNX Momentics Tool Suite plug-in for Flash Builder 4

� Sample Applications� Reference Applications

– Youtube, audio and video player, browser, weather� Widget use demonstration

� Documentation� ActionScript extension API� PPS documentation� Installation guide� Sample applications� Developers guide� API documentation

The framework also includes a QNX Momentics plug-in for Flash Builder 4 along with a number of sample

applications and documentation to increase developer productivity right from the start.

12

13

13

Adobe Flash in Embedded

� 1 billion mobile devices in 2009

� 400+ device models

Number constantly changing (800 million as of March ‘09, 500 million in Oct ‘08)

Even Apple who initially said they would refuse to support Flash on the iPhone has announced they would be

partnering with Adobe to release it on the iPhone.

14

14

Why Adobe Flash?

� Flash is the de facto HMI standard

� Flash player installed on*:

– 98% of US Web users

– 99.3% of all Internet desktop users

� Over one million graphics designers

worldwide

� Flash realizes the Java promise of Write Once, Run Anywhere

� Class library dependency problems do not exist as in Java

� Content developed for web or PC can run on embedded without change

� Embedded FlashLite player uses less memory and provides faster rendering

� Flash is ideal for HMI creation

� Graphical environment by nature

� Used by industrial designers to design look & feel

� Ability to create “executable specifications”*Numbers are estimates from Adobe, and depend on demographical & statistical info

Numbers are Adobe estimates, depends on demographics, etc., but still impressive

Java requires many assorted class libraries, so not all APIs are available on any arbitrary target.

Also graphical capability of Java isn’t consistent (AWT, Swing), or must hand build what Adobe has built-in.

15

15

From Design to Deployment, no Detours

Flash HMI Development1) Designer creates HMI in Adobe Flash

on PC2) .swf file from design process runs

directly on the target system3) Done.

Standard HMI Development1) Designer creates HMI in Adobe Flash on PC

2) .swf file from design process run by software engineers on PC as mock-up of system

3) Engineers attempt to duplicate behavior by recreating screens and graphics

a) Design all state logic from scratch

b) Use C/C++ code or another toolkit

c) Graphical assets not directly reusable

4) Engineers debug and get C/C++ code running on target

5) Design difficulties encountered in HMI are sent back to designers for rework with real hardware constraints

6) Engineers make adjustments to software & graphics for new design

7) Pass running system by design for verification

8) Errors in HMI duplication reworked and recoded

9) Repeat steps 4 – 8 as necessary

10) Done.

Flash greatly simplifies the development process, since most HMI designers use Flash as a design tool to capture

and disseminate their ideas. This cuts out time and cost from the development cycle.

16

16

Adobe CS3/CS4 Development Environment

� Timeline controls “movies” with object motion, state control, and interaction

� Many pre-scripted actions (effects, transitions, tweens, etc.)

� Both scripted (code) and non-scripted (timeline) behavior

� Graphical resources are primarily vector graphics

� Can include videos (.flv)

� Very well suited for Internet or connected development� remote connection, datasets

� web services� wsdl, xml, urls, etc.

Example screenshot of Adobe Creative Suite 3 in action. There is now a CS4 (announced in December 08), which could also be used. The development environment for flash is a different paradigm for your normal programmer, since it’s timeline driven—all actions are “frames” and the resulting program is a “movie”. They’re stored in an .fla, which is the flash extension for source, compiled down to a .swf (compiled flash). SWF actually stands for “Shock Wave Flash”, which is a historical name, I believe being the company that originally created Flash before it was sold off to Macromedia.

Flash uses a virtual machine (like the JVM) to run code, and that VM is the “Flash Engine” that we’ve ported to QNX. It is vector-based and designed around graphical animation, which is what guides many of the design decisions.

Flash development is not very linear, and is well suited to a graphics designer because of this “broken” programmer model. What I mean by “broken” is that the code is scattered all over the place, little bits of event driven stuff or different frames, and it’s not easy to get a view on someone else’s code just by looking—you need to poke around in the IDE to find all the little tidbits and discover why it behaves as it does.

You can do development also mostly in a “code” model, and in fact, that’s what our guys do. They use CS3 to create the framework (graphics, libraries consisting of code and images, resources, etc.) and then use third party compilers on “real source” to manipulate the resources (load, display, move them around, alter them, etc.). This is a little easier for a true programmer to handle, but is unlikely the way that a graphical designer would work.

For a good example of ActionScript and writing simple programs (as well as pointing out the difs between AS2 and AS3), check out this:

http://www.senocular.com/flash/tutorials/as3withflashcs3/#differences

1717

Challenges with HMI Frameworks Performance

18

Flash Lite in Embedded

� Adobe Flash Lite 3� Highly optimized implementation of the Flash runtime

� Specifically for embedded market

� Requirements in line with embedded� 32-bit data bus, 200MHz� 380KB core player DLL size (without video codecs)

� RAM requirements:– For user interface: 4–6MB– For standalone content: 2–4MB (excludes video)

– For web browsing: 32MB

� Advantages� Web/desktop Flash content to mobile and back again, with minimal effort

� Vector graphics adjusts to different screen sizes without loss of quality

� Vector graphics creates very small content files, compared to bitmaps

QNX has worked directly with Adobe to create a highly optimized implementation of the Flash runtime,

specifically for the embedded market. In keeping with the nature of resource constrained embedded

devices, this version minimizes the required processing while providing the full Flash experience.

18

19

How do you squeeze more speed out of Flash?

�Real-time Optimizations

�Use hardware acceleration

�Optimize Flash movies

�Move to native code

There are several ways to further optimize Flash Performance.

19

20

20

Non-Real Time Behavior

� Jerky HMI (Flash unable to keep up with frame rate)

� May need to raise priority of Flash Player thread or lower priority of other system activities

� Can move portions of HMI into separate native applications and use hardware graphics layers to merge into one display surface

� Can use Adaptive Partitioning to reserve processing time for HMI

� What about inability to meet real-time deadlines for other components?

� Won’t happen if thread priorities correctly set

� Robust OS should include:

– Automatic priority inversion protection

– Priority-based graphics synchronization

This is what happens when your Flash component may be taking up more CPU power than you anticipated, and

you're causing issues with other real-time components in the system. Well, this really shouldn't happen. The

reason it shouldn't happen is because you have the ability to organize the rest of your system based on priorities.

That means that the other components of your system, they should be set with adequate priority to make sure that

they're getting enough CPU time. The Flash environment is a single-threaded environment. That single-threaded

environment does mean that there is a single thread that that Flash is running. There are other threads that are

doing other things, but all the HMI stuff is being done with that one thread. As long as you have processes that are

running at a higher priority than that thread, then those processes will get the CPU if they need it. You really

shouldn't see any of this behavior in a properly configured system. The only reason that you might potentially see it

is because you haven't gone through the effort of changing your system priorities appropriately.

You can move portions of your HMI display into separate native applications and use that to actually like drive the

display. The example of the instrument cluster where we implemented the needles in OpenGL, we're able to get

real-time performance regardless of what the Flash engine does, even if it's doing out to sleep or you're doing

something very long and time-consuming, or if it's doing garbage collection or whatever else. We can guarantee

that the needles will be updated promptly and up-to-date, because they're being implemented and rendered by a

separate process. That separate process is merging the display using GF or graphics framework component as a

separate layer on that API. The other thing that you can do is use adaptive partitioning to reserve processing time

for the HMI. So adaptive partitioning does let you take this CPU time and carve it up into different chunks. You

can say I want 25% of it reserved for like the needle display or indicators or whatever else. Maybe another 30 or

40% is actually reserved for HMI development. So that Flash thread will get 40%. And then you can reserve the

remaining 25, 30, whatever percent to other system processes. And that way, you'll be guaranteed, even if you

haven't done the work of setting all your priorities appropriately, you'll be guaranteed of getting the appropriate

CPU amount of time.

One other thing to point out with regard to graphics is that because the graphics synchronization primitives are

based on priorities, so the way that IO display, which is our graphics component gateway basically, synchronizes

multiple applications right into the display is through Mutex. And that Mutex inherits the priority of the processes

that are running to it. And the reason that this is important is if you do for example have needles sharing the

display with Flash components, the needles are at a higher priority. What that means is that the needles will

automatically be the next in line to get that resource as it's required. So the automatic use of the graphics system is

pretty much dictated by how you've assigned the priorities. That just sort of falls out from that.

21

21

Graphics Hardware Acceleration

� QNX Aviage HMI Suite adds OpenVG hardware acceleration

to Adobe Flash

� OpenVG drawing accelerated by graphics hardware

� Transparent application—no changes to flash code

� Improvement depends on type of operation

� Some operations 14 times faster! (Continuous rescale of transparent vector image)

� Some operations slightly slower (alpha-blended bitmaps, which aren’t supported hardware)

� Improvement is expressed in either faster frame rate, less CPU used, or both

� Overall improvement is ~150%

Adobe Flash with OpenVG

-4

-2

-

2

4

6

8

10

12

14

16

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75

Benchmark Test

Pe

rfo

rma

nc

e Im

pro

ve

me

nt

Note that although OpenVG acceleration is transparent to the application (no changes to source or flash .fla’s

required), it doesn’t necessarily generate pixel-for-pixel identical output. Slight differences in scaling or anti-

aliasing may result, although there is no way that any subtle changes will be noticeable by a casual observer.

22

22

Minimize Software Alpha Blends

� Use a minimum of alpha blends (transparency)

� Byte operations per pixel: 6 loads, 6 multiplies, 3 adds, 3 stores

� Separate alpha channel (in addition): 1 load, 1 subtract

• α +

• α +

• α +

• (255-α)

=• (255-α)

=• (255-α)

=

23

23

Minimize Gradients

� Gradients require interpolation between multiple colors

� Determination of colors on either side of the pixel

� Calculation of interpolation percentage

� Multiply percentage against each of two colors

� Adobe provides complicated gradients

� Radial, linear (multiple point), bitmap, rainbow, angled, etc.

� Choose the simplest gradient that looks acceptable

� Use bitmap caching if your platform supports it

Performance can be increased by ensuring that excessive gradients are not used. Supporting gradients is

computationally intensive and by selecting the simplest gradient that is visually acceptable the CPU load can be

minimized.

24

24

Minimize Framerate

� Use the lowest acceptable frame rate

� 40-60 gives very fluid animation

� 20-25 provides acceptable

� 15 typical “lower-end” web recommendation

� 10-12 may be best target for embedded

� Frame rate determines speed of “movie” playback, and number of tweens (e.g. between key frame) generated per second

� More frames = more CPU, and more RAM/Flash

� Using multiple SWF movies (one that calls another), the framerate used is the caller’s

� Can lead to unanticipated (i.e. higher than expected) frame rate

� Easiest solution: use consistent frame rate across project

Obviously increasing framerate will consume more CPU cycles and will be more resource intensive. 60 frames per

minute provides very fluid animation but in many cases this is not a requirement for an embedded application.

2525

Integrating the HMI to Real-time ComponentsActionScript Extensions

26

Persistent Publish Subscribe (PPS)

� Subscriber and Publisher� Can be unknown from each other at compile time

� Can be dynamically substituted at runtime� Can have one-to-one, one-to-many, many-to-one relationships

� Do not use direct calling API; available to all languages, scripts, shells

� Can be in different languages, virtual machines, sandboxes

� Are asynchronous from each other

� Publisher� Can be configured how to persist data� Can support on-demand publishing (pull rather than push)

� Persists attribute data over reboots

� Subscriber� Are notified when data changes� Do not poll (block until delta update received)

� Can receive notification on changes within a publishing tree

Heat sensor

pll

EtherCAN

/fs/pps/heat, ethercan, pll

Temperature

MonitorMotor

Persistent Publish Subscribe provides a mechanism for processes to share data elements. The publisher

publishes its data and any process subscribing to the publisher is notified when the data changes.

Alternately subscribers can poll if you cannot afford to have a thread block awaiting a delta update). It is

an asynchronous scheme. Any number of subscribers can subscribe to any given publisher and any

number of publishers can exist at a given time (m:n relationship).

26

27

27

PPS Benefits

Attribute Feature Benefit

Applications loosely coupled to services

Object-based data publication

Asynchronous data push/pull

No polling required; unblocking notification interface

POSIX file namespace

Less brittle connection

Modifications to client or server won’t break interface

Implementation isolation (client doesn’t need to know who publisher is)

Language independent interface

Multi-view publishing Multiple objects can give differing views on same data source

Can have more than one publisher update different attributes of the same object

Easier for client’s most appropriate use

Can publish derived data

Can extend system and retain logical coherence

Client can get delta updates

Persistent state save Save to media of choice (NAND flash, NOR flash, Harddisk, USB, etc.)

Save on shutdown and system controlled points

Save attributes across system reboot

Retain object state for intermittent clients

The many benefits of a Persistent Publish Subscribe are detailed here.

2828

Blending Multiple Apps into a Seamless InterfaceGraphical Composition

29

29

Combining Flash and Native Applications

� Flash as the HMI Manager � Flash normally lives in a browser environment� For embedded applications, use Flash as the main graphical environment� Control of display pane� Foremost HMI layer

� Launch and control multiple native apps

� Applications blended with OpenKODE composition manager

Browser

OS

index.html

Flash Plug-in

main.swf

OS

Standalone Flash Player

main.swf

Flash and native applications can be combined in the HMI environment. Flash can be used as the main control

environment and launch and control native applications

30

30

Example HMI Decomposition

Here a Flash player provides a control interface with a number of buttons along the bottom and a status bar at the

top. A large window in the middle is left transparent. A 3D visualization program rendering a view of gears is

running. By overlaying the Flash control interface over the visualization program a unified view is created.

31

31

OpenKODE Window Transforms

� OpenKODE supports multiple ways to merge window content

� Positioning

� Rotation

� Scaling

� Alpha blend

� Chroma key

•EGL is another standard API (like OpenKODE), that we partially support in order to provide all the functions

needed to create windows. OpenKODE by itself is not sufficient.

•“Allows separation of application,…” What and/or why? This is to allow a critical process to share screen real

estate with other processes. An example is the Digital Instrument cluster in QNX CAR. The underlying app is

Flash (the gauges, indicators, and gadgets), where the needles are a separate OpenGL-ES process. Separation like

this allows you to make a safety critical system, even if not all parts of the system are of equal trust.

•QNX provides Core Graphics comprised of the GF 2D (QNX proprietary) and OpenGL-ES libraries, the

Composition Manager (OpenKODE and EGL pieces), Open VG (on some platforms) and Photon.

•In the diagram, the composition manager is represented by the “+” , since it combines the screen windows

together.

32

32

Why Develop an HMI Using Multiple Processes?

� Different safety/criticality requirements

� 99.999% uptime required for some pieces; lesser guarantee required on rest of system

� Separate into multiple processes for validation

� Require process isolation

� Create sandbox around untrusted code

� Take full advantage of existing software blocks

� Browser, 3D visualization, etc.

The importance of separating systems into multiple distinct processes is well understood in embedded development

and is often a key architectural consideration in the design phase. HMIs are no different.

3333

Smart Meter Reference Implementation

34

Smart Energy Reference Design – Menu Tree

Home Screen Weather Info

Lighting Control Video Surveillance

This is one example of how one can use Flash to build a comprehensive HMI. In this example QNX has

built a complete Smart Energy Home control panel to manage multiple zones within the home. This is

done using touchscreen technology and supports Apple like gestures.

34

35

Smart Energy Reference Design – More Screens

Energy Usage Detailed Monthly Energy Usage

Energy Management Editing Energy Rules

Here we are showing year over year energy usage, carbon offset (2016 trees) and a way to manage high

consumption appliances in a way that reduces costs by only running them in non-peak hours wherever

possible.

35

36

System Configuration

Network

Powerline

Communications

Insteon

Server

Dimmer

Controllers

IP Camera

Zigbee

Gateway

Utility

Simulator

Zigbee Sensors

Temperature & Humidity

Smart Meter

Load

Switches

Weather, Flickr,

Pandora, etc.

This provides a view of the overall home system. Communicating via a network (could be 802.11) the

home panel can control dimmers, load switches, remote surveillance cameras and access internet

services all from one location.

36

37

Reference Design Software Architecture

Flash Lite 3 Player

PPS SQL

Power

Scheduler

SQL

SQLite

Energy Usage

Database

PPS

Flash Player Extensions

Insteon

Lighting

PPS

HVAC

Controller

PPS

MPG4Mediaplayer

PPS

Load

Switches

PPS

Smart

Meter

PPS

Flash

Action Script

C/C++

Drivers &

Applications

Zigbee

Sensors

PPS

38

Thank You