open standards for today’s gaming industry · functions for mobile vision use external camera api...
TRANSCRIPT
© Copyright Khronos Group 2013 - Page 1
Open Standards for Today’s Gaming Industry
Erik Noreke VP of Business Development, Khronos Group
© Copyright Khronos Group 2013 - Page 2
The Topics for this Afternoon • Why do we need open standards for gaming and animation?
- What makes successful standards?
• Graphics Acceleration Standards
- Khronos acceleration API standards for low power and high performance gaming and
animation
• Bringing accelerated graphics into the Browser
- WebGL – revolutionary access to the GPU in standard browsers
• 3D Content Standards
- How to make content re-usable without stifling innovation
© Copyright Khronos Group 2013 - Page 3
Why Do We Need Standards? • Standards are interoperability interfaces so that compelling user experiences
can be created inexpensively to build mass markets
- Don’t slow growth with functionality fragmentation that adds no value
• E.g. Wireless and IO standards
- GSM/EDGE, UMTS/HSPA, LTE, IEEE 802.11, Bluetooth, USB …
Standards drive mobile market
growth by expanding device
capabilities
© Copyright Khronos Group 2013 - Page 4
Standards in the Real World
Vendor differences adding no value -
fragmentation is slowing growth – clear
goals emerge for a standard
REFINE BY COMMITTEE
Industry agrees on what to standardize
– cooperative refinement from multiple
viewpoints creates a robust solution
A good standard enables
implementation innovation
Darwinian industry is still
experimenting with what works
and what doesn’t
DESIGN BY COMMITTEE
Experimentation and design by
committee can be
slow and unfocused
A bad standard stifles innovation
and causes commoditization
Right time to Standardize?
Every successful open standard has a de facto proprietary
competitor and is open to competitive evolution
Ecosystems seem to work best when both are
healthy and evolving
© Copyright Khronos Group 2013 - Page 5
Busting Some Standardization Myths • “Standards are slow to develop”
- Time to productive ecosystems is the key …
… rather than minimizing time to a proprietary specification
- OpenCL 1.0 took just 6 months – intensive cooperation
• “If I particpate in standards I ‘lose’ my IP”
- Good IP Framework fully protects Members IP and the specification
- Members agree not to assert IP claims against other Members or Adopters
- License grant is VERY narrow – but protects the specification in practice
• “Standards are boring”
- A good standard is the industry coming together to solve real issues
© Copyright Khronos Group 2013 - Page 6
Khronos Connects Software to Silicon
ROYALTY-FREE, OPEN STANDARD APIs for
advanced hardware acceleration
Low level silicon to software interfaces needed on every platform
Graphics, video, audio, compute,
vision, sensor and camera processing
Defines the forward looking roadmap for
the silicon community
Shipping on billions of devices across
multiple operating systems
Rigorous conformance tests for
cross-vendor consistency
Khronos is OPEN for any company to
join and participate
Acceleration APIs BY the Industry
FOR the Industry
© Copyright Khronos Group 2013 - Page 7
Courtesy of PC Gaming Alliance
www.pcgamingalliance.org
© Copyright Khronos Group 2013 - Page 8
OpenCL – Heterogeneous Computing
• Native framework for programming diverse
parallel computing resources
- CPU, GPU, DSP – as well as hardware blocks(!)
• Powerful, low-level flexibility
- Foundational access to compute resources for
higher-level engines, frameworks and languages
• Embedded profile
- No need for a separate “ES” spec
- Reduces precision requirements
A cross-platform, cross-vendor standard for
harnessing all the compute resources in an SOC
OpenCL
Kernel
Code
OpenCL
Kernel
Code
OpenCL
Kernel
Code
OpenCL
Kernel
Code
GPU
DSP
One code tree can be executed on
CPUs, GPUs, DSPs and hardware.
Dynamically interrogate system load
and load balance work across
available processors
CPU
CPU HW
© Copyright Khronos Group 2013 - Page 9
OpenCL Overview • C Platform Layer API
- Query, select and initialize compute devices
• Kernel Language Specification
- Subset of ISO C99 with language extensions
- Well-defined numerical accuracy - IEEE 754 rounding with specified max error
- Rich set of built-in functions: cross, dot, sin, cos, pow, log …
• C Runtime API
- Runtime or build-time compilation of kernels
- Execute compute kernels across multiple devices
• Memory management is explicit
- Application must move data from
host global local and back
- Implementations can optimize data movement
in unified memory systems
© Copyright Khronos Group 2013 - Page 10
OpenCL SPIR 1.2 Provisional released!
OpenCL Roadmap
OpenCL 2.0
Significant enhancements to memory and execution models to
expose emerging hardware capabilities and provide increased
flexibility, functionality and performance to developers
OpenCL SPIR (Standard Parallel Intermediate Representation)
LLVM-based, low-level Intermediate Representation for IP Protection and as
target back-end for alternative high-level languages
OpenCL HLM (High Level Model)
High-level programming model, unifying host and device execution environments through
language syntax for increased usability and broader optimization opportunities
OpenCL 2.0 Provisional released!
© Copyright Khronos Group 2013 - Page 11
Mobile OpenCL Shipping • Android ICD extension released in latest extension specification
- OpenCL implementations can be discovered and loaded as a shared object
• Multiple implementations shipping in Android NDK
- ARM, Imagination, Vivante, Qualcomm, Samsung …
© Copyright Khronos Group 2013 - Page 12
OpenGL 3D API Family Tree
OpenGL ES 1.0
OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
OpenGL 1.5 OpenGL 2.0 OpenGL 4.3 OpenGL 2.1
OpenGL 3.0
OpenGL 3.1
OpenGL 3.2
OpenGL 3.3
OpenGL 4.0
OpenGL 4.1
OpenGL 4.2
2002
OpenGL 1.3
ES-Next
GL-Next
OpenGL ES 2.0
Content OpenGL ES 1.1
Content
OpenGL ES 3.0
Content
ES3 is backward compatible
so new features can be
added incrementally Fixed function
3D Pipeline
Programmable vertex
and fragment shaders
WebGL 1.0
OpenGL 4.4 is a
superset of DX11
WebGL-Next
Desktop 3D
Mobile 3D
OpenGL 4.4
© Copyright Khronos Group 2013 - Page 13
OpenGL ES 3.0 Highlights • Better looking, faster performing games and apps – at lower power
- Incorporates proven features from OpenGL 3.3 / 4.x
- 32-bit integers and floats in shader programs
- NPOT, 3D textures, depth textures, texture arrays
- Multiple Render Targets for deferred rendering, Occlusion Queries
- Instanced Rendering, Transform Feedback …
• Make life better for the programmer
- Tighter requirements for supported features to reduce implementation variability
• Backward compatible with OpenGL ES 2.0
- OpenGL ES 2.0 apps continue to run unmodified
• Standardized Texture Compression
- #1 developer request!
© Copyright Khronos Group 2013 - Page 14
DirectX 11.1
2004 2006 2008 2009 2010 2005 2007 2011
Accelerating OpenGL Innovation
DirectX 10.1
OpenGL 2.0 OpenGL 2.1 OpenGL 3.0
OpenGL 3.1
DirectX 9.0c DirectX 10.0 DirectX 11
OpenGL 3.2
OpenGL 3.3/4.0
OpenGL 4.1
Bringing state-of-the-art functionality to cross-platform graphics
2012
OpenGL 4.2
OpenGL 4.4
2013
OpenGL 4.3
© Copyright Khronos Group 2013 - Page 15
OpenGL 4.3 Compute Shaders • Execute algorithmically general-purpose GLSL shaders
- Can operate on uniforms, images and textures
• Process graphics data in the context of the graphics pipeline
- Easier than interoperating with a compute API IF processing ‘close to the pixel’
• Standard part of all OpenGL 4.3 implementations
- Matches DX11 DirectCompute functionality
Physics AI Simulation Ray Tracing Imaging Global Illumination
© Copyright Khronos Group 2013 - Page 16
OpenGL Conformance Test Suite released!
Conformance submissions are required for GL 4.4 implementations encouraged for earlier driver versions
Shared codebase with OpenGL ES 3.0 CTS additional desktop-specific tests
Core profile functionality
Enhancements underway to add more coverage
© Copyright Khronos Group 2013 - Page 17
OpenCL and OpenGL Compute Shaders • OpenGL compute shaders and OpenCL support distinctly different use cases
- OpenCL provides a significantly more powerful and complete compute solution
Enhanced 3D
Graphics apps
“Shaders++”
Pure compute
apps touching
no pixels
Compute Shaders
1. Full ANSI C programming of
heterogeneous CPUs and GPUs
2. Utilize multiple processors
3. Precisely defined IEEE accuracy
1. Fine grain compute operations
inside OpenGL
2. GLSL Shading Language
3. Execute on single GPU only
Imaging
Video
Physics
AI
© Copyright Khronos Group 2013 - Page 18
Active Gamers by Platform
Courtesy of PC Gaming Alliance
www.pcgamingalliance.org
© Copyright Khronos Group 2013 - Page 19
Visual Sensor Revolution • Single sensor RGB cameras are just the start of the mobile visual revolution
- IR sensors – LEAP Motion, eye-trackers
• Multi-sensors: Stereo pairs -> Plenoptic array -> Depth cameras
- Stereo pair can enable object scaling and enhanced depth extraction
- Plenoptic Field processing needs FFTs and ray-casting
• Hybrid visual sensing solutions
- Different sensors mixed for different distances and lighting conditions
• GPUs today – more dedicated ISPs tomorrow?
Dual Camera LG Electronics
Plenoptic Array Pelican imaging
Capri Structured Light 3D Camera PrimeSense
© Copyright Khronos Group 2013 - Page 20
OpenVX • Vision Hardware Acceleration Layer
- Enables hardware vendors to implement
accelerated imaging and vision algorithms
- For use by high-level libraries or apps
• Focus on enabling real-time vision
- On mobile and embedded systems
• Diversity of efficient implementations
- From programmable processors, through
GPUs to dedicated hardware pipelines
Open source sample
implementation
Hardware vendor
implementations
OpenCV open
source library
Other higher-level
CV libraries
Application
Dedicated hardware can help make vision
processing performant and low-power enough
for pervasive ‘always-on’ use
© Copyright Khronos Group 2013 - Page 21
OpenVX - Power Efficient Vision Acceleration • Create vision processing graph for power and performance efficiency
- Each Node can be implemented in software or accelerated hardware
- Nodes may be fused by the implementation to eliminate memory transfers
• EGLStreams can provide data and event interop with other APIs
- BUT use of other Khronos APIs are not mandated
• VXU Utility Library provides efficient access to single nodes
- Open source implementation – easy way to start using OpenVX
OpenVX Node
OpenVX Node
OpenVX Node
OpenVX Node
Heterogeneous
Processing
Native
Camera
Control
© Copyright Khronos Group 2013 - Page 22
OpenVX and OpenCV are Complementary
Governance Open Source
Community Driven No formal specification
Formal specification and full conformance tests
Implemented by hardware vendors
Scope Very wide
1000s of functions of imaging and vision Multiple camera APIs/interfaces
Tight focus on hardware accelerated functions for mobile vision Use external camera API
Conformance No Conformance testing
Every vendor implements different subset Full conformance test suite / process
Reliable acceleration platform
Use Case Rapid prototyping Production deployment
Efficiency Memory-based architecture
Each operation reads and writes memory Sub-optimal power / performance
Graph-based execution Optimized nodes and data transfer
Highly efficient
© Copyright Khronos Group 2013 - Page 23
Typical Imaging Pipeline • Pre- and Post-processing can be done on CPU, GPU, DSP…
• ISP controls camera via 3A algorithms
Auto Exposure (AE), Auto White Balance (AWB), Auto Focus (AF)
• ISP may be a separate chip or within Application Processor
Pre-processing Image Signal Processor
(ISP)
Post-
processing
CMOS sensor
Color Filter Array
Lens
Bayer RGB/YUV
App
Lens, sensor, aperture control 3A
Need for advanced
camera control API!
© Copyright Khronos Group 2013 - Page 24
Camera Control API Goals • Provide functional portability for advanced camera applications
- Reduce extreme fragmentation for ISVs wanting more than point and shoot
• Generate image bursts with parameterized camera control and ISP control
- For downstream processing by flexible combination of CPU, GPU and DSP
• Control multiple sensors with multi-sensor synch and alignment
- Stereo pairs, Plenoptic arrays, Depth Cameras
• Enable system-wide sensor time-stamping
- Synchronize MEMS and image sensor samples
• This functionality is not available on any current platform APIs
- Make this API align with future platform direction for easy adoption
© Copyright Khronos Group 2013 - Page 25
Camera API Design Philosophy • C-language API starting from proven designs
- e.g. FCAM, Android camera platform
• Design alignment with widely used hardware standards
- e.g. MIPI CSI
• Focus on mobile, power-limited devices
- But do not preclude other use cases such as automotive, surveillance, DSLR…
• Minimize overlap and maximize interoperability with other Khronos APIs
- But other Khronos APIs are not required
• Provide support for vendor-specific extensions
Apr13
Jul13
Group charter approved
4Q13
Provisional specification
1Q14
First draft specification
2Q14
Sample implementation and
tests
3Q14
Specification ratification
© Copyright Khronos Group 2013 - Page 26
Low Power Environment Scanning • Many sensor use cases would consume too much power to be running 24/7
- Environment aware use cases have to be very low power
• ‘Scanners’ - very low power, always on, detect things in the environment
- Trigger the next level of processing capability
ARM 7 1 MIP and accelerometers can
detect someone in the vicinity
DSP Low power activation of camera
to detect someone in field of view
GPU GPU acceleration for precision
gesture processing
© Copyright Khronos Group 2013 - Page 27
Sensor Industry Fragmentation …
© Copyright Khronos Group 2013 - Page 28
StreamInput Sensor Fusion Stack
OS Sensor OS APIs (E.g. Android SensorManager or
iOS CoreMotion)
Low-level native API defines access to
fused sensor data stream and context-awareness
…
Applications
Sensor Sensor
Sensor
Hub Sensor
Hub
StreamInput implementations
compete on sensor stream quality,
reduced power consumption,
environment triggering and context
detection – enabling sensor
subsystem vendors to increased
ADDED VALUE
Middleware (E.g. Augmented Reality engines,
gaming engines)
Platforms can provide
increased access to
improved sensor data stream
– driving faster, deeper
sensor usage by applications
Middleware engines need platform-
portable access to native, low-level
sensor data stream
Mobile or embedded
platforms without sensor
fusion APIs can provide
direct application access
to StreamInput
Hardware transport
interfaces are defined
by each system, e.g.
IIO or HID sensor
© Copyright Khronos Group 2013 - Page 29
Khronos APIs for Virtual Reality
Advanced Camera Control and stream
generation
3D Rendering and Video
Composition
On GPU
Audio
Rendering
Application
on CPUs, GPUs
and DSPs
Sensor
Fusion
Feature
Tracking
MEMS
Sensors
Camera Control
API
EGLStream Stream frames between APIs
Precision timestamps
on all sensor samples
VR needs not just advanced sensor processing, vision
acceleration, computation and rendering - but also for
all these subsystems to work efficiently together
© Copyright Khronos Group 2013 - Page 30
Worldwide Game Users - 2012
0
100
200
300
400
500
N.America Europe Asia Other
WORLDWIDE GAME USERS BY REGION AND PLATFORM (IN MILLIONS)
Console PC Dedicated Handheld Mobile
Courtesy of PC Gaming Alliance
www.pcgamingalliance.org
© Copyright Khronos Group 2013 - Page 31
Leveraging Proven Native APIs into HTML5 • Khronos and W3C liaison
- Leverage proven native API investments into the Web
- Fast API development and deployment
- Designed by the hardware community
- Familiar foundation reduces developer learning curve
Native APIs shipping
or Khronos working group
JavaScript API shipping,
acceleration being developed
or work underway
WebVX? Vision
Processing
WebCAM(!) Camera
control and
video
processing
Possible future
JavaScript APIs or
acceleration
WebStream? Sensor Fusion
Native
JavaScript Canvas
Path Rendering
Camera
Control
HTML
© Copyright Khronos Group 2013 - Page 32
Content
JavaScript, HTML, CSS, ...
WebGL Implementation Anatomy
JavaScript Middleware
HTML5
JavaScript CSS
Browser provides WebGL functionality
alongside other HTML5 technologies
- no plug-in required
OS Provided Drivers. WebGL on Windows
can use Google Angle to create conformant
OpenGL ES 2.0 over DX9
OpenGL ES 2.0 OpenGL
DX9/Angle
Content downloaded from the Web.
Middleware can make WebGL accessible to
non-expert 3D programmers
© Copyright Khronos Group 2013 - Page 33
WebGL Availability in Browsers
- Microsoft – “where you have IE11, you have WebGL – turned on by default and working all the time” - Microsoft - WebGL also enabled for Windows applications - web app framework and web view - Apple - WebGL must be explicitly turned on MAC Safari and only exposed on iOS for iAds - Chrome OS - WebGL is the only cross-platform API to program the GPU - Google IO announcement - Chrome on Android will soon launch with WebGL
Much WebGL content uses three.js library:
http://threejs.org/
© Copyright Khronos Group 2013 - Page 34
C/C++
SDK Dalvik (Java)
Objective C C#
DirectX
HTML/CSS HTML/CSS HTML/CSS
Cross-OS Portability
HTML5 provides cross
platform portability. GPU
accessibility through
WebGL available soon on
~90% mobile systems
Preferred development
environments not
designed for portability
Native code is portable-
but apps must cope with
different available APIs
and libraries
© Copyright Khronos Group 2013 - Page 35
WebGL First Wave Application Categories • Maps and Navigation
• Modeling Tools and Repositories
• Games
• 3D Printing
• Visualization
• Music Videos and Promotion
• Education
• Photo Editors
• Music Visualizers
• Vision/Video Processing
© Copyright Khronos Group 2013 - Page 36
Google Maps • All rendering (2D and 3D) in Google Maps uses WebGL
© Copyright Khronos Group 2013 - Page 37
WebCL – Parallel Computing for the Web • JavaScript bindings to OpenCL APIs
- Enables initiation of Kernels written in OpenCL C within the browser
• Bindings stay close to the OpenCL standard
- Maximum flexibility to provide a foundation for higher-level middleware
- Minimal language modifications for 100% security and app portability - E.g. Mapping of CL memory objects into host memory space is not supported
• API definition underway – public draft released - https://cvs.khronos.org/svn/repos/registry/trunk/public/webcl/spec/latest/index.html
• Compelling use cases
- Physics engines for WebGL games
- Image and video editing in browser River Trail
JavaScript binding to OpenCL APIs to enable initiation of OpenCL C kernels
Data-parallel language extensions to JavaScript
for offload at run-time
© Copyright Khronos Group 2013 - Page 38
3D Needs a Transmission Format! • Compression and streaming of 3D assets becoming essential
- Mobile and connected devices need access to increasingly large asset databases
• 3D is the last media type to define a compressed format
- 3D is more complex – diverse asset types and use cases
• Needs to be royalty-free
- Avoid an ‘internet video codec war’ scenario
• Eventually enable hardware implementations of successful codecs
- High-performance and low power – but pragmatic adoption strategy is key
Audio Video Images 3D
MP3 H.264 JPEG ? !
An effective and widely adopted codec ignites previously
unimagined opportunities for a media type
© Copyright Khronos Group 2013 - Page 39
glTF Goals • Binary file format for efficient transmission for 3D assets
- Reduce network bandwidth and minimize client processing overhead
• Run-time neutral - DO NOT IMPLY OR MANDATE ANY RUN-TIME BEHAVIOR
- Can be used by any app or run-time – usually WebGL accelerated
• Scalable to handle compression and streaming
- Though baseline format does not include compression
• ‘Direct load efficiency’ for WebGL
- Little or NO processing to drop glTF data into WebGL client
• Carry conditioned data from any authoring format
- Prototyping and optimizing efficient handling of COLLADA assets
A standards-based
content pipeline for
rich native and Web 3D
applications Playback Authoring
© Copyright Khronos Group 2013 - Page 40
COLLADA and glTF Open Source Ecosystem
Tool Interop
Three.js glTF Importer. Rest3D initiative
COLLADA2GLTF
Translator
OpenCOLLADA
Importer/Exporter
and COLLADA
Conformance Tests
On GitHUB
Pervasive WebGL deployment
Other
authoring
formats
Web-based Tools
https://github.com/KhronosGroup/glTF
https://github.com/KhronosGroup/OpenCOLLADA
https://github.com/KhronosGroup/COLLADA-CTS
© Copyright Khronos Group 2013 - Page 41
Conclusion • Gaming and Animation is a complex and expanding application domain and
multiple standards across multiple domains are needed to enable the market
• Advances in SOC silicon processing and associated APIs are about to enable
Generated Reality to truly meet user expectations on all platforms
• Now is a great time to get involved with the standards initiatives
that effect your business
• www.khronos.org