adobe session on flash online conference #12
TRANSCRIPT
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Video and Camera Support for Stage3D using Video TextureNimisha/Chandra Prakash | AIR Engineering
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Overview
2
How Stage3D fits within the Flash display model? Problem at hand: using video as a content in 3D scene Current method to fix the problem at hand New approach to solve the problem How to use the new approach? Use cases
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Stage3D: The stage behind the Stage
3
Set of special Stages that are specifically dedicated to 3D
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
How to fix : Bitmap representation of video
5
Problems Requires multiple copying of
bitmap Slow and Resource Intensive
Video rendered in RGB on a Video Object
Video rendered in RGB on a Video Object
Bitmap Representation of Video Bitmap Representation of Video
Create a TextureCreate a Texture
Update the texture with the bitmapUpdate the texture with the bitmap
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
New Approach - Video Texture
6
Directly access texture from netstream or camera
Directly access texture from netstream or camera
Advantages Source texture in stage 3D
pipeline Fast and Hardware accelerated
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Video Texture – How to use?
Confirm if VideoTexture is supported by Context3D class
7
if( stage.stage3Ds.length > 0 ){ var stage3D:Stage3D = stage.stage3Ds[0]; stage3D.addEventListener( Event.CONTEXT3D_CREATE, Context3DHandler ); stage3D.requestContext3D( ); } function Context3DHandler ( event : Event ) : void{ if (Context3D.supportsVideoTexture) { // We can create VideoTexture Object with Context3D object. }}
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Video Texture – How to use?
Create VideoTexture object and attach NetStream/Camera object to created VideoTexture object
8
var ns:NetStream;
var context3D:Context3D;
var texture:VideoTexture;
texture = context3D.createVideoTexture();
texture.attachNetstream(ns);
texture.addEventListener(Event.TEXTURE_READY, renderFrame);
ns.play("http://www.adobe.com/AIR/videoTexture/demo.mp4");
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Video Texture – How to use?
Video frame corresponding to current audio playback Callback for an event Event.TEXTURE_READY is fired Use an updated VideoTexture object with currently available video frame
10
var context3D:Context3D;
function renderFrame(e:Event):void
{
// Render on stage3D with VideoTexture
}
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Video Texture – How to use?
VideoTextureEvent object that works same with the StageVideoEvent event
11
texture.addEventListener(VideoTextureEvent.RENDER_STATE, onRenderState);
function onRenderState (event:VideoTextureEvent)
{
if ( event.status == VideoStatus.SOFTWARE) {
// Indicates software video decoding works.
}
if ( event.status == VideoStatus.ACCELERATED ) {
// Indicates hardware-accelerated (GPU) video decoding works.
}
if ( event.status == VideoStatus.UNAVAILABLE ) {
// Indicates Video decoding is not supported.
}
}
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Video Texture – Use Cases
12
Camera Input
In-game video call with shader effects
Efficient embedding of camera streams
Video Ads
In place Video Ads
In game video streams containing streamed content or live content or commercials
Credit: Waste Invaders
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Roadmap
Implementation of video texture on iOS We have already signed off the feature for iOS and the developer community is using it
Implementation of video texture on Android Feature is ready for release and is available in labs for developers to try out video texture on Android The feature would most probably be usable in a full fledged way in our next release
13
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
ETC2 Texture Compression (Ericsson Texture Compression)Ridam Batra | AIR Engineering
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
GPU TEXTURE COMPRESSION
16
CPU GPU
Upload of textures from CPU to GPU is a very slow operation
GPU can understand block compressed texture formats like etc1,pvrtc,etc2,dxt etc.
Compressed Textures
Texture Compression is the ANSWER.
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
ADOBE TEXTURE FORMAT
17
• Container format for compressed images. • AIR or Flash Player automatically extracts the appropriate texture depending on platform.
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Problem
18
ETC1
(android) Size on CPU – 170KB Size after GPU upload - 170KB
PVRTC (ios)Size on CPU – 170KBSize after GPU upload - 170 KB
Original png image Size on CPU – 256KB Size after GPU upload - 1048KB
DIFFERENT RENDERING RESULT ON IOS AND ANDROID !!!!
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
SOLUTION
OpenGL ES 3.0 introduced high quality ETC2 compression as a standard feature. It works on ios and android both.
19
Original png image
1048 KB
ATF with ETC1 + PVRTC
340 KB
ATF with ETC2170 KB
SIMILAR RENDERING RESULT ON IOS AND ANDROID SIMILAR RENDERING RESULT ON IOS AND ANDROID
SIZE OF TEXTURES FURTHER REDUCED
SIZE OF TEXTURES FURTHER REDUCED
AND AND
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Comparing The Two Applications
21
Size of APK with PNG files - 37.7 MB Size of APK with ATF Files - 21.2 MB
App With PNG Images GPU Memory for textures– 73,738 KB
App With Atf Files GPU Memory for textures– 9,216 KB
GPU MEMORY
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
64-bit Support & Parallel Compilation for iOSAbhinav Dhandh | AIR Engineering
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
64-bit support for iOS applications
Work on supporting 64-bit binary feature for iOS applications Support for newer A7+ processor devices:
Additionally, Apple issued new guidelines for app submission
iPhone 5s iPhone 6 iPhone 6+ iPad Air Mini iPad Air iPad Air 2
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Universal binary was the way..
SWFs ABC Files
32-bit Compilation and Linking
Executable File
IPA Combine
ADT
Additionally, stability of New compiler has been improved Major bug fixes for real world applications
64-bit Compilation and Linking
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
But..
Average Packaging Time Increase : 93%
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Parallel Compilation for packaging time reduction:
27
SWFs
ABC Files
ABC Files
32-bit Compilation and
Linking
Executable File
IPA
Combine
ADT
Multi-threaded Compilation
Executable File
Combine
32-bit Compilation and
Linking32-bit
Compilation and Linking
64-bit Compilation and
Linking64-bit
Compilation and Linking64-bit
Compilation and Linking
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Packaging time comparison
Average reduction in packaging time : 44%
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Standard Extended ProfileMayank Bhagya | AIR Engineering
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Features
Enhanced register set
Shader size
31
AGAL v2 AGAL v3
Vertex Attributes
8 16
Fragment Constants
64 200
AGAL v2 AGAL v3
Tokens 1024 2048
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Usecases
33
Complex shaders
© Jojo Nicado
Water© Dom Crossley
Fog
© 2015 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Graphics Assembly Language - v3
Standard Extended Profile Mobile – AIR v17 Desktop – FP & AIR v18
Mini Assembler @ GitHub https://github.com/adobe-flash/graphicscorelib
35