advanced graphics workshop - gfx2011

80
IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011 Advanced Graphics Workshop - Prabindh Sundareson, Texas Instruments GFX2011 Dec 3 rd 2011 Bangalore Note: This slide set is a public version of the actual slides presented at the workshop

Upload: prabindh-sundareson

Post on 06-May-2015

2.667 views

Category:

Technology


1 download

DESCRIPTION

This was presented at GFX2011 - conducted by IEEE Consumer Electronics society, Bangalore chapter. This is the public version of the presentation.

TRANSCRIPT

Page 1: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Advanced Graphics Workshop- Prabindh Sundareson, Texas Instruments

GFX2011Dec 3rd 2011Bangalore

Note: This slide set is a public version of the actual slides presented at the workshop

Page 2: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

GFX20118.30 AM [Registration and Introduction, Equipment setup]

9.00 AM Why Graphics ? Present and Future – Prof. Vijay Natarajan, Assistant Professor, Department of Computer Science and Automation, IISc, Bangalore

9.45 AM

Introduction to the OpenGL/ES Rendering pipeline, and algorithms

Detailed walkthrough of the OpenGL ES2.0 spec and APIs – Part 1

1.00 PM [Lunch]

Detailed walkthrough of the OpenGL ES2.0 spec and APIs – Part 2

- Break -

Framework and platform integration - EGL, Android (SurfaceFlinger)

Tools for performance benchmarking, and Graphics Development

Q&A, Certificate presentation to participants – Networking

Page 3: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Detailed Agenda Inaugural Talk – Dr. Vijay Natarajan – “Graphics – Present and Future” - Break GPU HW Architectures, and the GL API The CFF APIs

Lab 1 Texturing objects

Lab 94 – Rectangle wall Vertices and Transformations

Lab 913 – Eyeing the eyes Lunch Break Real-life modeling and loading 3D models

Lab 9 – Review .obj file loading method Shaders – Vertex and Fragment

Lab 96 – Squishing the slices - Break Rendering targets

Lab 910 – 3D in a 2D world Creating special effects EGL – and Platform Integration

Overview of GLES2.0 usage in Android/ iOS

*This PPT is to be used in conjunction with the labs at http://www.gpupowered.org

Page 4: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

GPU HW Architectures CPUs are programmed with sequential code

Typical C program – linear code Well defined Pre-fetch architectures, cache mechanisms Problem ?

Limited by how fast “a” processor can execute, read, write

GPUs are parallel Small & same code, multiple data Don’t care - control dependencies

If used, drastically reduces throughput “Output” is a result of a matrix operation (n x n)

Graphics output – color pixels Computational output – matrix values

Page 5: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

GPU integrated SOCs The A5 chipset

CPU size ~= GPU size

Page 6: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Integrated GPU architectures (Samsung SRP)

From – Khronos presentation

Page 7: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Vivante GPU Architecture

GPUs vary in Unified vs

separate shader HW architecture

internal cache size

Bus size rendering

blocks Separated 2D

and 3D blocks

From - http://www.socip.org/socip/speech/pdf/2-Vivante-SoCIP%202011%20Presentation.pdf Spec evolution

Page 8: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

OpenGL specification evolution

Reading the spec

Page 9: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

A note on reading the GLES specifications It must be clear that GLES specification is a

derivative of the GL specification It is recommended to read the OpenGL 2.0 specification

first Then read the OpenGL ES2.0 specification

Similarly, for shading language It is recommended to read the OpenGL SL specification

first Then read the OpenGL ES SL specification

Extensions

Page 10: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

What are Extensions ? Extension types

OES – Conformance tested by Khronos EXT – Extension supported by >1 IP vendor Proprietary (vendor_ prefix) – Extension from 1 IP vendor

How to check for extensions ? getSupportedExtensions (WebGL), getExtension() glGetString (openGL ES)

Number of extensions OpenGL 400 + OpenGL ES 100+

Dependencies

Page 11: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

OpenGL Dependencies OpenGL depends on a number of external systems to

run A Windowing system (abstracted by EGL/ WGL/ XGL …) External inputs – texture files, 3D modelling tools, shaders,

sounds, … OpenGL is directly used by

OS/ Driver developers (ofcourse!) HW IP designers Game studios (optimisation) Researchers (Modelling, Realism, ) … Tools developers

Application developers do not generally program on OpenGL, but rather do it on an Android API binding, or Java binding

GL vs ES

Page 12: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

OpenGL 2.0 vs OpenGL ES2 All Fixed function functionality is removed

Specific drawing calls like Fog etc Matrix setup

Replaced with programmable entities, GLES SL is ~= GL SL

Compatibility issues GL GLES Shaders

GLSL does not enable fixed function state to be available ex, gl_TexCoord

To enable compatibility, Architecture Review Board (ARB) extension – “GL_ARB_ES2_compatibility”

http://www.opengl.org/registry/specs/ARB/ES2_compatibility.txt

Good reference http://developer.amd.com/gpu_assets/GDC06-GLES_Tutorial_Day-

Munshi-OpenGLES_Overview.pdf

GLES API

Page 13: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

The OpenGL ES API From the Khronos OpenGL ES Reference Card

“OpenGL® ES is a software interface to graphics hardware.

The interface consists of a set of procedures and functions that allow a programmer to specify the objects and operations involved in producing high-quality graphical images, specifically color images of three-dimensional objects”

Keep this document handy for API reference http://www.khronos.org/opengles/sdk/docs/

reference_cards/OpenGL-ES-2_0-Reference-card.pdf

Client server

Page 14: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

The GL Client – Server Model Client (application on host), server (OpenGL

on GPU) Server can have multiple rendering contexts,

but has a global state Client will connect to one of the contexts at any point of

time

Client can set the states of the server by sending commands Further API calls will thus be affected by the previous

states set by the client

Server expects not-to-be interrupted by the client during operation Inherent nature of the parallel processor

GL,SL, EGL spec versions

Page 15: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

OpenGL Specifications

OPENGL Full versionOPENGL Full version

ES versionES version

CommonCommon

Common-LiteCommon-Lite

GLSL companionGLSL companion

GLSL-ES companionGLSL-ES companion

What we miss in ES compared to desktop version:Polygons, Display lists, Accumulation buffers,…

Currently in 4.0+

Currently in 2.0Currently in 1.0.16

Currently in 1.20

EGLEGLCurrently in 1.3

Core GL SpecCore GL Spec Shader SpecShader Spec Platform IntegrationPlatform Integration

EGLEGLCurrently in 1.3

Programming Flow

Page 16: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming in OpenGL / ES

Step1: Initialise EGL for rendering – context, surface, window

Step2: Describe the scene (VBOs, Texture coordinates) – objects with

Triangles, lighting Step3:

Load the textures needed for the objects in the scene Step4:

Compile the Vertex and Fragment Shaders Step 5:

Select the output target (FBO, Fb, Pixmap …) Step5:

Draw the scene Step 6

Run this in a loop

Frames/vertex/basics

Page 17: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Preliminaries Pixel Throughput

Memory bandwidth for a 1080P display @ 60 fps Did we forget Overdraw ?

Vertex Throughput Vertex throughput for a 100k triangle scene

Tearing

Frame switch(Uniform)

Driver frame draw(non-uniform)

Real frame switch happens here

Triangles

Page 18: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Why Triangles ? Connectivity, Simplicity, Cost

+

+???

pipeline

Page 19: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

ES2.0 Pipeline

What do the APIs look like ? …

Page 20: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

The GLES API – Overall view

Global Platform Management

Vertex Operations

Texture Operations

Shader Operations

Rendering Operations

-VBOs-Attaching attributes

-Attaching textures-Loading texture data-Mipmaps

-Loading, compiling,-Linking to program-Binary shaders

-Rendering to Framebuffer-Rendering to FBO-RTT

State Management

- CFF-Front/Back facing-Enable/Disable (culling, ..)-Get/ Set uniforms

-egl, wgl, glx ..-Antialiasing, -Configuration

ContextManagement

-Surface - window-Threading models-Context sharing

EGL

GL

Page 21: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Starting on the GPU

Page 22: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Flush, and Finish Several types of rendering methods adopted by GPUs

Immediate rendering Deferred rendering

Tiled rendering (ex, QCOM_tiled_rendering)

Immediate rendering – everyone is happy Except the memory bus!

Deferred – the GPU applies its “intelligence” to do/not do certain draw calls/ portions of draw calls Used most commonly in embedded GPUs

Flush() – The call ensures pending operations are kicked off, returns

Finish() – The call ensures pending operations are kicked off, “waits for completion”, returns

Page 23: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

A note on WebGL In native code, need to handle Surface, context

For example, refer to this native code eglGetDisplay eglInitialize – for this display eglBindAPI – bind to GLES/ VG context eglChooseConfig – configure surface type, GLES1/2/VG,

Depth/Stencil buffer size eglCreateWindowSurface / eglCreatePixmapSurface eglCreateContext eglMakeCurrent

Platform initialisation in WebGL is handled by browser Only configurations are – stencil, depth, AA, Alpha, preserve No EGL calls in JS application code No multi-threading issues (not yet, but “workers” are coming)

The hands on labs will focus on GL, not EGL Note : - GL context gets lost, when user account is locked/screen saver mode

etc – Restart browser as required.

Page 24: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming ClearColor Clear – clears with color

specified earlier Flush/ Finish

“setup”, “vertex”, “fragment”, “animate” functions in class

Will be called by framework

Clear web cache first time Login Open “Default Lab #1” - Copy Open “My labs” #1 – Paste

code Change code in setupFunc() Save, Run again

Lab #1

Below link contains introductory video for starting the labs:http://www.youtube.com/watch?v=TM6t2ev9MHk

Page 25: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Texturing

Page 26: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

A note on Binding, Buffer Objects What is “Binding” ?

Binding a server to a client – ex, VBO to a texture All objects are associated with a context state Binding an object is ~ copying the object state context Removes clientserver movement everytime “Xfer-once-to-server, keep the token, Use-multipletimes-

later” Good practice to “unbind” after operations– set binding to

0/null to avoid rogue programs changing state of bound object

Buffer-Objects Allows data to be stored on the “server” ie, the GPU

memory, rather than client memory (via pointer) GPU can decide where to place it for the fastest performance

Page 27: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Correct approach of data transfers

1. Generate Object (ex, glGenBuffers, glGenTextures)

2. Bind Object to an ID “xyz” (glBindBuffer(xyz), ..)3. Transfer data to Object (glBufferData,

glTexImage2D)4. Unbind (glBindBuffer(0))

After this point, the data remains bound to “xyz” and is managed by GPU.

Can be accessed later by referencing “xyz” Applies to VBOs, Textures, … Note the implicit “no atomicity” – needs locking

Page 28: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Texturing basics Texture Formats available

RGB* formats, Luminance only formats Relevance of YUV

Texture Filtering Maps texture coordinates to object coordinates – think of

wrapping cloth over object Mipmaps

Local optimisation – use pre-determined “reduced” size images, if object is far away from viewer – as compared to filtering full image

Objective is to reduce bandwidth, not necessarily higher quality Application can generate and pass through TexImage2D() GPU can generate using GenerateMipMap() Occupies more memory

Uv mapping

Page 29: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Texturing 3D objects Mapping from a bitmap to a 3D object

involves matching the texture coordinates to the object surface

Texture coordinates are calculated along with the vertex coordinates

3D tools output Texture coordinates along with vertex information, for the scene

Lab 12 (Sphere), Lab 13

compression

Page 30: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Texture Compression types GLES spec supports RGBA textures, Luminance …

To reduce memory bandwidth, compression used Texture Compression major types

PVRTC ETC1 Others

Android primarily supports ETC1 iOS supports PVRTC (and no other) Extension support queryable using GL API queries

How to store this information in an uniform manner ? Texture file formats PVRTC (using Textool converter from IMG) commonly used KTX file format

KTX

Page 31: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Khronos KTX file format To render a texture, steps to be used today:

Application needs apriori knowledge of texture type, format, storage type, mipmap levels, and filename or the buffer data itself

Then load into server using TexImage2D()

Proprietary formats exist to separate this application+texture dependency – ex, PVRT from IMG

KTX file format from Khronos is a standard way to store texture information, and the texture itself See next slide for structure of the file

Page 32: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

KTX format …

Passing coords

Page 33: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Texture coordinates to GPU Texture coordinates are passed to GPU as

“Attributes” along with the vertices Gen-bind-bufferdata, then bindAttrib

WebGL/Textures

Page 34: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Note on WebGL and Textures Because of the way file loads work on

browsers (asynchronous), texture loading may not happen before the actual draw Expect black screen for a very short-while till the Texture

image loads from the website

On native applications, due to the synchronous nature of loading the texture this issue will not be present

Programming

Page 35: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming with Textures bindTexture pixelStorei (webGL only)

UNPACK_FLIP_Y_WEBGL texImage2D texParameteri

TEXTURE_MAG_FILTER TEXTURE_MIN_FILTER

Note: WebGL “null” binding instead of “0”

Lab #2 (why not square ?)

Lab #94 – The wall

Page 36: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Note on the lab hints

Each session ends with a lab. The lab sessions online intentionally have errors that the reader has to debug to show the rendered object on screen.

Keys are provided for each such lab at the end of the section in this PPT

36

Page 37: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Lab 94 – Texturing (Keys to remove errors) var indexArray = new Uint16Array([0, 1, 2, 2,

1, 3]); var texCoordArray = new

Float32Array([0,0, 10,0, 0,10, 10,10]); context.enableVertexAttribArray(1); context.vertexAttribPointer(1, 2,

context.FLOAT, context.FALSE, 0, 0);

Page 38: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Vertices, Transformations

Page 39: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

What are vertices ? Vertices –

Points defined in a specific coordinate axes, to represent 3D geometry

Atleast 3 vertices are used to define a Triangle – one of the primitives supported by GLES

Page 40: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Vertex operations Where do vertices come from ?

Output of Modelling tools Mesh rendering / transforms – optimisations

For 2D operations (ex Window systems), just 2 triangles

Attributes

Page 41: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Vertex Attributes A vertex is characterised by its position

{x,y,z} {x,y,z} are floating point values

Additionally, normals are required for directional lighting calculations in shader 3D Tools output the normal map also along with vertex

information

Additionally, texture coordinates are required Again, 3D tools output the texture coordinates

Each HW implementation must support a minimum number of vertex attributes Maximum number can be queried using

MAX_VERTEX_ATTRIBS

CPU to GPU xfer

Page 42: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Vertices – CPU to GPU Optimising Vertex operations

A 3D object will have a lot of “common” vertices Ex – Cube has 6*2 triangles, (6*2)*3 vertices, but only 8

“points” So rather than passing vertices, pass 8 vertices, and 36

indices to the vertices to reduce Bandwidth Indices can be 16bit, so reduce BW by ~50%

GL_ELEMENT_ARRAY_BUFFER and GL_ARRAY_BUFFER STATIC_DRAW, DYNAMIC_DRAW Not uploading again and again but re-use

What are Vertex Buffer Objects ? genBuffers (createBuffer in WebGL), binding,

bufferData/offset and usage Usage of Index Buffers (ELEMENT_ARRAY_BUFFER)

Cartesian

Page 43: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Translation matrix1.translate(X,0.0,0);

X = 0

X = 0.4

Translation applied to all objects (effect is not dependent on depth of object)

Page 44: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Rotation

x

y

z

Rotation Observe effect of x offset!Refresh M,V,P after every rotate

-0

Lookat

Page 45: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Getting the eye to see the object “Model” Matrix made the object “look” right Now make the object visible to the “eye” – The

“View” Eye is always at the origin {0,0,0} So using matrices, move the current object to the eye

“LookAt” is implemented in many standard toolkits The LookAt transformation is defined by

Viewpoint - from where the view ray starts (eye) A Reference point (where the view ray ends) – in middle of scene

(center) A look-”up” direction (up)

ex – gluLookAt Utility function

Significant contributor of grey-hair Viewport

Page 46: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Viewport Transformation Convert from the rendering to the final screen

size ie physical screen

Define the viewport using glViewport() Viewport can be an area anywhere within the physical

screen

This takes care of aspect ratio Ex, square becomes rectangle in laptop

After the transformation, successful triangles get to the rasterisation HW, and then to the Fragment shader

HW optimisations

Page 47: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Summary - The Transformation Sequence

Translation example

mathematical step - w

Page 48: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

HW Optimisations Not all triangles are visible

HW can reject based on depth coverage Front-facing or back-facing (Culling)

Culling is disabled by default per specification However, most HW do this optimisation by default to save

on bandwidth/ later pixel processing

Programming

Page 49: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming ! Recall the Bandwidth needs

for the vertex transfers / frame

Passing Vertices Create Buffer Object bindBuffer bufferData Indices are passed as type

ELEMENT_ARRAY Passing Attributes

bindAttribLocation enableVertexAttribArray vertexAttribPointer

matrix.getAsArray()

Lab #913 – Eyeing the eyes

Page 50: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Lab 913 – Keys for “Eyeing the eyes” var totalArcs = 36; //shadervertexsetup_tunnel

texCoords.push(10*numArcs / totalArcs); texCoords.push(10*zslice / numZSlices);

matrix1.scale(1.0, 1.0, 1.0); //not 15.0

Page 51: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Real life 3D models

Page 52: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Real-life modelling of objects 3D models are stored in a combination of

Vertices Indices / Faces * Normals Texture coordinates

Ex, .OBJ, 3DS, STL, FBX … f, v, v//norm, v/t, o Export of vertices => scaling to 1.0-1.0 Vertex normals vs face normals Materials (mtl), animations Problem of multiple indices not allowed in openGL

Tools and Models Blender, Maya, … http://assimp.sourceforge.net/ - tool for importing multiple types http://www.blendswap.com/ - Blender models

Tessellation of meshes can be aided by HW in GPUs

Page 53: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming Loading 3D models is an

application functionality No new APIs from OpenGLES

are needed A parser is required to

parse the model files, and extract the vertex, attribute, normal, texture coordinate information

Look through objdata.js in Lab #9

Page 54: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Shaders

Page 55: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Vertices, Fragments - Revisited Vertices –

Points defined in a specific coordinate axes, to represent 3D geometry

Atleast 3 vertices are used to define a Triangle – one of the primitives supported by GLES

Fragments The primitives are “rasterised” to convert the “area” under

the primitive to a set of color pixels that are then placed in the output buffer

Shader characteristics

Page 56: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Shader characteristics Uniforms – uniform for all shader passes

Can be updated at run time from application

Attributes – changes per shader pass Varying – Passed between vertex and

fragment shaders Ex, written by Vertex shader, and used by Fragment

shader gl_Position

Programs Why do we need multiple programs in an application

for offscreen animation, different effects

MAX VARYING VECTORS – enum

Inputs to shader

Page 57: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Inputs to the Shaders Vertex Shader

Vertices, attributes, Uniforms

Fragment Shader Rasterised fragments (ie, after rasteriser fixed function

HW) Varyings from vertex shader Uniforms

Shader types

Page 58: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Fragment Shaders A fragment is – a pixel belonging to an area of the

target render screen (on-screen or off-screen) Primitives are rasterised, after clipping

Fragment shader is responsible for the output colour, just before the post-processing operations

A Fragment shader can operate on “1” fragment at a time

Minimum number of “TEXTURE UNITS” is 8 Calculation of colors

Colors are interpolated across vertices automatically (Ref Lab 6 in the hands-on session) – ie, “varyings” are interpolated in Fragment shaders during rendering

Colors can be generated from a texture “sampler” Each HW has a specific number of “Texture Units” that need to be

activated, and textures assigned to it for operation in the shader Additional information from vertex shader through “varyings”

Outputs gl_FragColor

Sample Frag shader

Page 59: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Program Each program consists of 1 fragment shader,

and 1 vertex shader Within a program, all uniforms share a single

global space

Precision

Page 60: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Advanced Shaders Animation Environment Mapping Per-Pixel Lighting (As opposed to textured lighting) Bump Mapping Ray Tracers Procedural Textures CSS – shaders (HTML5 – coming up)

Page 61: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming with Shaders Pass in shader strings Compile, link, Use Set uniforms Do calculations

Lab #96

Page 62: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Lab 96 – Keys for “Squishing the slice” uniform mediump float skyline; vec4 tempPos; tempPos = MVPMatrix * inVertex; tempPos.y=min(skyline, tempPos.y); //or, try below – one of the 2 tempPos.y=min(sin(inVertex.x*5.0)+cos(inVertex.y*2.0), tempPos.y); gl_Position = tempPos;

var skylineLoc = context.getUniformLocation(sprogram,"skyline"); context.uniform1f(skylineLoc, -0.1);

context.drawArrays(context.TRIANGLES, 0, vertexparams[1]/3);

Page 63: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Rendering Targets

Page 64: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Rendering Targets A rendering context is required before

drawing a scene. And a correponding Framebuffer Recall bindFramebuffer()

It can be Window system Framebuffer (Fb) Offscreen buffer (Implemented in a Frame Buffer Object)

FBO is not a memory area – it is information about the actual color buffer in memory, depth/ stencil buffers

By default, rendering happens to the Window system framebuffer (ID ‘0’)

Need

Page 65: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Need for offscreen rendering Special effects

Refer the fire effect specified earlier (Multiple passes)

Interfacing to “non-display” use-cases Ex, passing video through GPU, perform 3D effects, then

re-encode back to compressed format Edge detection/ computation – output is sent to a memory

buffer for use by other (non-GL) engines

FBO

Page 66: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

FrameBuffer Object A Frame Buffer Object

Can be just a color buffer (ex, a buffer of size 1920x1080x 4)

Typically also has depth/ stencil buffer By default – FBO – ID “0” is never assigned to new FBO

It is assigned to Window system provided Frame Buffer (onscreen)

Renderbuffers and Textures can be “attached” to FBO For RB – application has to allocate storage For FBO, the GL server will allocate the storage

rtt

Page 67: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Render-To-Texture By binding a Texture to a FBO, the FBO can be

used as Stage 1 – target of a rendering operation Stage 2 – used as a texture to another draw This is “Render-To-Texture” (RTT)

This allows the flexibility of “discreetly” using the server to do 3D operations (not visible onscreen), then use this output as texture input to a visible object If not for RTT, we have to render to regular Framebuffer

then do CopyTexImage2D() or readPixels() which are inefficient

Offscreen rendering is needed for dynamic-reflections

APIs

Page 68: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Post-processing operations Blending with Framebuffer - enables nice

effects (Ref Lab #6) Standard Alpha-Blending

glEnable ( GL_BLEND ); glBlendFunc ( GL_SRC_ALPHA, GL_ONE );

Is a “bad” way of creating effects Reads back previous framebuffer contents, then

blend Makes application memory bound, specially at

larger resolutions Stalls parallel operations within the GPU Recommended way is to perform Render-To-Texture,

and blending where necessary in the shader But needed for medical image viewing – ex

Ultrasound images, > 128 slices blendingprogramming

Page 69: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming FBO and back to Fb glGenFramebuffers glBindFramebuffer

Makes this FBO used glFramebufferTexture2D(i

d) Indicate ‘id’ is to be used for

rendering to TEXTURE, so storage is different

glDeleteFramebuffers

Then, create separate object to texture with TEXTURE ‘id’

Then, use previous textureID id as input to texImage2D next

Switching to FB Change binding to screen FB Load different set of vertices

as needed, different program as needed

Set texture binding to FBO texture drawn previously

DrawElements call

FBOs are used to do post-processing effects

Page 70: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Programming Draw a textured

rectangle to a FBO Using this FBO as

texture, render another rectangle on-screen

CheckFramebufferStatus very important

Lab #910

Page 71: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Lab 910 – Keys for “Render to Texture” lab Location, location, location ! Also note that readpixels doesn’t show

anything!

// context.clearColor(1.0, 0.0, 0.0, 1.0); // context.clear(context.COLOR_BUFFER_BIT |

context.DEPTH_BUFFER_BIT); // context.flush();

Page 72: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Platform Integration

Page 73: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Setting up the platform - EGL Context, Window, Surface OpenGL ES –

EGL_SWAP_BEHAVIOR == “EGL_BUFFER_PRESERVED” Reduces performance

Anti-aliasing configurations EGL_SAMPLES (4 to 16 typically, 4 on embedded platforms)

WebGL - preserveDrawingBuffer – attribute Optimisations done if it is known that app is clearing the

buffer – no dirty region check and whole scene is drawn efficiently

Dirty region check made in some systems

Android

Page 74: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Android Integration Details

Android composition uses GLES2.0 mostly as a pixel processor, not a vertex processor Uninteresting rectangular windows, treated as a texture

6 vertices Blending of translucent screens/ buttons/ text

3D (GLES2.0) is natively integrated 3D Live wallpaper backgrounds Video morphing during conferencing (?) Use the NDK

Surfaceflinger

Page 75: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Android SurfaceFlinger architecture Introduction to OpenGL interface on Android

http://code.google.com/p/gdc2011-android-opengl/wiki/TalkTranscript

HW acceleration on Android 3.0 / 4.0 http://android-developers.blogspot.com/2011/11/android-40-

graphics-and-animations.html

composition

Page 76: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Optimising OpenGL / ES applications Graphics performance is closely tied to a specific

HW Size of interface to memory, cache lines HW shared with CPU – ex, dedicated memory banks Power vs Raw performance Intelligent Discarding of vertices/ objects (!)

Performance is typically limited by Memory throughput GPU pixel operations per GPU clock CPU throughput for operations involving vertices Load balancing of units – within the GPU

GPUs that are integrated into SOCs are more closely tied to the CPU for operations, than separate GPUs Ex, GPU drivers offload some operations to CPU

debugging

Page 77: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Debugging OpenGL Vanishing vertices, Holes Improper lighting Missing objects in complex scenes Windows Tools

Perfkit/ GLExpert / gDEBugger Intel GPA

Linux Tools PVRTune (IMG) GDebugger Standard kernel tools Intel GPA

Pixel vs Vertex throughput, CPU loading, FPS, Memory limited – tuning knobs

Page 78: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

References Specs - http://khronos.org/opengles CanvasMatrix.js

https://github.com/toji/gl-matrix Tools - http://www.iquilezles.org/apps/shadertoy/

http://www.inka3d.com/ (from Maya) http://assimp.sourceforge.net/ - Asset importer

ARM – Mali – Architecture Recommendations http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0363d/CJAFCCDE.html

Optimising games – simple tips http://glenncorpes.blogspot.com/2011/09/topia-optimising-for-opengles20.html

Page 79: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Appendix: Video and Graphics Graphics is computed creation

Video is recorded as-is

Graphics is object – based Video (today) is not

Graphics is computed every frame fully Video is mostly delta sequences

Motion-detection, construction, compensation But extensions like swap_region (Nokia) exist

Page 80: Advanced Graphics Workshop - GFX2011

IEEE Consumer Electronics Society, Bangalore Chapter – GFX2011

Q & A, Feedback Feedback

http://ewh.ieee.org/r10/bangalore/ces/gfx2011.html