mining the rendering power in web browsers

Mining the Rendering Power in Web Browsers

Jianxia XueJan. 28, 2014

Rendering Power in Browsers・J. Xue・physics.olemiss・1/28/14

Outline

• Web application as software deployment platform

• WebGL: Graphics API inside browsers

• Explore browser rendering capability through three case studies

• Looking into the future


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

web

app

Web Applications for Content Creation

• No download, direct manipulation through browser

• Easy collaborative content creation

• Easy sharing, publishing, cloud backing up

Autodesk AutoCAD 360Online editing of document, presentation, spreadsheet Diagram creator

online latex editor


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

web

app

• The 5th revision of the markup language for web content since 2008.

• HTML5 + CSS3 + Javascript as one

static content, and

wrapper of allstyling even

with variable support

functions that fetch, store, upload, process dynamic

content - The workhorse of web app logics


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

web

app Javascript

• web browser client-side scripting language

• prototype-based, dynamic typing, multi-paradigm (procedural, object oriented, functional)

• Rich libraries

Google Web Toolkit (GWT)Yahoo User Interface (YUI) 2D data visualizer

three.js r65

3D graphics libEasy UI and AJAX toolsServer-side engine for real-time high-traffic apps


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

Graphics API Inside Browsers

• A Javascript API for Web Graphics Library

• Web browser can use rendering capabilities from OpenGL ES 2.0 and GLSL directly

• Designed and maintained by Khronos Group who also manages OpenGL graphics API series

• This means that one can use GPU shaders directly inside a web app!


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App WebGL Browser Support

• Data from 2013 December

57%28%

9%

4%2%

ChromeFirefoxInternet ExplorerSafariOpera

✔v9 ✔v4.0

✔v6.0

✔v11

✔v11

http://www.w3schools.com/browsers/browsers_stats.asp


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

The OpenGL Graphics API

• OpenGL ES 2.0 made for GPU based rendering, enforcing the use of vertex and fragment shaders

WebGLVender 3D

libraries such as three.js

OpenGL ES 2.0

OpenGL with GLSL since 2.0

web graphics application

Slimmed, and optimized for embedded system

a close binding to Javascript

2004 2007 2011


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

Modern Graphics Pipeline

RGBAZ RGBAZ RGBAZ RGBAZ

Vertex Processor

Vertex Processor

Vertex Processor

Vertex Processor

Fragment Processor

Fragment Processor

Fragment Processor

Rasterizer

...

...

Fragment Processor

Application Data

Frame Buffer Blender

...

...

Rendered Frame


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

Reaching to the Coding Spot<html lang="en"><head><script>

var vertexShader = "...";var fragmentShader = "...";function initWebGL(canvas) {

if (!canvas) {canvas = document.createElement('canvas');

document.body.appendChild(canvas); }

var gl = null;try {

gl = canvas.getContext("experimental-webgl") || canvas.getContext("webgl");} catch(e) {}if (!gl) {

alert("Your browser does not support WebGL!");return gl;

}// creates, compile, attach, link, and activate a GLSL program object// setup application datareturn gl;

}function drawScene(gl) { ... }

</script></head><body><script> var gl = initWebGL(); drawScene(gl);</script></body></html>

canvas dom element as the

frame container for webgl to draw upon

canvas.getContext to reach WebGL API

shader programs are dynamically activated

using WebGL API

Application data and draw routine

Application data and draw routine

shader programs in GLSL stringsshader programs in GLSL strings


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App Example Apps


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

Case Studies to Explore the Rendering Power in Browsers

• Fragment shader capability: Mandelbrot Set Navigator

• Vertex shader capability: Color space morphing

• Realtime Image Processing and Computer Vision using WebRTC (Chrome and Firefox only)


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

Mandelbrot Set Navigator

• Vertex data are simply 4 corner points of a 2D rectangle, the graphics pipeline creates the fragments according to the viewport size

• Fragment shader carries the main computation of the escape method

• Application data manages the viewport pixel to complex value mapping according to user controlled zooming and translation factors


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

Inside the Fragment Shaderuniform float ccx;uniform float ccy;uniform vec2 fViewportSize;uniform float zoom;uniform int nMaxIter;

void main(void) { const int MaxIter = 512; // Rescale the coordinate float Real0 = (gl_FragCoord.x -

fViewportSize.x / 2.0);

// GLSL use lower left as the origin by default float Imag0 = -(gl_FragCoord.y -

fViewportSize.y / 2.0);

float x0 = ccx + zoom * Real0; float y0 = ccy + zoom * Imag0;

float R = 0.0; float I = 0.0;

float R2 = R*R; float I2 = I*I;

int LastIter=-1;for (int iter=0; iter < MaxIter; iter++) { I=(R+R)*I + y0; R=R2-I2 + x0; R2=R*R; I2=I*I; if (R2+I2 >= 4.0 ) { LastIter = iter; break; } if (iter >= nMaxIter) { LastIter = 0; break; }}

vec4 color;color = vec4(1.0 - (0.5+0.5*cos(angle*2.0)),

1.0 - (0.5+0.5*cos(angle*3.0)), 1.0 - (0.5+0.5*cos(angle*5.0)), 1.);

}gl_FragColor = color;

}

Per

Pixe

l Com

puta

tion


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

Profiling Method and Result

• Use console.time and console.timeEnd

• Real-time navigation speed on full HD resolution rendering

• < 1 millisecond rendering time for 1M pixel

• The pixelation at zoom-in scale of 2-20 real values are due to the GLSL support of float type only.


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App Color Space Morphing

• Vertex shader carries the update of every vertex’s position per frame

• Fragment shader simply copies the vertex color since only points exist

RGB HSV


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App Inside Vertex Shader

uniform float uPointSize;uniform float uTime;attribute vec3 aRgbPos;attribute vec3 aHsvPos;varying vec4 vColor;

void main(void) { gl_PointSize = uPointSize; float s = 0.5*(cos(uTime)+1.0); vec3 pos = aRgbPos * s + aHsvPos * (1.0-s); vColor = vec4(aRgbPos, 1.); gl_Position = projectionMatrix * modelViewMatrix * vec4(pos, 1.);}

Per Vertex Computation of position and color

per vertex color, will be relayed to fragment


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App Profiling Result

• use~0.25 ms per frame

• use more advance profiling tool provided by Chrome about:tracing

• Vertex processing is more expensive than fragment processing


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App ToyCam

• Use WebRTC getusermedia, streaming user webcam or mic data directly

• Video streaming can be used as a texture for GPU shaders to further process


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App WebRTC

• Open project that enables browsers with Real-Time Communication (RTC) via javascript

• p2p chat can be done directly in an webapp

• Supported by Chrome, Firefox, and Opera as of now


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App Web-RTC Architect


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

getUserMedia()<!DOCTYPE html><html lang="en"><head> <meta charset="utf-8"> <title>HTML5 getUserMedia Demo By Arunkumar Gudelli</title> <script src="http://code.jquery.com/jquery-1.10.2.min.js" type="text/javascript"></script> <script> function onFailure(err) { alert("The following error occured: " + err.name); } jQuery(document).ready(function () { var video = document.querySelector('#webcam'); navigator.getUserMedia = (navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia); if (navigator.getUserMedia) { navigator.getUserMedia ( { video: true }, function (localMediaStream) { video.src = window.URL.createObjectURL(localMediaStream); }, onFailure); } else { alert('OOPS No browser Support'); } }); </script></head><body> <div> <video id="webcam" width="500" autoplay></video> </div></body></html>


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App ToyCam Data Flow

• Texture mapping used in the application to apply video frames to arbitrary shapes

webrtc getusermedia

dom element video

GLSL sample2d texture

fragment shader process


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App Future Web App

• More UCIs without keyboard/mouse

• More 3D games with audio/visual interactions

• Strong client-side visualization and basic signal processing + server-side clouding computing using aggregated user data


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App

More Apps Using Realtime Client Audio-Visual Signal


Curr

ent

Web

App・

Web

GL・

Case

Stu

dies・

Fut

ure

Web

App WebCL on its Way

• Javascript binding to OpenCL

• Firefox (by Nokia) and Webkit (by Samsung) open sourced prototype released in mid 2011

• This makes computations that are not necessarily lined up with graphics pipeline possible in GPU, e.g. physics engines in 3D games, more advanced computer vision algorithms

• Web deployment platform will become more common for computation intensive applications


Conclude• Browser rendering power strongly supports

heavily graphics applications such as 3D games, scientific visualizations, etc

• Web app development is easy given the mutual and continuing web UI and browser engine optimization

• Javascript Client Computing + Cloud Server Computing will boom big data artificial intelligence inferred applications


Thank You!

Questions?

mining the rendering power in web browsers

Documents