unconventional webapps with gwt:elemental & html5

19
Unconventional webapps with GWT/Elemental, WebRTC & WebGL Alberto Mancini [email protected] Blog: http://jooink.blogspot.com Online Demo: http://www.jooink.com/experiments/DevFest2012

Post on 18-Oct-2014

2.324 views

Category:

Technology


0 download

DESCRIPTION

Talk di Alberto Mancini alla DevFest 2012 @ Firenze

TRANSCRIPT

Page 1: Unconventional webapps with gwt:elemental & html5

Unconventional webapps with GWT/Elemental, WebRTC &

WebGLAlberto Mancini

[email protected]

Blog: http://jooink.blogspot.com

Online Demo: http://www.jooink.com/experiments/DevFest2012

Page 2: Unconventional webapps with gwt:elemental & html5

GWT 2.5 Elemental

Elemental is a new library for fast, lightweight, and "to the metal" web programming in GWT.

Elemental includes every HTML5 feature, ... DOM access ... WebGL, WebAudio, WebSockets, WebRTC, Web Intents, Shadow DOM, the File API, and more. ... we generate Java code directly from the WebIDL files used by JavaScript engines. (like DART)

Page 3: Unconventional webapps with gwt:elemental & html5

Unconventional

... insomma qualcosa che non ci sembrava possibile fare con il solo browser, tipo accedere a devices attaccati al 'client', una webcam per esempio.

L'idea e' farsi una foto ...

Page 4: Unconventional webapps with gwt:elemental & html5

WebRTC

WebRTC (Web Real-Time Communication) is an API definition being drafted by W3C and IETF.

The goal of WebRTC is to enable applications such as voice calling, video chat and P2P file sharing without plugins.

Specification & Up to date info: http://www.webrtc.org

Support: Chrome 22+, ... well work-in-progress at least for firefox.

what we need: getUserMedia (chrome !!)

Page 5: Unconventional webapps with gwt:elemental & html5

WebRTC (with Elemental)Video in a canvas

final VideoElement videoElement = Browser.getDocument().createVideoElement(); final CanvasElement canvas =

Browser.getDocument().createCanvasElement(); final CanvasRenderingContext2D ctx2d = (CanvasRenderingContext2D)canvas.getContext("2d"); ... //repeatedly, e.g. in a Timer or RepeatingCommand ctx2d.drawImage(videoElement, 0, 0);

Take snapshot as easy as: Image img = new Image(canvas.toDataURL("png"));

Page 6: Unconventional webapps with gwt:elemental & html5

WebRTC (with Elemental) binding <video/> to userMedia (search for facelogin on googlecode)

public void bindVideoToUserMedia(final VideoElement video, final EventListener l) { final Mappable map = (Mappable) JsMappable.createObject(); map.setAt("video", true); Browser.getWindow().getNavigator().webkitGetUserMedia(map, new NavigatorUserMediaSuccessCallback() {

public boolean onNavigatorUserMediaSuccessCallback(LocalMediaStream stream) {

setVideoSrc(video, stream); video.play(); ... }, new NavigatorUserMediaErrorCallback() {...}); }

private static native void setVideoSrc( VideoElement v, LocalMediaStream s) /*-{ v.src = window.webkitURL.createObjectURL(s); }-*/;

Page 7: Unconventional webapps with gwt:elemental & html5

WebGL

WebGL (Web Graphics Library) is a JavaScript API for rendering interactive 3D graphics and 2D graphics within any compatible web browser without the use of plug-ins.

Specification: http://www.khronos.org/webgl/

Support: http://caniuse.com/webgl

Chrome 22+, FireFox 15+, Safari 6+, Opera 12+ (partial)

Page 8: Unconventional webapps with gwt:elemental & html5

WebGL (with Elemental)Video in a canvas (3d rendering context) ... WebGLRenderingContext ctx3d = (WebGLRenderingContext)canvas.getContext("experimental-webgl");

Drawing is a bit harder: create a texture, create a background rectangle, use the video as a texture ctx3d.texImage2D(WebGLRenderingContext.TEXTURE_2D, 0, WebGLRenderingContext.RGBA, WebGLRenderingContext.RGBA, WebGLRenderingContext.UNSIGNED_BYTE, videoElement);

Page 9: Unconventional webapps with gwt:elemental & html5

WebGL (with Elemental)shaders

Vertex Shader

precision mediump float;attribute vec2 a_position;attribute vec2 a_texCoord;uniform vec2 u_resolution;varying vec2 v_texCoord;void main() { vec2 zeroToOne = a_position / u_resolution; vec2 zeroToTwo = zeroToOne * 2.0; vec2 clipSpace = zeroToTwo - 1.0; gl_Position = vec4(clipSpace * vec2(1, -1), 1, 1); v_texCoord = a_texCoord;}

Fragment Shader

precision mediump float;uniform sampler2D u_image;varying vec2 v_texCoord;

void main() { gl_FragColor = texture2D(u_image, v_texCoord);}

from html5rocks.com's WebGL Fundamentals ... probably :o

Page 10: Unconventional webapps with gwt:elemental & html5

WebGL (with Elemental)shaders

Fragment Shader...// Sepia tone// Convert to greyscale using NTSC weightings float grey = dot(texture2D(u_image, v_texCoord).rgb, vec3(0.299, 0.587, 0.114)); gl_FragColor = vec4(grey * vec3(1.2, 1.0, 0.8), 1.0);...// Negative vec4 texColour = texture2D(u_image, v_texCoord); gl_FragColor = vec4(1.0 - texColour.rgb, 1.0);

Fragment Shader

precision mediump float;uniform sampler2D u_image;uniform sampler2D SECOND_u_image;uniform float alpha;varying vec2 v_texCoord; void main(void) { vec4 prev = texture2D(SECOND_u_image, vec2(v_texCoord.s,1.0-v_texCoord.t)); vec4 cur = texture2D(u_image, v_texCoord); gl_FragColor = alpha*prev + (1.0-alpha)*cur;}

from http://r3dux.org/2011/06/glsl-image-processing/

Page 11: Unconventional webapps with gwt:elemental & html5

WebGL (with Elemental)3D

WebGL e' una libreria 2D !!

Trasformazioni prospettiche (proiezioni), frustum, modelview vanno implementate.

VertexShader: applichera' le trasformazioni (matrici) che noi gli forniremo, vertice per vertice

FragmentShader: usera', pixel-per-pixel, le informazioni calcolate vertice-per-vertice (ed interpolate dal 'sistema') per assegnare il colore

Page 12: Unconventional webapps with gwt:elemental & html5

WebGL (with Elemental)3D

double a = (right + left) / (right - left); double b = (top + bottom) / (top - bottom); double c = (farPlane + nearPlane) / (farPlane - nearPlane); double d = (2 * farPlane * nearPlane) / (farPlane - nearPlane); double x = (2 * nearPlane) / (right - left); double y = (2 * nearPlane) / (top - bottom); double[] perspectiveMatrix = new double[]{ x, 0, a, 0, 0, y, b, 0, 0, 0, c, d, 0, 0, -1, 0 };

....ctx3d.uniformMatrix4fv(pMatrixUniform, false, Utilities.createArrayOfFloat32(perspectiveMatrix));...private native static Float32Array createFloat32Array(JsArrayOfNumber a) /*-{ return new Float32Array(a);}-*/;

Page 13: Unconventional webapps with gwt:elemental & html5

WebGL (with Elemental)3D/Wavefront-OBJ

Loading Wavefront-obj files ...

parser in java facile da trovare in rete

ClientBundle DataResource perfetto per il loading asincrono dei modelli

Javascript per calcolare le normali se non sono nel modello

Page 14: Unconventional webapps with gwt:elemental & html5

Unconventional

Tutto quello che abbiamo visto fino ad ora si poteva fare, probabilmente con lo stesso sforzo, direttamente in javascript.

Ma usare GWT ci da o no una marcia in piu' ?

Nota: non stiamo dicendo che quello che segue non si potesse fare in javascript, ndr: Turing completeness, se ce ne fosse bisogno !

Page 15: Unconventional webapps with gwt:elemental & html5

GWT

Compiler: Java -> Javascript

Javascript as a target language

Page 16: Unconventional webapps with gwt:elemental & html5

NyARToolkit

ARToolKit is a computer tracking library for creation of strong augmented reality applications that overlay virtual imagery on the real world.NyARToolKit is an ARToolkit class library released for virtual machines, particularly those that host Java, C# and Android.

Up to date infohttp://nyatla.jp/nyartoolkit/wp/?page_id=198

Support: Java ... "Write once, run anywhere" (WORA), or sometimes write once, run everywhere (WORE) ... in my browser ?!?!?!

Page 17: Unconventional webapps with gwt:elemental & html5

NyARToolkit con GWT

http://jooink.blogpsot.com

super-source tag per InputStream & altre parti della JRE non disponibili in GWT

byte[] raster;UInt8ClampedArray image = ImageData.getData();private static native byte[] toArrayOfBytes(Uint8ClampedArray a) /*-{ return a; }-*/;

Documented in JAPANESE !!!

Page 18: Unconventional webapps with gwt:elemental & html5

How It Works

video

cam WebRTC <video/> canvas ImageData

byte[]

Elemental/jsni

GWT(NyARToolkit)mv matrix

WebGL + model

Page 19: Unconventional webapps with gwt:elemental & html5

Demo http://www.jooink.com/experiments/DevFest2012