unconventional webapps with gwt:elemental & html5
Post on 18-Oct-2014
2.324 views
DESCRIPTION
Talk di Alberto Mancini alla DevFest 2012 @ FirenzeTRANSCRIPT
Unconventional webapps with GWT/Elemental, WebRTC &
WebGLAlberto Mancini
Blog: http://jooink.blogspot.com
Online Demo: http://www.jooink.com/experiments/DevFest2012
GWT 2.5 Elemental
Elemental is a new library for fast, lightweight, and "to the metal" web programming in GWT.
Elemental includes every HTML5 feature, ... DOM access ... WebGL, WebAudio, WebSockets, WebRTC, Web Intents, Shadow DOM, the File API, and more. ... we generate Java code directly from the WebIDL files used by JavaScript engines. (like DART)
Unconventional
... insomma qualcosa che non ci sembrava possibile fare con il solo browser, tipo accedere a devices attaccati al 'client', una webcam per esempio.
L'idea e' farsi una foto ...
WebRTC
WebRTC (Web Real-Time Communication) is an API definition being drafted by W3C and IETF.
The goal of WebRTC is to enable applications such as voice calling, video chat and P2P file sharing without plugins.
Specification & Up to date info: http://www.webrtc.org
Support: Chrome 22+, ... well work-in-progress at least for firefox.
what we need: getUserMedia (chrome !!)
WebRTC (with Elemental)Video in a canvas
final VideoElement videoElement = Browser.getDocument().createVideoElement(); final CanvasElement canvas =
Browser.getDocument().createCanvasElement(); final CanvasRenderingContext2D ctx2d = (CanvasRenderingContext2D)canvas.getContext("2d"); ... //repeatedly, e.g. in a Timer or RepeatingCommand ctx2d.drawImage(videoElement, 0, 0);
Take snapshot as easy as: Image img = new Image(canvas.toDataURL("png"));
WebRTC (with Elemental) binding <video/> to userMedia (search for facelogin on googlecode)
public void bindVideoToUserMedia(final VideoElement video, final EventListener l) { final Mappable map = (Mappable) JsMappable.createObject(); map.setAt("video", true); Browser.getWindow().getNavigator().webkitGetUserMedia(map, new NavigatorUserMediaSuccessCallback() {
public boolean onNavigatorUserMediaSuccessCallback(LocalMediaStream stream) {
setVideoSrc(video, stream); video.play(); ... }, new NavigatorUserMediaErrorCallback() {...}); }
private static native void setVideoSrc( VideoElement v, LocalMediaStream s) /*-{ v.src = window.webkitURL.createObjectURL(s); }-*/;
WebGL
WebGL (Web Graphics Library) is a JavaScript API for rendering interactive 3D graphics and 2D graphics within any compatible web browser without the use of plug-ins.
Specification: http://www.khronos.org/webgl/
Support: http://caniuse.com/webgl
Chrome 22+, FireFox 15+, Safari 6+, Opera 12+ (partial)
WebGL (with Elemental)Video in a canvas (3d rendering context) ... WebGLRenderingContext ctx3d = (WebGLRenderingContext)canvas.getContext("experimental-webgl");
Drawing is a bit harder: create a texture, create a background rectangle, use the video as a texture ctx3d.texImage2D(WebGLRenderingContext.TEXTURE_2D, 0, WebGLRenderingContext.RGBA, WebGLRenderingContext.RGBA, WebGLRenderingContext.UNSIGNED_BYTE, videoElement);
WebGL (with Elemental)shaders
Vertex Shader
precision mediump float;attribute vec2 a_position;attribute vec2 a_texCoord;uniform vec2 u_resolution;varying vec2 v_texCoord;void main() { vec2 zeroToOne = a_position / u_resolution; vec2 zeroToTwo = zeroToOne * 2.0; vec2 clipSpace = zeroToTwo - 1.0; gl_Position = vec4(clipSpace * vec2(1, -1), 1, 1); v_texCoord = a_texCoord;}
Fragment Shader
precision mediump float;uniform sampler2D u_image;varying vec2 v_texCoord;
void main() { gl_FragColor = texture2D(u_image, v_texCoord);}
from html5rocks.com's WebGL Fundamentals ... probably :o
WebGL (with Elemental)shaders
Fragment Shader...// Sepia tone// Convert to greyscale using NTSC weightings float grey = dot(texture2D(u_image, v_texCoord).rgb, vec3(0.299, 0.587, 0.114)); gl_FragColor = vec4(grey * vec3(1.2, 1.0, 0.8), 1.0);...// Negative vec4 texColour = texture2D(u_image, v_texCoord); gl_FragColor = vec4(1.0 - texColour.rgb, 1.0);
Fragment Shader
precision mediump float;uniform sampler2D u_image;uniform sampler2D SECOND_u_image;uniform float alpha;varying vec2 v_texCoord; void main(void) { vec4 prev = texture2D(SECOND_u_image, vec2(v_texCoord.s,1.0-v_texCoord.t)); vec4 cur = texture2D(u_image, v_texCoord); gl_FragColor = alpha*prev + (1.0-alpha)*cur;}
from http://r3dux.org/2011/06/glsl-image-processing/
WebGL (with Elemental)3D
WebGL e' una libreria 2D !!
Trasformazioni prospettiche (proiezioni), frustum, modelview vanno implementate.
VertexShader: applichera' le trasformazioni (matrici) che noi gli forniremo, vertice per vertice
FragmentShader: usera', pixel-per-pixel, le informazioni calcolate vertice-per-vertice (ed interpolate dal 'sistema') per assegnare il colore
WebGL (with Elemental)3D
double a = (right + left) / (right - left); double b = (top + bottom) / (top - bottom); double c = (farPlane + nearPlane) / (farPlane - nearPlane); double d = (2 * farPlane * nearPlane) / (farPlane - nearPlane); double x = (2 * nearPlane) / (right - left); double y = (2 * nearPlane) / (top - bottom); double[] perspectiveMatrix = new double[]{ x, 0, a, 0, 0, y, b, 0, 0, 0, c, d, 0, 0, -1, 0 };
....ctx3d.uniformMatrix4fv(pMatrixUniform, false, Utilities.createArrayOfFloat32(perspectiveMatrix));...private native static Float32Array createFloat32Array(JsArrayOfNumber a) /*-{ return new Float32Array(a);}-*/;
WebGL (with Elemental)3D/Wavefront-OBJ
Loading Wavefront-obj files ...
parser in java facile da trovare in rete
ClientBundle DataResource perfetto per il loading asincrono dei modelli
Javascript per calcolare le normali se non sono nel modello
Unconventional
Tutto quello che abbiamo visto fino ad ora si poteva fare, probabilmente con lo stesso sforzo, direttamente in javascript.
Ma usare GWT ci da o no una marcia in piu' ?
Nota: non stiamo dicendo che quello che segue non si potesse fare in javascript, ndr: Turing completeness, se ce ne fosse bisogno !
GWT
Compiler: Java -> Javascript
Javascript as a target language
NyARToolkit
ARToolKit is a computer tracking library for creation of strong augmented reality applications that overlay virtual imagery on the real world.NyARToolKit is an ARToolkit class library released for virtual machines, particularly those that host Java, C# and Android.
Up to date infohttp://nyatla.jp/nyartoolkit/wp/?page_id=198
Support: Java ... "Write once, run anywhere" (WORA), or sometimes write once, run everywhere (WORE) ... in my browser ?!?!?!
NyARToolkit con GWT
http://jooink.blogpsot.com
super-source tag per InputStream & altre parti della JRE non disponibili in GWT
byte[] raster;UInt8ClampedArray image = ImageData.getData();private static native byte[] toArrayOfBytes(Uint8ClampedArray a) /*-{ return a; }-*/;
Documented in JAPANESE !!!
How It Works
video
cam WebRTC <video/> canvas ImageData
byte[]
Elemental/jsni
GWT(NyARToolkit)mv matrix
WebGL + model
Demo http://www.jooink.com/experiments/DevFest2012