WebRTC... GWT & in-browser computation

Alberto Mancini - alberto@jooink.com

Francesca Tosi - francesca@jooink.com

WebRTCPlug-in free real-time communication …

WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs.

source: webrtc.org

WebRTCJust another JS API

or

WebRTC is a new front in the long war for an open and unencumbered webBrendan Eich (Mozilla CTO and inventor of JavaScript)

?

WebRTCJust another JS API

or

WebRTC is a new front in the long war for an open and unencumbered webBrendan Eich (Mozilla CTO and inventor of JavaScript)

?

Once upon a time … browsers

(js-)applications were sandboxed and kept far away from the hosting computer’s resources.

Once upon a time … browsers

(js-)applications were sandboxed and kept far away from the hosting computer’s resources.

Browser Piercing (aka HTML5) a Javascript Application cannot

➔ access the filesystem

➔ open f/d socket connections

➔ use graphics accelerator

➔ span multiple threads

Browser Piercing (aka HTML5) a Javascript Application cannot

➔ access the filesystem → and then FileAPI

➔ open f/d socket connections → and then WebSockets

➔ use graphics accelerator → and then WebGL

➔ span multiple threads → and then WebWorkers

Browser Piercing

a Javascript Application cannot

➔ acquire audio and video

➔ communicate P2P

Browser Piercing

a Javascript Application cannot

➔ acquire audio and video

➔ communicate P2P

WebRTC … secure enough?

“… WebRTC has encryption baked right into it; there's actually no way to send unencrypted media using a WebRTC implementation … both Chrome and Firefox implement.”

WebRTC … secure enough?But ...

“If the developers fail to carefully consider the security implications of their choices, the safeguards mandated by the specification will not be enough to guarantee the security of their WebRTC-based applications and the users.”

WebRTC … secure enough?Do not ...

“... it would be very easy to inadvertently click on something that gave camera or microphone control to someone I don't know and don't care to know.”

Courtney Sato - nerd queen @ConstellationRG

WebRTC … secure enough?See also:WebRTC: APIs, Protocols and Security Considerations - Part 1 / Part 2

WebRTC Security and Confidentiality

Security Considerations for WebRTC

WebRTC Security Architecture

WebRTC across platforms● Chrome (mobile too)● Firefox (mobile too)● Opera● Native Java and Obj-C

WebRTC JS-API

Some code …

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints,

successCallback,

errorCallback);

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints, successCallback,

errorCallback);

WebRTC JS-APIConstraints

● Controls the contents of the MediaStream● Media type, resolution, frame rate

var constraints = {video: true};

video: { mandatory: { ... }, optional [{... }]}

WebRTC JS-APIConstraints

● Controls the contents of the MediaStream● Media type, resolution, frame rate

var constraints = {video: true};

video: { mandatory: { chromeMediaSource: ‘screen’ }, optional [{... }]}

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints,

successCallback,

errorCallback);

WebRTC JS-APIerrorCallback

function errorCallback(error) { console.log("error: ", error);}

;-)

WebRTC JS-APIAcquiring video and audio

navigator.getUserMedia(constraints,

successCallback,errorCallback);

WebRTC JS-APIsuccessCallback

function successCallback(stream) { ...}

stream: MediaStream; a flux of audio- or video-related data.

WebRTC JS-APIsuccessCallback

function successCallback(stream) { var video = ... video.src =

window.URL.createObjectURL(stream);

}

(W3C File API)

WebRTC JS-APIfull sample

var constraints = {video: true};

function successCallback(stream) { var video = document.querySelector("video"); video.src = window.URL.createObjectURL(stream);}

function errorCallback(error) { console.log("navigator.getUserMedia error: ", error);}

navigator.getUserMedia(constraints, successCallback, errorCallback);

WebRTC JS-APILive Demo - 1

http://goo.gl/7X91Ie

WebRTC JS-APIRTCPeerConnection

→getUserMedia

+RTCPeerConnection

←

WebRTC JS-APIRTCDataChannel

Bidirectional communication of arbitrary data

var position = {x: 3.0, y: -2.0};

...var message = {txt: ‘...’, timestamp: ...};

RTCPeerConnection samplepc = new RTCPeerConnection(null);pc.onaddstream = gotRemoteStream;pc.addStream(localStream);pc.createOffer(gotOffer);

function gotOffer(desc) { pc.setLocalDescription(desc); sendOffer(desc);}

function gotAnswer(desc) { pc.setRemoteDescription(desc);}

function gotRemoteStream(e) { attachMediaStream(remoteVideo, e.stream);}

RTCPeerConnection

Live Demo - 2

https://apprtc.appspot.com

RTCPeerConnection

RTCPeerConnection

RTCPeerConnection

STUN

TURN

WebRTC made easy...

● Video chat:○ SimpleWebRTC○ easyRTC○ webRTC.io

● Peer-to-peer data:○ PeerJS○ Sharefest

Our target

Our target

● getUserMedia: gives us real time stream from the camera

● we can draw the <video/> into a <canvas/>

● from the canvas we can extract pixel data getImageData()

and then elaborate.

Computing in the browsergoals- Almost native performance- Manage complex algorithms and applications on the browser

Our target

cam WebRTC <video/> canvas ImageData

UInt8ArrayGWT(NyARToolkit)mv matrix

WebGL OBJ Model Loader3D Shaders

WebGL video

effects

Video

Computing in the browserconstraints- reuse existing code- cross-browser (as much as possible)

Computing in the browserour approach- javascript as a target language (GWT) - hand written optimized pieces of code

GWTGWT Web Toolkit

- java to javascript (optimizing) compiler- emulated jre- widget set

- develop in java and execute in javascript

JooinK’s WebRTC sample

Sample

http://picshare.jooink.com

Demo …

PicShareTecnologies

● WebGL for 3D graphics● WebRTC for MediaSteams● ImgUR for pictures store● AppEngine to store indices

● GWT as the glue

FOSS

gwt-nyartoolkit.googlecode.com

gwt-webgl.googlecode.com

The strategy: compile a java-lib with GWT into javascript

What’s nextChallanges

● real-time on mobile apps● real world recognition: computer vision

(BoofCV)

What’s nextStrategy

● use TypedArrays everywhere ● offload computation to the graphic

accelerator through WebGL● help the JIT compiler/optimizer with asm.js

References

● http://io13webrtc.appspot.com/#1(where we got inspiration and most of the pictures)

● http://www.youtube.com/watch?v=p2HzZkd2A40● http://www.webrtc.org (spec)

References

● http://www.jooink.com

● http://jooink.blogspot.com

● http://www.mokabyte.it

That’s all folks!!!

Francesca Tosifrancesca@jooink.com

Alberto Mancinialberto@jooink.com

Markers - use at picshare.jooink.com