webrtc: a front-end perspective

WebRTCA front-end perspective

Opera SoftwareFronteers2014

@shwetank

About me

Shwetank DixitOpera Software

Fronteers2014

@shwetankDeveloper Relations, Opera Software

http://www.thehindu.com/news/national/other-states/using-pcos-in-mumbai-show-id-proof/article4241422.ece

Source: https://www.flickr.com/photos/rambow/858719954

Source: https://www.flickr.com/photos/rambow/858719954Source: https://www.flickr.com/photos/rambow/858717314

What is WebRTC?

Real Time Peer-to-Peer

Communication

but …So far I’ve seen very

few web people talking about it

Lack of Awareness

Not supported by MS

Not supported by Apple

Other

0% 5.5% 11% 16.5% 22%

Whats the biggest barrier to WebRTC adoption?*

* WebRTC World Outlook Survey 2014

WebRTC is for Web Developersbut …

Mediastream

Mediastream

RTCPeerConnection

Mediastream

RTCPeerConnection

RTCDataChannel

` `

MicrophoneDevice Camera

mediaStream

+

` `

MicrophoneDevice Camera

getUserMedia

+

Sqwiggle

gUM + CSS(Filters, Masks, Blend Modes)

Create interesting visual effects from camera output in real-time using CSS

gUM + Web Audio

Sound powered actions and navigation

World’s first webcam

http://en.wikipedia.org/wiki/Trojan_Room_coffee_pot

- Motion Detection (Baby Monitors, Security Cam) - Hotspots - Gesture Recognition - Emotion Recognition - Sound Detection and Visualisation - Voice / Sound Activated Controls - Capture video (Kind of) (whammy.js, gifshot)

What can you achieve with gUM?

gUM: Current way to do it

navigator.getUserMedia*

*with prefixes

gUM: Broken Error Handling

Exceptions are not caught by the current API. Promises can fix this.

Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Sep/0171.html

gUM: Current Way

var failure = function(reason){ log("Failed to show camera: " + reason.message); } !navigator.getUserMedia(constraints, function(stream) { try { $video.src = window.URL.createObjectURL(stream); $video.play(); } catch(e) { failure(e); } //failure must tolerate non-MediaStreamError }, failure);

Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Sep/0171.html

gUM: In the future ?

navigator.mediaDevices.getUserMedia

gUM: In the future ? Promises Proposal

navigator.mediaDevices.getUserMedia(constraints) .then( function(stream) { videoElement.src = window.URL.createObjectURL(stream); videoElement.play(); }).catch(function (reason){ log(“Failed to show camera: ” +reason.message); });

Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Sep/0171.html

Debate

This is still up for debate. Some are eager to just ship v1.0 and add support for promises in gUM at a later stage. !

Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Oct/0003.html

gUM: Other concerns

Impossible for a gUM app to use the device flash for low-light

!

Use (auto) zoom Use (auto) focus

`

gUM: Other concerns

“If you look further into this API then it doesn’t offer any advanced features

that you expect in a good camera app: focus, flash, zoom etc. It is clear that …(getUserMedia).. has been designed

for use in P2P applications and not as a dedicated camera experience.”

- Paul Kinlan

Mediastream Image-Capture

!

Will provide a deeper level of control for taking photos

!

w3.org/TR/image-capture/

`

Mediastream Image-Capture

White Balance ISO

Red-Eye Reduction Exposure

Brightness Contrast

Saturation Sharpness

Zoom takePhoto()

Focus? Flash? ¯\_(ツ)_/¯

Mediastream Image-Capture

Getting permission is not easy

Some users don’t seem to notice the permissions dialog

in certain browsers

Getting permission is not easy

Some users don’t seem to notice the permissions dialog

in certain browsers

Some users don’t seem to notice the permissions dialog

in certain browsers

Some users don’t seem to notice the permissions dialog

in certain browsers

Some users don’t seem to notice the permissions dialog

in certain browsers

`

Getting permission is not easy

Permissions dialogs are different on different platforms and browsers …

especially mobile

HTTPS helps …

If the app is on HTTPS, then the browser will remember the last

granted permission.

RTCPeerConnection

webrtcglossary.com

RTCPeerConnection

* May or may not wax a chump like a candle.

ICE - Interactive Connectivity Establishment

STUN

Whats my external IP Address?

Your external IP Address is x.x.x.x

TURN around…

Total eclipse of the (p2p) call …

TURN

Basically relays stuff from one party to the other.

Big Fat TURN Server

TURN

Basically relays stuff from one party to the other.

In this case, connection won’t be p2p, but at least the

connection will be made!

TURN

Basically relays stuff from one party to the other.

In this case, connection won’t be p2p, but at least the

connection will be made!

Generally speaking, a TURN server will include a STUN server too.

Free ICE Project

github.com/DamonOehlman/freeice(List of free STUN/TURN servers to use in your projects)

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Session Description (SDP)

v=0 o=- 680121471469462884 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS GUKF430Khp9jEQiPrdYe0LbTAALiNAKAIfl2 Audio Lines m=audio 54278 RTP/SAVPF 111 103 104 0 8 106 105 13 126 c=IN IP4 180.6.6.6 a=rtcp:54278 IN IP4 180.6.6.6 !ICE Candidates a=candidate:4022866446 1 udp 2113937151 192.168.0.197 36768 typ host generation 0 a=candidate:4022866446 2 udp 2113937151 192.168.0.197 36768 typ host generation 0 a=candidate:2706108158 1 tcp 1509957375 192.168.0.197 0 typ host generation 0 a=candidate:2706108158 2 tcp 1509957375 192.168.0.197 0 typ host generation 0 a=candidate:1853887674 1 udp 1845501695 46.2.2.2 36768 typ srflx raddr 192.168.0.197 rport 36768 generation 0 a=candidate:1853887674 2 udp 1845501695 46.2.2.2 36768 typ srflx raddr 192.168.0.197 rport 36768 generation 0 a=candidate:2157334355 1 udp 33562367 180.6.6.6 54278 typ relay raddr 46.2.2.2 rport 38135 generation 0 a=candidate:2157334355 2 udp 33562367 180.6.6.6 54278 typ relay raddr 46.2.2.2 rport 38135 generation 0

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Creates Offer

local description

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Receives Offer

local description

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Receives Offer

remote description

local description

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Creates and sends Answer

local description

remote description

local description

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Receives Answer

local description

remote description

local description

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Receives Answer

local description

remote description

local description

remote description

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

Connectivity Checks

local description

remote description

local description

remote description

The WebRTC Signalling Process

WebSocket || XHR || SIP || Carrier Pigeon

P2P Connection Established

Libraries

!

720p : 1-2 Mbps // VP8 at 30fps 360p : 0.5 - 1 Mbps // VP8 at 30fps 180p : 0.1 - 0.5 Mbps // VP8 at 30fps

Audio : .06 - 0.51 Mbps // Opus

Bandwidth Numbers

Debugging

opera://webrtc-internals chrome://webrtc-internals

about:webrtc !

The getStats() method !

callstats.io !

Data Channels

Once RTCPeerConnection is established, we can use Data Channels to transfer data P2P.

Data can be text or binary data

Whats the advantage?

Data Channels provide a * High Performance * P2P * Low Latency Connection to others

Working with Data

datachannel .onopen // Data Channel opened .onclose // Data Channel closed .onerror // Error occurred

.onmessage // New message arrived

.send() // Send a message

DataChannels can work exactly like WebSockets if you want, but they don’t

have to …

SCTP // Transport Protocol DTLS // Encryption

UDP

notable options

ordered send data packets in ordered fashion or not

!

maxRetransmitTime Max time to try to send a failed message

!

maxRetransmits Max number of tries to deliver failed message before it

gives up

In the wild

Peerflix PeerCDN WebTorrent

In the wild

Peerflix !

PeerCDN WebTorrent

@mafintosh

@feross

Two ModesIsWebRTCReadyYet.com

ORTC

- WebRTC 1.1 (ORTC) - SDP exchange not required - Advanced stuff like simulcast - More low level than WebRTC - Shim for WebRTC to work on top of ORTC - Important goal is compatibility. Existing

apps should not be affected

Communication

Opera SoftwareFronteers2014

Cheers!@shwetank

shwetankd@opera.com