kranky geek sao paulo 2016 - webrtc statistics and analytics

WebRTC Statistics and Analytics

Varun Singh CEO, @callstatsio

twitter: @vr000m São Paulo, 22-11-2016

Multimedia Systems

• connected to a network

• capture and transmit media

• receive and render media

• video codecs: e.g., H.264, VP8, VP9, …

• audio codecs: e.g., G.711, Opus, …

2

Real-time Transport Protocol (RTP)

3

Sender Receiver

• Dejittering, sync, playout• Monitoring + reporting

RTP media stream(encoded media, FEC, repair)

RTP Control Protocol (RTCP)

4

Sender Receiver

RTCP Sender Reports (SRs)

RTCP Receiver Reports (RRs)

• Dejittering, sync, playout• Monitoring + reporting

RTP media stream(encoded media, FEC, repair)

RTCP Sender Report (SR)

5

Sender Receiver

RTP media stream(encoded media, FEC, repair)

RTCP Sender Reports (SRs)• Timing, synchronisation• Sending rate, packet count

RTCP Receiver Reports (RRs)

• Dejittering, sync, playout• Monitoring + reporting

RTCP Receiver Report

6

Sender Receiver

RTP media stream(encoded media, FEC, repair)

RTCP Sender Reports (SRs)• Timing, synchronisation• Sending rate, packet count

RTCP Receiver Reports (RRs)• Rough statistics• Congestion cues

• Dejittering, sync, playout• Monitoring + reporting

Adaptation

7

Sender Receiver

RTP media stream(encoded media, FEC, repair)

RTCP Sender Reports (SRs)• Timing, synchronisation• Sending rate, packet count

RTCP Receiver Reports (RRs)• Rough statistics• Congestion cues

• Dejittering, sync, playout• Monitoring + reportingShort-term adaptation

• Error-resilience (NACK, PLI)• Congestion control• Adaptive source coding

http://geek-and-poke.com/

8

It is not all about throughput*

… it is about latency†! *)capacity bandwidth

speed

audio only call

9

†)delay, one-way delay

time (sec)

Measuring User Experience

10

collect metrics and user actions C

all E

xper

ienc

e

Measuring User Experience

11

measure setup time

Cal

l Set

up

Cal

l Exp

erie

nce

Measuring User Experience

12

Ask the right question!

Cal

l Set

up

Cal

l Exp

erie

nce

Cal

l Fee

dbac

k

Measuring User Experience

13

Cal

l Set

up

Cal

l Exp

erie

nce

Cal

l Fee

dbac

k

Cal

l Ini

t (C

onte

xt)

Pag

e Lo

ad

represents overall quality

What to Measure?• Network metrics

• bits per second, RTT, jitter, packet losses, …

• Multimedia pipeline metrics• playout delay, frames metrics, …

• Quality Models from metrics

• Annoyances• resolution/frame rate changes, interface changes, …

• failure to setup (NATs, incompatibility)

• dropped calls (insufficient capacity, high latency, high losses)

14

getStats() API

15

Track

pc.getStats(mst, …)

IceTransportDtlsTransport

PeerConnection

Tracks

Datachannels

RtpReceiver

SctpTransport

caninvoketheAPIasoften(>150ms)

mostmetricsarecumulativemetrics

pc.getStats(null, …)

partial interface RTCPeerConnection { Promise<RTCStatsReport> getStats(optional MediaStreamTrack? selector = null);};

getStats() APIExample: var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];

pc.getStats(selector).then(function (report) {

processStats(report);

})

.catch(function (error) {

log(error.toString()); //if something went wrong

});

function processStats(report) {

for (var i in report) {

var now = report[i];

if (now.type == "outbound-rtp") {

//do something

}

}

16

Output:

{

timestamp:1479753828

ssrc:3938144725

packetsSent:15656

bytesSent:1562150

roundTripTime:31

…

}

Example

17

Internet

PeerConnection

PeerConnection

Alice

Bob

media direction

Sender-side pipeline

18

Internet

IceTransportDtlsTransport

PeerConnection

Tracks RtpReceiver

DtlsTransport IceTransport

PeerConnection

Tracks RtpSender

media direction

Receiver-side pipeline

19

IceTransportDtlsTransport

PeerConnection

Tracks RtpReceiver

DtlsTransport IceTransport

PeerConnection

Tracks RtpSender

Internetmedia

direction

Frame height

20

DtlsTransport IceTransport

PeerConnection

Tracks RtpSender

RTP Stats

21

DtlsTransport IceTransport

PeerConnection

Tracks RtpSender

ICE Stats

22

DtlsTransport IceTransport

PeerConnection

Tracks RtpSender

Simplified E-model (audio)

23ITU-TREC-G.114https://en.wikipedia.org/wiki/Mean_opinion_score

Example (1/3)

var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];var rttMeasures = [];

setTimeout(function () { pc.getStats(selector, function (report) {

// do something }, logError);}, 1000);

24

Example (2/3)var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];var rttMeasures = [];

setTimeout(function () { pc.getStats(selector, function (report) { for (var i in report) { var now = report[i]; if (now.type == "outbound-rtp") { //do something }

} }, logError);}, 1000);

25

Example (3/3)var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];var rttMeasures = [];

setTimeout(function () { pc.getStats(selector, function (report) { for (var i in report) { var now = report[i]; if (now.type == "outbound-rtp") { rttMeasures.append(now.roundTripTime); var avgRoundTripTime = average(rttMeasures); var emodel = simplemodel(avgRoundTripTime); console.log ("e-model: "+str(emodel));

//can add jitter, losses, framesDiscarded, etc }

} }, logError);}, 1000);

26

What more can you do?

27

ICE States

28

https://w3c.github.io/webrtc-pc/

Disruption: loss of connectivity when network interfaces change, low available capacity,or high delay

The light grey vertical lines show disruption, highlighted by the red bounding boxes.29

Network Disruptions

Disruptions and user behaviour

User Behaviour: The user tries to correct for the disruption by turning on and off video

30

31

Lessons learnt

• Deploy TURN servers with TCP support

• Detect crashes, disruptions and re-setup connections

• media pipelines sometimes crash

• screen sharing plugin crash

• network connectivity re-establishment

32

Summary

• RTP and RTCP

• getStats() API

• Simplified E-model

33