kranky geek sao paulo 2016 - webrtc statistics and analytics
TRANSCRIPT
WebRTC Statistics and Analytics
Varun Singh CEO, @callstatsio
twitter: @vr000m São Paulo, 22-11-2016
Multimedia Systems
• connected to a network
• capture and transmit media
• receive and render media
• video codecs: e.g., H.264, VP8, VP9, …
• audio codecs: e.g., G.711, Opus, …
2
Real-time Transport Protocol (RTP)
3
Sender Receiver
• Dejittering, sync, playout• Monitoring + reporting
RTP media stream(encoded media, FEC, repair)
RTP Control Protocol (RTCP)
4
Sender Receiver
RTCP Sender Reports (SRs)
RTCP Receiver Reports (RRs)
• Dejittering, sync, playout• Monitoring + reporting
RTP media stream(encoded media, FEC, repair)
RTCP Sender Report (SR)
5
Sender Receiver
RTP media stream(encoded media, FEC, repair)
RTCP Sender Reports (SRs)• Timing, synchronisation• Sending rate, packet count
RTCP Receiver Reports (RRs)
• Dejittering, sync, playout• Monitoring + reporting
RTCP Receiver Report
6
Sender Receiver
RTP media stream(encoded media, FEC, repair)
RTCP Sender Reports (SRs)• Timing, synchronisation• Sending rate, packet count
RTCP Receiver Reports (RRs)• Rough statistics• Congestion cues
• Dejittering, sync, playout• Monitoring + reporting
Adaptation
7
Sender Receiver
RTP media stream(encoded media, FEC, repair)
RTCP Sender Reports (SRs)• Timing, synchronisation• Sending rate, packet count
RTCP Receiver Reports (RRs)• Rough statistics• Congestion cues
• Dejittering, sync, playout• Monitoring + reportingShort-term adaptation
• Error-resilience (NACK, PLI)• Congestion control• Adaptive source coding
It is not all about throughput*
… it is about latency†! *)capacity bandwidth
speed
audio only call
9
†)delay, one-way delay
time (sec)
Measuring User Experience
10
collect metrics and user actions C
all E
xper
ienc
e
Measuring User Experience
11
measure setup time
Cal
l Set
up
Cal
l Exp
erie
nce
Measuring User Experience
12
Ask the right question!
Cal
l Set
up
Cal
l Exp
erie
nce
Cal
l Fee
dbac
k
Measuring User Experience
13
Cal
l Set
up
Cal
l Exp
erie
nce
Cal
l Fee
dbac
k
Cal
l Ini
t (C
onte
xt)
Pag
e Lo
ad
represents overall quality
What to Measure?• Network metrics
• bits per second, RTT, jitter, packet losses, …
• Multimedia pipeline metrics• playout delay, frames metrics, …
• Quality Models from metrics
• Annoyances• resolution/frame rate changes, interface changes, …
• failure to setup (NATs, incompatibility)
• dropped calls (insufficient capacity, high latency, high losses)
14
getStats() API
15
Track
pc.getStats(mst, …)
IceTransportDtlsTransport
PeerConnection
Tracks
Datachannels
RtpReceiver
SctpTransport
caninvoketheAPIasoften(>150ms)
mostmetricsarecumulativemetrics
pc.getStats(null, …)
partial interface RTCPeerConnection { Promise<RTCStatsReport> getStats(optional MediaStreamTrack? selector = null);};
getStats() APIExample: var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];
pc.getStats(selector).then(function (report) {
processStats(report);
})
.catch(function (error) {
log(error.toString()); //if something went wrong
});
function processStats(report) {
for (var i in report) {
var now = report[i];
if (now.type == "outbound-rtp") {
//do something
}
}
16
Output:
{
timestamp:1479753828
ssrc:3938144725
packetsSent:15656
bytesSent:1562150
roundTripTime:31
…
}
Example
17
Internet
PeerConnection
PeerConnection
Alice
Bob
media direction
Sender-side pipeline
18
Internet
IceTransportDtlsTransport
PeerConnection
Tracks RtpReceiver
DtlsTransport IceTransport
PeerConnection
Tracks RtpSender
media direction
Receiver-side pipeline
19
IceTransportDtlsTransport
PeerConnection
Tracks RtpReceiver
DtlsTransport IceTransport
PeerConnection
Tracks RtpSender
Internetmedia
direction
Frame height
20
DtlsTransport IceTransport
PeerConnection
Tracks RtpSender
RTP Stats
21
DtlsTransport IceTransport
PeerConnection
Tracks RtpSender
ICE Stats
22
DtlsTransport IceTransport
PeerConnection
Tracks RtpSender
Simplified E-model (audio)
23ITU-TREC-G.114https://en.wikipedia.org/wiki/Mean_opinion_score
Example (1/3)
var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];var rttMeasures = [];
setTimeout(function () { pc.getStats(selector, function (report) {
// do something }, logError);}, 1000);
24
Example (2/3)var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];var rttMeasures = [];
setTimeout(function () { pc.getStats(selector, function (report) { for (var i in report) { var now = report[i]; if (now.type == "outbound-rtp") { //do something }
} }, logError);}, 1000);
25
Example (3/3)var selector = pc.getRemoteStreams()[0].getAudioTracks()[0];var rttMeasures = [];
setTimeout(function () { pc.getStats(selector, function (report) { for (var i in report) { var now = report[i]; if (now.type == "outbound-rtp") { rttMeasures.append(now.roundTripTime); var avgRoundTripTime = average(rttMeasures); var emodel = simplemodel(avgRoundTripTime); console.log ("e-model: "+str(emodel));
//can add jitter, losses, framesDiscarded, etc }
} }, logError);}, 1000);
26
What more can you do?
27
Disruption: loss of connectivity when network interfaces change, low available capacity,or high delay
The light grey vertical lines show disruption, highlighted by the red bounding boxes.29
Network Disruptions
Disruptions and user behaviour
User Behaviour: The user tries to correct for the disruption by turning on and off video
30
31
Lessons learnt
• Deploy TURN servers with TCP support
• Detect crashes, disruptions and re-setup connections
• media pipelines sometimes crash
• screen sharing plugin crash
• network connectivity re-establishment
32
Summary
• RTP and RTCP
• getStats() API
• Simplified E-model
33