kranky geek 2015 - decisions & considerations in building your webrtc app

© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER

RUNNING RTCOperational lessons learned at Twilio.

mailto:[email protected]


TWILIO?



Communication Channels

T W I L I O C O M M U N I C AT I O N S P L AT F O R M

VO I C E S M S M M S

W E B M O B I L E

Your Business Applications

I P M ESS A G I N G V I D E O



750,000 Developers & Businesses

50 BILLION

API Interactions a year

O P E R AT I N G AT S C A L E



TWILIO AND WEBRTC?



2011 2012 2013 2014 Present

TWILIO CLIENT 1.1: WEBRTC

TWILIO CLIENT: WEBRTC CALL VOLUME

WEBRTC RELEASED IN CHROME 23

TWILIO CLIENT 1.0: FLASH



2011 2012 2013 2014 Present

TWILIO CLIENT 1.1: WEBRTC

TWILIO CLIENT 1.2: GLOBAL LOW LATENCY

TWILIO CLIENT: WEBRTC CALL VOLUME

WEBRTC RELEASED IN CHROME 23

WEBRTC RELEASED IN FIREFOX 22

TWILIO CLIENT 1.0: FLASH



COMMON USE CASES



COMMON USE CASES

CRM and SaaS application integration

Call center / customer service

Sharing economy

Education / expertise on-‐demand

Recruiting

Anonymous calling



A HANDFUL OF OUR WEBRTC CUSTOMERS



WEBRTC COMPLEXITY DOES NOT INCREASE LINEARLY.



Signaling Server



Signaling Server

TURN Server



TURN Server

Signaling Server Media Server

Conference State



TURN Server

Signaling Server Media Server SIP Gateway

Conference State



TURN Server

Signaling Server SIP Gateway

Conference State

Media Server



TURN Server

Signaling Server SIP Gateway

Conference State

PSTN GatewayMedia Server

Carrier



TS SC PM CDC1

DC5



DC3

TS SC PM C

DC7

DC6DC1

DC5

DC2

DC4



SOMETHING WILL BREAK.



Stuff you can’t control

Random host failure

Network degradation

Spurious network traffic

Stuff you can control

Resource management

Operational mishaps

Bugs / crashes



Tens of failures / week.

100% up-‐time two quarters running.

99.95% up-‐time for the year.



TEST CONSTANTLY.



END-TO-END TESTS. CONTINUALLY TEST THE PRODUCT THE

SAME WAY THE CUSTOMER WILL.



END-TO-END TESTING

node-‐webrtcsignaling

media



END-TO-END TESTING


media

Original approach:

-‐ Test app built using node-‐webrtc.

-‐ Acted like a full-‐fledged browser client.

-‐ Ran a suite of tests at high frequency (some tests multiple times / min, other tests every few minutes).



END-TO-END TESTING


media



END-TO-END TESTING


media

Hard to maintain:

-‐ Difficult to keep up-‐to-‐date with WebRTC changes.

-‐ Build challenges (CentOS 6 on AWS).

-‐ (node-‐webrtc has since removed MediaStream API)



END-TO-END TESTING

node signaling

mediarobocallerssignaling



END-TO-END TESTING

node signaling

media

Revised approach:

-‐ Frequent testing for service monitoring through simple node websocket client.

-‐ Automated browser regression testing and “thick” scenario testing with robocallers.

robocallerssignaling



Rollbar



END-TO-END TESTING



END-TO-END TESTING

Manual testing tools:

-‐ Automated tests won’t catch everything.

-‐ Simple, reliably manual testing tools for rapidly running standard scenarios are invaluable.



End-‐to-‐end tests

Anomaly detection

Host monitoring (Nagios)

Manual tools



End-‐to-‐end tests

Anomaly detection

Host monitoring (Nagios)

Manual tools

Layered monitoring

-‐ Event aggregated in Rollbar (and assorted other tools).

-‐ Alerts delivered through PagerDuty.



PLAN FOR FAILURE, PLAN FOR RECOVERY.



Load Balancer A Load Balancer B

Gateway A Gateway B Gateway C

DNS distributionwss://www.mywebrtcservice.com

Registrar


http://www.mywebrtcservice.com




Client

Registrar



x Many



Client

Registrar



x 10,000,000



Client

Registrar



x 10,000,000



Client

Registrar

Rate limit here to protect downstream services



x 10,000,000



Client

Registrar

Be careful about locking here



EXPECT LOUSY NETWORKS.



Network Link Conditioner



https://github.com/vadymy/networkThrottler



.GETSTATS() IS YOUR FRIEND.



pc.getStats()

AMAZON KINESIS

MONITORING

REDSHIFT



SOME OF THE STATS WE CAPTURE…



audioInputLevel

audioOutputLevel

browser

jitter

rtt

received_packets

sent_packets

received_packets_lost

SOME OF THE STATS WE CAPTURE…



SOME OF THE PATTERNS WE LOOK FOR…



High percentage of packets lost

Excessive jitter or rtt

Calls > 5 seconds with no packets sent but > 0 audio-‐input level (or the inverse; evidence of one-‐way audio)

…look for patterns by Twilio account, endpoint identity,

browser version, etc.

SOME OF THE PATTERNS WE LOOK FOR…



OPERATIONAL EXCELLENCE IS A CULTURE ISSUE.



SOME CULTURAL TOOLS



SOME CULTURAL TOOLS

Write the SLA first

Build it, operate it

Five-‐Whys Analysis

Fire Drills

Runbooks

Post-‐Mortems

Time to Recovery Reports



COMMUNICATIONS TOOLS ARE ONLY AS GOOD AS THEIR AVAILABILITY.



ROB BRAZIER [email protected]



kranky geek 2015 - decisions & considerations in building your webrtc app

Technology