Building our own CDN

How to serve millions of viewers efficiently

Zoltán Németh

at a glance Live streaming

Free broadcastingSoftware as a Service

Big events• Sony Playstation• Nintendo• Lady Gaga• Festivals

Citizen journalism• Rescue of Chilean miners• Earthquake in Japan• Revolutions in Egypt, Syria• Protests in Ukraine

• Obama campaign• Climate Reality• Football• Justin Bieber

Concurrent viewership peaks at almost 2 million

Streaming Definitions

Broadcaster, viewer

Media

Buffering

History

FMS, RTMP

Edge

UHS, HTTP

How we use the network

Live streaming properties

Continuous connection vs

chunk download

Big bandwidth

Stability critical

Large audience

Global delivery

Providers performance

Mobile networks

First approach: CDN

Pros

Scaling, flexibility

Bigger capacity

Cons

Costs

Origin requests

Limitations

Regional and provider

differences

Capacity race conditions

Multiple CDNs Global coverage solved

Costs high

Unified configuration problem

How to select which one to use

Own capacity Why

Cost-effectiveness

Keep CDNs for peaks

Transit lines

Peering

Billing model

95% / Flat rate

How to utilize own network

Basic decision logic

Peering first

Transit second

CDNs last

Cache layer

Varnish

Monitoring

POPs, Edge clusters

Viewer side switching - QoS

List all providers

In case of problem – switch

Challenges:

How to detect a problem

Sync

Broadcaster side problems

Server side prediction

Initial guess for QoS

Minimize useless switches (UX)

Cost optimization

How?

Viewers report metrics

Server side processing and

aggregation

UMS Ustream Media Server

Continuous connection to all

clients

Real-time push-based updates

First use: viewer number

display

Channel status poll

Viewer authentication and

authorization

Stream information

Java, kernel tweaks

UMS Frontend layer

Up to 150.000 clients /

machine

3 IP addresses per machine

1.5 billion outgoing messages

per machine per day (10

billion for the live cluster)

Aggregation / logic layer

Current: 400.000 clients per

media limit

Future: distributed

The data source: Streamstat

Viewer reports

UMS process

Log

Historic data processing

From log

Hadoop + Tableau

Realtime processing in UMS

Chandra

{"appId":1,"appVersion":1,"application":"channel","as":"AS3462 Data Communication Business Group","brandId":"1","city":"Taipei","clientIP":"220.142.6.108","clientId":1421873992,"country":"TW","createTime":1399223703493,"eventType":"STAT_UPDATE","mediaId":"17913321","rpin":"rpin.07680805512286006","rsid":"fvwfnxzx:y1wowjfr","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtocol":"http","size":471654,"streamChunkHash":"1108464946","streamChunkId":"1399223036","streamName":"live_1","streamNetworkProvider":"ntt","streamProtocol":"uhs","time":405,"url":"http://sjc-ucdn03.ntt.tcdn.ustream.tv/sjc-uhs22/streams/httpflv/ustreamVideo/17913321/streams/live_1_1399223036_1108464946.flv"}]}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931444,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://www.ustream.tv/channel/thdudf4"}

{"appId":1,"appVersion":1,"application":"channel","as":"AS15377 ISP Fregat Ltd.","brandId":"1","city":"Dnepropetrovsk","clientIP":"46.98.73.39","clientId":1188395153,"country":"UA","createTime":1399224734039,"eventType":"STAT_UPDATE","mediaId":"13166013","rpin":"rpin.23043611550692003","rsid":"0qx26uw2:671wyhbv","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"akamai","networkProtocol":"http","size":214689,"streamChunkHash":"225119325","streamChunkId":"1399223580","streamName":"live_1","streamProtocol":"uhs","time":241,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223580_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":180117,"streamChunkHash":"225119325","streamChunkId":"1399223581","streamName":"live_1","streamProtocol":"uhs","time":259,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223581_225119325.flv"},{"cdnProvider":"akamai","networkProtocol":"http","size":198214,"streamChunkHash":"225119325","streamChunkId":"1399223582","streamName":"live_1","streamProtocol":"uhs","time":164,"url":"http://uhs-akamai.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223582_225119325.flv"},{"cdnProvider":"level3","networkProtocol":"http","size":192575,"streamChunkHash":"225119325","streamChunkId":"1399223579","streamName":"live_1","streamProtocol":"uhs","time":6044,"url":"http://uhs-level3.ustream.tv/ams/ams-uhs03/streams/httpflv/ustreamVideo/13166013/streams/live_1_1399223579_225119325.flv"}]},"common":{"pvdname":"uhs_akamai"}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 11,6,602,180","webUrl":"http://www.ustream.tv/channel/new-odessa"}

{"appId":1,"appVersion":1,"application":"channel","as":"AS5615 Koninklijke KPN N.V.","brandId":"1","city":"Eindhoven","clientIP":"82.170.189.202","clientId":469374750,"country":"NL","createTime":1399224671784,"eventType":"STAT_UPDATE","mediaId":"17926037","rpin":"rpin.5570720779755052","rsid":"4uv3mjhk:naytle8n","stats":{"benchmark":{"bwrpt":[{"cdnProvider":"ucdn","networkProtocol":"http","size":343420,"streamChunkHash":"118845438","streamChunkId":"1399206465","streamName":"live_1","streamNetworkProvider":"ntt","streamProtocol":"uhs","time":2083,"url":"http://sjc-ucdn05.ntt.tcdn.ustream.tv/ams-uhs03/streams/httpflv/ustreamVideo/17926037/streams/live_1_1399206465_118845438.flv"}]}},"swfUrl":"http://static-cdn1.ustream.tv/swf/live/viewer.rsl:633.swf","time":1399224931445,"umsId":"sjc-ums01","userAgent":"WIN 13,0,0,206","webUrl":"http://goodcast.tv/e/5.html"}

Streamstat reports

Types of reports

Playing

Buffering

Bandwidth

Errors, QoS events

Verticals

Geo

AS number

Media

Chandra Real-time data aggregation layer

Replaceable data store

Redis

Simple logic: based on increments

Easily scalable

Chandra demo

Decisions Ways of routing viewers

Initial setup

Updates

Stream priority

Stream module

Ways of decision

Rule-based

Dynamic

Stream module generation flow

Filter the Stream Priority list

Dedications / whitelists

Stream module generation flow

Filter the Stream Priority list

Dedications / whitelists

Stream module generation flowRule blocking UCDN

Routing on own network

Provider resolution service

Reject to CDN if full

Capacity monitoring

Lines

Clusters

What we have now

3 big CDN providers

~5 smaller providers

UMS clusters

Core in SJC

POPs: NRT, AMS, BUD

Exchanges

QoS and static rules

Future plans Growth

Dynamic rules

QoS and MBR

UMS Elastic Logic

Open sourcing

Framework

Chandra

Questions

syntaxerror@ustream.tv