the next wave of massive disruptions to the peering ecosystem
DESCRIPTION
The Next Wave of Massive Disruptions to the Peering Ecosystem. William B. Norton Co-Founder & Chief Technical Liaison Equinix, Inc. Asia Pacific Peering Forum Singapore, October 5, 2006. Slide set v0.9. Internet Operations White Papers. Name:. William B. Norton - PowerPoint PPT PresentationTRANSCRIPT
The Next Wave of
Massive Disruptions to the Peering Ecosystem
Asia Pacific Peering ForumSingapore, October 5, 2006
William B. NortonCo-Founder & Chief Technical Liaison
Equinix, Inc.
Slide set v0.9
Internet Operations White Papers
Name: William B. Norton Internet Researcher
What is Peering? When does it make sense?
On the Internet Everyone is a Publisher
Internet Operations White Papers1) “Interconnection Strategies for ISPs”2) “Internet Service Providers and Peering”3) “A Business Case for Peering”4) “The Art of Peering: The Peering Playbook”5) “The Peering Simulation Game”6) “Do ATM-based Internet Exchanges Make Sense
Anymore?” 7) “Evolution of the U.S. Peering Ecosystem”8) “The Asia Pacific Internet Peering Guidebook”9) “The Folly of Peering Traffic Ratios?”
Freely available. See Web site or send e-mail to [email protected] makes anyone a publisher, similar effect now emerging for video
Massive Disruption in U.S. Peering Ecosystem Short Videos
• YouTube – founded 2005– Short video clips – 50 million view per day!– 20Gbps of peering traffic Feb 2006– $1M/month in Sept 2006!– Entering Peering Ecosystem– 30 Other competitors600Gbps peerable?
• DoveTail• Video may dwarf current peered traffic
– 2010 – 80-90% Internet is Video– Inculcate video guys into peering ecosystem
On the Internet Everyone is a Broadcaster
Short video clips…Full TV shows…
Source: http://www.nanog.org/mtg-0606/norton.html
Source: http://digg.com/tech_news/YouTube_Gets_Bandwidth_Boost_from_Level_3
Massive Disruption in U.S. Peering Ecosystem Full Episodes
• “Desperate Housewives” – 210MB/hour– For 320x240 H.264 Video iTunes image
• 10,000,000 households
• 2,100,000,000 MB = 2.1 peta-Bytes
• How long will that take to download?
Source: http://www.pbs.org/cringely/pulpit/pulpit20060302.html
Historical Perspective…review 5yr disruptions…
3 days @ 64Gbps non-stop !Just one showTry 250M*180 Channels*HDTV
Evolution #1
1) Volume of traffic is huge2) Cable Cos Open Peering3) “Kazaa Effect” amplifies peering benefits
Scale: O(20Gbps) peered
Significant Evolution…
2002: Evolution #1Cable Companies Peer
T1 ISPs
T2 ISPs
Content
1) Volume of traffic is huge2) Content is Open Peering3) Improves End-User Experience4) Leading Players are paving the way
Scale: O(100Gbps) peered
Significant Evolution…
T2 ISPs
Content
T1 ISPs
T2 ISPs
Content
2002: Evolution #2Large Scale
Content Players Peer
1) Volume of traffic pulled away from T1s is huge2) Reduces perceived need forT1s (for local delivery anyway)3) T1s still needed for distance
Content Literally directly onThe Cable Company NetworkScale: O(100Gbps)
Significant Evolution…
2002: Evolution #3 Cable Cos Peer w/Large Scale Content Players
T2 ISPs
Content
T2 ISPs
Content
T1 ISPs
Content
T2 ISPs
1) Volume of traffic is huge2) Most Traffic is Regional3) Massive Growth4) Many Emerging players5) Video size growth
Scale: > O(600Gbps)
Significant Evolution…
2006: Evolution #4 VideoPeering
T2 ISPs
Content
T2 ISPs
Content
T1 ISPs
Content
T2 ISPs
Video ServiceProviders
Notes: Questionable if aggregate capacity existsTBD Impact of CDN/P2P/Satellite/caching/etc.Net Neutrality Issues not considered here
Research Topic
• Massive Wave of Internet Traffic– 90% of all Internet bits by 2010
• How will Video Service Providers distribute this massive amount of Video Traffic over the Internet?
Modeling the Video Service Provider Distribution Networks
Four Models
1. Commodity Transit
2. CDN
3. Transit/Peering/DIY CDN
4. Peer2Peer
Four Load ModelsA: Small LoadB: Medium LoadC: Large Load
Models A:10 videos B: 100 C: 1000
1: Transit Model 1A Model 1B Model 1C
2: CDN Model 2A Model 2B Model 2C
3: Hybrid Model 3A Model 3B Model 3C
4: P2P Model 4A Model 4B Model 4C
Load Model A
• Load Model A – Light Load: Every 5 minutes, 10 customers each start to download a 1.5 GB movie, resulting in an average 15GB five minute load.
MbpsGbpss
GB4004.0
Gbyte
Gbits8*
sec
GB0.05
sec300
15
Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean
Average Load 400 Mbps95th Percentile Load 4 1,600 MbpsPeak Load 6.6 2,640 Mbps
Load Model B
• Load Model B – Medium Load: Every 5 minutes, 100 customers each start to download a 1.5 GB movie, resulting in an average 150GB five minute load.
Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean
Average Load 4,000 Mbps95th Percentile Load 4 16,000 MbpsPeak Load 6.6 26,400 Mbps
Load Model C
• Load Model C – Large Load: Every 5 minutes, 1000 customers each start to download a 1.5 GB movie, resulting in an average 1500GB five minute load.
Adjust load to sinusoidal customer traffic demand curve Jeff Turner: 6.6:1 peak-to-mean
Average Load 40,000 Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 Mbps
Assumptions
• 1000 full length 1.5GB videos• Equipment Costs• Transit Costs• Colo Costs• Staff Costs• Software=LAMP• Colo@IX• Multi-homed
Model 1: Commodity Transit
Business Premise:• VSP focuses on core competence• Transit Providers handle traffic better and cheaper
– Economies of scale, Aggregation, Expertise, Billing, Peering, etc. Model A: 1,600Mbps @ $25 /Mbps
Model B: 16,000Mbps @ $18 /Mbps
Model C:160,000Mbps@$10 /Mbps
(Note: the pricing here has generated a lot of controversy. About half the reviewers say the prices are about right or a bit lower than the market price, and others stating that these prices are higher than the current market price.)
Model 1A: Transit Light Load
Server GigESwitch
2gigE
Upstream ISPs
Server
Server
2 * 10GE to upstreams
::
1U ServerDual Core Opteron4GB RAM4 500GB disks2 gigE uplinks$4000 each Sustain 1Gbps each
Distribution GigE Switch48 port GigE for servers2 10GE for upstream$10,000Add another at 48Gbps
RouterCisco 6509Sup720-3bxl4-port 10GE,48 port GE, $60,000Scales to 100Gbps
Router
Server-Distribution-Core Model
10G
Average Load 400 Mbps95th Percentile Load 4 1,600 MbpsPeak Load 6.6 2,640 MbpsModel 1A - Simple Commodity Transit for Video Distribution
monthlyTransit Fee 1,600 mbps@ $25 perMbps $40,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6509 $60,000 3yrAmort $1,667AggregationSwitch 1 $10,000 3yrAmort $2781U 1G Servers 3 $4,000 3yrAmort $333Staff 0.5 $180,000 $7,500Total $51,278
# videos downloaded per month 86,400Cost per video downloaded $0.59
Model 1B: Transit Medium Load
Server1 GigESwitch
Upstream ISPs
:
Server24
3 * 10GE to upstreams
::
1U ServerDual Core Opteron4GB RAM4 500GB disks2 gigE interfaces$4000 each Sustain 1Gbps each
Distribution GigE Switch48 port GigE for servers2 10GE for upstream$10,000Add another every 24 servers
RouterCisco 6509Sup720-3bxl2*4-port 10GE, $80,00030Gbps from switches 30Gbps to Upstreams
Router
Server-Distribution-Core Model
10G
GigESwitch
Server25
Server26
Server27
:
:
Average Load 4,000 Mbps95th Percentile Load 4 16,000 MbpsPeak Load 6.6 26,400 MbpsModel 1B - Simple Commodity Transit for Video Distribution
monthlyTransit Fee 16,000 mbps@ $18 perMbps $288,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6509 $80,000 3yrAmort $2,222AggregationSwitch 2 $10,000 3yrAmort $556Servers 27 $4,000 3yrAmort $3,000Staff 0.75 $180,000 $11,250Total $306,528
# videos downloaded per month 864,000Cost per video downloaded $0.35
Model 1CServer1 GigE
Switch1
Upstream ISPs
:
Server24
8 * 10GE to upstreams each
::
Distribution GigE Switch48 port GigE for servers2 10GE for upstream$10,000Add another every 24 servers
RoutersCisco 6509Sup720-3bxlw/4*4-port 10GE, $150,00080Gbps from switches, 80Gbps to upstreams
Router1
10G
Server262
Server263
Server264
:
:
GigE Switch14
:
Router2Router2
Router4
10G
Average Load 40,000 Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 1C - Simple Commodity Transit for Video Distribution
monthlyTransit Fee 160,000 mbps@ $10 perMbps $1,600,000Colo 14 rack@ $2,000 perRack $28,000Network Equip 4 6509 $150,000 3yrAmort $16,667AggregationSwitch 14 $10,000 3yrAmort $3,889Servers 264 $4,000 3yrAmort $29,333Maintenance 15% $7,483Staff 3 $180,000 $45,000Total $1,730,372
# videos downloaded per month 8,640,000Cost per video downloaded $0.20
Model 2: Content Delivery Networks (CDNs) for the
Distribution of Video Content • Business Premise: Single-site transit traffic
traverses potentially many network devices, increasing latency and the potential of packet loss:– By spreading web objects closer to the eyeball
networks latency is reduced– Fewer network elements are traversed so reliability is
improved– Congestion points in the core of the Internet are
avoided– CDNs have the expertise, deployed infrastructure,
economies of scale from aggregation efficiencies.
Assumption CDN Price Points
Model A: 1,600Mbps @ $35 /Mbps
Model B: 16,000Mbps @$23 /Mbps
Model C:160,000Mbps@$13 /Mbps
Model 2A: CDN Light Load
Last MileEyeball Networks
P P P P PP
TT
TTTT
Server
6503
CDN Pod
CDN Pod
CDN Pod
Aggregate Transit measured hereCustomer maintains“seed”, CDN distributes it
IX IX IX
End-users are sent toCDN local cache, instead of the “seed”
Entire “seed”(1000 videos)cached atedge?
Model 2A: CDN Light LoadAverage Load 400 Mbps95th Percentile Load 4 1,600 MbpsPeak Load 6.6 2,640 MbpsModel 2A - Content Delivery Network for Video Distribution
monthlyTransit Fee 1,600 mbps@ $35 perMbps $56,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $66,311
# videos downloaded per month 86,400Cost per video downloaded $0.77
Model 2B: CDN Medium LoadAverage Load 4,000 Mbps95th Percentile Load 4 16,000 MbpsPeak Load 6.6 26,400 MbpsModel 2B - Content Delivery Network for Video Distribution
monthlyTransit Fee 16,000 mbps@ $23 perMbps $368,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $378,311
# videos downloaded per month 864,000Cost per video downloaded $0.44
Model 2C: CDN Large LoadAverage Load 40,000 Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 2C - Content Delivery Network for Video Distribution
monthlyTransit Fee 160,000 mbps@ $13 perMbps $2,080,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $2,090,311
# videos downloaded per month 8,640,000Cost per video downloaded $0.24
Model 3: Transit/Peering/DIY CDN
• Business Model Premise: Operation of the Internet distribution is seen as strategic to the VSP:– End-user experience is mission-critical so outsourcing the end
user experience to a transit provider or CDN puts the VSP at risk.
– The VSP has visibility into what video are being released, which ones are likely to be hot and which ones don’t require special infrastructure adjustments.
– Internet Video distribution is so new that the VSP prefers control. This is a strategic focus of the VSP: ensuring reliability, scalability, through the constant monitoring and evolution of the infrastructure to ensure the end user experiences during these early phases of Internet Video Distribution.
– The traditional CDN may be ill-suited to distribute very large video object, therefore we have to do it yourself.
Model 3: Transit/Peering Light Load
S R
S R
R S
Last MileEyeball Networks
P P P P PP
TT
TTTT
Model 3A: Peering/Transit Light Load
Average Load 400 Mbps95th Percentile Load 4 1,600 MbpsPeak Load 6.6 2,640 MbpsModel 3A - Blended Transit and Peering for Video distribution3 site 25% peering monthlyTransit Fee 1,200 mbps@ $25 perMbps $30,000Colo 3 rack@ $3,000 rack+port $9,000Network Equip 3 6509 $90,000 3yrAmort $7,500AggregationSwitch 1 $10,000 3yrAmort $278Servers 3 $4,000 3yrAmort $333Maintenance 15% $1,217Staff 0.75 $180,000 $11,250Total $59,578
# videos downloaded per month 86,400Cost per video downloaded $0.69
Model 3B: Transit/Peering Medium Load
Average Load 4,000 Mbps95th Percentile Load 4 16,000 MbpsPeak Load 6.6 26,400 MbpsModel 3B - Blended Transit and Peering for Video distribution3 site 25% peering monthlyTransit Fee 12,000 mbps@ $18 perMbps $216,000Colo 3 rack@ $10,000 rack+port $30,000Network Equip 3 6509 $90,000 3yrAmort $7,500AggregationSwitch 2 $10,000 3yrAmort $556Servers 12 $4,000 3yrAmort $1,333Maintenance 15% $1,408Staff 1 $180,000 $15,000Total $271,797
# videos downloaded per month 864,000Cost per video downloaded $0.31
Model 3C: Transit/Peering Heavy Load
Average Load 40,000 Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 3C - Blended Transit and Peering for Video distribution3 site 25% peering monthlyTransit Fee 120,000 mbps@ $10 perMbps $1,200,000Colo 42 rack@ $2,000 rack+port $84,000Network Equip 12 6509 $150,000 3yrAmort $50,000AggregationSwitch 42 $10,000 3yrAmort $3,889Servers 792 $4,000 3yrAmort $88,000Maintenance 15% $21,283Staff 3 $180,000 $45,000Total $1,492,172
# videos downloaded per month 8,640,000Cost per video downloaded $0.17
Model 4: Peer2Peer
• Business Model Premise: The current Internet Service Providers and CDNs at the core can not handle the load across single or even multiple locations:– Backbone, peering interconnects, and the hundreds of
thousands of routers deployed can not handle the load of today and tomorrows video.
– the leaf nodes (i.e. Grandma’s PCs left on) in aggregate have the cycles and network capacity, if shared, to distribute popular content today.
– Popular content can be chopped up into small chunks such that many downloaders become sources, and topologically close downloaders will prefer the topologically close sources. This ‘swarmcasting’ requires only a source ‘seed’, and a lookup mechanism for the first downloaders to find the seed, and then to direct future downloaders to topologically closer sources.
Model 4A: P2P Light Load
S R
Last MileEyeball Networks
T
T
Model 4A: P2P Light LoadAverage Load 400 Mbps95th Percentile Load 4 1,600 MbpsPeak Load 6.6 2,640 MbpsModel 4A - Peer-to-Peer Network for Video Distributionsingle-site stormcasting monthlyTransit Fee 100 mbps@ $50 perMbps $5,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $15,311
# videos downloaded per month 86,400Cost per video downloaded $0.18
Model 4B: P2P Medium Load
S R
Last MileEyeball Networks
T
T
Model 4B: P2P Medium LoadAverage Load 4,000 Mbps95th Percentile Load 4 16,000 MbpsPeak Load 6.6 26,400 MbpsModel 4B - Peer-to-Peer Network for Video Distributionsingle-site stormcasting monthlyTransit Fee 100 mbps@ $50 perMbps $5,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $15,311
# videos downloaded per month 864,000Cost per video downloaded $0.0177
Model 4C: P2P Large Load
S R
Last MileEyeball Networks
T
T
Model 4C: P2P Large LoadAverage Load Mbps Mbps95th Percentile Load 4 160,000 MbpsPeak Load 6.6 264,000 MbpsModel 4C - Peer-to-Peer Network for Video Distributionsingle-site stormcasting monthlyTransit Fee 100 mbps@ $50 perMbps $5,000Colo 1 rack@ $1,500 perRack $1,500Network Equip 1 6503 $30,000 3yrAmort $833Servers 1 $4,000 3yrAmort $111Maintenance 15% $367Staff 0.5 $180,000 $7,500Total $15,311
# videos downloaded per month 8,640,000Cost per video downloaded $0.0018
SummaryInternet Video Distribution Methods
$0.00
$0.10
$0.20
$0.30
$0.40
$0.50
$0.60
$0.70
$0.80
$0.90
A B C
Load Models
$/V
ideo
Do
wn
load
Model 1:Transit
Model2: CDN
Model3: Transit/Peering
Model 4: P2P
Models A:10 videos B: 100 C: 1000
1: Transit 1A: $0.60 1B: $0.36 1C: $0.20
2: CDN 2A: $0.77 2B: $0.44 2C: $0.24
3: Hybrid 3A: $0.69 3B: $0.31 3C: $0.17
4: P2P 4A:$0.18 4B: $0.0177
4C: $0.0018
Per Video Cost Of delivery
Acknowledgements
• Vish Yelsangikar (NetFlix), Peter Harrison (NetFlix), Aaron Weintraub (Cogent), Jon Nistor (TorIX), Barrett Lyon (BitGravity), Dave Knight (ISC), Aaron Hughes (Caridien), David Filo (Yahoo!), Jim Goetz (Sequoia Capital), Jason Holloway (DoveTail), Matt Peterson, Richard Steenbergen (nLayer), Lane Patterson (Equinix), Eric Schwartz (Equinix), Pete Ferris (Equinix), David Cheriton (Sun), Andy Bechtolsheim (Sun), Jeffrey Papen (Peak Web Consulting), KC Broberg (Rackable), Henk Goosen (Sun), Geoffrey Noer (Rackable), Jeff Turner (InterStream/nuMetra), Vab Goel (NorWest Venture Partners), Phil Thomas (Quad)
