Traffic delivery evolution in the InternetENOG 4 – Moscow – 23rd October 2012

Christian KaufmannDirector Network ArchitectureAkamai Technologies, Inc.January 29th, 2008

©2012 Akamai

way-back machineway-back machine

FASTER FORWARD

©2012 Akamai

way-back machineway-back machine

Web 1998

FASTER FORWARD

©2012 Akamai

Web 1998

FASTER FORWARD

©2012 Akamai

Web 2012

FASTER FORWARD

©2012 Akamai

Web and Cloud 2012

FASTER FORWARD

©2012 Akamai

So what’s the difference?

• Lots of “high definition” content being pushed to end users

• Read “large files”

• Can the Internet scale to support this?

-> Short answer: NO

FASTER FORWARD

©2012 Akamai

Key Issues

• Problems with a centralized approach, especially for large media files

• Problems with Peering

• Problems with routing protocols

• Inter AS Multicast not really existent

• QoS not really existent and consistent

FASTER FORWARD

©2012 Akamai

Media

Gaming

eCommerce

Mobile

Blogging

Online Banking

Social Networking

Trends 2012

Immersive

FASTER FORWARD

©2012 Akamai

Media

Gaming

eCommerce

Mobile

Blogging

Online Banking

Social Networking

Simple on the outside …

Immersive

FASTER FORWARD

©2012 Akamai

… complicated on the inside

Media

Gaming

eCommerce

Mobile

Blogging

Online Banking

Social Networking

Immersive

L3

Telia DT

IX

IX

Verizon

FASTER FORWARD

©2012 Akamai

•Centralized sites create an inherent bottleneck and target for attackers

•Worldwide user population = huge infrastructure problem

•Not scalable

•Long latency between server and end-user

The Centralization Bottleneck

FASTER FORWARD

©2012 Akamai

•Content Distribution Networks solve the centralization problem by distributing content.

•Greater distribution means greater performance and reliability

The Centralization Bottleneck

FASTER FORWARD

©2012 Akamai

The Edge is Highly Distributed

% ofAccessTraffic

ASes (29,000+)

4.4% Network A3.6% Network B1.8% Network C1.7% Network D1.7% Network E1.7% Network F1.6% Network G1.4% Network H

• No one Autonomous System has more than 4.4% of the access traffic.

• The top 50 ASes add up to only 48%.

FASTER FORWARD

©2012 Akamai

Edge Proximity From 1 Location

3000

1

Locations

Average Miles to Edge

FASTER FORWARD

©2012 Akamai

Edge Proximity From 30 Locations

3000

1500

1 30

Locations

Average Miles to Edge

FASTER FORWARD

©2012 Akamai

Edge Proximity From 1000 Locations

3000

1500

250

1 30 1000

Locations

Average Miles to Edge

FASTER FORWARD

©2012 Akamai

Does Latency Matter for Large File Downloads?

…who cares if the latency to

download a 2-hour DVD

is 10ms or 100ms?

FASTER FORWARD

©2012 Akamai

Big Data Center

Approach

Akamai

The Fat-file Paradox: Latency Limits Throughput…

0.6%

1.4%

1.0%

0.7%

1.6 ms

96 ms.

48 ms.

16 ms

12.2 min.

20 hrs.

8.16 hrs.

2.2 hrs.

Local <100 mi.

Different Continent <6,000 mi.

Cross Continent <3,000 mi.

Regional 500-1000 mi.

…and throughput limits the time to download large files (e.g., 4 GB DVDs)

Download Time

Packet Loss

Network Latency

Distance from Server to User Throughput

44 Mbs

0.4 Mbs

1 Mbs

4 Mbs

FASTER FORWARD

©2012 Akamai

Centralized Approach with “Big Data Centers”

Clusters of Web servers in largedata centers at the core of the Internet

• Tens of data centers

• Transit provided by large backbone ISPs

FASTER FORWARD

©2012 Akamai

• Economic considerations limit peering capacity – results in congestion and poor performance

• Routing algorithms (BGP) ignore congestion

• BGP ignores latency

• Data used to determine routes is subject to intentional inaccuracies and human error

The Problems with Peering

FASTER FORWARD

©2012 Akamai

BGP picks the “best” route and all packets flow over that path

Normal traffic flow

CW

Web Server

Level 3

SprintEnd User

FASTER FORWARD

©2012 Akamai

CW

Web Server

Level 3

SprintEnd User

If there is congestion on a link, BGP will continue to send packets down that link

Normal traffic flow

FASTER FORWARD

©2012 Akamai

Web Content

Web Users

$

Infrastructure

50X50X

20X20X

6X6X

BottleneckEconomics

$$

$$$

$$$

$$$

First Mile

Hosting

Broadband

Last Mile

MiddleMile

FASTER FORWARD

©2012 Akamai

FirstMile

FirstMile

Edge ISPs

Edge ISPs

FirstMile

Edge ISPs Edge ISPs Edge IPSs

Edge ISPs Edge ISPs

The Big Data Center Approach to Content Delivery

Data Centers(10s)

Local ISPs(1,000s)

FASTER FORWARD

©2012 Akamai

FirstMile

FirstMile

Edge ISPs

Edge ISPs

FirstMile

Edge ISPs Edge ISPs Edge IPSs

Edge ISPs Edge ISPs

The Big Data Center Approach to Content Delivery

Problem 1 Data centers are on the wrong side of the bottleneck

Data Centers(10s)

Bottleneck

Users

Local ISPs(1,000s)

FASTER FORWARD

©2012 Akamai

The Big Data Center Approach to Content Delivery

Problem 1 Data centers are on the wrong side of the bottleneck

Edge ISPs

Edge ISPs Edge ISPs Edge ISPs Edge IPSs

Edge ISPs Edge ISPs

Tier 1IPS’s

Tier 1IPS’s

Tier 1IPS’s

Solution Caching servers are on the right side of the bottleneck

Users

Bottleneck

Data Centers(10s)

Local ISPs(1,000s) Cache Servers

FASTER FORWARD

©2012 Akamai

FirstMile

FirstMile

FirstMile

CDNCDNCDN

The Big Data Center Approach to Content Delivery

Edge ISPs

Edge ISPs Edge ISPs Edge ISPs Edge IPSs

Edge ISPs Edge ISPs

Problem 2 Data centers are far from end users

Local ISPs (1,000s)

Data Centers(10s)

Users

500-5,000 miles

FASTER FORWARD

©2012 Akamai

Tier 1IPS’s

Tier 1IPS’s

Tier 1IPS’s

CDNCDNCDN

The Big Data Center Approach to Content Delivery

Edge ISPs

Edge ISPs Edge ISPs Edge ISPs Edge IPSs

Edge ISPs Edge ISPs

Problem 2 Data centers are far from end users

Local ISPs (1,000s)

Data Centers(10s)

Users

Solution Cache servers are close to end users

Cache Servers

10-100 miles

500-5,000 miles

FASTER FORWARD

©2012 Akamai

The Akamai System

The world’s largest on-demand, distributed computing platform

delivers all forms of Web content and applications for over 130,000 domains.

Resulting in traffic of:13 Tbps peak traffic100+ petabytes / day1,468+ billion hits / day 560+ million unique clients IPs / day

The Akamai EdgePlatform:

120,000+Servers

1928POPs

83Countries

1069Networks

660+ Cities

FASTER FORWARD

©2012 Akamai

Conclusion: How to deliver HD traffic in the future?

• Overlay CDN networks are state of the Art

• Akamai, Google, Netflix, using on-net servers

• Protocols like Multicast and QoS are not the solution

• CDN Interconnects and Federations are not the solution either

• Peer to peer networks - unclear so far

• Waiting for the next big idea

FASTER FORWARD

©2012 Akamai

Questions?

• ck@akamai.com

FASTER FORWARD