martin levy, director, hurricaine electric "the background of internet interconnection"

The background of Internet interconnection

Hurricane Electric IPv6 Native Backbone – Massive Peering!

BEREC expert workshop on IP-Interconnection in cooperation with OECD

Brussels – 2nd November 2011

Martin J. Levy, Director IPv6 Strategy Hurricane Electric

Ask me about Internet peering!!

Agenda – Internet interconnections (Peering?)

!! Defining terms "! Long understood within the industry "! Subject to change and interpretation as needed "! Useful numbers

!! Reality "! It’s not as clear-cut as it seems

!! The cost model "! Highly simplified

!! Dissecting a peering agreement "! Some serious hidden gotcha’s

2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection

2

PEERING & TRANSIT


3

Redefining the terms

!! Peering "! The exchange of traffic between two networks "! Only for traffic associated with the network, or its customers "! Can be via an IX/NAP or a private interconnect "! Subject to a bilateral or unilateral contract "! Either zero-cost, charged-for service or as part of other contracts "! Majority of negotiations done on a bilateral basis

!! Transit "! A network providing another network with a full connection to the Internet "! Sometimes provided as a “default route” or as full routes

!! Partial Transit "! A network providing access to only some of the global Internet "! Sometimes sold as access to a specific provider or destination


4

Internet peering points (IXs/NAPs) exist globally


5

Tokyo

JPNAP JPIX DIX-IE EQUINIX

Los Angeles

EQUINIX ANY-2 LAIIX

Seattle SIX

EQUINIX

San Jose EQUINIX

ANY-2 AMES

São Paulo

NAP do Brasil PTTMetro

Buenos Aires CABASE

Miami

PAIX NOTA Virginia area

EQUINIX ANY2

Atlanta

TIE TELX EQUINIX

Chicago EQUINIX

Chicago IX

Toronto EQUINIX

TORIX Dublin

INEX

NJ & NY EQUINIX TIE TELX NYIIX BigApe

London LINX

LONAP

Paris EQUINIX FRANCEIX SFINX FreeIX

Brussels BNIX FreeBiX

Amsterdam AMSIX NL-IX

Oslo NIX

Boston MXP

Johannesburg JINX

Nairobi KIXP

Singapore

EQUINIX SOX

Auckland

APE V6IX

Beijing National IX

Lisbon GigaPix

Wellington WIX

Sydney EQUINIX PIPE

Santiago NAP Chile

Madrid ESPANIX

Geneva CERN

Moscow MSK-IX

Frankfurt DE-CIX KleyReX

Stockholm NetNod STHIX SOL-IX

Jakarta IIX

Zurich

SwissIX EQUINIX

Prague NIX CZ

Vienna VIX

Milan

MIX MINAP

Hong Kong HKIX

EQUINIX Seoul

KIDC KINX

Taipei TWIX

Mumbai NIXI

New Delhi NIXI

Kuala Lumpur MyIX

Dallas

EQUINIX TIE TELX

Guangzhou National IX

Shanghai National IX

Major IX

s/NA

Ps represented; plenty m

ore exist

Internet IXs/NAPs – fundamental to the Internet


6

!! Internet peering points exist globally "! High traffic densities in the US, EU & parts of Asia

!! London, Amsterdam, Frankfurt, Bay Area, Virginia, Tokyo, etc.

!! Corporate structure: "! Member-based or Commercial

!! Nearly always public stats "! Member list & traffic stats vital

!! Highly unregulated "! Slight exceptions: India, Chile, etc.

Back of the envelope design for

MA

E-W

est in San Jose C

alifornia

Is the Internet growing? Is there more peering?

!! Internet Growth still happening "! More growth within IPv6; which makes sense "! More networks joining the global Internet

!! This is just a measure of destinations on the Internet "! It’s not measure of peering or transit relationships


http://bgp.he.net/report/prefixes

Network Prefixes ASNs

10% Growth in IPv4 table

78% Growth in IPv6 table

Who are the big Internet peering players?

!! This is a measure of BGP adjacencies "! Customers are adjacencies "! Peers are adjacencies

!! A high adjacencies means there’s a short path "! Short as in “AS PATH” vs. physical path


http://bgp.he.net/report/peers

IPv6 IPv4

Can one diagram peering and transit?


Tele2 (Sweden)

Sprint – providing transit

AOL Level3 NTT/Verio Verizon AT&T Qwest/CenturyLink Savvis TATA Global Crossing

IIJ DTAG AboveNet Telia TInet Cogent Telecom Italia Hurricane Electric C&W

Telecom Italia

Verizon– providing transit? (nah!)

IIJ

Can one diagram peering and transit?


BICS (Belgacom Intl’)

Level3 – providing transit

NTT – providing transit

Global Crossing – providing transit

TATA – providing transit

REALITY


11

The reality within the peering and transit world

!! Peering as a donut model "! Smaller players peer around the core "! This bypasses “Tier1 players” – and works somewhat well

!! Peering can be more expensive than transit "! Over the last few years the cost of building a peering-based

network has grown to be greater than buying transit "! Peering requirements force applicants to expand to match those

requirements "! Peering requirements chance

!! Peering sometimes means non-peering related costs "! You can get peering if your paying for something else


12

Peering – “I felt a great disturbance in the Force”


13

1997 was a good year for the peering world to “wake up” …

HOW WE GOT HERE


14


15

1981!RFC791!

J a n u a r y 1 s t !

1983!

DNS!

1984! 1987!10,000 hosts!

UUnet founded!

1988!First computer worms!distributed via the Internet!

1990!ARPAnet name goes away!

1991!CIX! 1992!

WWW!NSFnet upgrades to!

DS3’s (45 Mbps)!

1993!InterNIC!Mosaic!

1994!IETF kicks off IPng (IPv6)!

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010!

EXPLOSIVE GROWTH!!!!!

2011!CONTENT! EYEBALLS!

What’s all the fuss about?


16

Commercial pressure and classic business sense

vs.

TRANSIT COSTS


17

Transit costs – but how much depends on where


18

Cheap

Expensive

Cheap

Expensive

Expensive

Expensive

Cheap

Moderate

Very Expensive

Moderate

THE INVERSE COST MODEL


19

Transit costs – counter intuitive – inverse distance


20

!

!!!!!!

!!!!

!!!!! !!!!!

Country-Based ISP/Operator

Content

Content

Content

Content

Cheaper transit cost for content / more expensive for operator

Higher latency (and potential packet loss)

Content closer to customer (perceived to be better)

Proven better performance

PEERING AGREEMENT REVIEW


21

Peering Agreement – The review

!! What follows is a sentence-by-sentence review of the Comcast Settlement-Free Interconnect (SFI) Policy

!! Can be found at: http://www.comcast.com/peering/ "! Text dated Jan/2008 June/2009 July/2011

!! Anything with strikethrough is from an earlier version

!! This peering language can change at any time

!! Not actually called “peering” but “SFI”


22

Sample peering agreement (public information)


23

Comcast Communications, Inc., Comcast Cable Communications Management, LLC, through its operating subsidiaries ("Comcast"), seeks to interconnect its IP network with other Internet backbone providers on a settlement-free basis when such interconnection provides tangible benefits to Comcast and its customers.

For purposes of this SFI policy, an Internet Network must be a single Autonomous System (“ASN”).

The criteria involving traffic measurement below applies to IPv4 IPv4 and IPv6 and must be met by the Applicant for three consecutive months. To apply for IPv4 or IPv6 settlement-free interconnection with Comcast an email must be sent to [email protected] containing evidence of meeting all criteria listed in the policy. Applicants will be responded to within a reasonable timeframe to discuss their request.

Networks that do not meet the requirements of the settlement-free interconnection policy, but remain interested in Comcast dedicated internet access services may also address their requests to [email protected]. In this case the subject line of the e-mail should indicate a need for sales assistance. The body of the message should include as many details as possible, such as location and telephone number, to expedite the redirect for assistance.



24

•! IPv4 IPv4 and IPv6 Settlement-Free Interconnection Requirements for Comcast AS7922

•! Applicant must operate a US-wide IP backbone whose links are primarily 10 Gbps or greater.

•! Applicant must meet Comcast at a minimum of four mutually agreeable geographically diverse points in the US. Interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast, and must currently be chosen from Comcast peering points in the following list of metropolitan areas: New York City/Newark NJ, Ashburn, Atlanta, Miami, Chicago, Denver, Dallas, Los Angeles, Palo Alto/San Jose, and Seattle.

•! Applicant’s traffic to/from the Comcast network must be on-net only and must amount to at least 3 Gbps peak 7 Gpbs peak 10 Gbps average in the dominant direction. Interconnection bandwidth must be at least 1 Gbps 10 Gbps at each interconnection point.

•! A network (ASN) that is a customer of a Comcast network for any dedicated IP services may not simultaneously be a settlement-free network peer.

•! Applicant must have a professionally managed 24x7 NOC and agree to repair or otherwise remedy any problems within a reasonable timeframe. Applicant must also agree to actively cooperate to resolve security incidents, denial of service attacks, and other operational problems.



25

•! Applicant must maintain responsive abuse contacts for reporting and dealing with UCE (Unsolicited Commercial Email), technical contact information for capacity planning and provisioning and administrative contacts for all legal notices.

•! Applicant must agree to participate in joint capacity reviews at pre-set intervals and work towards timely augments as identified.

•! Applicant must maintain a traffic scale between its network and Comcast that enables a general balance of inbound versus outbound traffic. The end-to-end costs of network cost burden for carrying traffic between networks shall be similar to justify SFI.

•! Applicant must abide by the following routing policy: •! Applicant must use the same peering AS at each US interconnection point and must

announce a consistent set of routes at each point, unless otherwise mutually agreed. •! No transit or third party routes are to be announced; all routes exchanged must be

Applicant's and Applicant's customers' routes. •! Applicant must filter route announcements from their customers by prefix. •! Neither party shall abuse the SFI network peering relationship by engaging in activities such

as, but not limited to: pointing a default route at the other or otherwise forwarding traffic for destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to others.



26

•! Applicant should be willing to enter into an NDA before formal discussions begin.

•! Applicant should be advised that the SFI processes will start with a 90 day trial. On successful completion of that trial, a formal interconnect agreement will be processed. This agreement will renew annually, subject to the then current SFI Policy. During the year if there is a violation of the policy, the agreement and interconnections may be terminated upon written notice to the contacts specified in the agreement.

•! Applicant shall not be permitted to offer or sell any IP transit services providing only AS7922.

•! Applicant must be financially stable.

•! Comcast requires that Applicants seeking SFI in the United States agree to provide reciprocal SFI arrangement with Comcast in the Applicant’s home market.



27

•! Ongoing SFI Reviews with Comcast AS7922

•! Meeting the SFI network peering guidelines set forth herein is not a guarantee that an SFI network peering relationship with Comcast will be established. Comcast shall evaluate a number of business factors and reserves the right not to enter into a SFI network peering agreement with an otherwise qualified applicant. Applicants may apply for settlement-free interconnection once every six (6) months.

•! Existing SFI network peers of Comcast will have their status reviewed periodically to ensure that joint capacity planning intervals are sufficient for growth, contacts are refreshed for operations purposes, and all criteria continue to be met.

•! Comcast reserves the right to terminate SFI network peering, upon a notice period as determined by the parties' agreement, with peers who do not meet the criteria described above.

•! Periodic review of the policies contained here will be conducted to ensure that the criteria and eligibility requirements are consistent with Comcast's business needs. Comcast may modify this SFI policy at any time.

SUMMARY


28

Summary


29

!! Starting in the peering world is easy !! Fraught with “landmines” (once you

play big) !! Financially challenging !! It’s the road to “Tier-1” status

!! Wholesale transit has evolved as a commodity-play with very low margins !! Enterprise pricing is still good !! Bulk hosting sales – not so good

!! Complaining to the regulator? !! Jimmy Wales said it best!

“the greatest threat to the internet was misguided and over-reaching government policy”

…

Financial Times – N

ovember 2

nd 2011 – http://ft.com/

Q&A

Contact: Martin J. Levy Director, IPv6 Strategy Hurricane Electric 760 Mission Court Fremont, CA 94539, USA http://he.net/ martin at he dot net +1 (510) 580 4167

?


Want peering? - The answer is!

always “yes” at Hurricane Electric!

martin levy, director, hurricaine electric "the background of internet interconnection"

Technology

internet growth

global internet

internet growing

big internet peering

internet ixsnaps fundamental

peering transit2

diagram peering

peering agreement