The background of Internet interconnection
Hurricane Electric IPv6 Native Backbone – Massive Peering!
BEREC expert workshop on IP-Interconnection in cooperation with OECD
Brussels – 2nd November 2011
Martin J. Levy, Director IPv6 Strategy Hurricane Electric
Ask me about Internet peering!!
Agenda – Internet interconnections (Peering?)
!! Defining terms "! Long understood within the industry "! Subject to change and interpretation as needed "! Useful numbers
!! Reality "! It’s not as clear-cut as it seems
!! The cost model "! Highly simplified
!! Dissecting a peering agreement "! Some serious hidden gotcha’s
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
2
PEERING & TRANSIT
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
3
Redefining the terms
!! Peering "! The exchange of traffic between two networks "! Only for traffic associated with the network, or its customers "! Can be via an IX/NAP or a private interconnect "! Subject to a bilateral or unilateral contract "! Either zero-cost, charged-for service or as part of other contracts "! Majority of negotiations done on a bilateral basis
!! Transit "! A network providing another network with a full connection to the Internet "! Sometimes provided as a “default route” or as full routes
!! Partial Transit "! A network providing access to only some of the global Internet "! Sometimes sold as access to a specific provider or destination
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
4
Internet peering points (IXs/NAPs) exist globally
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
5
Tokyo
JPNAP JPIX DIX-IE EQUINIX
Los Angeles
EQUINIX ANY-2 LAIIX
Seattle SIX
EQUINIX
San Jose EQUINIX
ANY-2 AMES
São Paulo
NAP do Brasil PTTMetro
Buenos Aires CABASE
Miami
PAIX NOTA Virginia area
EQUINIX ANY2
Atlanta
TIE TELX EQUINIX
Chicago EQUINIX
Chicago IX
Toronto EQUINIX
TORIX Dublin
INEX
NJ & NY EQUINIX TIE TELX NYIIX BigApe
London LINX
LONAP
Paris EQUINIX FRANCEIX SFINX FreeIX
Brussels BNIX FreeBiX
Amsterdam AMSIX NL-IX
Oslo NIX
Boston MXP
Johannesburg JINX
Nairobi KIXP
Singapore
EQUINIX SOX
Auckland
APE V6IX
Beijing National IX
Lisbon GigaPix
Wellington WIX
Sydney EQUINIX PIPE
Santiago NAP Chile
Madrid ESPANIX
Geneva CERN
Moscow MSK-IX
Frankfurt DE-CIX KleyReX
Stockholm NetNod STHIX SOL-IX
Jakarta IIX
Zurich
SwissIX EQUINIX
Prague NIX CZ
Vienna VIX
Milan
MIX MINAP
Hong Kong HKIX
EQUINIX Seoul
KIDC KINX
Taipei TWIX
Mumbai NIXI
New Delhi NIXI
Kuala Lumpur MyIX
Dallas
EQUINIX TIE TELX
Guangzhou National IX
Shanghai National IX
Major IX
s/NA
Ps represented; plenty m
ore exist
Internet IXs/NAPs – fundamental to the Internet
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
6
!! Internet peering points exist globally "! High traffic densities in the US, EU & parts of Asia
!! London, Amsterdam, Frankfurt, Bay Area, Virginia, Tokyo, etc.
!! Corporate structure: "! Member-based or Commercial
!! Nearly always public stats "! Member list & traffic stats vital
!! Highly unregulated "! Slight exceptions: India, Chile, etc.
Back of the envelope design for
MA
E-W
est in San Jose C
alifornia
Is the Internet growing? Is there more peering?
!! Internet Growth still happening "! More growth within IPv6; which makes sense "! More networks joining the global Internet
!! This is just a measure of destinations on the Internet "! It’s not measure of peering or transit relationships
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
Page 7
http://bgp.he.net/report/prefixes
Network Prefixes ASNs
10% Growth in IPv4 table
78% Growth in IPv6 table
Who are the big Internet peering players?
!! This is a measure of BGP adjacencies "! Customers are adjacencies "! Peers are adjacencies
!! A high adjacencies means there’s a short path "! Short as in “AS PATH” vs. physical path
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
Page 8
http://bgp.he.net/report/peers
IPv6 IPv4
Can one diagram peering and transit?
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
Page 9
Tele2 (Sweden)
Sprint – providing transit
AOL Level3 NTT/Verio Verizon AT&T Qwest/CenturyLink Savvis TATA Global Crossing
IIJ DTAG AboveNet Telia TInet Cogent Telecom Italia Hurricane Electric C&W
Telecom Italia
Verizon– providing transit? (nah!)
IIJ
Can one diagram peering and transit?
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
Page 10
BICS (Belgacom Intl’)
Level3 – providing transit
NTT – providing transit
Global Crossing – providing transit
TATA – providing transit
REALITY
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
11
The reality within the peering and transit world
!! Peering as a donut model "! Smaller players peer around the core "! This bypasses “Tier1 players” – and works somewhat well
!! Peering can be more expensive than transit "! Over the last few years the cost of building a peering-based
network has grown to be greater than buying transit "! Peering requirements force applicants to expand to match those
requirements "! Peering requirements chance
!! Peering sometimes means non-peering related costs "! You can get peering if your paying for something else
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
12
Peering – “I felt a great disturbance in the Force”
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
13
1997 was a good year for the peering world to “wake up” …
HOW WE GOT HERE
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
14
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
15
1981!RFC791!
J a n u a r y 1 s t !
1983!
DNS!
1984! 1987!10,000 hosts!
UUnet founded!
1988!First computer worms!distributed via the Internet!
1990!ARPAnet name goes away!
1991!CIX! 1992!
WWW!NSFnet upgrades to!
DS3’s (45 Mbps)!
1993!InterNIC!Mosaic!
1994!IETF kicks off IPng (IPv6)!
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010!
EXPLOSIVE GROWTH!!!!!
2011!CONTENT! EYEBALLS!
What’s all the fuss about?
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
16
Commercial pressure and classic business sense
vs.
TRANSIT COSTS
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
17
Transit costs – but how much depends on where
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
18
Cheap
Expensive
Cheap
Expensive
Expensive
Expensive
Cheap
Moderate
Very Expensive
Moderate
THE INVERSE COST MODEL
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
19
Transit costs – counter intuitive – inverse distance
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
20
!
!!!!!!
!!!!
!!!!! !!!!!
Country-Based ISP/Operator
Content
Content
Content
Content
Cheaper transit cost for content / more expensive for operator
Higher latency (and potential packet loss)
Content closer to customer (perceived to be better)
Proven better performance
PEERING AGREEMENT REVIEW
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
21
Peering Agreement – The review
!! What follows is a sentence-by-sentence review of the Comcast Settlement-Free Interconnect (SFI) Policy
!! Can be found at: http://www.comcast.com/peering/ "! Text dated Jan/2008 June/2009 July/2011
!! Anything with strikethrough is from an earlier version
!! This peering language can change at any time
!! Not actually called “peering” but “SFI”
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
22
Sample peering agreement (public information)
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
23
Comcast Communications, Inc., Comcast Cable Communications Management, LLC, through its operating subsidiaries ("Comcast"), seeks to interconnect its IP network with other Internet backbone providers on a settlement-free basis when such interconnection provides tangible benefits to Comcast and its customers.
For purposes of this SFI policy, an Internet Network must be a single Autonomous System (“ASN”).
The criteria involving traffic measurement below applies to IPv4 IPv4 and IPv6 and must be met by the Applicant for three consecutive months. To apply for IPv4 or IPv6 settlement-free interconnection with Comcast an email must be sent to [email protected] containing evidence of meeting all criteria listed in the policy. Applicants will be responded to within a reasonable timeframe to discuss their request.
Networks that do not meet the requirements of the settlement-free interconnection policy, but remain interested in Comcast dedicated internet access services may also address their requests to [email protected]. In this case the subject line of the e-mail should indicate a need for sales assistance. The body of the message should include as many details as possible, such as location and telephone number, to expedite the redirect for assistance.
Sample peering agreement (public information)
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
24
•! IPv4 IPv4 and IPv6 Settlement-Free Interconnection Requirements for Comcast AS7922
•! Applicant must operate a US-wide IP backbone whose links are primarily 10 Gbps or greater.
•! Applicant must meet Comcast at a minimum of four mutually agreeable geographically diverse points in the US. Interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast, and must currently be chosen from Comcast peering points in the following list of metropolitan areas: New York City/Newark NJ, Ashburn, Atlanta, Miami, Chicago, Denver, Dallas, Los Angeles, Palo Alto/San Jose, and Seattle.
•! Applicant’s traffic to/from the Comcast network must be on-net only and must amount to at least 3 Gbps peak 7 Gpbs peak 10 Gbps average in the dominant direction. Interconnection bandwidth must be at least 1 Gbps 10 Gbps at each interconnection point.
•! A network (ASN) that is a customer of a Comcast network for any dedicated IP services may not simultaneously be a settlement-free network peer.
•! Applicant must have a professionally managed 24x7 NOC and agree to repair or otherwise remedy any problems within a reasonable timeframe. Applicant must also agree to actively cooperate to resolve security incidents, denial of service attacks, and other operational problems.
Sample peering agreement (public information)
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
25
•! Applicant must maintain responsive abuse contacts for reporting and dealing with UCE (Unsolicited Commercial Email), technical contact information for capacity planning and provisioning and administrative contacts for all legal notices.
•! Applicant must agree to participate in joint capacity reviews at pre-set intervals and work towards timely augments as identified.
•! Applicant must maintain a traffic scale between its network and Comcast that enables a general balance of inbound versus outbound traffic. The end-to-end costs of network cost burden for carrying traffic between networks shall be similar to justify SFI.
•! Applicant must abide by the following routing policy: •! Applicant must use the same peering AS at each US interconnection point and must
announce a consistent set of routes at each point, unless otherwise mutually agreed. •! No transit or third party routes are to be announced; all routes exchanged must be
Applicant's and Applicant's customers' routes. •! Applicant must filter route announcements from their customers by prefix. •! Neither party shall abuse the SFI network peering relationship by engaging in activities such
as, but not limited to: pointing a default route at the other or otherwise forwarding traffic for destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to others.
Sample peering agreement (public information)
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
26
•! Applicant should be willing to enter into an NDA before formal discussions begin.
•! Applicant should be advised that the SFI processes will start with a 90 day trial. On successful completion of that trial, a formal interconnect agreement will be processed. This agreement will renew annually, subject to the then current SFI Policy. During the year if there is a violation of the policy, the agreement and interconnections may be terminated upon written notice to the contacts specified in the agreement.
•! Applicant shall not be permitted to offer or sell any IP transit services providing only AS7922.
•! Applicant must be financially stable.
•! Comcast requires that Applicants seeking SFI in the United States agree to provide reciprocal SFI arrangement with Comcast in the Applicant’s home market.
Sample peering agreement (public information)
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
27
•! Ongoing SFI Reviews with Comcast AS7922
•! Meeting the SFI network peering guidelines set forth herein is not a guarantee that an SFI network peering relationship with Comcast will be established. Comcast shall evaluate a number of business factors and reserves the right not to enter into a SFI network peering agreement with an otherwise qualified applicant. Applicants may apply for settlement-free interconnection once every six (6) months.
•! Existing SFI network peers of Comcast will have their status reviewed periodically to ensure that joint capacity planning intervals are sufficient for growth, contacts are refreshed for operations purposes, and all criteria continue to be met.
•! Comcast reserves the right to terminate SFI network peering, upon a notice period as determined by the parties' agreement, with peers who do not meet the criteria described above.
•! Periodic review of the policies contained here will be conducted to ensure that the criteria and eligibility requirements are consistent with Comcast's business needs. Comcast may modify this SFI policy at any time.
SUMMARY
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
28
Summary
2 Nov 2011 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
29
!! Starting in the peering world is easy !! Fraught with “landmines” (once you
play big) !! Financially challenging !! It’s the road to “Tier-1” status
!! Wholesale transit has evolved as a commodity-play with very low margins !! Enterprise pricing is still good !! Bulk hosting sales – not so good
!! Complaining to the regulator? !! Jimmy Wales said it best!
“the greatest threat to the internet was misguided and over-reaching government policy”
…
Financial Times – N
ovember 2
nd 2011 – http://ft.com/
Q&A
Contact: Martin J. Levy Director, IPv6 Strategy Hurricane Electric 760 Mission Court Fremont, CA 94539, USA http://he.net/ martin at he dot net +1 (510) 580 4167
?
2 Nov 2011 Page 30 Martin J. Levy - Hurricane Electric - BEREC in cooperation with OECD - The background of Internet interconnection
Want peering? - The answer is!
always “yes” at Hurricane Electric!