The newly completed Internet2 network upgrade: capabilities, services and how to connect with them

Heather BoylesDirector, International Relations, Internet2heather@internet2.edu

APAN Network Engineering WorkshopAPAN 24th Meeting, Xi’An China27-31 August 2007

Outline of Talk• Brief Recap of Overall Internet2 infrastructure

• Physical Plant• Networks running on the Internet2 infrastructure

• ESnet• Internet2

– Internet2 IP Network– Internet2 Dynamic Circuit Network (DC Network)

• Other capabilities that can be provisioned over the Internet2 infrastructure

• Connections between the Internet2 infrastructure and non-US networks

• Connectivity details specific to APAN and APAN member networks

Internet2 Infrastructure

• Dedicated fiber pair on national footprint of Level3 Communications• 13,500 route miles

• Infinera optical gear• Owned by Level3, *controlled* by Internet2

NOC

• Have lit 10 waves• Additional waves can be lit on same equipment

4

Internet2 InfrastructureInternet2 Infrastructure

Internet2 Network Optical Switching Node

Level3 Regen Site

Internet2 Redundant Drop/Add Site

ESnet Drop/Add Site

Networks running on the Internet2 infrastructure

• ESnet• ESnet4 – US Dept. of Energy• 2 of the (10)10Gbps waves lit on Internet2

infrastructure• ESnet IP Core and ESnet Science Data

Network (SDN)

• Internet2 networks• Internet2 IP Network• Internet2 Dynamic Circuit Network

Internet2 IP Network

• Replaces the old “Abilene” network• Same AS number 11537• Same routing policies for non-US peers

• Currently uses 1 of the (10)10G wavelengths

• Highly reliable, high-performance IP network• Best-effort packet-based• Native IPv4, IPv6 and multicast for both

*Commercial Peering Service

• New (sort of) to Internet2 IP Network• Old: peering with commercial IPv6 and multicast

networks (this continues)• New: settlement-free peering with commercial IPv4

networks and content providers

• Optional for Internet2 access networks (domestic connectors)

• Implemented via MPLS VPN, uses separate VRF• No change in BGP peering with International

Partners

*Available to domestic (US) paying connectors only

Dynamic Circuit Network (DC Network)

• A new Internet2 network• Although predecessor is the HOPI testbed

• “network” because once connected, can set up circuit to any other connector

• “circuit” because circuit-switched (not packet-switched)

• “dynamic” because operates under a control plane that supports dynamic setup of circuits

DC Network

• Provisioned on Internet2 infrastructure with Ciena CoreDirectors• Allow circuits of bandwidth 50mpbs up to 10Gbps

• Characteristics:• Subject to blocking based on available capacity• Short in duration, no longer than 2 weeks

• Connectors must utilize control plane to set up circuits• Built on DRAGON control plane (NSF-funded

project at U Maryland and USC-ISI-East)

Internet2 Dynamic Circuit Network

Recap: Internet2 networks

• 2 networks with which a non-US network can connect• Internet2 IP Network• Dynamic Circuit Network

• Putting the 2 different networks back together: the hybrid model• Ultimately, transparent to user – application will

decide on packet or circuit-switched• Much investigation yet to do over next several

years• Continue to use HOPI testbed (now provisioned over

Internet2 infrastructure) to try out these ideas

Other capabilities of the Internet2 infrastructure

• Transport Services (Static Circuit Services)• 2 fixed endpoints across the Internet2

infrastructure• 50Mbps to full 10Gbps or multiple 10Gbps waves

• Provisioned over Ciena Core Directors or directly over Infinera

• Set up manually by Internet2 NOC• Can be reserved up to a year in advance• Can be ordered as protected• Duration is weeks to years• Fee is based on distance, bandwidth, duration and

protection scheme

International connections with the Internet2 infrastructure and its networks

• Internet2 IP Network

• Internet2 Dynamic Circuit Network

• Ordering and utilizing Transport Services over the Internet2 infrastructure

Internet2 IP Network

• Very little change from “Abilene” network• Most existing connections and peerings

have transitioned seamlessly to new Internet2 IP Network

• 25 direct* non-US peers• *including via multi-hop BGP peering

How to interconnect and peer with Internet2 IP Network

• In US, prefer to interconnect via international exchange points• PacificWave (north and south)• StarLight• NGIX-East• MAN LAN• Atlantic Wave (via NGIX-East and MAN LAN)

• Outside US, Internet2 has no IP network infrastructure• Transit via established peers• E.g. TEIN2 transits to/from Internet2 via TransPAC2• E.g. India’s ERNET transits to/from Internet2 via GEANT2

• For direct peering, contact heather@internet2.edu

Internet2 IP Network International PeeringMAN LAN

32 Ave. of the AmericasNew York, NY

10GE

NGIX-EastCollege Park,

MD

10GE

StarLight710 North LakeShore DriveChicago, IL

10GEPacificWave-N2001 Sixth Ave Seattle, WA

10GE

PacificWave-S624 S. Grand, 600 W. 7th or 818 W. 7th

10GE

Internet2 IP Network International PeeringMAN LAN

32 Ave. of the AmericasNew York, NY

10GE

NGIX-EastCollege Park,

MD

10GE

StarLight710 North LakeShore DriveChicago, IL

10GEPacificWave-N2001 Sixth Ave Seattle, WA

10GE

PacificWave-S624 S. Grand, 600 W. 7th or 818 W. 7th

10GE

Internet2 IP Network International PeeringMAN LAN

32 Ave. of the Americas

New York, NY

10GE

NGIX-EastCollege Park,

MD

10GE

StarLight710 North LakeShore DriveChicago, IL

10GEPacificWave-N2001 Sixth Ave Seattle, WA

10GE

PacificWave-S624 S. Grand, 600 W. 7th or 818 W. 7th

10GE

AtlanticWaveMANLAN

NGIX-EAST

Southern Crossroads56 Marietta St

Atlanta, GA

AMPATH50 NE 9th St

Miami, FL

10GE via MAN LAN10GE via NGIX-East

Internet2 Dynamic Circuit Network

• New!• We’re all in this together figuring out what this

means• Physical connection• Control plane ‘peering’ – inter-domain control of

circuits

• Internet2-GEANT2 service trials underway to learn more about how to do this• Interested in other service trials: JGN2,

AARNET

Physical connection to the Internet2 DC Network

• In US, again prefer via exchange points where available:• MAN LAN: DC Network is connected to Nortel HDXc and

OME6500• StarLight: DC Network is connected to Force 10 Ethernet

switch• PacificWave: DC Network not connected yet in Seattle or

Los Angeles• Talking with John Silvester about connection to TransLight/PacificWave

ONS15454 that serves AARNET southern link• When PacificWave decides SONET/TDM/Layer 1 equipment, will

connect

• NGIX-East: DC Network not connected yet: will be to NGIX-East Ciena Core Director

MAN LAN 32 Ave. of the Americas

New York, NY

OC192c to MAN LAN Nortel HDXc

NGIX-EastCollege Park,

MD

None yet

StarLight710 North LakeShore DriveChicago, IL

10GE to Force10 Ethernet Switch

PacificWave-N2001 Sixth Ave Seattle, WA

None Yet

PacificWave-S624 S. Grand, 600 W. 7th or 818 W. 7th

None yet

Internet2 DC Network International Connectivity

The Internet2 DC Network control plane

• Internet2 DC Network adopting control plane architecture based on NSF-funded DRAGON project• Jerry Sobieski, MAX and Tom Lehman, USC-ISI

• DRAGON using GMPLS to build layer 2 circuits

• More information on DRAGON/Internet2 DC Network control plane at: http://events.internet2.edu/2007/spring-mm/sessionDetails.cfm?session=3165&event=267

“Peering” control planes

• Working jointly with ESnet and DANTE (GEANT2, Autobahn) on inter-domain control plane development

• Also University of Amsterdam

• Pursuing jointly developed web services-based solution to tie together heterogeneous control planes

• Initial release real soon now; official release Fall 2007

• For good overview, see presentation from Joint Techs at: http://www.internet2.edu/presentations/jt2007jul/20070717-lehman-et-all.ppt

Internet2 DC Network summary

• Stay tuned!• Let us know if you have a similar

network to our Dynamic Circuit Network• What control plane are you using for

dynamic circuit setup across your network?

• Contact heather@internet2.edu to discuss more

Ordering and Utilizing Internet2 Transport Services (Static Circuit Services)

• Think of this as if you were going to buy a circuit from a vendor on that vendor’s infrastructure• In this case, that vendor is Internet2• The circuits available are on the fiber

footprint we have available to us

• Endpoints at any of the yellow dots in the following map

Regen DTN Core NodeDrop DTNOther Level 3

Denver1850 Pearl Kansas City

GPN1100 Walnut

Seattle1000 Denny WayLevel 3

Portland1335 NW NorthropLevel 3

Sunnyvale1380 Kifer

Pacific Northwest GP2001 6th AveWestin Bldg

San Diego

PortlandOregon GP

707 SW WashingtonQwest

San Francisco

Sacramento

Los AngelesCENIC

600 W 7thEquinix

Los Angeles818 W 7th

Level 3

Salt LakeInter-Mountain GP

Front Range GP572 S DeLong

AlbuquerqueNew Mexico GP

104 Gold Ave SE

Tulsa18 W Archer

HoustonLEARN1201 N I-45

Dallas

San Antonio Baton RougeLONI9987 Burbank

Chicago600 W Chicago

Detroit

ChicagoCIC/MREN

BOREASInternet2

710 N Lakeshore

Starlight

Atlanta180 Peachtree St NW

Birmingham

CharlotteRaleigh5301 Departure Dr

PittsburghPittsburgh GP143 S 25th

WashingtonMAX1755 Old Meadow

PhiladelphiaMAGPI401 N Broad

New YorkNYSERNET32 A of the A

New York111 8th

Level 3

CambridgeNOX300 Bent St

JacksonvilleFAMU&USF4814 Phillips Hwy

IndianapolisIU1902 S East St

LouisvilleUniv Louisville848 S 8th St

via 1005 N B St

El Paso501 W Overland

NashvilleUniv Memphis2990 Sidco Dr

ClevelandTFN/MERIT4000 Chester

Albany316 N PearlLevel 3

Ogden

Rieth

Boise

Tionesta

Rawlins Omaha

St. Louis

EurekaReno

Valentine

Sanderson

Raton

Phoenix

San Luis Obispo

Mobile

Tallahassee

Syracuse

Tucson

Buffalo

Rancho De La Fe(tentative)

Edison

Atlanta SLR 345 Courtland

Using Internet2 Transport Services

• Example:• Los Angeles to Chicago circuit• 50Mbps to 10G full wave• You arrange for connectivity to both ends

• We can help arrange ‘tail circuits’ in some cases, depending on destination

• You pay according to distance, bandwidth, duration and protection scheme• Contract with Internet2 will be required

• Contact heather@internet2.edu if you are interested

Information specifically for APAN Network and APAN Member Networks

• In Asia:• TransPAC2 is the US NSF-funded project linking US

R&E networks to APAN network and others in Asia• Internet2 is a partner in TransPAC2

• Internet2 IP Network: peer with TransPAC2 in Tokyo at the APAN XP• 10Gbps capacity across Pacific and with Internet2 IP

Network

• Internet2 DC Network: TransPAC2 looking to implement similar capabilities• Connecting to Internet2 DC Network in Los Angeles

(hopefully ultimately via PacificWave)

Info for APAN Networks, cont’d

• In the US:• PacificWave is the US-based international

exchange point on the US west coast• Interconnect here for peering with the Internet2

IP Network• Future: Interconnect here for connectivity with

the Internet2 DC Network– Now: may entertain direct connections on LA Core

Director

We Have Renamed

• NOC URL:http://noc.net.internet2.edu/

• NOC email:noc@net.internet2.edu

• Internet2 Network URLhttp://www.internet2.edu/network/

• Internet2 Network contact address: network@internet2.edu

Questions?

• Heather Boyles

• heather@internet2.edu