1

ESnet Update Winter 2008

Joint Techs Workshop

Joe Burrescia ESnet General Manager

January 21, 2008

Energy Sciences Network Lawrence Berkeley National Laboratory

Networking for the Future of Science

2

TWC

SNLL

YUCCA MT

PNNL LIGO

INEEL

LANL

SNLA Allied Signal

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPL

Lab DC Offices

MIT

ANL

BNL

FNAL AMES

NR

EL

LLNL

GA

DOE-ALB

OSC GTN NNSA

International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (≥ 10 G/s) Lab supplied links OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less

NNSA Sponsored (12) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6)

42 end user sites

SINet (Japan) Russia (BINP) CA*net4

France GLORIAD (Russia, China) Korea (Kreonet2

Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren

ESnet IP core: Packet over SONET

Optical Ring and Hubs

ELP

commercial peering points

MAE-E

PAIX-PA Equinix, etc.

PNW

GPo

P/

PAci

ficW

ave

ESnet 3 with Sites and Peers (Early 2007)

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

CERN (USLHCnet

DOE+CERN funded)

GÉANT - France, Germany, Italy, UK, etc

SNV

JGI LBNL

SLAC NERSC

SNV SDN

SDSC

Equinix

SNV

ALB

ORNL

CHI

MREN Netherlands StarTap Taiwan (TANet2, ASCC)

NA

SA A

mes

AU

AU

SEA

CH

I-SL MA

N L

AN

A

bile

ne

Specific R&E network peers Other R&E peering points

AMPATH (S. America) AMPATH

(S. America)

ESne

t Sc

ienc

e Dat

a Net

work

(SD

N) co

re

R&E networks

Office Of Science Sponsored (22)

ATL

NSF/IRNC funded

Equinix

3 ESnet 3 Backbone as of January 1, 2007

Sunnyvale

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) MAN rings (≥ 10 G/s) Lab supplied links

Seattle

San Diego Albuquerque

El Paso

Chicago New York

City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

4 ESnet 4 Backbone as of April 15, 2007

Clev.

Boston

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

5 ESnet 4 Backbone as of May 15, 2007

SNV

Clev.

Boston

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Future ESnet Hub

ESnet Hub

6 ESnet 4 Backbone as of June 20, 2007

Clev.

Boston

Houston

Kansas City

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

7 ESnet 4 Backbone August 1, 2007 (Last JT meeting at FNAL)

Clev.

Boston

Houston

Los Angeles

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

8 ESnet 4 Backbone September 30, 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Future ESnet Hub

ESnet Hub

9 ESnet 4 Backbone December 2007

Clev.

Boston

Houston

Boise

Los Angeles

10 Gb/s SDN core (NLR) 2.5 Gb/s IP Tail (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

New York City

Washington DC

Atlanta

Denver

Nashville

Future ESnet Hub

ESnet Hub

Chicago

10

DC

ESnet 4 Backbone December, 2008

Houston 10 Gb/s SDN core (NLR) 10/2.5 Gb/s IP core (QWEST) 10 Gb/s IP core (Level3) 10 Gb/s SDN core (Level3) MAN rings (≥ 10 G/s) Lab supplied links

Clev.

Houston

Kansas City

Boston

Sunnyvale

Seattle

San Diego Albuquerque

El Paso

Chicago

New York City

Washington DC

Atlanta

Denver

Los Angeles Nashville

Future ESnet Hub

ESnet Hub

X2

X2

X2

X2

X2

X2

X2

YUCCA MT

PNNL LIGO

LANL

SNLA Allied Signal

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPL

Lab DC Offices

MIT/ PSFC

BNL

GA

DOE GTN NNSA

NNSA Sponsored (13+) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6)

~45 end user sites

SINet (Japan) Russia (BINP) CA*net4

France GLORIAD (Russia, China) Korea (Kreonet2

Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren

ELPA

commercial peering points

PAIX-PA Equinix, etc.

ESnet Provides Global High-Speed Internet Connectivity for DOE Facilities and Collaborators (12/2007)

ESnet core hubs

CERN (USLHCnet:

DOE+CERN funded)

GÉANT - France, Germany, Italy, UK, etc

SUNN

Abilene

JGI

LBNL

SLAC NERSC

SNV1

SDSC

Equinix

ALBU

MREN StarTap Taiwan (TANet2, ASCC)

NA

SA A

mes

AU

AU

SEA

CH

I-SL

Specific R&E network peers

AMPATH (S. America)

AMPATH (S. America)

R&E networks

Office Of Science Sponsored (22)

ATLA

NSF/IRNC funded

PacW

ave

KAREN/REANNZ ODN Japan Telecom America NLR-Packetnet Abilene/I2

KAREN / REANNZ Internet2 SINGAREN ODN Japan Telecom America

NETL

ANL

International (1-10 Gb/s) 10 Gb/s SDN core (I2, NLR) 10Gb/s IP core MAN rings (≥ 10 Gb/s) Lab supplied links OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less

Salt Lake

PacWave

Equinix

DOE

SUNN Geography is

only representational Other R&E peering points

USH

LCN

et

to G

ÉAN

T

12 ESnet4 Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),

500-600 Gbps by 2011-2012 (100 Gb/s circuits)

Cle

vela

nd

Europe (GEANT)

Asia-Pacific

New York

Washington DC

CERN (30+ Gbps)

Albuquerque

Aus

tral

ia

San Diego

LA

Denver

South America (AMPATH)

South America (AMPATH)

Canada (CANARIE)

CERN (30+ Gbps) Canada (CANARIE) Asia Pacific

GLORIAD (Russia and

China)

Boise

Jacksonville

Tulsa

Boston

Science Data Network Core

IP Core

Aus

tral

ia

Core network fiber path is ~ 14,000 miles / 24,000 km

1625

mile

s / 2

545

km

2700 miles / 4300 km

Production IP core (10Gbps) SDN core (20-30-40-50 Gbps) MANs (20-60 Gbps) or backbone loops for site access International connections

IP core hubs

Primary DOE Labs SDN hubs

High speed cross-connects with Ineternet2/Abilene Possible hubs

13 A Tail of Two ESnet4 Hubs

MX960 Switch

T320 Router

6509 Switch

T320 Routers

Sunnyvale Ca Hub Chicago Hub ESnet’s SDN backbone is implemented with Layer2 switches; Cisco 6509s and Juniper MX960s each present their own unique challenges.

14 ESnet 4 Factoids as of January 21, 2008

•  ESnet4 installation to date: o  32 new 10Gb/s backbone circuits

-  Over 3 times the number from last JT meeting

o  20,284 10Gb/s backbone Route Miles -  More than doubled from last JT meeting

o  10 new hubs -  Since last meeting

–  Seattle –  Sunnyvale –  Nashville

o  7 new routers 4 new switches o  Chicago MAN now connected to Level3 POP

-  2 x 10GE to ANL -  2 x 10GE to FNAL -  3 x 10GE to Starlight

15 ESnet Traffic Continues to Exceed 2 PetaBytes/Month

2.7 PBytes in July 2007

1 PBytes in April 2006

ESnet traffic historically has increased 10x every 47 months

Overall traffic tracks the very large science

use of the network

When A Few Large Data Sources/Sinks Dominate Traffic it is Not Surprising that Overall Network Usage Follows the

Patterns of the Very Large Users - This Trend Will Reverse in the Next Few Weeks as the Next Round of LHC Data Challenges Kicks Off

17 ESnet Continues to be Highly Reliable; Even During the Transition

“5 nines” (>99.995%) “3 nines” (>99.5%) “4 nines” (>99.95%)

Dually connected sites Note: These availability measures are only for ESnet infrastructure, they do not include site-related problems. Some sites, e.g. PNNL and LANL, provide circuits from the site to an ESnet hub, and therefore the ESnet-site demarc is at the ESnet hub (there is no ESnet equipment at the site. In this case, circuit outages between the ESnet equipment and the site are considered site issues and are not included in the ESnet availability metric.

18 OSCARS Overview

Path Computation •  Topology

•  Reachability •  Contraints

Scheduling •  AAA

•  Availability

Provisioning •  Signalling •  Security

•  Resiliency/Redundancy

OSCARS Guaranteed Bandwidth

Virtual Circuit Services

On-demand Secure Circuits and Advance Reservation System

19 OSCARS Status Update •  ESnet Centric Deployment

o  Prototype layer 3 (IP) guaranteed bandwidth virtual circuit service deployed in ESnet (1Q05)

o  Prototype layer 2 (Ethernet VLAN) virtual circuit service deployed in ESnet (3Q07)

•  Inter-Domain Collaborative Efforts o  Terapaths (BNL)

-  Inter-domain interoperability for layer 3 virtual circuits demonstrated (3Q06) -  Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)

o  LambdaStation (FNAL) -  Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)

o  HOPI/DRAGON -  Inter-domain exchange of control messages demonstrated (1Q07) -  Integration of OSCARS and DRAGON has been successful (1Q07)

o  DICE -  First draft of topology exchange schema has been formalized (in collaboration with NMWG)

(2Q07), interoperability test demonstrated 3Q07 -  Initial implementation of reservation and signaling messages demonstrated at SC07 (4Q07)

o  UVA -  Integration of Token based authorization in OSCARS under testing

o  Nortel -  Topology exchange demonstrated successfully 3Q07 -  Inter-domain interoperability for layer 2 virtual circuits demonstrated at SC07 (4Q07)

20

•  ESnet o  About 1/3 of the 10GE bandwidth test platforms & 1/2 of the

latency test platforms for ESnet 4 have been deployed. -  10GE test systems are being used extensively for acceptance

testing and debugging -  Structured & ad-hoc external testing capabilities have not been

enabled yet. -  Clocking issues at a couple POPS are not resolved.

o  Work is progressing on revamping the ESnet statistics collection, management & publication systems

-  ESxSNMP & TSDB & PerfSONAR Measurement Archive (MA) -  PerfSONAR TS & OSCARS Topology DB -  NetInfo being restructured to be PerfSONAR based

•  LHC and PerfSONAR o  PerfSONAR based network measurement solutions for the

Tier 1/Tier 2 community are nearing completion. o  A proposal from DANTE to deploy a perfSONAR based

network measurement service across the LHCOPN at all Tier1 sites is being evaluated by the Tier 1 centers

Network Measurement Update

21

End