1

ESnet Update

ESnet/Internet2 Joint Techs Madison, Wisconsin

July 17, 2007

Joe Burrescia

ESnet General ManagerLawrence Berkeley National Laboratory

2

OutlineESnet’s Role in DOE’s Office of Science

ESnet’s Continuing Evolutionary DimensionsCapacityReach ReliabilityGuaranteed Services

3

DOE Office of Science and ESnet

• “The Office of Science is the single largest supporter of basic research in the physical sciences in the United States, … providing more than 40 percent of total funding … for the Nation’s research programs in high-energy physics, nuclear physics, and fusion energy sciences.” (http://www.science.doe.gov)

• The large-scale science that is the mission of the Office of Science is dependent on networks for

o Sharing of massive amounts of datao Supporting thousands of collaborators world-wideo Distributed data processingo Distributed simulation, visualization, and computational steeringo Distributed data management

• ESnet’s mission is to enable those aspects of science that depend on networking and on certain types of large-scale collaboration

4

The Office of Science U.S. Community

Institutions supported by SC Major User Facilities

DOE Multiprogram LaboratoriesDOE Program-Dedicated LaboratoriesDOE Specific-Mission Laboratories

Pacific NorthwestPacific NorthwestNational LaboratoryNational Laboratory Ames LaboratoryAmes Laboratory

Argonne National Argonne National LaboratoryLaboratory

BrookhavenBrookhavenNationalNational

LaboratoryLaboratory

Oak RidgeOak RidgeNational National

LaboratoryLaboratoryLos AlamosLos Alamos

National National LaboratoryLaboratory

Lawrence Lawrence LivermoreLivermore

&&SandiaSandia

National National LaboratoriesLaboratories

LawrenceLawrenceBerkeley Berkeley NationalNational

LaboratoryLaboratory

FermiFermiNationalNational

Accelerator Accelerator LaboratoryLaboratory

PrincetonPrincetonPlasmaPlasmaPhysicsPhysics

LaboratoryLaboratory

Thomas Jefferson Thomas Jefferson National National

Accelerator FacilityAccelerator Facility

NationalNationalRenewable Energy Renewable Energy

LaboratoryLaboratory

StanfordStanfordLinearLinear

Accelerator Accelerator CenterCenter

Idaho National Idaho National LaboratoryLaboratory

SC Program sites

General General AtomicsAtomics

SandiaSandiaNational National

LaboratoryLaboratory

5

Footprint of SC Collaborators - Top 100 Traffic Generators

Universities and research institutes that are the top 100 ESnet users• The top 100 data flows generate 30% of all ESnet traffic (ESnet handles about 3x109 flows/mo.)• 91 of the top 100 flows are from the Labs to other institutions (shown) (CY2005 data)

6

Changing Science Environment New Demands on Network

• Increased capacityo Needed to accommodate a large and steadily

increasing amount of data that must traverse the network

• High-speed, highly reliable connectivity between Labs and US and international R&E institutionso To support the inherently collaborative, global nature

of large-scale science• High network reliability

o For interconnecting components of distributed large-scale science

• New network services to provide bandwidth guaranteeso For data transfer deadlines for remote data analysis,

real-time interaction with instruments, coupled computational simulations, etc.

7

0

200

400

600

800

1000

1200

Feb, 90Jul, 90

Dec, 90May,91Oct, 91Mar, 92Aug, 92Jan, 93Jun, 93Nov, 93Apr, 94Sept,

Feb, 95Jul, 95

Dec, 95May,

Oct, 96Mar, 97Aug, 97Jan, 98Jun, 98Nov, 98Apr, 99Sep, 99Feb, 00Jul, 00

Dec, 00May,

Oct, 01Mar, 02Aug, 02Jan, 03Jun, 03Nov, 03Apr, 04Sep, 04Feb, 05Jul, 05

Dec, 05May,

Network Utilization

ESnet Accepted Traffic (Bytes)Jan 1990 to Jun 2006

ESnet is Currently Transporting over 1.2 Petabytes/monthand this volume is increasing exponentially

TByt

es/M

onth

1.04 Petabyte/mo April 2006

1.20 Petabyte/mo June 2006

R 2 = 0.9898

0.0

0.1

1.0

10.0

100.0

1000.0

10000.0

Jan, 90 Jan, 91 Jan, 92 Jan, 93 Jan, 94 Jan, 95 Jan, 96 Jan, 97 Jan, 98 Jan, 99 Jan, 00 Jan, 01 Jan, 02 Jan, 03 Jan, 04 Jan, 05 Jan, 06

ESnet traffic has increased by10X every 47 months, on average, since 1990

Tera

byte

s / m

onth

Log Plot of ESnet Monthly Accepted Traffic, January, 1990 – June, 2006

Oct., 19931 TBy/mo.

Aug., 1990100 MBy/mo.

Jul., 199810 TBy/mo.

38 months

57 months

40 months

Nov., 2001100 TBy/mo.

Apr., 20061 PBy/mo.

53 months

9

High Volume Science Traffic Continues to Grow

• Top 100 flows are increasing as a percentage of total traffic volume

• 99% to 100% of top 100 flows are science data (100% starting mid-2005)

• A small number of large-scale science users account for a significant and growing fraction of total traffic volume

2 TB/month

2 TB/month

2 TB/month

1/05

6/05

1/06

2 TB/month

7/06

10

Traffic coming into ESnet = GreenTraffic leaving ESnet = BlueTraffic between ESnet sites% = of total ingress or egress traffic

Traffic notes• more than 90% of all traffic is Office of Science• less that 10% is inter-Lab

Who Generates ESnet Traffic?ESnet Inter-Sector Traffic Summary for June 2006

Peering Points

Commercial

R&E (mostlyuniversities)

7%

30%

31%

5%

5%

14%

ESnet

~7%

DOE collaborator traffic, inc. data

76%

31%

DOE is a net supplier of data because DOE facilities are used by universities and commercial entities, as well as by DOE researchers

DOE sites

International(almost entirelyR&E sites)

11

CA*net4 FranceGLORIAD (Russia, China)Korea (Kreonet2)

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ATL

DC

MAE-EPAIX-PAEquinixMAE-West

ESnet’s Domestic, Commercial, and International Connectivity

Abilene

Abilene

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

Starlight CHI-SL

Abilene

SNV

SDSC

MAXGPoP

SoXGPoP

High Speed International ConnectionCommercial and R&E peering points

Abilene High-speed peering points with Abilene

ESnet core hubs IP SDN

CHI

MRENNetherlandsStarTapTaiwan (TANet2)UltraLight

NGIX-W

Australia

SEA

SINet (Japan)Russia (BINP)

AMPATH(S. America)

AMPATHS. America

MA

N LA

NAb

ilene

ESnet provides:• High-speed peerings with Abilene, CERN,

and the international R&E networks• Management of the full complement of global

Internet routes (about 180,000 unique IPv4 routes) in order to provide DOE scientists rich connectivity to all Internet sites

Australia

Equinix

Equinix

PNWGPoP/PacificWave

NGIX-E

PacificWave

UNM ALB

SNV

USN

USN

12

NLR

Sup

plie

d Ci

rcui

ts

LVK

SNLL

YUCCA MT

BECHTEL-NV

PNNLLIGO

INEEL

SNLAPANTEX

ARM

KCP

NOAA

OSTI ORAU

SRS

JLAB

PPPL

MIT

ANL

BNLFNAL

AMESLLNL

GA

OSC GTNNNSA

International (high speed)Lab Supplied10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

Office Of Science Sponsored (22)NNSA Sponsored (12)Joint Sponsored (4)Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)

42 end user sites

ESnet IP core

CA*net4 FranceGLORIAD (Russia, China)Korea (Kreonet2

Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren

ESnet IP core: Packet over SONET Optical Ring and

Hubs

ELP

ATL

DC

commercial and R&E peering points

MAE-E

PAIX-PAEquinix, etc.

PNW

GPo

P/PA

cific

Wav

e

ESnet’s Physical Connectivity (Summer 2006)

ESnet core hubs IP

Abilene high-speed peering points with Internet2/Abilene

Abilene

CERN(USLHCnet

CERN+DOE funded)

GÉANT - France, Germany, Italy, UK, etc

NYC

StarlightChi NAP

CHI-SL

SNV

Abilene

SNV SDN

JGILBNL

SLACNERSC

SDSC

Equinix

MAXGPoP

SoXGPoP

SNV

ALB

ORNL

CHI

MRENNetherlandsStarTapTaiwan (TANet2)UltraLight

AU

AU

SEA

SINet (Japan)Russia (BINP)

LANL

NREL

AMPATH(S. America)

AMPATH

MA

N LA

NAb

ilene

MAN rings

Abile

ne

USN

USN

IARC

NA

SAA

mes

KCP-ALB UNM

LLNL/LANLDC Offices

LBNL DC

ORAU DC

DOE-ALB

13

ESnet LIMAN with SDN Connections

14

LIMAN and BNL• ATLAS (A Toroidal LHC ApparatuS), is one of four

detectors located at the Large Hadron Collider (LHC) located at CERN

• BNL is the largest of the ATLAS Tier 1 centers and the only one in the U.S, and so is responsible for archiving and processing approximately 20 percent of the ATLAS raw data

• During a recent multi-week exercise, BNL was able to sustain an average transfer rate from CERN to their disk arrays of 191 MB/s (~1.5 Gb/s) compared to a target rate of 200 MB/so This was in addition to “normal” BNL site traffic

15

Chicago Area MAN with SDN Connections

16

CHIMAN: FNAL and ANL

• Fermi National Laboratory is the only US Tier1 center for the Compact Muon Solenoid (CMS) experiment at LHC

• Argonne National Laboratory will house a 5-teraflop IBM BlueGene computer part of the National Leadership Computing Facility

• Together with ESnet, FNAL and ANL will build the Chicago MAN (CHIMAN) to accommodate the vast amounts of data these facilities will generate and receiveo Five 10GE circuits will go into FNALo Three 10GE circuits will go into ANLo Ring connectivity to StarLight and to the Chicago ESnet

POP

Jefferson Laboratory Connectivity

Eastern LITE

(E-LITE)

Old Dominion University

W&MJLAB

JTASC

VMASC

Bute St CO

MATPNYC

Atlanta

ODU

NASA

Lovitt

MATP Virginia Tech

MAX GIGAPOP

ESnet Router

10GEOC192OC48

ES

net c

ore

JLAB Site Switch

DC - MAX Giga-POP

T320T320

18

10-50 Gb/s circuitsProduction IP coreScience Data Network coreMetropolitan Area NetworksInternational connections

MetropolitanArea Rings

Primary DOE Labs

IP core hubs

possible hubs

SDN hubs

Europe(GEANT)

Asia-Pacific

New York

Chicag

o

Washington, DC

Atla

nta

CERN

Seattle

AlbuquerqueAus

.A

ustr

alia

San Diego

LA

Sunn

yval

e

Denver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

CERN

Loop off Backbone

Canada(CANARIE)

Europe(GEANT)

SDN Core

IP Core

ESnet Target Architecture: IP Core+Science Data Network Core+Metro Area Rings

19

Reliability

“5 nines” “3 nines”“4 nines”

Dually connected sites

20

Guaranteed Services Using Virtual Circuits• Traffic isolation and traffic engineering

– Provides for high-performance, non-standard transport mechanisms that cannot co-exist with commodity TCP-based transport

– Enables the engineering of explicit paths to meet specific requirements• e.g. bypass congested links, using lower bandwidth, lower latency paths

• Guaranteed bandwidth [Quality of Service (QoS)]– Addresses deadline scheduling

• Where fixed amounts of data have to reach sites on a fixed schedule, so that the processing does not fall far enough behind that it could never catch up – very important for experiment data analysis

• Reduces cost of handling high bandwidth data flows– Highly capable routers are not necessary when every packet goes to the

same place– Use lower cost (factor of 5x) switches to relatively route the packets

• End-to-end connections are required between Labs and collaborator institutions

21

OSCARS: Guaranteed Bandwidth VC Service For SC Science

• ESnet On-demand Secured Circuits and Advanced Reservation System (OSCARS)

• To ensure compatibility, the design and implementation is done in collaboration with the other major science R&E networks and end sites

o Internet2: Bandwidth Reservation for User Work (BRUW) - Development of common code base

o GEANT: Bandwidth on Demand (GN2-JRA3), Performance and Allocated Capacity for End-users (SA3-PACE) and Advance Multi-domain Provisioning System (AMPS)

- Extends to NRENso BNL: TeraPaths - A QoS Enabled Collaborative Data Sharing Infrastructure for Peta-

scale Computing Research o GA: Network Quality of Service for Magnetic Fusion Research o SLAC: Internet End-to-end Performance Monitoring (IEPM) o USN: Experimental Ultra-Scale Network Testbed for Large-Scale Science

• In its current phase this effort is being funded as a research project by the Office of Science, Mathematical, Information, and Computational Sciences (MICS) Network R&D Program

• A prototype service has been deployed as a proof of concepto To date more then 20 accounts have been created for beta users, collaborators, and

developerso More then 100 reservation requests have been processed

22

OSCARS - BRUW Interdomain Interoperability Demonstration

BRUW OSCARS

1

23 3

Source

LSP

Sink

LSPIndianapolis

INChicago

ILChicago

ILSunnyvale

CA

4

• The first interdomain, automatically configured, virtual circuit between ESnet and Abilene was created on April 6, 2005

23

A Few URLs• ESnet Home Page

o http://www.es.net

• National Labs and User Facilitieso http://www.sc.doe.gov/sub/organization/organization.htm

• ESnet Availability Reportso http://calendar.es.net/

• OSCARS Documentationo http://www.es.net/oscars/index.html