SCinet:  The Annual Convergence of

Advanced Networking and High Performance Computing

Steve Corbató, Internet2

MasterWorks track

14 November 2001

SC99 GNAP Demo Network15-18 November, 1999

Portland, Oregon

Outline

• SCinet • Wide area connectivity• Fiber• Wireless• Infrastructure• Operations, Measurement, & Security• Events

– Xnet, Bandwidth Challenge, SC Global

• Trends• Q&A

SCinet is 4 networks

• Production commodity network

• Ubiquitous wireless network

• High-performance/availability exhibit floor network

• Bleeding-edge testbed - Xnet

Scinet is people (and employers)• Basil Decina • Bill Iles • Bill Kramer• Bill Nickless• Bill Wing • Bob Stevens • Brad Pope • Brent Sweeny• Caren Litvanyi• Chris Wright • Chuck Fisher• Dave Koester• Davey Wheeler • David Mitchell • David Richardson • Debbie Mantano • Dennis Duke • Doug Luce • Doug Nordwall • Eli Dart • Erik Plesset • Gayle Allen • Greg Goddard • Hal Edwards • Hoan Mai • James Patton • Janet Hull • Jeff Carrell • Jeff Mauth • Jerry Sobieski • Jim Rogers • John Dysert • John Jamison (JJ• Jon Dugan • Kevin Oberman Walsh • Kim Anderson

• Linda Winkler • Martin Swany • Marvin Drake• Matt Zekauskas • Paola Grosso • Patrick Dorn• Paul Daspit• Paul Love • Paul Reisinger • Rex Duncan • Rick Bagwell• Rick Mauer • Riki Kurihara • Rob Jaeger • Robert Riehl • Roland Gonzalez • Russ Wolf • Seth Viddal • Stanislav Shalunov • Steve Corbato • Steve Kapp • Steve Shultz • Steve Tenbrink • Thomas Hutton • Tim LeMaster • Tim Toole • Tom Kile • Tom Lehman • Tony Rimovsky • Tracey Wilson • Warren Birch • Will Murray • Derek Gassen • Paul Fernes • Steve Pollock        

SC2001 Leadership

• Bill Wing, ORNL – chair• Jim Rogers, CSC – vice chair• Dennis Duke, FSU – incoming

chair

• Chuck Fisher, ORNL – hardware• Jeff Mauth, PNNL – fiber• Martin Swany, UTK –monitoring• Eli Dart, NERSC – security• Bill Nickless, ANL – routing• Tim Toole, SNL – wireless• David Koester, Mitre – Xnet

• Jon Dugan, NCSA – net mgmt• Bill Kramer, NERSC –

Bandwidth Challenge• Greg Goddard, UFl –

monitoring• Kevin Oberman, LBL – Denver

fiber• Steve Corbató, Internet2 –

WAN• Debbie Montano, Qwest –

Denver connectivity• Linda Winkler, ANL – SC

Global

SCinet Committee process

• Conference calls – biweekly weekly• Planning meetings (x3)• Venue recon trips (fiber, wireless)• Staging (~3 weeks before SCxy)• Build (starts Monday before SCxy)• Booth drops (~36 hours before gala reception)• Operate network for ~6 days• Tear down (starts Thursday 4:01p)• Rest & do day job for four months and then start

again…

Staging

Wide area connectivity

• Denver: 15 Gbps– 2xOC-48c: Abilene (Denver) – 2xGigE: STAR LIGHT (Chicago) – 1xOC-48c: Pacific/Northwest Gigapop (Seattle)– 2xOC-48c: ESnet (Sunnyvale & Chicago)

• Level(3) provided wide area connectivity

• Qwest provided local dark fiber

WAN Bandwidth trends

• SC98 (Orlando): 200 Mbps

• SC99 (Portland): 13 Gbps

• SC2000 (Dallas): 10 Gbps

• SC2001 (Denver): 15 Gbps

• SC2002 (Baltimore): Nx10-Gbps ’s??

• Increasing focus on BW utilization

Abilene & SCxy

Escalating bandwidth– SC99 Portland: OC-12c SONET (622 Mbps)– SC2000 Dallas: OC-48c SONET (2.5 Gbps)– SC2001 Denver: 2xOC-48c SONET (5 Gbps)

SCxy transit connectivity offered to domestic & international R&E nets

Backbone MTU raised to 9K bytesTraffic engineering for SC2001End-to-End Performance: GigaTCP testingSC2002 Baltimore: 10-Gbps (planned)

Abilene traffic engineering – SC2002

Fiber (Jeff Mauth)

• 60+ miles of fiber deployed in exhibit hall– 0.3+ FTE-year– ~1.5 fiber-miles/hour

• 120 fiber drops (90% multimode)• Pirelli 24 strand MM fiber used since ’98• Deployment custom engineered to the venue

selected for SCxy• ST fiber connectors standard

– Will review choice for SC2002

Fiber timeline – SC2002

• 5 scouting trips• Tue 11/6 9p – gained access to 2/3 of hall• Thu 11/8 6p – gained access to rest of hall• Fri 11/9 a.m. – fiber done• Sun 11/11 a.m. – equipment patching• Sun 11/12 p.m. – booth drops start

– wireless & HP Jornada

• Mon 11/12 noon – drops complete• Mon 11/12 7p – gala opening (D-DAY)• DANGER: carpet layers (20-30 cuts this year)

Wireless (Tom Hutton)

• Significant 802.11b effort this year• 35 Cisco wireless access points (13 in exhibit hall)

– One on DCC roof pointed at Embassy Suites

• Wireless still requires a lot of wires & work– 5000’ of wiring in exhibition hall– Several site surveys over the year

• Totally flat LAN (3.5 Gbps switched BW)• Wireless really helps show set-up

– Booth drop teams, booth connectivity prior to fiber

• Clients seen: 618 peak, 246 average

Infrastructure (Chuck Fisher)

• SC98– Core Routing provided by traditional Cisco 7500 series

routers

– First "production" use of gigabit Gigabit Ethernet (only 1 customer drop requested)

– Most booth service was 10Base-FL and 100-FX provided via Fore Power Hubs

– Limited use of network monitoring and statistics

An earlier topology…

Infrastructure trends - II

• SC99– Core Routing provided by Cisco GSR series

routers– Concept of a routing core and a layer of L3

distribution switches adopted– Extensive use of DWDM hardware to provide

WAN badwidth– Xnet introduced as a showcase for "bleeding

edge" hardware

Infrastructure trends - III

• SC2000– Core routing provided by Cisco and Juniper– Increased focus on network monitoring and

statistics– First Xnet demonstration of 10 Gigabit Ethernet– Bandwidth Challenge introduced to SC

SCinet 2001 Network Topology

Infrastructure trends -IV• SC2001 Contributing Hardware Vendors

– Cisco– Juniper– Marconi– Nortel– Spirent– Force10– Foundry– ONI– LuxN

• Equivalent to 3-5 bldg advanced campus network on major R&E backbones

Operations

• Servers– DNS, DHCP, NTP, Performance, beacons

• Database

• Network monitoring

• Help desk

• Trouble ticket system

• Routing support (unicast, multicast, v6)

Measurement and Security

Security monitoring

Local Nimbda infections 2

Clear Text Root Logins 68

Clear Text Passwords 1483

Code Red 1 Infections 1

External Nimbda sources 857

Passwordless accounts 59

Scans 137

Xnet

TeraGrid Distributed Backplane -

NCSA, ANL, SDSC, Caltech

NCSA/UIUC

ANL

UICMultiple Carrier Hubs

Starlight / NW Univ

Ill Inst of Tech

Univ of Chicago Indianapolis (Abilene NOC)

I-WIRE

StarLightInternational Optical Peering Point

(see www.startap.net)

Los Angeles

San Diego

DTF Backplane (4x: 40 Gbps)

Abilene

Chicago

IndianapolisUrbana

OC-48 (2.5 Gb/s, Abilene)

Multiple 10 GbE (Qwest)

Multiple 10 GbE (I-WIRE Dark Fiber)

• Solid lines in place and/or available by October 2001• Dashed I-WIRE lines planned for summer 2002

Source: Charlie Catlett, Argonne

Xnet

Trends

… or what we might see in Baltimore?

Optical networking •Dense Wave Division Multiplexing (DWDM)

–Current systems can support >160 10-Gbps ’s (1.6 Tbps!)

–Optical growth can overwhelm Moore’s Law (routers)

•Costs scale dramatically with distance•Three possible scenarios for the future

–Enhanced IP transport (higher BW and circuit multiplicity)–Fine-grained traffic engineering

• p2p links between campuses, HPC centers, & Gigapops

–Physical e2e switched circuits (a la ATM SVCs)

•Evolution of optical switching will be critical–Don’t write off OEO

Future of Abilene

• Extension of Qwest’s original commitment to Abilene for another 5 years – 10/01/2006

–Originally expired March, 2003

• Upgrade of Abilene backbone to optical transport capability - ’s

–x4 increase in the core backbone bandwidth• OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM

–Capability for flexible provisioning of 10-Gbps ’s to support future point-to-point experimentation & other projects

• Emphasis on v6, network measurement, & measurement capabilities

SC2002/Baltimore crystal ball

• Strong local networking community– MAX Gigapop (University of Maryland) – DARPA Supernet (ISI-East, NRL)

• Dark fiber & network presences in region• Abilene is aiming for 10-Gbps connectivity• Increased focus on e2e performance & multicast

reliability• More wireless (add 802.11a); less ATM?• 10 Gigabit Ethernet should be standardized• Optical switch in Xnet?

Conclusion

• Scinet is…… a diverse group of very committed and

talented people and companies working very hard under extreme time constraints and trying conditions to make both the expected and the new and impossible in SCxy networking happen for one week in November and then return to do it again the next year