Scalable Bandwidth on Scalable Bandwidth on Demand: A New Model Demand: A New Model

for the Era of for the Era of Entitlement?Entitlement?

Educause Mid-Atlantic Regional ConferenceEducause Mid-Atlantic Regional ConferenceJanuary 17, 2003January 17, 2003

Gregory D. PalmerGregory D. PalmerJennifer MacDougallJennifer MacDougall

Copyright Gregory D. Palmer/Jennifer MacDougall Copyright Gregory D. Palmer/Jennifer MacDougall 2003. This work is the intellectual property of the 2003. This work is the intellectual property of the authors. Permission is granted for this material to be authors. Permission is granted for this material to be shared for non-commercial, educational purposes, shared for non-commercial, educational purposes, provided that this copyright statement appears on provided that this copyright statement appears on the reproduced materials and notice is given that the the reproduced materials and notice is given that the copying is by permission of the authors. To copying is by permission of the authors. To disseminate otherwise or to republish requires disseminate otherwise or to republish requires written permission from the authors.written permission from the authors.

Topics For TodayTopics For Today

1.1. An Overview of the MAGPI GigaPoPAn Overview of the MAGPI GigaPoP

2.2. Problem: Matching the transport to Problem: Matching the transport to the Payloadthe Payload

3.3. University of Pennsylvania Beta SiteUniversity of Pennsylvania Beta Site

4.4. Importance to other institutionsImportance to other institutions

5.5. Future implicationsFuture implications

6.6. Q & AQ & A

MAGPI Overview - MAGPI Overview - VisionVision Striving to serve the needs of the Striving to serve the needs of the

research and education research and education communitiescommunities– Enabling ApplicationsEnabling Applications– Responsive to the Community’s NeedsResponsive to the Community’s Needs– Proactively Introducing TechnologyProactively Introducing Technology– Creatively Deploying SolutionsCreatively Deploying Solutions– Innovative Financial ModelsInnovative Financial Models– Self Sustaining Over the Long TermSelf Sustaining Over the Long Term

MAGPI Overview - MAGPI Overview - StaffingStaffing Two employees dedicated to MAGPITwo employees dedicated to MAGPI

– AdministrativeAdministrative– ApplicationsApplications

Penn ISC Networking & Penn ISC Networking & TelecommunicationsTelecommunications– EngineeringEngineering– Network Operations CenterNetwork Operations Center– Budget CoordinationBudget Coordination

MAGPI Overview - MAGPI Overview - TopologyTopology

How to meet the How to meet the growing bandwidth growing bandwidth requirements of the requirements of the research community research community efficiently and cost efficiently and cost effectively.effectively.

Problem Statement:Problem Statement:

Have you ever heard…Have you ever heard…

“I need ……. TOMORROW!”

“My grant requires………….”

“But my NSF demo is in two weeks!”

“There is no line item for telecom in my budget”

“But the University should pay for this!”

Offsetting Market Offsetting Market TrendsTrends

IT BudgetsIT Budgets InfrastructureInfrastructure

New TechnologyNew Technology Bandwidth Bandwidth CostsCosts

$

$

$

$

$$$

0

100

200

300

400

500

600

700

Bandw

idth

in M

bps

1 365

Bandwidth Events in Days

Typical Day

ResearchDemo Near FutureNeed

Efficiency… (or lack Efficiency… (or lack of )of )

Challenge: Most Challenge: Most Efficient, Cost-Effective Efficient, Cost-Effective PathPath

Researcher A

Researcher B

?- - - - - - - - - - - - - - - - - - -

Challenge: The WAN Challenge: The WAN Connectivity ExpenseConnectivity Expense

Researcher A

- - - - - - - - - - - - - - - - - - - - - - - - - - - - Researcher B

Campus Network

Campus Network

Cross-Campus

Regional

National

International

$$$ WANs $$$

Problem SummaryProblem Summary

Research and education will require Research and education will require increasing amounts of bandwidth in increasing amounts of bandwidth in an un-sustained manneran un-sustained manner

It is neither efficient nor cost-It is neither efficient nor cost-effective to create independent effective to create independent WAN connectionsWAN connections

A general lack of flexibility from the A general lack of flexibility from the vendor/providersvendor/providers

Match the Transport to the Match the Transport to the PayloadPayload

A Solution in Beta A Solution in Beta TestingTesting

NTII – National TeleImmersion InitiativeNTII – National TeleImmersion Initiative– 3D Videoconferencing between UNC Chapel 3D Videoconferencing between UNC Chapel

Hill and the Univ of PennHill and the Univ of Penn– Have accomplished data streams of 700 Have accomplished data streams of 700

MbpsMbps– Pittsburgh SuperComputing Center used for Pittsburgh SuperComputing Center used for

interim processinginterim processing– Experiments in progress 30 days per yearExperiments in progress 30 days per year– Normal traffic is < 3MbpsNormal traffic is < 3Mbps

3-D Videoconferencing3-D Videoconferencing

TeleImmersionTeleImmersion Penn->PSC->UNCPenn->PSC->UNC 300 to 700 Mbps300 to 700 Mbps

Network InfrastructureNetwork Infrastructure

High density Ethernet switch with High density Ethernet switch with 100 Mbps ports and Gig-E uplink 100 Mbps ports and Gig-E uplink interfaceinterface

Single mode fiber to the campus Single mode fiber to the campus router that also has the Internet2 linkrouter that also has the Internet2 link

Yipes! provided scalable Gig-E WAN Yipes! provided scalable Gig-E WAN link to MAGPI routing corelink to MAGPI routing core

Yipes! provided web interface to Yipes! provided web interface to control rate limits on a Gig-E circuitcontrol rate limits on a Gig-E circuit

Connection DiagramConnection Diagram

Extreme Summit 48i

Workstation

TM

LT

M10

Jun iperNETW ORKS

AUX/MODEM

CONSOLE

MGMT

PIC 0/3

PIC 1/3

PIC 0/2

PIC 1/2

PIC 0/1

PIC 1/1

PIC 0/0

PIC 1/0

PI nternetrocessor

R

GBIC-LX

LGBC Single ModeFiber

Juniper LX-Gig PIC

Juniper M-10Campus Router

VagelosLaboratories

Univ. of Penn.

TM

LT

M10

Jun iperNETW ORKS

AUX/MODEM

CONSOLE

MGMT

PIC 0/3

PIC 1/3

PIC 0/2

PIC 1/2

PIC 0/1

PIC 1/1

PIC 0/0

PIC 1/0

PI nternetrocessor

R

ETHERNET 1000 BASE-LX

LINE

RX

AC

TI V

ITY

RX

TX

ST

AT

US

ETHERNET 1000 BASE-LX

LINE

RX

AC

TI V

ITY

RX

TX

ST

AT

US

ETHERNET 1000 BASE-LX

LINE

RX

AC

TI V

ITY

RX

TX

ST

AT

US

ETHERNET 1000 BASE-LX

LINE

RX

AC

TI V

ITY

RX

TX

ST

AT

US

Juniper M-10MAGPI Router

Internet2Project Abilene

Carrier Hotel401 N. Broad

Street

UNCChapel Hill

PittsburghSupercomputing

Center

Video

YIPES!Scalable 200 Mbps - 1 Gbps

Video

MAGPI/AbileneConnection

OC12

GRASPLaboratories3401- C Wing

The ProcessThe Process

Researcher sends an email request Researcher sends an email request to the MAGPI NOC for the amount to the MAGPI NOC for the amount of additional bandwidth and the of additional bandwidth and the required daterequired date

A confirmation is returned by a A confirmation is returned by a technician and the event is technician and the event is scheduled through a standard work scheduled through a standard work orderorder

Through the Yipes! Web interface, Through the Yipes! Web interface, bandwidth is increased the evening bandwidth is increased the evening before the eventbefore the event

The ProcessThe Process

24 hours later, the NOC 24 hours later, the NOC technician returns the bandwidth technician returns the bandwidth to normalto normal

A copy of the work order is sent A copy of the work order is sent to MAGPI with the date, amount to MAGPI with the date, amount of bandwidth, and duration of the of bandwidth, and duration of the eventevent

Researcher is invoicedResearcher is invoiced

Financial AspectsFinancial Aspects

Building to Building infrastructure Building to Building infrastructure upgrades are a University functionupgrades are a University function

Equipment at the laboratory end Equipment at the laboratory end are the responsibility of the are the responsibility of the researcherresearcher

Risers, fiber, wallplates are the Risers, fiber, wallplates are the school, researcher’s responsibilityschool, researcher’s responsibility

Router, interface cards are a joint Router, interface cards are a joint responsibilityresponsibility

Financial AspectsFinancial Aspects

Cost ComponentsCost Components– NOC techniciansNOC technicians

40 minutes per event40 minutes per event

– YIPES! Scalable WAN serviceYIPES! Scalable WAN service $1.00/Mb/day$1.00/Mb/day

– GigaPoP Bandwidth availabilityGigaPoP Bandwidth availability $ 1.65/Mb/day (minimum)$ 1.65/Mb/day (minimum)

– Grant Telecom CostsGrant Telecom Costs $ 2.65/Mb/day$ 2.65/Mb/day

Comparative Costs – 300 Comparative Costs – 300 MbpsMbps

OC12 DirectOC12 Direct– $ 270,000 – Abilene$ 270,000 – Abilene– $ 24,000 – Qwest$ 24,000 – Qwest– $ 60,000 – loop $ 60,000 – loop

chgchg

____________________________________– $ 354,000 – Basic $ 354,000 – Basic

TotTot– Not IncludedNot Included

EquipmentEquipment StaffStaff

OC12 Shared (30 OC12 Shared (30 events)events)– $ 23,850 – GigaPoP$ 23,850 – GigaPoP– $ 9,000 – YIPES!$ 9,000 – YIPES!

__________________________________________– $ 32,850 – Operations$ 32,850 – Operations– Not includedNot included

Infrastructure BuildsInfrastructure Builds EquipmentEquipment

0100200300400500600700800

136

5

Bandwidth Events in Days

Bandw

idth

in M

bps

ResearchDemo

TypicalDay

NearFutureNeed

Efficiency - RevisitedEfficiency - Revisited

OutcomesOutcomes

High end research is appropriately High end research is appropriately supportedsupported

Campus networking costs maintainedCampus networking costs maintained Cost-recovery budget models reinforcedCost-recovery budget models reinforced Needs met on a usage basis–not paying Needs met on a usage basis–not paying

for idle, unused bandwidthfor idle, unused bandwidth Baseline bandwidth expectation is set Baseline bandwidth expectation is set

(10 Mbps)(10 Mbps)

Importance to OthersImportance to Others

Within the regionWithin the region– GigaPoP efficiency in bandwidth GigaPoP efficiency in bandwidth

allocationallocation To the UniversityTo the University

– Research enablingResearch enabling– Cost sharingCost sharing

To the research communityTo the research community– Bandwidth as neededBandwidth as needed– AffordableAffordable

Subjective BenefitsSubjective Benefits

Builds trusted relationshipsBuilds trusted relationships– Research and central ITResearch and central IT– University IT and the GigaPoPUniversity IT and the GigaPoP– IT, GigaPoP, and the vendor/providerIT, GigaPoP, and the vendor/provider

Sets appropriate expectation Sets appropriate expectation levelslevels– Between research, central IT, and Between research, central IT, and

the Universitythe University

Future GigaPoP Future GigaPoP ServicesServices Vendor negotiated scalable Vendor negotiated scalable

uplinks for Institutions needing uplinks for Institutions needing “on demand” bandwidth“on demand” bandwidth

Scheduling services to prevent Scheduling services to prevent conflicts between applicationsconflicts between applications

Internal research to determine Internal research to determine applicability of DWDM applicability of DWDM implementationimplementation

Thank You – Q&AThank You – Q&A

Greg PalmerGreg Palmer– gpalmer@magpi.netgpalmer@magpi.net

Jennifer MacDougall Jennifer MacDougall – jmacdoug@magpi.njmacdoug@magpi.n

etet