Abilene UpdateFall Member Meeting ’05Philadelphia, PA

Abilene UpdateFall Member Meeting ’05Philadelphia, PA

Steve Cotter

Director, Network Services

scotter@internet2.edu

Steve Cotter

Director, Network Services

scotter@internet2.edu

2

WelcomeWelcome

• The Abilene Network

• Hurricane Katrina

• Advanced Services Across Abilene

• Network Research Across Abilene

• Other Network Services

• Network Security

• Abilene Network Futures

The Abilene NetworkThe Abilene Network

4

Abilene PartnershipsAbilene Partnerships

• Indiana University• Juniper Networks• Nortel Networks• Qwest Communications• ITECs• NC ITEC• Ohio ITEC• San Diego ITEC• Texas ITEC

• Internet2 Staff

5

Abilene Network TopologyAbilene Network Topology

6

Abilene Network TopologyAbilene Network Topology

7

Abilene Network TopologyAbilene Network Topology

8

Abilene Network TopologyAbilene Network Topology

9

Abilene Network TopologyAbilene Network Topology

10

Abilene Network TopologyAbilene Network Topology

11

Abilene Network TopologyAbilene Network Topology

12

Abilene CommunityAbilene Community

• 38 direct connections (OC-3c 10 Gbps)• 3 10 GE connections (OC-192c SONET also supported)• 7 OC-48c connections & 3 GE connectors• 26 connected at OC-12c (622 Mbps) or higher

• 240 Primary Participants – research universities and labs• Claremont Colleges, New World Symphony, Manhattan

School of Music, Cleveland Museum of Art, Cleveland Institute of Music, Los Alamos National Lab and Qwest are the most recent additions

• 130 Sponsored Participants - Individual institutions, K-12 schools, museums, libraries, research institutes

• 34 Sponsored Educational Group Participants - state-based education networks

See: http://abilene.internet2.edu/

13

Abilene R&E PeeringsAbilene R&E Peerings

14

Abilene International Peerings Abilene International Peerings

September 2005

15

Abilene Connector FeesAbilene Connector Fees

Original Fee

2003 2004 2005

OC-3c

(155 Mbps)$110k (1998)

($110k) ($110k) ($110k)

OC-12c

(622 Mpbs)$320k (1998)

$270k $240k $220k

Gig E

(1 Gbps)$325k (2001)

$325k $280k $250k

OC-48c

(2.5 Gbps)$495k

(2000)

$430k $360k $340k

10 Gbps

(SONET/ Ethernet)

$490k $480k $480k

16

Abilene Participation FeesAbilene Participation Fees

Effective January 1, 2006:

• Abilene Primary Participation - $21,000

Effective January 1, 2007:

• Abilene Primary Participation - $22,000

First increase since Abilene was launched in 1998

Hurricane KatrinaHurricane Katrina

18

Hurricane KatrinaHurricane Katrina

19

Hurricane KatrinaHurricane Katrina

• Hurricane Katrina strikes the Gulf Coast on August 29th, 2005.• Abilene’s unprotected lambda network link from Houston to

Atlanta goes down. The IGP (IS-IS) automatically reroutes around the fault.

• On September 1st, 2005 the damage to the carrier network was fully assessed and estimated to take days to repair.

• During this time, Abilene was operating with the risk of network isolation if there is a loss of the Chicago to Kansas City link.

• A redundancy plan was formulated and approved by Internet2 to have Abilene traffic route over the HOPI wave from Chicago to Seattle in the event that Chicago to Kansas link fails. The Abilene NOC engineers implement the redundancy plan.

• Service is restored to the Houston to Atlanta link on September 8th, 2005. No Abilene outages occurred during this period.

20

Hurricane KatrinaHurricane Katrina

8

88

88

8 88 8

8 H

H

H

H

21

Hurricane KatrinaHurricane Katrina

We would like to thank our partner Qwest for the extraordinary efforts they made to repair the network. Great job!

We also appreciate the support we received from the Abilene NOC and NLR. Thanks!

22

Abilene RedundancyAbilene Redundancy

• Responding to requests of our members, Internet2 has pursued redundancy options with our partner Qwest Communications.

• Qwest has agreed to provide on a per port basis, redundant connections to the Abilene router, at the node, for a cost of $400 per month regardless of speed as long as the redundant circuit speed is equal to or less than the primary circuit.

• This option is available to any active Abilene Connector who delivers their redundant circuit to the Abilene node. SONET and Ethernet framing methods would be supported under this option.

23

Abilene RedundancyAbilene Redundancy

Most Abilene Connectors Today:

What We Can Offer:

24

Redundancy OfferingRedundancy Offering

• We can make the following redundant connections available to our members who bring their circuits to an Abilene node:• VLAN connections through an existing exchange point:

• 1 GE $50,000.00• Physical connections to the router:

• OC3 $70,000.00• OC12 $75,000.00• OC48 $90,000.00• OC192 $125,000.00• 1 GE $80,000.00• 10GE $125,000.00

• A redundant circuit must be equal to or less than the primary circuit in speed and will not carry traffic unless the primary circuit fails.

• Each request will be evaluated on a case basis. The above figures are for budgetary purposes and are subject to change.

25

Redundancy OfferingRedundancy Offering

• Member’s requests for redundant circuits carried back to an Abilene node over the Qwest network will have their requests evaluated on a case-by-case basis for available capacity and pricing.

• These types of connections currently must be SONET.

Advanced Services Across AbileneAdvanced Services Across Abilene

27

IPv6 Peerings IPv6 Peerings

• IPv6 Deployment• Significant number of peers and connectors now have native

connections:• Roughly 2/3 of the connectors are IPv6 enabled• Roughly 1/2 of the peers are IPv6 enabled

• Connected to Palo Alto PAIX peering fabric at 333 Mbps for IPv6 and IPv4-Multicast experimental, non-production peering • 10 new experimental, non-production IPv6 peerings at the

PAIX so far in 2005

• Connected to MCI MAE-West at OC-3 for IPv6-only experimental, non-production peering• Qwest and MCI collaborated in providing the connection

28

IPv6 Addressing IPv6 Addressing

• Abilene has /32 that it can distribute to its members

• However, a number of connectors and members have or are acquiring their own address space: • 2001:4e0::/32 Wiscnet• 2001:5e8::/32 Pittsburgh Supercomputing Center• 2001:1860::/32 Pacific Northwest Gigapop• 2001:18e8::/32 Indiana University

29

IPv6 Security IPv6 Security

• Abilene NOC activities: • Limiting the v6 prefixes connectors send us

(as we do for IPv4) • Limited filtering for peer networks

• v6-security@internet2.edu is a mailing list for v6 security topics

30

Internet2 Involvement with the NAv6TF Internet2 Involvement with the NAv6TF

• Internet2 is active in the North American IPv6 Task Force (NAv6TF). • Rick Summerhill is on NAv6TF advisory

committee

• Abilene is key network component of the NAv6TF's Moonv6 national test network

31

Internet2 IPv6 Member Activities Internet2 IPv6 Member Activities

• North Carolina State University and Centaur Labs -- IPv6 streaming audio feeds from radio stations WCPE and WZYC

• IPv6 H.323 at Georgia Tech

• Abilene IPv6-enabled hosts • http://ipv6.internet2.edu/ipv6hosts.shtml

32

Internet2 Member Multicast Activities Internet2 Member Multicast Activities

• DVGuide - http://db.arts.usf.edu/dvguide/listings.asp • Several campus radio stations multicasting across

Abilene • ConferenceXP, a Microsoft Research initiative, relies

on multicast and has been deployed at several schools

• Access Grid continues to grow • More activity requiring "bridging" to multicast in

challenged environments, using the rcBridge software from ANU

• NYSERnet, Abilene and Internet2 deploying native IPv6 multicast• IPv6 Multicast demo live at Fall Member Meeting

33

Multicast Security Multicast Security

• Basic measures on Abilene: • Not allowing multicast streams with RFC1918 source

addresses • Not allowing multicast streams to "site local" group

addresses (239.0.0.0/8) which is a similar idea to RFC1918 addresses, but for group addresses.

• Blocking group addresses which are used for application which only have local significance. A good example of this is Norton Ghost.

• Other measures are under consideration, such as: • Blocking all IANA reserved multicast group addresses • Place a limit on the number of MSDP SAs each Abilene

Connector/Peer can originate

34

Internet2 Hands-on Multicast Workshops Internet2 Hands-on Multicast Workshops

Upcoming Workshops:• Hartford, CT – 4-6 October 2005 • Ann Arbor, Michigan - 17-19 October 2005 • Albuquerque, New Mexico - 2-4 February

2006

http://multicast.internet2.edu/workshops/

35

Other Advanced ServicesOther Advanced Services

• MPLS VPN testing – NC-ITEC experimenting (with ITEC Ohio) with creating a multipoint Layer 2 VPN using inter-domain MPLS tunnels and Virtual Private LAN Service (VPLS). Working in both a lab environment and between the ITECs using Abilene.

• The goal is to examine multipoint alternatives for a possible Abilene private network service offering.

Network Research Across AbileneNetwork Research Across Abilene

37

Network Research PhilosophyNetwork Research Philosophy

• Internet2 today does not do network research per se, but seeks to facilitate and support research projects led by faculty at member institutions• Make accessible network resources readily

available to this community• Participate in research collaborations and provide

support for proposals• Integrate research findings into the evolution of

Internet2 network initiatives and services

38

Network Research Resources Network Research Resources

• Resources available to researchers:• Abilene Observatory• MAN LAN Exchange Point • HOPI testbed• Collaboration with NLR, Regional Optical

Networks and other testbeds

39

Abilene ObservatoryAbilene Observatory

The Abilene Observatory is a program that supports the collection and dissemination of network data associated with the Abilene Network.

Provides researchers:• Operational data associated with a large-scale network • Data associated with the fundamental properties of basic

network protocols.

Two components of the Observatory:• Data collected by Abilene engineers using equipment

located in the router nodes and operated by the Abilene NOC

• Data collected by separate research projects using equipment collocated in the Abilene racks

40

Abilene ObservatoryAbilene Observatory

There are more than 30 research projects currently using Observatory data. Some of the more recent additions are:

• Flow Sampling and Anomaly Detection , Paul Barford, University of Wisconsin• Assess the Presence and Incidence of Alpha Flows in

Backbone Links, Vincenzo Liberatore, Case Western Reserve University• Traffic Management and QoS Provisioning in IP Networks,

Hassan Peyravi, Kent State University• Spatio-Temporal Network Analysis, Mark Crovella and Eric

Kolaczyk, Boston University.• MINDS Project, Vipin Kumar, University of Minnesota• Study of the Temporal-spatial Correlations in Network

Traffic, Don Towsley, University of Massechusetts

For a more comprehensive list, see: http://abilene.internet2.edu/observatory/research-projects.html

41

Project Highlight: PlanetLabProject Highlight: PlanetLab

• PlanetLab Upgrade• PlanetLab nodes currently located at all Abilene router

nodes, connected to the IP network• Upgrade will add connection to an MPLS L2VPN

configuration forming a layer2 network where the PlanetLab nodes will provide the routing engines

• Abilene becomes the layer2 circuit provider for PlanetLab• Normal users on Abilene don't have direct access to this

new "backbone network"• The PlanetLab network can peer with the commodity

network• Provides an infrastructure for network research that has

national scope

Other Network Services: FiberCo & MAN LANOther Network Services: FiberCo & MAN LAN

43

FiberCo OverviewFiberCo Overview

• Tool designed to support optical initiatives in the regions or nationally• Spun off from NLR governance discussions• Internet2 took responsibility for forming the LLC • Operates on behalf of U.S. higher education and affiliates –

Internet2 and NLR membership• Not an operating entity

• Will not light the fiber – only a holding company• Functions

• Market maker• Assignment vehicle for both national & regional optical initiatives

• Dark fiber provider: Level3 Communications• 3 year pricing agreement ends March 06• Intercity and metro fiber, new builds, consulting services• Exploring more formal relationships with other providers

44

State and Regional Optical NetworksState and Regional Optical Networks

• Alabama*• Arizona (CENIC)• Arkansas*• California (CALREN)• Colorado (FRGP/BRAN)• Connecticut (Conn. Education

Network)• Florida (Florida LambdaRail)• Georgia (Southern Light Rail)• Great Plains Network* (MIDnet)• Indiana (I-LIGHT)• Illinois (I-WIRE)• Louisiana* (LONI)• Massachusetts*• Maryland, D.C. & northern Virginia

(MAX)• Michigan (MiLR)• Minnesota* (BOREAS)

• National LambdaRail• New England Region (NEREN)• New Mexico (NMSU, UNM)• New York (NYSERNet*, Cornell)• North Carolina (NC LambdaRail)• Ohio (Third Frontier Network)• Oklahoma (OneNet)• Oregon• Pacific Northwest (Lariat – NIH BRIN,

PNNL)• Rhode Island (OSHEAN)• SRON* (southeastern U.S.)• Tennessee* (OneTN)• Texas (LEARN)• Virginia (MATP)• Wisconsin (WiscNet)• Wyoming

(RONs in red have made dark fiber acquisitions through FiberCo)

(*RONs with RFx’s issued or in process of acquiring fiber)

45

States with Regional Optical NetworksStates with Regional Optical Networks

States with a RON

46

Dark Fiber PlacementDark Fiber Placement

• Aggregate dark fiber assets acquired by U.S. R&E optical initiatives • CENIC (for CalREN & NLR) 6,200+ • FiberCo (via Level 3 for NLR & RONs) 8,600 • SURA (via AT&T) 6,000

• Plus 2,000 route-miles for research

• NLR Phase 2 (WilTel & Level3) 5,000 • OARnet 1,500 • ORNL (via Qwest)

900• NEREN 670 • Other projects (IN,IL,OR,CT…) 2,200+

Total (conservative estimate) 30,000+

• Over 60% of these assets are now held by RONs • Remainder held by NLR (~11,250 route-miles)

47

MAN LAN Exchange PointMAN LAN Exchange Point

• Manhattan Landing in New York City - partnership with NYSERNet, Indiana University, and the IEEAF

• Provides a high performance exchange facility for research and education networks

• Located at 32 AoA in NYC - easy interconnection to many national and international carriers and other research and education networks

• Peering model is open and bilateral• Cost recovery model - minimal connection charges

for layer 2 facility, none for layer 1 connections• Working with AtlanticWave on future distributed

exchange point along U.S. East Coast (NYC↔Miami)

48

MAN LAN ServicesMAN LAN Services

• Layer 2 - Ethernet switch for IPv4/v6 peering with 1GigE and 10 GigE interfaces

• Layer 1 - TDM based optical equipment (SONET / Ethernet interfaces)• Cisco 15454• Nortel OME 6500• Nortel HDXc

• Layer 0 – Glimmerglass optical cross connect to facilitate changes

Network SecurityNetwork Security

50

Network SecurityNetwork Security

Basic Premise: Abilene Security Policy is determined by the properties of an IP network• Control is at the edge

• Hosts determine when and where to send packets and initiate flows

• This control often leads to vulnerabilities• Hosts can become compromised• Hosts may be used to compromise other hosts• Can lead to large amounts of traffic sent to other hosts

As a backbone network, we view Abilene as a ‘pipe’ and not a controlling entity

51

Network ControlNetwork Control

The Abilene backbone does have the means to apply some control across the network:• It is possible to block traffic on some ports• It is possible to block all traffic from a particular IP address

Abilene does not unilaterally filter traffic on a network wide basis unless the network itself is under attack.

52

Filtering TrafficFiltering Traffic

Abilene will filter traffic in some situations:• If one or more hosts on a connector or peer were under

attack• If requested by an institution, peer, or connector

(noc@abilene.iu.edu, 317-278-6622)

Abilene will filter traffic to a connector or peer if requested by that particular connector or peer network, filtering the appropriate traffic through the connection in question.

• Abilene’s method for blocking this traffic is our BGP Discard Routing procedure

53

Filtering TrafficFiltering Traffic

Abilene reserves the right to protect itself and its connectors / peers from other connectors and peers.• If a threat to the network exists through a particular

connector, Abilene reserves the right to filter that traffic• Ultimately, Abilene could disconnect the offending connector

or peer

Abilene reserves the right to filter all traffic or terminate any connection if it is under attack.

• Note: Every attempt will be made to contact the network in question to discuss various options and alternatives.

54

Research and Education Information Sharing Analysis Center (REN-ISAC)Research and Education Information Sharing Analysis Center (REN-ISAC)

The REN-ISAC supports higher education and the research community by:• Provides advanced security services to national supporting

networks• Supports efforts to protect the national cyberinfrastructure by

participating in the formal sector ISAC infrastructure

Abilene will report all known incidents of security threats to the REN-ISAC.

55

Data CollectionData Collection

Abilene collects flow statistics on a sampling basis that potentially could identify source and destination addresses and ports• This data is anonomyzed (11 lower order bits of all IP

addresses are zeroed out) before it is saved to disk • For privacy reasons: Abilene does not collect data pertaining

to communications between identifiable hosts• However, this information could identify compromised hosts

During times of security attacks, the REN-ISAC can unanonomyze data, but only that data related to the attack itself. The resulting data is anonomyzed as soon as possible after the attack is understood.

56

Data AnalysisData Analysis

Information derived from analysis of the flow data that identifies specific institutions or hosts is treated as confidential information.

Institutions may request specific sources of cyber security attacks located on their respective networks. Only security related information will be reported to the institutions.

Abilene data is meant to supplement, not replace, data collected by individual institutions or connectors. Internet2 strongly encourages institutions to collect their own data, potentially providing a greater degree of specificity to particular security problems.

57

BGP Discard RoutingBGP Discard Routing

Connectors can advertise routes to Abilene via BGP for which all traffic to those routes will be discarded by the Abilene routers. This is useful during a DoS attack because the traffic can be dropped before it crosses the link to the connector.

Here are a few important points: • Discard routes will NOT be accepted for routes larger than a /24 • There is no way to place a limit on the number of discard routes

a connector can advertise. The limit on the total number of routes a Connector can advertise is currently 3,000.

• Abilene's default policy is to not accept routes smaller than a /27. There have been some exceptions made to this policy. For those /28 and smaller routes, it will not be possible to announce more specific discard routes.

58

Abilene Network SecurityAbilene Network Security

Actions underway/planned:• Updated the Abilene Transit Security Policy.• Planning periodic Operational Security Assessment Excercises• Work more closely with the REN-ISAC on investigating, validating,

and resolving ongoing security issues.• Work with the REN-ISAC, Abilene NOC, Advanced Network

Management Lab (ANML) and Arbor Networks to enhance our security capabilities.• Deploying Arbor Networks Peekflow SP equipment

• Work with industry and researchers to gather information about threats and attacks and disseminate this information to the community.• Developing portal views for Abilene Connectors and Peers• Web publish traffic statistics• Fingerprint detection and sharing with other networks• Disseminate alerts when worms and anomalies detected

Abilene Network FuturesAbilene Network Futures

60

Next Generation AbileneNext Generation Abilene

Mission of Internet2: To build leading-edge R&E networking capabilities. 

This mission rests on belief that evolving new technologies will drive new network architectures with a broader set of services and capabilities. 

61

Next Generation AbileneNext Generation Abilene

• Internet2 is focused on integrating and rapidly deploying innovative new capabilities

• Working to understand how the next generation architecture will evolve over the next 5-7 year timeframe• Numerous discussions with researchers, carriers and

equipment vendors• Examining how a hybrid of shared IP packet switching and

dynamically provisioned optical lambdas can meet the needs of the community.

• Continue to engage the GigaPoPs, state/regional networks and campus environments

62

Next Generation Abilene Design ConsiderationsNext Generation Abilene Design Considerations

Architectural Design Considerations• NLR, RON and international integration • Advanced service support - Multicast, v6, High Performance

Throughput, Measurement• Enhanced network research facilitation• Network and end-user security• The applications that will ride across the network• Options for increased reliability and additional servicesProcess• Hybrid architecture evaluation (HOPI)

• Production IP core network• Dedicated point-to-point capabilities (’s, MPLS tunnels)

• Evaluation of optical transport capabilities - • NLR, commercial providers & RONs

• Design & planning collaboration• U.S. & int’l partners (ESNet, TeraGrid, SURFnet, GEANT-2)

63

HOPI ResourcesHOPI Resources

Resources available to the HOPI team:• Abilene Network – 10 Gbps IPv4/IPv6 + MPLS

tunnels• 10-Gbps on the NLR footprint• MAN LAN Exchange Facility

• 10-Gbps λ NYC – London to provide connectivity to the European testbeds

• Layers 1 and 2 switching gear

• Collaborations with Regional Optical Networks (RONs) and other related efforts (GLIF, UltraLight, DRAGON, etc.)

64

Next Generation Abilene TimelineNext Generation Abilene Timeline

• October 2007 - End of recent 1-year Abilene transport MoU extension• Sets next-generation network planning timeline

• Architecture definition: 1/1/2006• Transport selection: 4/1/2006• Equipment selection: 7/1/2006• Backbone deployed: 1/1/2007• Connector transition: 2007

• Concurrently, review overall business plan and management model

• Network design time frame: 2007-2012

• HOPI testbed is expected to be in place for 2-3 years, to experiment with future protocols• Refine and evolve next generation architecture

65

Next Generation Network RoadmapNext Generation Network Roadmap

• 2005-2007• ‘WaveCo’ – complementary relationship for carrier provided

wavelengths to augment backbone• Collocation and dark fiber services via FiberCo• Layer 1 measurement / monitoring• Interdomain control plane & AAA

• 2008• Wavelength services

• Static ‘Core’ wavelengths for IP backbone• Point-to-point unprotected & protected variable duration waves

• GMPLS dynamic provisioning: dynamic set up on the order of minutes• 40G transport / switching on selected routes• Optical layer security

• 2009-2010• GMPLS dynamic provisioning: near real-time dynamic set up• Alien / transparent wave service

66

Many Thanks to the Abilene TeamMany Thanks to the Abilene Team

• Heather Bruning – Program Manager, Business Operations

• Andrea Blome – Asst. Prog. Manager, Business Operations

• Bill Cerveny – Internet Engineer • Christian Todorov – Network Engineer• Ana Preston – Program Manager, International

Relations and RONs• Members of the Indiana NOC, Abilene Planning

Team, Abilene TAC

And other Internet2 staff and member volunteers who help make Abilene run.

67

Abilene InformationAbilene Information

• For more Information:• http://abilene.internet2.edu• http://abilene.internet2.edu/observatory/ • http://www.nationallambdarail.org• http://hopi.internet2.edu

• Or contact us at:• scotter@internet2.edu • heather.bruning@internet2.edu • abilene@internet2.edu

68

Questions / Comments?

Thank you for coming.