history of 100g and internet2
DESCRIPTION
Internet2 will be aggressive in its deployment of its Innovation Platform in order to allow its members to capitalize on the groundswell of support for high-speed software defined networking - summer of networking, Chris Robb, Indiana University/Internet2TRANSCRIPT
HISTORY OF 100G AND INTERNET2Chris Robb, Indiana University/Internet2
• Introduction• Technical Background• History of Internet2’s 100G Deployment• Current Progress• Lessons Learned
Topics
• Why, I’m Chris Robb!• Director of Engineering and Operations for Internet2• But I work at IU. It’s all horribly confusing. • Started in the trenches of UITS computer lab support• Joined campus networking division as a programmer in 1998• GlobalNOC engineer in 2001. First assignment: Hawaii!• Managed the 2006 Internet2 transition to Level3• Subcontracted to Internet2 in 2008• My day consists of video conferencing, lunch, then more video
conferencing- usually in that order• I also occasionally haunt the halls of the CIB
Who’s this guy?
• “Internet2 is an advanced networking consortium led by the research and education community”
• Formed in 1996• More than just a network
– Middleware– Net+– Security– Research– Measurement
• Based in Ann Arbor with offices in Washington DC, New York and (soon) California
What’s Internet2?
Technical Background
National Backbone Organization
• Fiber• Optical Equipment• Routers and Switches• Network Operations Center
Dark Fiber• Someone has to lay fiber in the ground• Commonly laid along railway lines and interstate roads• A lot of investment in the late 90s• Multiple conduits installed
– Ability to blow fiber later• Multi-strand cable (144ct is common, per conduit)• Hut spacing defined at installation time• Huge investment
– Nationally cost prohibitive– Regionally expensive, but doable
• Sometimes not very well documented
Cable conduit is laid
Fiber is blown through empty conduit
Photos courtesy of Steve Wallace
Access to fiber via handholes
Fiber is terminated in huts to house optical amplification equipment
Who can buy fiber?• Depends on when you’re asking, who you’re asking, and who you are• Early 2000s: plentiful supply of fiber (and companies that are no
longer around)• Now: not so much
– Consolidation– Not a lot of investment in fiber (exception: high frequency trading)
• Mostly carrier swaps and internal use• Sold as an Indefeasible Right to Use (IRU), typically on a 20 year term• Expensive!
• Activate Google Earth
• What do you put on the fiber once it’s laid? – Too expensive to just put a single signal on each fiber
• Client interfaces can only go so far– 10GBase-SR – up to 300m (depending on fiber type)– 10Gbase-LR – 1310nm – 10km– 10GBase-ER – 1550nm – 40km– 10GBase-ZR – 1550nm – 80km
• Put a switch every 80km?– You saw those huts! Too little space.– Who wants to manage that many devices? Too complex.– There are only so many coal plants in the US. Too much power.
OK, we have some fiber….
• Multiplex!• Convert signals from client equipment (routers & switches) to specific
wavelengths of light (ITU grid)• Combine the different wavelengths onto a common fiber on one end,
and separate them on the other end• Uh oh! Light doesn’t travel far enough!
– Amplify!– Periodically shoot the signal through erbium doped fiber spools that
amplify the signal• The noise! The noise! Amplification also amplifies noise.
• Periodically regenerate the signal• Break down the analog signal back into its digital form, then send it
out again as a newly cleaned up analog signal
DWDM Optical Equipment
15
DWDM History• Early WDM (late 80s)
–Two widely separated wavelengths (1310, 1550nm)
• “Second generation” WDM (early 90s)–Two to eight channels in 1550 nm window–400+ GHz spacing
• DWDM systems (mid 90s)–16 to 40 channels in 1550 nm window–100 to 200 GHz spacing
• Next generation DWDM systems–64 to 160 channels in 1550 nm window–50 and 25 GHz spacing
ITU Wavelength Grid
• ITU-T l grid is based on 191.7 THz + 100 GHz• It is a standard for laser in DWDM systems
l1530.33 nm 1553.86 nm0.80 nm
195.9 THz 193.0 THz100 GHz
Freq (THz) ITU Ch Wave (nm) 15201/252 15216 15800 15540 15454192.90 29 1554.13 x x x x x192.85 1554.54192.80 28 1554.94 x x x x x192.75 1555.34192.70 27 1555.75 x x x x x192.65 1556.15192.60 26 1556.55 x x x x x
• Fiber• Erbium Doped Fiber Amplifier (EDFA) (“Amp”)• Optical Add/Drop Multiplexor (“OADM”)• Reconfigurable Optical Add/Drop Multiplexor (“ROADM”)• Tributary (“Trib”) card• And a few more concepts
– Directionless– Colorless
• To the whiteboard!!
Modern Optical Components
Optical Multiplexer
Optical De-multiplexer
Optical Add/Drop Multiplexer(OADM)
Transponder
DWDM Componentsl1
l2
l3
l1
l2
l3
850/1310 15xx
l1
l2
l3
l1...n
l1...n
Optical Amplifier(EDFA)
Optical AttenuatorVariable Optical Attenuator
Dispersion Compensator (DCM / DCU)
More DWDM Components
Optical Amplifier
Optical Add/Drop
Transmission Effects• Attenuation:
– Reduces power level with distance
• Dispersion and nonlinear effects: – Erodes clarity with distance and speed
• Noise and Jitter:Leading to a blurred image
• How do you get millions of people connected to this capacity?• Routers direct your data transfers around the country• Generally large boxes at the national level ($500K-$1M each)
– Higher interface speeds (10Gbps+)– Not a lot of edge features (traffic shaping, policing, etc.)
National Routed Network
Internet2 Network Topology Evolution
• 1998-2003– Cisco GSR routers– Qwest OC-12/OC-48 backbone– Average Connector Speed: 450-600Mbps
• 2003-2007– Juniper T640 routers– Qwest OC-192 backbone– Average Connector Speed: 1-4Gbps
• 2007-2009– Juniper T640/T1600 routers– Level(3) OC-192/10GigE backbone– Average Connector Speed: 8.3Gbps
• 2009-2011– Juniper MX960 Routers– Level3 Multiple 10GigE Backbone– Average Connectors Speed: 10+ Gbps
• 2011 and beyond– Juniper T-1600 Routers– Owned Infrastructure with 100GigE Backbone
Potential Edge Bandwidth Growth
4/1/
99
4/1/
00
4/1/
01
4/1/
02
4/1/
03
4/1/
04
4/1/
05
4/1/
06
4/1/
07
4/1/
08
4/1/
09
4/1/
10
4/1/
110
2
4
6
8
10
12 11.210.3
Measured Perabytes into Internet2 per Month
1997 1998 2002 2008 2010 2011+0
10
20
30
40
50
60
70
80
90
100
0.6 2.410
20
30
100
Internet2 Network Backbone Speeds (in Gbps)
Internet2’s McLean, VA POP
Internet2’s McLean, VA POP
History of 100G Deployment
A Community Defines Its Future
• In June 2009, Internet2 and the R&E community author the “Internet2 Architectural Directions” Document….
• Multiples of 10GigE will be the primary transport to Regional and State Networks over the next 3-5 years.– 10G cost low compared to the cost of 40 or 100G– Multiple large sub-10G flows the norm
• Internet2 Network access will be divorced from physical interface speeds and available for apportionment across the network– Flexibility for connectors an important success factor.
Architectural Principles
• Native 100GigE at the optical layer is an important technology to adopt today– Take advantage of current opportunities to lay the
groundwork for future expansion.
• Collapsing Layer2 and Layer3 services onto a single delivery platform is an important step toward the hybridization of the network – Reduce overall operating expenses to the Connectors– Candidate technologies include MPLS L2 VPNs, Layer2
Ethernet VLANs and Virtual Private LAN Service (VPLS).
Architectural Principles
• The Internet2 IP and Layer2 Networks need a migration path to 40G and 100G in the next few years – Backbone must be able to efficiently handle multiple
simultaneous 7-10 G flows and individual flows >10Gbps
• The Internet2 Network emphasis should be on additional services and technologies that will drive transport bandwidth requirements – The use case for the network drives the technology of
the network.
Architectural Principles
• Internet2 will coordinate with the Regional and State Network partners to determine the most optimal node quantity and locations – Offer a flexible partnership with the connectors.– Create more options for connections.
• As mission-critical applications become more integral to the Regional cost-recovery model, the Internet2 Network must focus on enhanced redundancy where needed – Many recent services and uses of the network require
increasingly reliable/redundant/resilient connectivity
Architectural Principles
• The Internet2 Network will continue to be instrumented and operated in a transparent fashion that supports the end-to-end model – The more information that is available about
the network the better everyone understands the need for and requirements of the network.
Why is 100G So Important?
• Edge speeds are outpacing backbone speeds– mid 2009 potential edge bandwidth was 17x the edge
bandwidth of 2001
• Traffic is steadily growing• Large flows are becoming increasingly
commonplace• 100Gbps will scale and become economically
cost effective• Drive to innovate!
History of 100G and Internet2
• June 2009: Internet2 staff began polling the community for set of technical principles to guide staff efforts over the next several years– Resulted in the “Internet2 Architectural Directions”
document that specifically called out 100Gbps networking as a strategic direction
– AOAC approval of document on October 5th, 2009• August 2009 Internet2 released an RFI to the
industry that sought industry feedback on a 100Gbps optical platform
Funding History
• In late summer of 2009, the NTIA released a Notice of Funding Availability (NOFA) for broadband funding– Broadband Technologies Opportunities Program
(BTOP)– $7.2B in funding
• Internet2 decided not to respond to the first round of funding so as not to compete with the regional networks that were applying for funds
Funding
• Second round of BTOP funding in early 2010• Internet2 submits a joint proposal with NLR, IU and
the Northern Tier Networking Consortium (NTNC)– Obtain a partial national footprint and equip it with an
optical network– Leverage the existing NLR and Internet2 optical paths– Upgrade the NTNC network between Chicago and Seattle– Upgrade all Internet2 routers to be 100G capable– Upgrade the TR-CPS routers to newer hardware– Create a “low-latency” layer of small routers at 27 sites
Award and Change of Plans
• Internet2/NLR/NTNC were selected as the only “national middle mile” network in the BTOP program in late spring 2010
• During the due diligence phase, NLR opted to step out of the partnership– Network topology was refactored to a national
footprint to cover those sections of the country that had been covered by NLR’s in-kind contribution
• Project officially started in July, 2010
Preliminary Optical RFI
• February 2010: meetings were scheduled with several companies to review their offerings– Evaluation was based on a number of different factors:
• Optical properties• Control plane development • Physical properties• Road map for product availability and others
– From this review it was determined that there were optical vendors with viable options for building the U.S. UCAN network
– Accordingly they were given the option to respond to the RFP for optical equipment in September 2010
RFP Issuance
• Based on this research potential vendors were identified: – Cisco– Infinera– Ciena
• Internet2 released the U.S. UCAN Optical Network Request for Proposal (RFP) on September 13, 2010, in collaboration with Indiana University
• Vendor responses were due on September 22– The response due date was extended three times:
• on September 17 Internet2 granted an extension to September 29 • on September 23 Internet2 granted an extension to October 5th • on October 5 Internet2 granted an extension to October 9th
Review Team
• At the September NTAC meeting, Internet2 solicited volunteers from the community to review the RFP responses
• A team of community members, Internet2 staff and IU NOC staff was assembled to evaluate the responses
Team Outcome
• All three responses were outstanding and each vendor included compelling and creative support for this community effort. – Infinera will provide an upgrade path for the Northern Tier
Networking Consortium on their next generation platform– Ciena will provide the national footprint on its Activeflex
(formerly OME) 6500 platform– Cisco is being engaged on a variety of fronts for parallel efforts
• Recommendation presented to the AOAC on November 10th
– AOAC endorsement and forward to Internet2 senior management for final approval
Ciena Platform
• 100Gbps capable 80-channel DWDM system– 100G cards shipping today
• ROADM-based solution at most or all add/drop facilities
• Directionless capability in metro areas• Non-Dispersion-Shifted approach provides
economical approach that reduces CAPEX• Compact, scalable footprint that adapts to the
changing needs of our community
Current Progress
• Network Infrastructure– Dark Fiber on a national footprint– Wave Capacity
• Optical equipment to light new fiber with 100Gbps-capable equipment• Upgrade of Northern Tier Networking Consortium Infinera equipment
to be 100G-capable– 100Gbps IP Backbone
• 100G and 10G routers for R&E network• 10G routers for TR/CPS
• Enhanced Services– Increased connectivity to Commercial Exchange Points– Regional interconnects
• Research Opportunity– Stronger collaboration– Enhanced capability
Upgraded Internet2 Infrastructure benefits
48 – 04/09/2023, © 2009 Internet2
Internet2 Optical Network Topology
Phase 1 Optical Build - COMPLETE
• All fiber acquired and accepted• All Optical Equipment Installed• BER Testing complete• System Commissioning
– EMS system installed and populated with nodes– Internet2 NOC database population nearly complete
• First 100GigE IP circuit between New York and Washington DC live and integrated into the Internet2 IP Network
• First Trans-Continental 100G configured and passing pings for Internet2 Fall Member Meeting in early October
Phase 1 Progress – 100% Complete
Phase 2 – Complete
52 – 04/09/2023, © 2009 Internet2
Phase 3 – January 2012 – July 2013
53 – 04/09/2023, © 2009 Internet2
Current Status
• Optical is great, but what about the routers?• Well, that’s a tricky story. • We’re at a crossroads• Routers are expensive, big, power hungry monsters• Did I mention expensive?• We want them everywhere• We want them cheap• We want them small• We want control• We want them powerful
• Is there a white knight?
Will someone please think of the routers?
Openflow?
• Faster– 2002: One of the first 10G IP national backbones deployed on Juniper T640
platform– Mid-2009: Architectural Directions document cites need for 100G IP backbone– Mid-2010: $96M Broadband Technologies Opportunities Program (BTOP)
National Middle Mile Award– Late-2011: First national 100GigE circuit lit between New York and Sunnyvale
• Smarter– 2005: Hybrid Optical Packet Infrastructure (HOPI) network deployed in first
national trial of reconfigurable network– 2007: Interoperable On-demand Network (ION) deployed at 21 locations
around the US– 2011: DYNES campus deployment project begins pushing dynamic deployment
to the edge– Mid-2011: Open Networking Foundation formed by a coalition of research and
industry members– Late-2011: 5-node NDDI deployment and development of OSE software
History of implementation
57 – 04/09/2023, © 2011 Internet2
• Demand from Internet2 Members to Innovate• Equipment Refresh Cycle• Available Funding
Why now?
58 – 04/09/2023, © 2011 Internet2
• This is a very high-level restructure of the Internet2 network, so it’s important that stakeholders at all levels participate in the discussion– Researchers– Campuses– Regionals– International Peers– Funding Agencies– Governance Councils– Technical Bodies
• Internet2 will be aggressive in its deployment of its Innovation Platform in order to allow its members to capitalize on the groundswell of support for high-speed software defined networking
Engagement Plan
59 – 04/09/2023, © 2011 Internet2
• Goal: Integrate SDN functionality into the Internet2 Network– Preferred Goal: platform integration between two sets of services to
provide initial network effect and migration path• A few different models to consider• Nothing is set in stone, but we do have to move aggressively
Implementation possibilities
60 – 04/09/2023, © 2011 Internet2
Separate Infrastructure: What we do today
61 – 04/09/2023, © 2011 Internet2
Combined Infrastructure: What we might do tomorrow
62 – 04/09/2023, © 2011 Internet2
Upgraded Layer2 Network
• Pre-project– Know your goals– Know your limitations, then ignore them– Community Input
• Negotiations– Have a good lawyer– Understand what you’re buying– Budget a lot of time
• Remember: this is for 20+ years
• Implementation– Documentation is critical– Inventory management needs to be worked out ahead of time– It isn’t plug and play– Budget a lot for space and power
Lessons Learned
QUESTIONS?