ethernet evolution: the path to 100 gigabit ethernet · 2006-12-06 · ethernet evolution: the path...
TRANSCRIPT
Ethernet Evolution: The Path to 100 Gigabit Ethernet
John D’Ambrosia -Scientist, Components TechnologyChair, IEEE 802.3 HSSG
Version 2.0 16 Oct 2006
2
Per IEEE-SA Standards Board Operations Manual, January 2005
At lectures, symposia, seminars, or educational courses, an individual presenting information on IEEE standards shall make it clear that his or her views should be considered the personal views of that individual rather than the formal position, explanation, or interpretation of the IEEE.
3
Acronym Cheat SheetCFI – Call for InterestDWDM – Dense Wavelength Division MultiplexingEMI – Electro-magnetic InterferenceGbps – Gigabit per SecondHSSG – Higher Speed Study GroupITU – International Telecommunications UnionIETF – Internet Engineering Task ForceJEDEC - Joint Electron Device Engineering CouncilMAC – Media Access ControlMDI – Medium Dependent InterfaceMSA – Multi Source AgreementOIF – Optical Internetworking ForumPCS – Physical Coding SublayerPMA – Physical Medium AttachmentPMD – Physical Medium DependentPHY – Physical Layer DeviceSERDES – Serialize / De-serializeSMF / MMF – Single Mode Fiber / Multi Mode FiberTbps – Terabit per SecondWIS – WAN Interface SublayerXGMII – 10 Gigabit Media Indpendent Interface
4
Agenda
The Ethernet EcoSystemIEEE 802.3 Higher Speed Study GroupConsiderations for System Development
5
Agenda
The Ethernet EcoSystemIEEE 802.3 Higher Speed Study GroupConsiderations for System Development
6
The Ethernet Ecosystem
Research, Education and Government
Facilities
ResearchNetworks
Broadband Access
BroadbandAccess Networks
Data Centers and Enterprise
EnterpriseNetworks
Content Providers
ContentNetworks
Internet BackboneNetworks
Internet BackboneNetworks
IEEE 802.3 HSSG “Call-For-Interest”, 7/2006
Internet eXchange andInterconnection Points
7
What is Driving the Need for Higher Speed Ethernet?
Research network trends– 10 GbE WANs– Cluster and grid computing
Data center trends– Lots of GbE and 10 GbE servers
ISP / IX trends– Personalized content – Consumer broadband– 10 GbE peering
Aggregating 10 GbE links with LAG is an interim solution
Service ProviderService Provider
Data CenterData Center
Government/ResearchGovernment/Research
8
802.3ad Link Aggregation (LAG)
It is being usedTemporary fix for increased bandwidth demandIncreased complexity– Difficult to plan for capacity and traffic engineering– Harder to manage & troubleshoot multiple physical links based
on a single logical interface– Cable & link management
Uneven distribution of traffic– Limitations in the standard– Inefficient distribution of large flows – Load balancing requires packet inspection or other knowledge
IEEE 802.3 HSSG “Call-For-Interest”, 7/2006
9
Bandwidth and Growth Projections
Research, Educationand Government
Facilities
Consumer Broadband Access
Content Providers
Corporate Data Centers and Enterprise
ResearchNetworks
BroadbandAccess Networks
EnterpriseNetworks
ContentNetworks
Internet BackboneNetworks
Internet BackboneNetworksLevel 3: 8x10 GbE LAG today,
BW growth 15x in 5 years (~70%/year)
Internet Exchanges: Up to 8x10 GbE LAG today,
BW growth 50-75% per year for next 3 – 5 years
Cisco: 10GbE today, 40+ GbE (100 GbE
preferred) in 5 years
Cox: 10 GbE today, BW growth 50-75% per
year for next 3 – 5 years
Comcast: 4x10 GbELAG today, 3X BW
increase in 3 to 5 years
Yahoo!: 4x10 GbE LAG today,BW doubling in <12 months
LLNL: 4x10 GbE LAG and 500x10 GbE ports
today, 10x speed requirement in 5 years
on deployed ports
ESnet: 10 GbE today, 10 Gbps on 20+ links 5 years from now; 5-
10 locations will require more than 40
Gbps
IEEE 802.3 HSSG “Call-For-Interest”, 7/2006
10
Agenda
The Ethernet EcoSystemIEEE 802.3 Higher Speed Study GroupConsiderations for System Development
11
IEEE 802.3 July, 2006 Plenary
Higher Speed Study Group (HSSG) “Call-For-Interest”– http://grouper.ieee.org/groups/802/3/cfi/0706_1/CFI_01_0706.pdf– Led by John D’Ambrosia / Joel Goergen, Force10 Networks
Broad Industry Support– End Users– System Vendors– Component Vendors
“Higher Speed Study Group” formation approved
Sept 2006 - John D’Ambrosia, Force10 Networks, Confirmed as HSSG Chair
12
IEEE 802.3 HSSG Reflector and Web
To subscribe to the HSSG reflector, send an email to:[email protected]
with the following in the body of the message:subscribe stds-802-3-hssg <your first name> <your last name>end
HSSG web page URL:http://grouper.ieee.org/groups/802/3/hssg/index.html
13
Developing a Standard
Call for Interest
Study Group
Task Force
Working Group Ballot
Sponsor Ballot
Standards Board Approval
Publication
Feasibility and Research
Ideas From Industry
IEEE~4 Years
Industry Pioneering
1 Year
Higher Speed Ethernet is
Here
Ad Hoc Efforts
CFI July 18, 2006
14
Normal Study Group Output
Project Authorization Request5 Criteria– Broad Market Potential– Compatibility – Distinct Identity– Technical Feasibility– Economic Feasibility
Project Objectives
15
MDI
Objectives – Speed?
Higher Speed Ethernet– Means faster MAC data rate
Options Discussed– 40 Gbps– 80 Gbps– 100 Gbps– 120 Gbps– Scaleable
IEEE 802.3 HSSG adopted 100 Gb/s (Nov 2006)
64B/66B PCS
MAC Client
Higher Layers
MAC ControlMAC
Reconciliation Sublayer (RS)
WIS64B/66B PCS
PMA PMAPMD PMD
MDIMEDIUM MEDIUM
XGMII XGMII
10GBASE-R 10GBASE-W
MAC = Media Access ControlMDI = Medium Dependent InterfacePCS = Physical Coding SublayerPMA = Physical Medium AttachmentPMD = Physical Medium DependentWIS = WAN Interface SublayerXGMII = 10 Gigabit Media Independent Interface
MAC Rate = 10GPHY Rate < 10G
16
Objectives –PHY Types and Reach?
10 Gb Ethernet Historical Perspective
To date, IEEE 802.3 HSSG has adopted (Nov 2006)– Support at least 10km on SMF.– Support at least 100 meters on OM3 MMF
Serial4 lanes
Twisted Pair
4 lanes
SerialSerial
WDMSerialSerial
50 um / 62.5 um MMF220 m1310 nm10GBASE-LRM
1 m1 m
100 m15 m
40 km10 km
300 m 10 km
300 / 33 m
1550 nm1310 nm
1310 nm
850 nm
Improved FR-410GBASE-KX4Backplane
Copper
Fiber
Improved FR-410GBASE-KR
UTP10GBASE-TTwinaxial10GBASE-CX4
Single Mode Fiber10GBASE-ER / W Single Mode Fiber10GBASE-LR / W
50 um / 62.5 um MMFSingle Mode Fiber
10GBASE-LX4
50 um / 62.5 um MMF10GBASE-SR / W
17
Summary of Adopted Objectives
To date, IEEE 802.3 HSSG has adopted:– Support full-duplex operation only.– Preserve the 802.3/Ethernet frame format at the MAC Client
service interface.– Preserve minimum and maximum FrameSize of current
802.3 Std.– Support a speed of 100 Gb/s at the MAC/PLS interface.– Support at least 10km on SMF.– Support at least 100 meters on OM3 MMF.
All objectives adopted at IEEE 802 November 2006 Plenary
18
Existing Backbone Networks
ResearchNetworks
BroadbandAccess
Networks
EnterpriseNetworks
ContentNetworks
Internet BackboneNetworks
Internet BackboneNetworks
ResearchNetworksResearchNetworks
BroadbandAccess
Networks
EnterpriseNetworksEnterpriseNetworks
ContentNetworksContent
Networks
Internet BackboneNetworks
Internet BackboneNetworks
10 Gbps λ DWDM
40 Gbps λ DWDM
10 Gbps λ DWDM – wide deployment
40 Gbps λ DWDM– Being evaluated– Initial deployments– Return on Investment?– Interim solution?
Rate Choice– 100 Gbps maps into 10 Gb/s λ
DWDM – 120 Gbps maps into 40 Gb/s λ
DWDM“Ethernomics”– The Ecosystem is cost sensitive– Choice of 120 vs 100 Gbps
– Faster?– Wider?
IEEE 802.3 HSSG choose 100 Gbps
Ethernet Ecosystem
19
Future Backbone Networks?
“Ethernomics”– The Ecosystem is cost sensitive
20+ Gbps λ DWDM?– Lower cost than 40 Gb/s– Re-use of existing infrastructure?– Return on Investment?
100 Gbps λ DWDM– Return on Investment?– Future capacity requirements
ResearchNetworks
BroadbandAccess
Networks
EnterpriseNetworks
ContentNetworks
Internet BackboneNetworks
Internet BackboneNetworks
ResearchNetworksResearchNetworks
BroadbandAccess
Networks
EnterpriseNetworksEnterpriseNetworks
ContentNetworksContent
Networks
Internet BackboneNetworks
Internet BackboneNetworks
20+ Gbps λ DWDM?
100 Gbps λDWDM?
Future Ethernet Ecosystem
20
Thoughts Related to Backbone
40 Gbps λ DWDM– Not widely deployed– Interim solution to bandwidth crunch– Support seen by
– Transport equipment providers – Service providers
Choice of 100 Gbps– Service providers are evaluating Ethernet for transport– Maps into existing deployed 10 Gbps λ DWDM– Support for 100 Gbps λ DWDM seen by
– Transport equipment providers– Service providers
– Recent decision by IEEE– Future discussions and work needed
21
Other Bodies
The Ethernet AllianceOptical Internetworking Forum – 20Gbps to 25Gbps electrical interface project.– 100Gbps to 160Gbps system interface project.
ITUIETFJEDEC
22
Agenda
The Ethernet EcoSystemIEEE 802.3 Higher Speed Study GroupConsiderations for System Development
23
Chassis Slot Density
400Gbps to 500Gbps2009
120Gbps2006/7 – in design now
Based on max back plane thickness of 300mils, 20TX and 20RX differential pipes.
60Gbps2004
40Gbps2000
Slot densityYear System Introduced
24
Chassis Design
Chassis design issues:– Backplane and channel
signaling– Lower system BER– Connectors– N+1 switch fabric – Reduced EMI– Clean power routing
architecture– Thermal and cooling– Cable management
All design aspects must also meet local regulatory standards
25
Backplane Considerations
Slot CapacitySwitching CapacityPerformance– Signal coding– BER– Impacts SerDes Design
Design and technology drives scalability– Advanced fiberglass materials– Unique conductor layers– Engineered trace geometries
26
Anatomy of a 100 Gbps Solution: Designing for EMI Compatibility
EMI is Electro-magnetic Interference, i.e. noiseToo much noise interfere can cause bit errorsConcerned about two types of EMI– Conducted through
power– Radiated through air
0
10
20
30
40
50
60
30M 50 60 80 100M 200 300 400 500 800 1G
Leve
l in
dBµV
/m
Frequency in Hz
EN 55022 Class A Electric Field S
E300 Power Supply Radiated Emission for 30 MHz to 1 GHz
27
Power Considerations
End User Restrictions?Total system wattage?Input power quality not specifiedHigher speeds require lower noise
28
Thermal Management Considerations
Cooling capacity per slot?Front to back filtered airflow for carrier deploymentsCooling redundancyHeat can affect material performance which affects high-speed signaling performance
29
Line Card Components
PHY Network Processor
SRAMCAM
MACFabric
Interface
SwitchFabric 1
SwitchFabric N
MediaInterface
DRAMLine Card
Pluggable or fixed media interfaces
Ethernet PHYsical Layer
• Line drivers/receivers• Encoders/decoders• Timing
Ethernet Media Access Control
• Framing• Addressing• Error handling• Flow control
Packet lookup memory
Packet buffer memory
Packet lookup, classification and forwarding functions
Interfaces to switch fabrics
30
High Speed SignalingInterfaces will use SERDES
– Short Reaches– Long Reaches
SERDES will replace parallel busing ASIC’s need new higher speed SERDES
– 6.25 Gbps today– 10 Gbps emerging
Discrete higher speed SERDES for 20 Gb/s?
PHY Network Processor
SRAMCAM
MACFabric
Interface
SwitchFabric 1
SwitchFabric N
MediaInterface
DRAMLine Card
SERDES
31
Higher Speeds Drive Switch/Router Requirements
Driving architectural requirementsMassive hardware and software scalability– >200 Gbps/slot switch fabric capacity – Support for several thousand interfaces– Multi-processor, distributed architectures
Fast packet processing at line-rate– 100 GbE is ~149 Mpps or 1 packet every 6.7 ns(10 GbE is only ~14.9 Mpps or 1 packet every 67 ns)
32
Higher Speeds Drive Density
100 Gbps Ethernet will benefit allDrives 10 GbE port density up and cost downPossible line-rate combinations– 1 x 100 GbE port– 10 x 10 GbE ports– 100 x 1 GbE ports– And even more oversubscribed port density…
The more things change the more they stay the same….
33
System Port Count Cycle
2002 2004 2006 2010*2008
100 GE 56 PortsStandard In Development
GE 336 Ports 672 Ports 1260 Ports 2240 Ports ?
10 GE 28 Ports 56 Ports 224 Ports 560 Ports 2240 Ports
34
Conclusions
Aggregation is the Killer Application– From carrier to the consumer– Not talking desktop
100G Ethernet is coming!The more things change, the more they stay the same– Lower speeds (1G and 10G)
– Higher density– Integration and cost optimization
– Port density of 100G– Higher scalability– Integration and cost optimization
35
Thank You