100gbe electrical backplane / cu cabling...

100GbE Electrical Backplane / Cu Cabling100GbE Electrical Backplane / Cu Cabling Call-For-Interest

IEEE 802.3 Working Groupg pDallas, TX

November 9, 2010

Objectives for this meetingTo measure the interest in starting a study group for 100GbE Electrical Backplane and Cu Cabling

We don’t need toFully explore the problemDebate strengths and weaknesses of solutionsChoose any one solutionCreate project documentation (i e PAR 5 Criteria Project Objectives)(i.e. PAR, 5 Criteria, Project Objectives)Create a standard or specification

Anyone in the room may speak / voteAnyone in the room may speak / vote

RESPECT… give it, get it

November 9, 20102 100GbE Electrical Backplane/Cu Cable CFI IEEE 802 Plenary, Dallas, TX, Nov 2010

Agenda

PresentationsPresentations“The Need”, Howard Frazier, Broadcom.“Technical Viability”, Adam Healey, LSI.“Why Now?”, John D’Ambrosia, Force10 Networks.

Q&AStraw PollsFuture Work

3 100GbE Electrical Backplane/Cu Cable CFI IEEE 802 Plenary, Dallas, TX, Nov 2010 November 9, 2010

The Need

Presented by Howard FrazierHoward Frazier

Broadcom

IEEE 802 3 Working GroupIEEE 802.3 Working GroupDallas, TX

November 9, 2010

Historical perspective

10GBASE-KR: 10Gb/s serial backplane operationBlade servers justified the effortBlade servers justified the effortDefined a backplane channel model

During IEEE P802.3ba project10GBASE-KR: basis for 40 / 100GbE PMDs

40GBASE-KR440GBASE-CR4100GBASE-CR10

No solution for 100GbE backplane


Speed and capacities

Higher speed electrical signalingOther industry bodies exploring 20 to 28Gb/s per laneEthernet has a historical preference for x4 interfaces Note – using 4 x 25Gb/s as an example interface for this presentation

Front panel I/OsFront panel I/Os10GbE, 40GbE, 100GbE OptionsHigher density offerings at all speeds

Front panel can provide 100’s of Gigabits of I/OFront panel can provide 100 s of Gigabits of I/O


Front panel I/O driving backplane capacities

Line card illustrationsa.48 ports SFP+ @ 10GbE = 480Gb/sp @b.44 ports QSFP @ 40GbE = 1.76 Tb/sc.4 ports CFP @ 100GbE= 400 Gb/sd 32 ports CXP@ 100GbE= 3 2 Tb/sd.32 ports CXP@ 100GbE= 3.2 Tb/s

Potential backplane bandwidth pcapacities

• 8 Line Cards: 3.2 Tb/s to 25.6 Tb/s• 14 Line Cards: 5 6 Tb/s to 44 8 Tb/s

b d

• 14 Line Cards: 5.6 Tb/s to 44.8 Tb/s

ca

7

b dca

100GbE Electrical Backplane/Cu Cable CFI IEEE 802 Plenary, Dallas, TX, Nov 2010 November 9, 2010

Front panel I/O driving backplane capacities

4000

Required Traces for Varied Effective BandwidthTypical configurations for A+B and N+1 Topologies

# of signals will impact

3000

3500

TX

& R

X)

10G - Typ. A+B

10G - Typ. N+1 Max

10G - Typ. N+1 Min

25G - Typ. A+B

25G T N+1 M Based on

# of signals will impact

• Feasibility• Support target

backplane bandwidth?Achievable

i ?

2000

2500

Diff

eren

tial P

airs

(T 25G - Typ. N+1 Max

25G - Typ. N+1 Min

Based on 10 Gb/s

backplane bandwidth?• Connector density

(Pair / inch)?• Can it be built?

Capacity?

500

1000

1500

# H

igh

Spee

d D

Based on 25 Gb/s

• Cost• # of connector pins• # of PCB layers

0

0 5000 10000 15000 20000

Effective Switch Capacity (Gb/s)

Feasibility?

Effective Switch Capacity (Tb/s)

5 10 15 20

N t D h d bl k li t l f id ti

8

Note – Dashed black lines represent examples for consideration


Server I/O BW drivers

Higher system processing capability Multi-core processorsHi h d t b d t t h l iHigher speed memory, systems bus, and next gen. process technologies

Server virtualizationConsolidation of multiple logical servers in a single physical server

Converged networking and storageMultiple I/O connections converging to single connection with fabric virtualization

Clustered serversScientific, financial, oil/gas exploration, engineering workloads

Internet applicationsIPTV, Web 2.0


Server bandwidth trends

100,000

1,000,000

100 Gigabit Ethernet

10,000

Rat

e M

b/s

CoreNetworkingDoubling≈18 mos

10 Gigabit Ethernet

g

40 Gigabit Ethernet

1,000

R

ServerI/O

Gigabit Ethernet

1001995 2000 2005 2010 2015 2020

Date

Doubling≈24 mos

Source: An overview: Next Generation of Ethernet ─Nov 2007 IEEE 802 HSSG tutorial

Server I/O bandwidth Doubles ≈ every 24 mos100G Need: 2017

Next Gen blade systemsForward migration path:

10G → 40G → 100GBackplanes supporting 100G will be shipped prior to 100G Servers

Development cyclesStandards: 2-3 yearsSystem and silicon: 2-3 yearsExpect 5 years from start of 802 3 j t t fi t d t

10

shipped prior to 100G Servers Standards for new speeds needed

802.3 project to first products


x86 servers by Ethernet connection speed(2010 forecast)

12

( )

For server applications 100Gb/s solution h ld b t ti i d f 2017 t h l

Based on IDC (2010) Server Forecast and hays_01_0407 ratios of Ethernet port speed

8

10 should be cost-optimized for 2017 technology envelope.

er U

nits

4

6

ns o

f Ser

ve

2

4

Mill

ion

0

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

100M 1G 10G 40G 100G


Trends in server form factorsSource: IDC (Sep 2010)

Blade ShareExtrapolated at 8% CAGR

100

70

80

90

100

serv

ers

40

50

60

70

%

10

20

30

40

0

10

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

k k bl d

12

non-rack rack blade


Backplane Ethernet & blade server architectures

• IEEE Std 802.3™-2008 defines GbE and 10GbE operation over a modular platform backplane (1 m objective)

• 1000BASE-KX (GbE)• 10GBASE-KX4 (10 GbE, 4 x 3.125 GBd)• 10GBASE-KR (serial 10 GbE)

• Blade servers: 2nd Gen backplanes• Based on 10GBASE-KX4 architecture...• ...but satisfy 10GBASE-KR channel

requirementsrequirements• IEEE Std 802.3baTM-2010 introduced 40

Gb/s operation on backplanes: 40GBASE-KR4 ( 4 x 10.3125 GBd)

• Blade servers: 3rd Gen backplanes• Backwards compatibility needed• 100GbE must support 4 lane approach

Source: Koenen, “Channel Model Requirements for Ethernet Backplanes in Blade Servers”, May 2004.


Cu Cabling: Why 4x25Gb/s vs 10x10Gb/s? • 8 lanes/pairs versus 20 lanes/pairs

• Smaller form factor enables higher port densities. • Cost factors

- Fewer cable assembly pairs reduces cost.+20-pair assembly ~1.5 times cost of 8-pair assembly.

- Reduction in number of Tx/Rx lanes simplifies host routing. - Reduction in cable assembly pairs enables reduction in cable diameterReduction in cable assembly pairs enables reduction in cable diameter.- Fewer pairs => higher levels of integration => higher port density

• Could define plug compatible port types to enable plug-and-play for 4x10Gb/s and 4x25Gb/s.

8 lanes 8 lanes

8 differential pairs

Transmit or 4x 4x Transmit or

Tx or Rx PCB Tx or Rx

PCB

Transmit or receive function

Tx or Rx PCB

4x

Tx or Rx PCB

4x

Transmit or receive function

Channel 100GbE Electrical Backplane/Cu Cable CFI IEEE 802 Plenary, Dallas, TX, Nov 2010

14 November 9, 2010

Media reach

~0.75 m 1.5 m

Intra-rack configurations Inter-rack configurations

2.4 m max 2.4 m max

Cabinet/rack Cabinet/rack Cabinet/rack

• at least 3m addresses • 3 to 7m addresses a meaningfulmajority of configurations

3 to 7m addresses a meaningful portion of configurations


SummaryBackplane application space

Front panel capacities are challenging backplane capacitiesServer requirements

Cu cablingI t k & i t k li tiIntra-rack & inter-rack applicationsCan leverage backplane solutionsPotential lower cost and wider application usage

Timing ConsiderationsNetwork equipment backplanes and inter / intra-rack connectionsconnectionsBlade servers / backplanes & rack servers


Technical Viability

Presented by Adam Healey

LSI Corporation

IEEE 802.3 Working GroupDallas, TX

November 9, 2010

Potential enablers for more Gb/s/lane

MAC Increase coding gain (trade-off against latency)

RECONCILIATION

CGMII

100GBASE R PCS

latency)

EqualizationCrosstalk cancellation100GBASE-R PCS

FEC

PMA

Crosstalk cancellationModulation (e.g. M-PAM)

Lower loss printed circuit board traces

PMD

AN

pLower loss cablingLow noise connectorsImproved impedance control

Combine (a subset of) technologies to increase the per-lane throughput

MDI

MEDIUM


Number of lanes and signaling rates

Number of lanes

Uncodedrate Gb/s Noteslanes rate, Gb/s

10 10 State of the art, e.g. 100GBASE-CR10

5 20 Lower signaling rate per lane

4 25 Better alignment with existing physical architectures

2 50 Technology not yet available

Spectrally efficient modulation reduces signaling rate per lane

1 100 Technology not yet available

Spectrally efficient modulation reduces signaling rate per laneOpportunities to re-use the 100GBASE-R PCS4 x 25 Gb/s will be used as the basis for subsequent examples


Survey of modular platform topologiesTopology Distance, in.

Dual-star mid-plane 11 to 22p

Torus-like 11 to 22(a) Dual-star mid-plane

Dual-star backplane (switches centered) 12 to 26

Dual-star backplane ( it h t d ) 22 to 39(switches at edges)

Full-mesh 22 to 39

(b) Dual-star backplane, switches centered

Dual-star predominant in the server marketplace10GBASE-KR targeted 1 m

(c) Dual-star backplane, switches at edges

g(~39 in.)

20

Source: Richard Mellitz, Intel


Backplane channel loss

0 000.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00

Frequency, GHz

0.00

5.00

10.00s, d

B

15.00

20.00

nser

tion

loss

10GBASE-KR attenuation limit23.3 dB over 1 m of “improved

FR 4” t 5 156 GH [1]

19.1 dB over 0.78 m at 12.891 GHz [2]

25.00

30.00

I

A max Tyco (fit)

FR-4” at 5.156 GHz [1]

A_max Tyco (fit)

[1] IEEE Std 802.3TM-2008, 69B.4.2[2] Source: Tyco Electronics, 0.51 m backplane, two 0.10 m line cards, two 0.035 m ideal connectors, 6-8-6

mil differential stripline, Nelco 4000-13SI dielectric, 1 oz copper


Backplane connector evolution

19.1 dB over 0.78 m at 12.891 GHz

Impro ements in impedanceImprovements in impedance and noise control

22

Source: Tyco Electronics


Passive cable assembly topologyTP4TP1

X = Y – 2 × (H – F)

Medium

Transmit function

Receive function

TP2 TP3 TP5TP0

H

Y = X + 2 × (H – F)

H

TP2 TP3 TP5TP0

TP2 (TP3)TP1 (TP4)Mated Test Fixtures

Module Compliance Board (MCB) and Host Compliance Board (HCB)

TP2 (TP3)TP1 (TP4)

23

F


Measured bulk cable loss, 3 mFrequency, GHz

26 AWG 10.82 dB @ 12.89 GHz including ≈ 1 dB cable text fixture = 9.82 dB24 AWG ≈ 17% less loss = 8.98 dB @ 12.89 GHz

on lo

ss, d

BIn

sert

io

Source: Chris DiMinico Leoni

24

Source: Chris DiMinico, Leoni


Twinaxial bulk cable insertion loss

0 000.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00

Frequency, GHz

0.00

5.00

10.00, dB

10.8 dB at 12.891 GHz (3.6 dB/m over 3 m) [2]

15.00

20.00

nser

tion

loss

25.00

30.00

In

40GBASE CR4 1 40GBASE CR4 2 DiMi i (fit)

40GBASE-CR4 bulk cable allocation13.3 dB at 5.156 GHz (1.9 dB/m over 7 m) [1]

40GBASE-CR4 - 1 40GBASE-CR4 - 2 DiMinico (fit)

[1] Derived from IEEE Std 802.3baTM-2010 Equations 85-19 (ILfitted) and 85-34 (ILcatf) assuming that the insertion loss of connector, paddle card, and wire termination is 1.2 × sqrt( f/5.1563 ) where the units of f are GHz.

[2] Chris DiMinico (Leoni) 26AWG includes test fixture loss (approximately 1 dB at 12 89 GHz)

25

[2] Chris DiMinico (Leoni), 26AWG, includes test fixture loss (approximately 1 dB at 12.89 GHz)


Host PCB insertion loss

0 000.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00

Frequency, GHz

0.00

1.00

2.00, dB

40GBASE-KR4 PCB allocation3.5 dB at 5.156 GHz over 4 inches of “FR-4” [1]

3.00

4.00

nser

tion

loss

4.7 dB at 12.891 GHz over 4 inches of Megtron 4 [2]

5.00

6.00

In

IL PCB 4" M t 4 ( )

over 4 inches of Megtron 4 [2]

IL_PCBmax 4" Megtron 4 (max.)

[1] IEEE Std 802.3baTM-2010, 85A.4.[2] Marco Mazzini and Gary Nicholl (Cisco), “CEI-28G-VSR -Channel Distance and Loss Budget Requirements”

Formulae for maximum insertion loss per inch extracted from channel measurements, W = 4 mils Megtron 4: IL = 0 0838 f + 0 0944

26

IL = 0.0838 f + 0.0944.


Front-panel connector evolutionParameter 40GBASE-CR4 Supplier A [2] Supplier B [3]

Signaling rate, GBd 10.3125 25.0 25.0

M t d t t fi t l [1] dB 2 80 6 4 7 0Mated test fixture loss [1], dB 2.80 ~6.4 ~7.0

MCB PCB loss, dB 0.67 1.8 2.1

HCB PCB loss, dB 1.26 3.5 4.0

Extracted connector loss, dB 0.87 ~1.1 ~0.9

MDFEXT[4] ICN[5], mV 3.5 1.3 1.5

MDNEXT[4] ICN, mV 1.0 0.5 0.5MDNEXT[4] ICN, mV 1.0 0.5 0.5[1] Losses are defined at the fundamental frequency for the cited signaling rate.[2] Source: Amphenol (http://www.ieee802.org/3/ad_hoc/OIF_VSR_liaison/public/AMPHENOL%20QSFP%20data.pdf)[3] Source: Molex/Tyco Electronics (http://www.ieee802.org/3/ad_hoc/OIF_VSR_liaison/public/fromm_01_0610.pdf)[4] MDFEXT = Multi-Disturber Far-End Crosstalk, MDNEXT = Multi-Disturber Near-End Crosstalk[5] ICN = Integrated Crosstalk Noise refer to IEEE Std 802 3baTM 2010 85 10 7

Data in this table represents solutions that are compatible with the QSFP mating interface

TP2 (TP3)TP1 (TP4)[5] ICN = Integrated Crosstalk Noise, refer to IEEE Std 802.3baTM-2010 85.10.7.

p g


Loss budget examples40GBASE-CR4 signal integrity

“Next generation” signal integrity

Uncoded rate, Gb/s 10.0 25.0 25.0 25.0Li d NRZ 4 PAM 4 PAM NRZLine code NRZ 4-PAM 4-PAM NRZSignaling rate, GBd 10.3125 12.8913 12.8913 25.7813SNR for BER < 10−12, dB [1] 17.0 26.6 26.6 17.0Cable length, m 7 7 7 3Host TX PCB (4”) [2], dB 3.50 4.33 2.54 4.70TX Connector, dB 2.07 2.31 [3] 1.41 [4] 2.00Bulk cable, dB 13.30 16.42 13.68 [5] 10.82RX Connector, dB 2.07 2.31 [3] 1.41 [4] 2.00Host RX PCB (4”), dB 3.50 4.33 2.54 4.70Total insertion loss, dB 24.44 29.70 21.58 24.22[1] Assumes fixed transmitter peak-to-peak differential output voltage.[2] Losses are defined at the fundamental frequency for the cited signaling rate.[3] Derived as 2.07 × sqrt( 6.4453/5.1563 )[4] Derived as 2 00 × sqrt( 6 4453/12 8913 )

28

[4] Derived as 2.00 × sqrt( 6.4453/12.8913 )[5] Derived as (7/3) × 7.06 × 0.83 where 0.83 is the reduction in loss for 24AWG cabling relative to 26AWG


Other industry efforts

Organization Project Notes

Optical InternetworkingForum (OIF) [1]

CEI-28G-(V)SR • 19.9 to 28.05 GBaud/lane• Chip-to-chip, chip-to-module

CEI 25G LR • 19.9 to 25.8 GBaud/laneCEI-25G-LR • Backplane

Infiniband Trade Association (IBTA) EDR • 25.78125 GBaud/lane

• Passive copper, active cables( ) pp ,

INCITS T11 Fibre Channel 32GFC• 28.05 GBaud (single lane)• Chip-to-module, multimode and

single-mode fiber

Broader industry movement towards higher throughput per lane

g[1] Refer to the proceedings of the IEEE 802.3 Ethernet Working Group OIF CEI-28G-VSR liaison response ad hoc

(http://www.ieee802.org/3/ad_hoc/OIF_VSR_liaison/index.html).

y g g p p


SummaryRich selection of technologies for a Study Group to exploreRepresentative of a broader industry movement


Why Now?Why Now?

Presented byJohn D’Ambrosia, Force10 Networks,

IEEE 802.3 Working GroupDallas, Tx

November 9 2010November 9, 2010

Why Now?There is no 100GbE backplane solution, but…..

Networking equipment - faceplate capacities are challenging current backplane solutionscurrent backplane solutions

o 10G I/O (SFP+, 10GBASE-T) densities are growingo Introduction of 40GbE / 100GbE & related MSAs

Blade Serverso 40GbE Backplanes need to support 100GbE

• 40GbE backwards compatibility needs40GbE backwards compatibility needs• 100GbE backplane design requirements

100GbE Cu Interfaces narrower than 100GBASE-CR10May leverage a 100GbE backplane solutionCould enable multiple benefits – backwards compatibility with 40GbE, higher port density, and lower cost


SupportersRick Rabinovich, Alcatel-Lucent

Steve Trowbridge Alcatel Lucent

Chris DiMinico, MC Communications/LEONI

O S l M ll

Chris Cole, Finisar

John D’Ambrosia Force10 NetworksSteve Trowbridge, Alcatel-Lucent

Mike Li, Altera

Shawn Searles, AMD

Brian Kirk, Amphenol TCS

Greg McSorley , Amphenol

Oren Sela, Mellanox

Howard Baumer, Mobius

Galen Fromm, Molex

Jay Neer, Molex

Paul Vanderlaan, Nexans

John D Ambrosia, Force10 Networks

Kapil Shrikhande, Force10 Networks

Uri Cummings, Fulcrum Microsystems

Ryan Latchman, Gennum

Francois Tremblay, Gennum

Brad Booth, Applied Micro

Matt Brown, Applied Micro

Rita Horner, Avago

Brian Misek, Avago

Charles Moore Avago

Fanny Minarsky, Octoscope

Shimon Muller, Oracle

Ron Nordin, Panduit

Pat Diamond, Semtech

George Eaton Semtech

Dave Koenen, HP

Steve Carlson, High Speed Design

Orlindo Savi, Hitachi

Hidehiro Toyoda, Hitachi

Jeff Lynch IBMCharles Moore, Avago

Jeff Slavick, Avago

Yakov Belopolsky, Bel Stewart

Paul Kish, Belden

Wael Diab, Broadcom

George Eaton, Semtech

Ed Cady, Siemon Company

Valerie Maguire, Siemon Company

George Zimmerman, Solarflare

Sanjay Kasturia, Teranetics

Jeff Lynch, IBM

Pravin Patel, IBM

Andy Moorwood, Infinera

Andre Szczepanek, Inphi

Dave Chalupsky, Intel

Howard Frazier, Broadcom

Ali Ghiasi, Broadcom

Arthur Marris, Cadence

Hecham Elkhatib, Cinch

Beth Donnay, Cisco

Karl Muth, Texas Instruments

Iain Robertson, Texas Instruments

Mike Fogg, Tyco Electronics

Nathan Tracy, Tyco Electronics

Jeff Lapak, UNH-IOL

Ilango Ganga, Intel

Rich Mellitz, Intel

Thananya Baldwin, IXIA

Jerry Pepper, IXIA

Jeff Maki, Juniper

Joel Goergen, Cisco

Mark Nowell, Cisco

Bhavesh A. Patel, Dell

Ghani Abbas, Ericsson

Vitt l B l b i FCI

p ,

Frank Chang, Vitesse

Ziad Hatab, Vitesse

George Noh, Vitesse

Atul Sharma, Volex

T k hi Ni hi Y i hi

p

David Ofelt, Juniper

Alan Flatman, LAN Technologies, UK

Mike Bennett, LBNL

Adam Healey, LSI

C th Li LSI

33

Vittal Balasubramanian, FCI


Takeshi Nishimura, Yamaichi

Marek Hajduczenia, ZTE

Cathy Liu, LSI

Marc Dupuis, Madison Cable

Q&AQ&A

S P llStraw Polls

Call-For-Interest

Should a Study Group be formed for “100GbE Electrical Backplanes and Cu Cable Interface”?

Y: 95 N: 0 A: 8


ParticipationI would participate in the “100GbE Electrical Backplane and Cu Cable Interface” Study Group in IEEE 802.3.

Tally: 64

My company would support participation in the “100GbE Electrical Backplane and Cu Cable100GbE Electrical Backplane and Cu Cable Interface” Study Group in IEEE 802.3

Tally: 43y


Future WorkAsk 802.3 to form 100GbE Electrical Backplane and Cu Cable SG on ThursdayBackplane and Cu Cable SG on Thursday

If approvedIf approved802 EC informed of 100GbE Electrical Backplane and Cu Cable Interface SG on FridayEmail reflector / web page to be createdEmail reflector / web page to be createdFirst 100GbE BP / Cu SG meeting, week of January 12 2011 IEEE 802.3 Interim.


Thank You!Thank You!


ContributorsHoward Frazier, BroadcomElizabeth Donnay, CiscoDave Chalupsky IntelDave Chalupsky, IntelJohn D’Ambrosia, Force10 NetworksAlan Flatman, LAN Technologies, UKJeff Lynch IBMJeff Lynch, IBMPravin Patel, IBMIlango Ganga, IntelRich Mellitz, IntelRich Mellitz, IntelAdam Healey, LSIMarc Dupuis, Madison Cable Chris DiMinico, MC Communications/LEONIC s co, C Co u cat o s/ OJay Neer, MolexMike Fogg, Tyco ElectronicsNathan Tracy, Tyco Electronics

40

y, y


100gbe electrical backplane / cu cabling...

Documents