inphi moves big data faster · 1/13/2021 · • electrical input: 8 x 26.5625 gbaud pam 4, ieee...
TRANSCRIPT
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 1Inphi Copyright
Inphi Moves Big Data Faster
Next-Gen Data Center Interconnects: The Race to 800G
Radhakrishnan Nagarajan, Ilya Lyubomirsky
Jan 13, 2021COBO Webcast
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 2Inphi Copyright
Data center interconnect evolution ➔ hierarchical to flatter architectures
• Elimination of layers
• Very high-speed interconnects between layers
• Elimination/minimization of over-subscription
• Datacenters distributed over large geographic distances
- Source: Microsoft
Interconnect-rich
Machine to machine traffic dominates
High radix connections
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 3Inphi Copyright
Why do we care about the datacenter radix?
Switch Generation Radix = 32 Radix = 64 Radix = 128
12.8T 400G 200G 100G
25.6T 800G 400G 200G
51.2T 1.6T 800G 400G
Network flattening
Optical
interconnects
-Source: Facebook
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 4Inphi Copyright
Ubiquitous 2x speed scaling
60Gbaud, QAM 16, 1λ
53Gbaud, PAM4, 4 λ’s
2x λ’s2x baud rate
Higher order
modulation format
… analog bandwidth … power, size, cost, not scalable
… complexity, SNR
400G, 4λ’s ➔ 800G, 8λ’s
QAM 256PAM 16
Engineering trade-offs
QAM 64, 90Gbaud
400GPAM 6, 90Gbaud
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 5Inphi Copyright
Technology choices in data center interconnects
IEEE 802.3
PSM4, CWDM4
NRZ
25G/λ,CWDM
PAM4
Coherent
100G to 600GDWDM
Coherent
QPSK/8QAM/16QAM
100G/200G/400GDWDM
Coherent
QPSK/8QAM
100G/200G/300GDWDM
Inside DC DCI-Campus/Edge DCI-Metro DCI-Long Haul
View at 100G
IEEE 802.3
LR4
NRZ
25G/λCWDM
IEEE 802.3
DR4, FR4
PAM4
100G/λCWDM
Coherent, OIF/IEEE/ITU-T
QPSK/8QAM/16QAM (400G ZR)
100G/200G/400G Pluggable ModuleDWDM
View at 400G
Distance (km)0.1 2.0 10 100 600 10000
IEEE 802.3
DR4, FR4
PAM4
200G/λCWDM
Coherent, OIF
QPSK/8QAM/16QAM (800G ZR)
100G/200G/400G Pluggable ModuleDWDM
View at 800G
CWDM LR
coherent
Distance
optimized,
low power
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 6Inphi Copyright
Inside data center PAM 4: 400G evolution to 800G
400G ➔ 4 λ 800G ➔ 8 λ 800G ➔ 4 λ
Host side interface 50G PAM4 100G PAM4 100G PAM4
Number of host side high speed lanes 8 8 8
Line side interface 100G PAM4 100G PAM4 200G PAM4
DSP CMOS node 7nm 7nm / 5nm 5nm
PAM4 DSP die area 1.0 2.0 (7nm) 1.X
PAM4 DSP power 1.0 2.0 (7nm) 1.4
Other optics, TIA, Driver power 1.0 2.0 1.X
Optics + electronics BW requirements ~ 40GHz ~40GHz ~65GHz
Optical laser source power 1.0 2.0 1.X
Smallest pluggable form factor QSFP-DD/OSFP QSFP-DD/OSFP QSFP-DD/OSFP
200G/λ is necessary to 1.6T modules
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 7Inphi Copyright
Inside data center: Coherent vs PAM
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 8Inphi Copyright
Pluggable module form factors for data center interconnects
QSFP-28 QSFP-DD 2
OSFP
CFP2
• QSFP-DD maintains the same width as the QSFP-28 and is backward compatible.
• OSFP is a slightly wider form factor.
• A 1RU (1.75” height) switch chassis, in a 19” (width) rack, will accommodate 32 modules of QSFP, QSFP-DD and OSFP form factors.
100G PAM 4 DCI module
400G QAM 16 DCI module
CFP2 module for size comparison
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 9Inphi Copyright
High speed optical interconnects enabled by silicon photonics
High speed Mach Zehnder modulator High speed Ge photodetector
Polarization beam splitter • Performance of key components in the integrated Silicon
photonics chip across C band.
• High speed Mach Zehnder modulator
• High speed Ge photodetector
• Low loss, high extinction ration polarization beam splitter.
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 10Inphi Copyright
Between data centers: silicon photonics based 400G ZR coherent modules
• QSFP-DD module ➔ similar form factor to DR4 and FR4 modules
• Advanced 7nm CMOS DSP, Canopus
• Advanced silicon photonics node from Bi-CMOS foundry
• Electrical Input: 8 x 26.5625 Gbaud PAM 4, IEEE 802.3bs
• Optical Output: 1 x 59.8 Gbaud QAM 16, 1 λ
• 80km-120km single span reach
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 11Inphi Copyright
Between data centers: silicon photonics based 400G ZR performance at 80km
• Polarization scrambled at 50krad/s
• < 0.5dB polarization penalty
400G switch
Multi-channel
spectrum
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 12Inphi Copyright
200G/λ: Modulation,
Equalization and FEC
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 13Inphi Copyright
Optical standards for data center interconnect
Standard ModulationBaud Rate
(Gbaud)
Wavelength
(nm)
Distance
(m)FEC
FEC NCG
(dB)
FEC Latency
(ns)
400GBASE-SR8 PAM4 26.56 850 100 RS(544,514) 6.8 100
400GBASE-FR8 PAM4 26.56 1300-LWDM 2,000 RS(544,514) 6.8 100
400GBASE-LR8 PAM4 26.56 1300-LWDM 10,000 RS(544,514) 6.8 100
400GBASE-ER8* PAM4 26.56 1300-LWDM 40,000 RS(544,514) 6.8 100
400GBASE-SR4* PAM4 53.125 850 50 RS(544,514) 6.8 100
400GBASE-DR4 PAM4 53.125 1300 500 RS(544,514) 6.8 100
400GBASE-FR4 PAM4 53.125 1300-CWDM 2,000 RS(544,514) 6.8 100
400GBASE-LR4-6 PAM4 53.125 1300-CWDM 6,000 RS(544,514) 6.8 100
400GBASE-LR4-10* PAM4 53.125 1300-CWDM 10,000 RS(544,514) 6.8 100
800GBASE** PAM4 (6) ~112-114 1300-CWDM ~1-2kmHigher Gain
FEC~9-10 < 300
* Under development
** IEEE 802.3 Beyond 400GbE Study Group starting work in Jan. 2021
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 14Inphi Copyright
Source: 800G Pluggable MSA, White Paper, “Enabling the Next Generation of Cloud & AI using 800 Gb/s Optical Modules”
▪ Transmitter analog bandwidth limitations and electrical reflections (see example below)
▪ Laser RIN noise (level dependent noise) - impacts PAM6 more than PAM4
▪ Fiber chromatic dispersion – penalty scales quadratically with baud rate
▪ Polarization mode dispersion, first order effect due to differential group delay (DGD)
▪ Receiver TIA noise, nonlinearity, and analog bandwidth limitations
Driver Laser/Modulator PD/TIA ADCDAC
S21 (dB)
Key challenges for 200G/λ PAM optical systems
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 15Inphi Copyright
PAM Rx DSP architectures for beyond 400GbE
ADC 10-30 tap FFE1 tap
DFE
FEC
Baud Rate
1/T
Analog Digital
PLL DCO TEDLF
Err Gen
LMS
MLSD
Mux
LMS
AFE
Slicer
1. High Speed ADC
enables
DSP Architectures
2. FFE, DFE, and
MLSD for stronger
EQ
3. Leverage DSP soft
information for
higher coding gain
FEC
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 16Inphi Copyright
Concatenated Codes: ADC Enables Soft Decision Inner Code
Source: Frank Kschischang, “Introduction to Forward Error Correction,” OFC Short Course, 2018
KP4
(e.g., Host FEC)
KP4
(e.g., Host FEC)Inside Module Inside Module
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 17Inphi Copyright
Low latency higher gain FEC simulation results
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 18Inphi Copyright
Baud Rate:
113Gbaud, PAM 4
90Gbaud, PAM 6
Optical System:
RX OMA = -4 dBm
Laser RIN = -145 dB/Hz
ER = 5 dB
TIA NEP = 12 pA/sqrt(Hz)
FEC Limit
Comparison of modulation formats for 200G/λ
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 19Inphi Copyright
>FEC threshold >FEC threshold
PAM4
113.3GBaud
PAM6
90.7GBaud
PAM4 vs PAM6 model with realistic component models
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 20Inphi Copyright
Tx chirp
FR4 2km
200G/λ PAM4 chromatic dispersion penalty
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 21Inphi Copyright
Conclusions
▪ Next generation 800G data center optical interconnects should be designed considering the
unique application requirements of data center operators.
▪ The choice of 200G/λ modulation format, DSP architecture, and FEC scheme requires careful
analysis on the engineering tradeoffs for analog component technology and optical channel
specifications.
▪ Optical system simulations show PAM4 200G/λ is a feasible technology for 800G-FR4/DR4,
synergistic with low latency AI, break out, and copper (DAC) applications.
▪ 800G coherent technology may be most useful in the longer reach applications such as 800G
for 40km, where coherent dispersion tolerance and lower fiber loss at 1550 nm are the key
advantages.
-
R. Nagarajan, Ilya Lyubomirsky, “Next-Gen Data Center Interconnects: The Race to 800G” 22Inphi Copyright
Inphi Moves Big Data Faster