data serialization, transmission and deserialization at 3...
TRANSCRIPT
Data Serialization, Transmission and Deserialization at 3.2 Gbps and Beyond
M.Matveev
Rice UniversityHouston, TX 77005 USA
April 3, 2006
4th CMS Workshop on Detectors and Electronics for the SLHC. April 3-4, 2006. Perugia, Italy
Outline
• Architectures of serializers (SER) anddeserializers (DES)
• Our experience at Rice University withSERDES @ 1.6Gbps and 3.2Gbps
• Programmable and 10Gbps SERDES
SERDES Architectures
• Discrete SERDES Low- and Mid- Rate SERDES (F < 160MHz)
- Parallel Clock SERDES- Embedded Clock (Start-Stop) Bits SERDES- SERDES with 8B/10B encoding
A detailed overview of these architectures is available in the articleDave Lewis. SerDes Architectures and Applicationshttp://www.national.com/appinfo/lvds/files/designcon2004_serdes.pdf#page=2
High Rate SERDES (F = 622/644MHz)
• Programmable SERDES FPGA (Xilinx, Altera, Lattice Semiconductor)
Parallel Clock SERDES
1
7
••MUX
7:117 6
LVDS Data Streams
14
••8
21
••15
MUX7:1
MUX7:1
PLL
•• 5 4 3 2• •
•• 14 13 12 11 810 9• •
• • 21 20 19 18 151617• •
LVDS Clock
• 21:3, 28:4, 48:8 multiplexing• National Semiconductor DS90CR2xx and DS90CR4xx Channel Links• Up to 133MHz clock • Latency (SER + DES) ~4 clock periods
Embedded Clock Bits SERDES
MUX
1
2 LVDS Data Stream3
16
•••
C0 C1
C0 C1 1 2 ••• C0 C116•••
• Two clock bits are embedded into the data stream every cycle, framingthe start and end of each serialized word
• Automatic receiver lock to random data • Several 10-, 16-, 18-bit devices available from National Semiconductor(DS92LVxx and SCAN 92xx families), Maxim MAX9205/9206/9207/9208
• Up to 80MHz parallel data rate• Latency (SER + DES) ~3 clock periods
8B/10B SERDES
8B/10Bencoder
Parallel to
Serial
ClockMultiplier
Serializer
TxCLK
Serial to
Parallel
10B/8Bdecoder
and clock recovery
Deserializer
RxCLK
Media
• 8B/10B encoding/decoding allows to maintain a proper number of “0” and “1” in the data stream for DC balance and:
• provide a reliable clock recovery• determine symbol and packet boundaries• detect transmission errors
• Used by Gigabit and 10G Ethernet, Fibre Channel, ATM, Infiniband transmission interfaces
Texas Instruments TLK 8B/10B SERDES
TLK1501
TLK2501
TLK3101125 156.25
75 125
30 75
F, MHz40 80 120 160
Assumptions and Requirements
Assumptions on Serial Data Links• Typical distance of optical links is < 100 m at the LHC experiments• Typical frequencies of parallel data are 40/80MHz at the LHC and will
likely increase to 160MHz at the SLHC• In most cases the source and target of data transmission links are the FPGA
Requirements • Low cost• Small size• Low power consumption• Radiation tolerance in many cases
Preferences • Low latency• Easy and quick synchronization• Simple control interface• Built-in Self Test (BIST) capability
Why 3.2 Gbps?
Serialization/Deserialization at 1.6 Gbps and below is (generally)straightforward
• 1.6 Gbps = 80MHz x (16 + 4) bits = 40MHz x (32 + 8) bits• Commercial and custom components are available (Texas Instruments
TLK1501/TLK2501, CERN GOL, Agilent HDMP-1022/1024, 1032/1034, Cypress HOTLink, National Semiconductor DS92LV16…)
• Widely used at the LHC (CMS EMU Trigger, LHCb muon trigger…)
Serialization/Deserialization at 3.2 Gbps is more challenging • 3.2 Gbps = 160MHz x (16 + 4) bits • Components need to be evaluated (most commercial parts are targeted
to Gigabit Ethernet, SONET OC-48, Fibre Channel applications whichutilize reference clocks of 106MHz, 125MHz, 155.52MHz, 156.25MHz)
CSC EMU/Trigger Electronics at CMS
80MhzLVDS
80Mhz GTLP80Mhz GTLP
40MhzLVDS
1.6Gbps
Cathode StripChambers(CSC)
1.6 Gbps CSC Trigger Optical Links
MPC
MPC
SP
TITLK2501
FinisarFTRJ8519
16-bit @ 80Mhzfrom FPGA
Muon Port Card
XilinxXCV
FPGA
FinisarFTRJ8519
TITLK2501
16-bit @ 80Mhzto FPGA
Sector Processor
XilinxXC2VFPGA
MPC
MPC
MPC
x 12180 optical links
• 50/125 um multimode fiber• 100 m long
• 2x5 SFF optical transceiver• 1.6 Gbps simplex transmission
• 16-bit SERDES• 8B/10B encoding• PRBS capability
Comparison of suitable SERDES Devices
85~10025Price, $ (small quantity)
Yes
64-pin VQFP
0.45 W
2.5
PRBS 27-1
Simple
3 consecutive IDLE cycles *
<45 ns @ 156MHz
Yes
125 - 156.25
16 LVTTL
Texas Instruments TLK3101
??Synchronization method
195-pin BGA256-pin BGAPackage
??
2.1 W
2.5
256-byte “counter”
Complex
< 63 ns @ 156MHz
Yes
30 – 39.6; 60-78.125; 120-156.25; 240-312.5
8 LVTTL*4 channels
Vitesse VSC7226-01
ModestControl interface
??
1.2 W
1.8
No
> 100 ns @ 160MHz
No
78-84.4;155-169
16 LVDS/LVPECL
Silicon Laboratories Si5100
Built-in Self Test
Latency (SER+DES)
Transmitter Reference Clock Frequency, MHz
Availability
Power consumption, (typ)
Power supply, V
8B/10B Encoder/Decoder
Number of parallel I/O
* As defined in IEEE 802.3 standard
SFP Plugguble Optical Transceivers
20-pin connector CageOptical transceiver
• Multi Source Agreement (MSA) compliant Small Form Factor Pluggable (SFP) footprint
• Up to 4.25 Gbps bidirectional links• 850 nm oxide VCSEL laser transmitter• Duplex LC connector• Up to 150 m on 50/125 um MMF @ 4.25 Gbps • Metal enclosure for lower EMI• 20-pin surface mounted connector• Several vendors, typical price < $100 per module
3.2 Gbps Optical Link
TITLK3101
TITLK3101
FinisarFTRJ8524
16-bit @ 160Mhzto/from FPGA
16-bit @ 160Mhzto/from FPGA
FinisarFTRJ8524
• 50/125 um multimode fibers
• 20-pin SFF Pluggable optical transceiver• Up to 4.25Gbps duplex transmission
• 16-bit SERDES• 8B/10B encoding• PRBS capability
~$80 ~$25 ~$80
Xilinx XC2V250
FPGA
Xilinx XC2V250
FPGA
Evaluation Board
233 x 160 mm
FTRJ8524
FTRJ8524
TLK3101
TLK3101
XilinxXC2V250
FPGA
TTCrqMezzanine
board
VME
Link Control Interface
XC2V250 FPGA
FIFO_A1
FIFO_B1
FIFO_A2
FIFO_B2
CONTROL
TXDTLK3101
TLK3101
RXD, RX_ER, RX_DV
TXD
RXD, RX_ER, RX_DV
RX_CLK
RX_CLK
CSR
TX_CLK
TX_CLK
LINK 116
18
LINK 216
18
• 160Mhz from QPLL on TTCrq mezzanine• On-board oscillator• External source
Optical Link
Top view
Bottom view
Total PCB space required for the link (SERDES + optical module): ~16 cm2 (2.5 sq in)
Test Configurations
TRRC
TRRCLink 1
Link 2
Board 1
TRRC
TRRCLink 1
Link 2
Board 1
TRRC
TRRCLink 1
Link 2100m 5.0m
100m
Board 1
5.0m
TRRC
TRRC
Link 2
Link 1
Board 2
100m100m100m
100m1 2
3
Test Configuration 3
Results of Measurements
< 10-14 (PRBS), <10-9 (Data)**
155.52;161.132
Epson EG-2101-CA oscillator(peak-to-peak jitter ~ 25 ps)
~ 10-9 (PRBS)40…160Xilinx XC2V250 Digital Clock Manager (peak-to-peak jitter ~ 250 ps)
~ 10-10 (PRBS)160.3147Cardinal Components CPPHV7-A7BC programmable oscillator (jitter ~ 80 ps)
~10-14 (PRBS)*, <10-9 (Data)**
160.3147QPLL ASIC on TTCrq mezzanine board (peak-to-peak jitter ~ 50 ps)
Bit Error Rate (BER)
Frequency, MHz
Clock Source
* 3 errors observed, ~2.3x1014 bits transmitted ** Random patterns transmitted from FIFO_A to FIFO_B
Power Supply and Grounding
• TLK3101 device is powered from Vdd=2.5V (2.3V < Vdd < 2.7V)Optoboard has an adjustable voltage regulator for +2.5V powerDoes the Bit Error Rate depend on power supply voltage? The error free PRBS data transmission over 100 m optical fibers was observed for 2.42V < Vdd < 2.65V
• The TLK3101 is housed in 64-pin VQFP PowerPAD package, which has an exposed die pad on the bottom of the device. It is important to solder this pad to thermal land connected to PCB ground for better thermal performance and device grounding (see TLK3101 specification).
125
80.00
190
125.00
230
155.52
240235Measured supply current for the TLK3101 @ Vdd = 2.5V, mA
161.1328160.3147Reference clock frequency, MHz
• Total power consumption for the TLK3101 + FTRJ8524 @ 160MHz is235mA x 2.5V + 150mA x 3.3V = 1.1W
Conditions: PRBS pattern transmitted and received
Latency MeasurementsFrom the TLK3101 specification:
So the total link (transmitter + receiver) latency may vary 110..145 bit times, or34..45 nanoseconds @ 160Mhz
Our measurements conducted at room temperature and 2.5V power for six TLK3101devices indicated that the total link latency is between 40 and 43 ns @ 160Mhz.
While the exact value is different at each link initialization in increments of the serialbit clock (310 ps), it has never varied more than the 3 nanoseconds total.
Radiation Tolerance
• Our 1.6Gbps link based on TLK2501 (same family as TLK3101)and Finisar FTRJ-8519 (850 nm VCSEL laser transmitter) was tested under irradiation, see report at the LECC2001http://lhc-electronics-workshop.web.cern.ch/LHC-electronics-workshop/2001/
posters/matveev2.pdf
Two TLK2501 devices were exposed up to ~270 kRad TID and produced 12 and 19 data errors respectively while transmitting the RRBS data through 100 m of fiber at 80Mhz. Similar results were reported inLHCb-2003-008 Technical Note.
While no errors were observed during the exposure of two Finisar FTRJ-8519 optical modules, both devices failed permanently at ~70 kRad.
Serial Transmission over Copper Cable
SFP solutions available from Molex
Cable directly terminated to module
HSSDC2 Copper SFP moduleand cable assembly• Industry standardHSSDC2 plug interface
• Passive or active transceiver
Optoboard with Molex 73929 Copper Cable
Molex 73929-0003 SFP Copper Patch Cable• Amphenol Spectra Strip Skewclear Cable• 5 m long, passive• 2 shielded parallel pairs• Skew (within pair) < 7.14 ps/m• Attenuation 0.57dB/m @ 1562.5 MHz (3.12Gbps)
Error free transmission (PRBS patterns @ 3.2Gbps) within 48 hours. BER < 10-14
Eye patters for 7 m and 17 m Skewclear cables (PRBS @ 3.1875Gbps) are available at http://www.t11.org/ftp/t11/pub/fc/10gfc/01-283v0.pdf. Look good.
10 Gigabit Industry Standards
SONET/SDH
Ethernet
Fibre Channel
1 2 3 4 5 6 7 8 9 10 11 Gbps
164.35 or 657.4210.5187510 Gigabit Fibre Channel155.52 or 622.089.9532 10GBase-W161.13 or 644.5310.312510GBase-RIEEE 802.3ae
10 Gigabit Ethernet
167.33 or 669.3310.709Optical Transport Network (OTN) G.709
155.52 or 622.089.9532Synchronous Optical Network SONET OC-192 (US, Canada) / Synchronous Digital Hierarchy SDH STM-64 (rest of the world)
Reference Clock Frequency, MHz
Serial Bit Rate, Gbps
Standard
10 Gbps SERDES Devices
??
~150
301-pin BGA
1.2 W
1.8/3.3
No
Modest
??
No
155/161/166/167; 622/644/666/669
16-bit LVDS
Broadcom BCM8152 SERDES
SimpleModestModestControl Interface
68-pin QFN244-pin BGA255-pin BGAPackage
~80>100175..210Price, $ (small quantity)
??
1.1 W
1.2&1.8/2.5/3.3
Yes
??
No
155/161;622/644
16-bit LVDS
AMCC S19235/S19237
SERDES
??
1.15 W
1.8/3.3
Yes
??
No
622-644
16-bit LVDS
Vitesse VSC8479 SERDES
27-1 PRBSBuilt-in Self Test
??Latency (SER+DES)
155/161; 622/644
Transmitter Reference Clock Frequency, MHz
YesAvailability (lead time)
1.15/1.5 WPower consumption, typ
3.3Power supply, V
NoEncoder/Decoder
16-bit LVDSNumber of parallel I/O
Maxim MAX3952 SERMAX3953 DES
Programmable SERDES
• RocketIO Transceivers available in Xilinx Virtex-2 Pro & Virtex-4- Full-Duplex SERDES
50Mhz - 156.25 MHz (600 - 3.125 Gbps) for Virtex-2 Pro 62.5Mhz - 425 MHz (2.488 - 6.25 Gbps) for selected Virtex-2 Pro X 106Mhz - 644MHz (622 - 10.3125 Gbps) for Virtex-4
- Up to 24 channels- 8-, 16-, 32-bit selectable FPGA interface- 8B/10B encoder and decoder with bypassing option on each channel- 50Ω/75Ω on-chip selectable transmit and receive terminators
• FPGA with embedded SERDES are available from other vendors as well- Altera Stratix II GX (6.375 Gbps)- Lattice Semiconductor ORLI10G (10Gbps/12.5Gbps)
• In 2003 Xilinx and Lattice announced a 10Gbps families of SERDES devices(RocketPHY and XPIO110GXS), but later both have been discontinued
Latency of SERDES in Xilinx FPGA
• Virtex-2 Pro Transmit latency (from TXDATA to TxP/TxN pins)
- 14..17 clock periods (with CRC)- 8..11 clock periods (without CRC)
Receive latency (delay from RxP/RxN to RXDATA pins)- 25..42 clock periods
• Virtex-4Transmit latency (from TXDATA to TxP/TxN pins) - min 6 clock periods (without 8B/10B encoding)- min 8 clock periods (with 8B/10B encoding)Receive latency (delay from RxP/RxN to RXDATA pins)- min 8 clock periods (without 8B/10B decoding)- min 9 clock periods (with 8B/10B decoding)
Reference Clock Requirements
2.5-3.125Gbps: <40ps1.06-2.5Gbps: <50ps< 1.06Gbps: <120ps
45..55
-100…+100
50 - 156.25
Xilinx Virtex-2 Pro
RocketIO
30..7040..6045..55Reference Clock duty cycle, %
-100…+100-100…+100-350…+350Reference Clock frequency tolerance, ppm
> 6.25Gbps: ?? UI;2.5-6.25Gbps: ?? UI;< 2.5Gbps: ?? UI
106 - 644
XilinxVirtex-4
RocketIO
??< 40 psReference Clock total jitter, peak-to-peak
155/161; 622/644
125 - 156.25Reference Clock frequency range, MHz
Maxim MAX3952/MAX3953
TI TLK3101
Examples of Oscillators for SERDES
55..215; Arrow40..50; Digikey30..40; DigikeyPrice, $ and availability
155 - 85053.125 - 50062.5 – 400Output frequency, MHz
3.3 +/- 0.33; 5.0 +/- 0.5
2.5 +/- 0.1253.3 +/- 0.15Power supply, V
-50 .. +50;-50 .. +50; -100 .. +100
-50 .. +50; -100 .. +100
Frequency tolerance, ppm
45 .. 5545 .. 5545 .. 55Output clock duty cycle, %
Recommended by Xilinx for Virtex-4 RocketIO
Used on evaluation board
Comments
< 5 ps25 ps (typ)25 ps (typ)Output clock total jitter, peak-to-peak
LVPECL
Epson 2101CA
ECL, PECL, LVPECL
LVPECL or LVDS
Output level
Vectron VS500Epson 2121CA
10 Gbps XFP Optical Transceiver
• XFP Multi Source Agreement (MSA), see http://www.xfpmsa.org 10+ vendors (Finisar, Intel, Picolight, Infineon, Tyco…)
• Protocol Agnostic (“any application, any rate”: OC-192, G.709, 10G Fibre Channel, 10G Ethernet)
• 850 nm laser (300 m), 1310 nm (up to 10 km)
850 nm VCSEL XFP & SFP Optical Modules
LC typeOptical Connector
< 82 m (500MHz*km bandwidth)
< 300 m (2000MHz*km bandwidth)
< 150 m (4.25 Gbps; 500MHz*km bandwidth)
< 270 m (4.25 Gbps; 2000MHz*km bandwidth)
Optical Link Distances on 50/125 um MMF
30-pin20-pin Electrical Connector78 x 18 x 8.556 x 13 x 8Dimensions, mm
+1.8; +3.3; +5.0+3.3Power supply, V
~400< 100Average Cost, $< 2.0< 0.6Power Dissipation, W
9.95 – 10.75Up to 4.2Bit Rate, Gbps
XFPSFPParameter
10 Gbps Over Copper Cable
• XFP form factor• Data transmission up to 15..30 m • Differential 100 Ohm twin-ax or 50 Ohm low loss coax cables
Example of the 10Gbps XFP copper transceiver from Vitesse
R&D on Programmable SERDES
Virtex-4XC4VFX20-11X FPGA
RocketIO
RocketIO
Optical XFP Module
• 30-pin XFP Pluggable transceiver
RocketIO
RocketIO
Virtex-4XC4VFX20-11X FPGA
Optical XFP Module
RocketIO
RocketIO
Virtex-4XC4VFX20-11X FPGA
Copper XFP Module
RocketIO
RocketIO
Virtex-4XC4VFX20-11X FPGA
Copper XFP Module
• Spectra Strip Skewclear Cable• HSSDC2 Connector
• Commercial or custom• With or without equalization
Conclusion
Based on our experience with three identical evaluation optoboards (6 links total) in laboratory conditions at room temperature we may conclude:
• The link built on TI TLK3101 SERDES can provide a reliable data transmission at 160Mhz over up to 100 m long multimode optical fiber(with Finisar FTRJ-8524 optical transceiver) and over 5 m of Spectra Strip Skewclear copper cable- low jitter reference clock for the transmitter is required- Virtex-2 FPGA works well at 160 MHz as a data source and target
• The board can be used for evaluation of several pluggable optical and copper modules and TI multigigabit transceivers TLK1501/TLK2501/TLK3101 with various clock options as well as for pre-selection of these devices
• We plan to continue R&D on evaluation of:- Programmable SERDES in Xilinx FPGA- 3.2Gbps…10Gbps components (SERDES, optical and copper transceivers) - High speed parallel and serial backplane technologies (GTLP, LVDS)