pluggable interfaces: passive copper, active copper, and

Pluggable Interfaces: Passive Copper, Active Copper,

and Optical DevicesAs the bandwidth and distance requirements on copper and optical links continue to steadily increase, a wide variety of advanced signal transmission technologies, on various transmission media, will be invented and developed. A key goal of the InfiniBand architecture is to encourage innovation in the market by simplifying delivery and adoption of these advanced transmission technologies, while ensuring interoperability and correct functionality of systems that use them.

To help achieve these goals, the IBTA Electromagnetic Working Group has recently expanded the definition for pluggable module interfaces. These pluggable module interfaces, described in Annex A5: Pluggable Interfaces: CATx, Copper and Optical, allow transmission technology and PHY-layer functionality to be logically and physically separated from the HCA, TCA, and switch devices that implement the higher levels in the InfiniBand layered protocol stack.

This document describes several of the many possible types of pluggable modules that may be built, and describes some of the characteristics and advantages of parallel pluggable interfaces relative to prior existing copper and optical cable interfaces.

The intent of this whitepaper is to clarify the potential uses of pluggable modules, and to expand the range of potential implementations for systems based on the InfiniBand architecture.

IBTA Electromechanical Working GroupEditor: Alan F. Benner

InfiniBandSM Trade Association March 6, 2007

1 INTRODUCTION AND OVERVIEW:

One of the explicit targets of the InfiniBandTM architecture has been to provide high performance connectivity between tightly-coupled systems. One part of that target has included providing the cabled links, at competitive costs, at high bandwidth, over long distances.

As system performance has improved, design targets have also increased for bandwidth-distance squared product (for copper links) and bandwidth-distance product (for optical links). Consequently, members of the data communications design community have developed a truly impressive variety of techniques for driving signals over copper and optical cables over longer distances at higher bitrates, with good signal integrity. A sampling of these techniques would in-clude improvements in cable materials and construction, improved connector designs with better impedance matching and lower crosstalk, equalization cir-cuitry at both transmitters and receivers, multi-level coding of various kinds, error detection codes including forward error correction codes, as well as op-tical transmission using various types of single-mode and multi-mode fiber, at a variety of wavelengths using various types of lasers and detectors and optical sub-assembly designs.

This list is by no means comprehensive, and these trends are continuing -- new and more sophisticated techniques for data transmission are being invented and put into practice on a regular basis, and will continue to be deployed as performance requirements for systems and networks continue increase.

As an architecture body, the InfiniBandSM trade association has attempted to incorporate the best available transmission technologies into the architecture, and has been successful in standardizing and propagating such features as high-performance shielded cabling and equalized data transmission. However, as the range and scope of InfiniBandTM technology deployments continue to in-crease, it will become increasingly difficult to encompass all the best available new transmission technologies into the architecture.

It is the intent of the InfiniBand architecture specification to allow for and en-courage innovation in the market. Therefore, the emphasis is on interopera-bility and minimum requirements needed to insure functionality without restricting the means of implementation, allowing suppliers to determine the optimum design for a given application.

To encourage innovation in transmission technology, and to incorporate a level of layering in the architecture which allows incorporation of improved data transmission technologies without impacting the design of chassis or ASICs using higher-level parts of the InfiniBandTM architecture, the Electromagnetic Working group has recently improved the definition for parallel pluggable inter-faces. Serial pluggable modules, such as SFP, XFP, XPAK, etc., have been


regularly used for optical transceivers, primarily to allow optical ports to be pop-ulated only as far as customers desire, and to allow for replacement or upgrade of optical components without impacting the rest of the system.

This whitepaper describes a range of potential pluggable modules that may be built using the parallel pluggable interfaces defined in Volume 2, Section 7.6 and in Annex A5, Pluggable Interfaces: CATx, Copper and Optical. This is not intended to be a complete list, but is intended to clarify some of the options available, and to highlight the advantages of moving from a fixed cabling defi-nition to one based on pluggable interfaces.

The motivations for extension of the pluggable interfaces include

• better support for InfiniBand links over twisted-pair cabling (e.g., CATx S/FTP),

• better configurability between copper cabling and optical cabling, • denser connector density than is possible with standard copper cable

plugs, and• incorporation of definitions for more modern pluggable devices than were

included in Volume 2, Release 1.2.1.A pluggable interface port is designed to be used with either copper or optical devices. Pluggable devices are designed to support transmission over cable. A pluggable device may either be part of the cable assembly, or may be detach-able from the cable assembly. Pluggable devices will typically incorporate ac-tive transceiver components in the pluggable device, but may also operate purely passively.

One of the motivations for the definition of pluggable interfaces is to allow in-corporation of advanced transmission technologies without defining and there-fore restricting the format of the data transmitted over the transmission medium. When interoperably-matched pairs of pluggable devices are used on both ends of a link, and the electrical interface requirements of a Type 1 or Type 2 port are met, the pluggable devices may use any transmission technology whatsoever. This allows innovation in transmission technology, without impact on the rest of the system.


2 EXAMPLES OF INTERCONNECTION USING PLUGGABLE INTERFACES

This section describes several cable structures that may be feasible and attrac-tive using the pluggable interface definitions. Since these are self-enclosed so-lutions as far as this specification goes, they are not part of the definitions in this specification, and are included here for informational purposes only.

Some potential pluggable device types are shown in Table 1 on page 4

Table 1 Potential Pluggable Device Types

Potential Device Types DescriptionTypical Max. Length Notes

Section 2.1 MicroGigaCN-Compatible

(4+4) differential pairs

0-10 m QSFP connector on one end, Micro-GigaCN connector on the other

Section 2.2Passive Copper

(4+4) differential pairs

0-10 m Standard IB 8b/10b-coded data for-mat

Active Copper Various 20-40m Vendor-dependent cabling and data

format

Section 2.3 & Section 2.4 Connectorized

Copper

4 differential pairs, CAT6 / Enhanced CAT6 or CAT7 (S/FTP) cable

20-30 m Examples include shielded RJ-45 connectors for CAT6 or enhanced CAT6 cable and Tera connector for CAT7 cable.

Section 2.5 Embedded optical

Various (vendor-dependent)

2m-100m No exposed optical connectors, fiber embedded inside pigtailed cable.

Section 2.6 Connectorized optical, parallel

Parallel fiber ribbon, MTP connectors

10m-100m Exposed 4x fiber connector & cable

Section 2.7 Connectorized optical, WDM

(1+1) fibers, multi-wavelength data

10 m to 2km, potential 10km

e.g., 4-wavelength multiplexed (DWDM or CWDM) data


2.1 MICROGIGACN-COMPATIBLE

Figure 1 on page 5 shows an example structure with a cable having the stan-dard InfiniBand 4x copper cable connector (MicroGigaCN) on one end, and a QSFP-compatible connector on the other end. This structure would provide in-teroperable plug-compatibility between a chassis made with QSFP sockets, and a chassis made with MicroGigaCN-compatible board connectors.

As shown in the figure, this cable assembly may be either passive or active, since the board connectors on both ends can supply active cable power.

Figure 1 Intermating cable between legacy InfiniBand copper connectors and QSFP

IB-4x (SDR, DDR, QDR)Electrical Interface

Circuit Board

Optional Electrical repeater or equalizer

Copper 4X Link

QSFP MicroGigaCN

QSFP Device

Circuit Board

Optional Electrical repeater or equalizer

Connector Shell


2.2 PASSIVE OR ACTIVE CABLE ELECTRICAL LINK USING A PLUGGABLE INTERFACE

Figure 2 on page 6 shows an potential structure with a copper cable structure using pluggable devices on both end. Such a structure would provide connec-tivity for short distances, with better density than standard InfiniBand electrical cable connectors.

The pluggable devices may be passive, or may contain active electronics to drive longer distances, or to drive signals well over less-expensive cable.

Figure 2 PASSIVE OR ACTIVE CABLE ELECTRICAL LINK USING PLUGGABLE INTERFACE


2.3 RJ45 / SHIELDED COPPER LINK USING 4X PLUGGABLE-QSFPFigure 3 on page 7 shows an example structure with an active QSFP device containing an outlet for an RJ-45 copper cable connector. This structure allows cables of various lengths to be driven from a pair of active pluggable devices.

Figure 3 on page 7 shows a link constructed using 4 pair foil twisted pair cable enclosed in an overall shield and jacket and fitted with RJ45 connectors. The cabling structure shown is available from many suppliers, and the RJ45 con-nectorization is ubiquitous throughout the data industry.

2.3.1 SHIELDED CABLING

The shielded cabling shown in Figure 3 on page 7 is constructed with 4 twisted pairs, where each pair is enhanced with a layer of aluminum / polyester foil tape shielding around each pair. The shielding gives a more consistent 100 ohm characteristic impedance, and significantly reduces crosstalk. This includes both crosstalk between pairs within the cable itself as well as crosstalk between cable assemblies and which is referred to as alien near-end crosstalk (ANEXT). Shielding is particularly important for this latter type of crosstalk, since signal processing is not able to compensate for ANEXT.

Figure 3 Copper Link in 4x pluggable QSFP modules, using shielded copper cable with RJ-45 connectors


2.3.2 RJ45 CONNECTOR

The RJ45 plug used on the cable assembly will generally to be shielded to give optimum performance. The outline of the shielded RJ45 plug, and the size of the connector outline relative to the size of the QSFP module outline, are shown in Figure 4 on page 8. The connector supports 4 differential pairs with a controlled impedance through the connector for low reflection and insertion loss. The connector can be field terminated.

2.3.3 ELECTRICAL DATA FORMAT FOR SHIELDED 4 PAIR CABLE

The InfiniBand specifications do not limit the format of data traveling over the cable assembly from a pluggable module to a matched pluggable module. This means that a variety of options are available to vendors for transporting data between modules. Some factors have to be considered, however, when de-fining the signaling format.

As shown in Figure 3 on page 7, shielded cabling provides 4 differential pairs per cable. Standard InfiniBand interfaces for 4X data transmission require 8 wire pairs, with 4 pairs allocated for each direction. The two-fold reduction in wire count allows for cheaper cables and connector solutions, but requires that, if high frequency components are to be minimized, the cables use bidirectional signaling with data traveling in both directions on a wire pair. Hybrid and echo cancellation circuitry will typically be used to ensure that received data is not corrupted or distorted at the receiver by the data being transported in the re-verse direction. Active electrical PHY signal processing circuitry or passive equalization components in the active pluggable modules may needed to re-duce NEXT caused at the connector.

Figure 4 Shielded RJ45 connector, and connector outline shown against outline of QSFP module


2.4 CAT7/TERA COPPER LINK USING 4X PLUGGABLE-QSFPFigure 5 on page 9 shows an example structure with an active QSFP device containing an outlet for a copper cable connector. This structure allows cables of various lengths to be driven from a pair of active pluggable devices.

Figure 5 on page 9 shows a link constructed using CAT 7/Class FA S/FTP (Screened, foil twisted-pair) cable, with Tera connectors. The cabling structure shown, which is available from various manufacturers, provides near-parity in price with the more common cabling structure using CAT5/CAT6 UTP (un-shielded twisted-pair) cabling with RJ-45 connectors, but provides enhanced high-speed electrical performance and somewhat smaller contact size.

2.4.1 CAT7 CABLING

As is shown in Figure 5, CAT7/ Class FA cables is constructed with 4 twisted pairs of conductors, as CAT5/CAT6 cables are, but is enhanced with a layer of aluminum/polyester foil tape shielding around each pair, and a further layer of tinned copper braid around the overall cable. These extra layers add little to the cost, but provide more consistent 100 Ω characteristic impedance, and dramat-ically reduce the crosstalk. This crosstalk reduction applies both to crosstalk between pairs within the cable, which may be equalized, and to alien near end

Figure 5 Copper Link in 4x pluggable QSFP modules, using CAT7 (S/FTP) copper cable with Tera differential connectors

IB-4x(SDR, DDR, ?QDR)Electrical Interface

Circuit Board

Electrical PHY chip

CAT7 S/FTP –copper cable

Copper S/FTP 4x Link

4-pair Tera Connector

QSFP Device

Tinned copper braid

Aluminum/ polyester foil tape

Jacket

Conductor

Insulation


crosstalk (ANEXT) and alien far end crosstalk (AFEXT) between cables, which can not be equalized away.

There are several different types of CAT7 / Class FA cables, including the fol-lowing:

• Stranded - 26 AWG - 6.0 mm (.238 inch) nominal o.d.• Solid - 22 AWG - 8.5 mm (.330 inch) nominal o.d. Electrical performance of this cable, with the Tera connectors described in Sec-tion 2.4.2 on page 11 is shown in Figure 6 on page 10. The Return Loss, Far-end Crosstalk (FEXT), Near-end Crosstalk (NEXT), and Insertion Loss (IL) are good enough to be comparable with the micro-coax cables typically used for copper InfiniBand links. The Alien Crosstalk (ANEXT & AFEXT from neigh-boring cables) is low enough to be nearly negligible in practice, in comparison to Return Loss/Echo (RL/ECHO), which is not typically the case with UTP ca-bles. Measurements are shown in a worst case orientation, with six disturber cables laid around a center victim cable. The lower right of Figure 6 shows that, under reasonable assumptions for crosstalk cancellation and residual noise, a CAT7 / Class FA cable should be able to support 8 and 16 Gbps signaling (4x SDR and DDR) and may even, with adequate signal processing, support the 32 Gbps of an IB-4x-QDR link over reasonably useful cable lengths.

Figure 6 CAT7 / Class FA Cable electrical performance

Insertion loss scaled to 35m

Shannon Capacity Simulation1,2

Superior Alien Crosstalk Suppression (includes powersum alien near and far end crosstalk

Theoretical Shannon Capacity = 40Gbps (over 4 pairs) achievable at Nyquist Freq of 625 MHz

Internal Link Parameters

Setup (m): (5-20-5 link)1

IL and PSFEXT scaled to 35m

Residual Noise Parameters

PS AFEXT Component factored in simulation

DSP Cancellation Factor for RL only:

PS NEXT: 0 dB

PS ANEXT: 0 dB

PS FEXT: 0 dB

RL/ECHO: 35 dB

1 Measurement data collected for this test run was applied as input to the Shannon Code. The PS ANEXT and PS AFEXT data is representative of a six around one alien crosstalk measurement setup. The powersum calculation for each respective pair is based on 24 disturber terms.2 Simulation assumes a Symbol Rate = 1250 Msymbols/sec and a Nyquist Freq = 625 MHz.

Data: courtesy Siemon Corp.


2.4.2 TERA CONNECTOR

Figure 7 on page 11 shows a mechanical drawing of the Tera connector shown in Figure 5. The connector supports 4 differential pairs, with controlled imped-ance through the connector for low reflection and insertion loss. As opposed to the RJ-45 connector, which was not originally designed as a multi-pair high-speed differential connector, and splits one pair of contacts around another pair at the middle of the connector, the Tera connector is designed to support high-speed differential signaling. The connector can be field-terminated, and has been built to support 1, 2, or 4 pairs, and could be extended to support more pairs.

2.4.3 ELECTRICAL DATA FORMAT FOR CAT7 CABLING

As described above in Section 2.3.3 on page 8, CATx cabling provides 4 differ-ential pairs to transport the (4+4) signal lanes in an InfiniBand 4x link, so typi-cally bidirectional signaling, with echo cancellation circuitry, will be used.

The cable and connector infrastructure described here allow use of technolo-gies from 10GBASE-T, but with enhanced distance capabilities vs. the CAT5/6 & RJ-45 connectors. Alternatively, more sophisticated signaling techniques could be used, for example to transport 16 Gbps of an IB-4x-DDR link, or 32 Gbps of an IB-4x-QDR link, over limited lengths of cable, using vendor-depen-dent techniques. This potential for supporting DDR and QDR data rates over CATx cabling is one of the strong motivators for the adoption of pluggable module interface in the system structure.

Figure 7 Tera S/FTP Cable Plug

1-, 2-, and 4-pair versions


2.5 ACTIVE EMBEDDED OPTICAL LINK USING A PLUGGABLE INTERFACE

Figure 8 on page 12 shows an potential structure with an active optical cable structure using Pluggable devices and embedded optics. Such a structure could provide connectivity with length equal to standard optical cables, without any exposed optical connectors or fibers. Also, optical transceivers would nat-urally be in matched pairs, allowing for better link margins. Another advantage of this structure is inherent eye safety, since no light would escape the cable.

Figure 8 ACTIVE EMBEDDED OPTICAL LINK USING 4X-PLUGGABLE-QSFP INTERFACE

IB-4x(SDR,DDR,?QDR)Electrical Interface

Circuit Board

QSFP Device

Lasers/drivers & PDs/amps

Circuit Board

Optical fiber cable


2.6 OPTICAL LINK USING PLUGGABLE OPTICAL TRANSCEIVER AND MULTI-FIBER CABLING

Figure 9 on page 13 shows an example structure with an active QSFP device with a receptacle for an MTP connector, which supports a (4+4)-fiber cable. This structure would typically be used for cables up to 100m long, with cable runs which may extend under raised floors or through building walls.


Circuit Board

QSFP Device


MTP/fiber ribbon - optical

Circuit Board

Optical 4x Link

MTP Receptacle

Figure 9 Multimode Multi-Fiber Cable Optical Link using 4x pluggable-QSFP


2.7 OPTICAL LINK USING PLUGGABLE WDM OPTICAL TRANSCEIVERS AND SERIAL FIBER CA-BLING

Figure 10 on page 14 shows an example structure with an active QSFP device with a receptacle for a connector for a (1+1)-fiber cable. This structure would typically be used for cables up to several kilometers long.

At the 10Gbps data rate, these devices would closely resemble an XFP trans-ceiver, with an internal SERDES capability to provide lower bit-rate (4+4)x2.5 Gbps electrical interfaces. Standard LR (1300 nm wavelength, 10 km reach) or ER (1550nm wavelength, 40 km reach) optical components could be used.In order to support 20, or 40 Gbps of data transmission per fiber at DDR and QDR rates, these devices would be likely to use wavelength-division multiplexing, with multiple laser transmitters each operating at 5 Gbps or 10Gbps.

Figure 10 WDM Optical Link using 4x pluggable-QSFP


Circuit Board

QSFP Device


Circuit Board

Optical 4x LR or ER Link


3 DENSITY COMPARISON

One of the primary advantages of pluggable module cabling, along with the ability to easily mix copper and optical cabling components, is the greater den-sity this structure affords at the bezel of the systems. Figure 11 on page 15shows a comparison of the size of the 4x cable plug compared to the I/O plate cutout required for QSFP pluggable devices. While the spacing between 4x cable plugs is unspecified, it’s clear that the 4x cable plug requires roughly 20% more linear space, and 50% more areal space, than the QSFP pluggable mod-ules do. Not shown is the possibility that multiple devices may be stacked, fur-ther increasing the linear density.

End of Document

Figure 11 Comparison of 4x cable plug and QSFP I/O plate density

25.0 Max.

4x Cable Plug

QSFP I/O Plate Cut-out

pluggable interfaces: passive copper, active copper, and

Documents