intel omni-path host fabric interface · intel® omni-path host fabric interface november 2015...

Doc. No.: H76466, Rev.: 1.0

Intel® Omni-Path Host Fabric InterfaceInstallation Guide

November 2015

Intel® Omni-Path Host Fabric InterfaceInstallation Guide November 20152 Doc. No.: H76466, Rev.: 1.0

Legal Lines and DisclaimersINFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.*Other names and brands may be claimed as the property of others.Copyright © 2015, Intel Corporation. All rights reserved.

http://www.intel.com/design/literature.htm

Intel® Omni-Path Host Fabric InterfaceNovember 2015 Installation GuideDoc. No.: H76466, Rev.: 1.0 3

Host Fabric Interface

Contents

1.0 Introduction ..............................................................................................................51.1 Intended Audience ..............................................................................................51.2 How this Guide is Organized .................................................................................51.3 Overview............................................................................................................51.4 Related Materials.................................................................................................51.5 Documentation Conventions .................................................................................61.6 License Agreements.............................................................................................71.7 Technical Support................................................................................................7

2.0 Hardware Installation Checklist.................................................................................9

3.0 Hardware Installation..............................................................................................113.1 Hardware Installation Requirements.....................................................................11

3.1.1 Hardware ..............................................................................................113.1.1.1 Form Factors ............................................................................11

3.1.2 Cabling and Switches ..............................................................................113.2 Safety with Electricity ........................................................................................113.3 Unpacking Information.......................................................................................12

3.3.1 Verify the Package Contents ....................................................................123.3.2 List of the Package Contents ....................................................................123.3.3 Unpacking the HFI..................................................................................14

3.4 Hardware Installation.........................................................................................143.4.1 Hardware Installation for HFI with PCI Express Riser ...................................14

3.4.1.1 Installation Steps......................................................................153.4.2 Hardware Installation for HFI Without a PCI Express Riser ...........................16

3.5 Cabling the HFI to the Switch..............................................................................173.6 Completing the Installation .................................................................................173.7 LED Link and Data Indicators ..............................................................................17

Figures1 Intel® Omni-Path Host Fabric Interface x8...................................................................132 Intel® Omni-Path Host Fabric Interface x16.................................................................133 PCIe Slot in a Typical Motherboard .............................................................................154 Assembled PCIe Host Fabric Interface with Riser ..........................................................16

Tables1 HFI Model and Related Platforms................................................................................11



§ §

Date Revision Description

November 2015 1.0 Document updated for Revision 1.0

September 2015 0.7 Document updated for Revision 0.7

June 2015 0.5 Initial release



1.0 IntroductionThis chapter describes the contents, intended audience, and organization of the Intel® Host Fabric Interface Installation Guide.

The Intel® Host Fabric Interface Installation Guide contains instructions for installing the HFI.

Note: This device has not been authorized as required by the rules of the Federal Communications Commission. This device is not, and may not be, offered for sale or lease, or sold or leased, until authorization is obtained.

1.1 Intended AudienceThis installation guide is intended for administrators responsible for installing the Intel® Omni-Path HFI in their Linux* cluster. Additional detailed information and instructions for administering an Intel® cluster can be found in the Intel® Omni-Path Host Software User Guide.

The Intel® Host Fabric Interface Installation Guide assumes that you are familiar with the specific hardware that you plan to use. Before installing the HFI, you should have basic knowledge of your host and target operating systems.

This document does not contain all the information you need to use basic Linux* commands or to perform all system administration tasks. For this information, see the software documentation you received with your system.

1.2 How this Guide is OrganizedThe Intel® Host Fabric Interface Installation Guide is organized into these sections:

• Chapter 1.0, “Introduction”, contains an overview of the HFI and software, describes interoperability with other products, lists all related documentation, and provides Intel contact information.

• Chapter 2.0, “Hardware Installation Checklist”, provides a high-level overview of the hardware installation procedures.

• Chapter 3.0, “Hardware Installation”, includes instructions for installing the Intel® HFI.

1.3 OverviewThe material in this documentation pertains to an OFED cluster. A cluster is defined as a collection of nodes, each attached to a fabric through the Intel® interconnect. The nodes are generally Linux*-based computers having multiple processors/cores. The Intel® HFI utilizes Intel® Omni-Path switches and cabling.

A cluster requires fabric management. Here are several options:• Use the Intel® Omni-Path Embedded Fabric Manager in one or more managed

Omni-Path switches. • Use a host-based subnet manager such as the Intel® Omni-Path Fabric Suite Fabric

Manager, as a part of the Intel® Omni-Path Fabric Suite download.

1.4 Related Materials• Intel® Omni-Path Switches GUI User Guide



• Intel® Omni-Path Fabric Software Installation Guide• Intel® Omni-Path Host Software User Guide• Intel® Omni-Path Fabric Suite FastFabric User Guide• Intel® Omni-Path Fabric Suite Fabric Manager User Guide• Intel® Omni-Path Fabric Suite FastFabric Command Line Interface Reference Guide• Intel® Omni-Path Host Software Release Notes• Intel® Omni-Path Fabric Suite Software Release Notes

1.5 Documentation ConventionsThis guide uses the following documentation conventions:

• Note: provides additional information.• Caution: indicates the presence of a hazard that has the potential of causing

damage to data or equipment.• Warning: indicates the presence of a hazard that has the potential of causing

personal injury.• Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this

guide, and links to Web sites are also shown in blue. For example:— Table 2 lists problems related to the user interface and remote agent.— See “Installation Checklist” on page 6.— For more information, visit www.intel.com.

• Text in bold font indicates user interface elements such as a menu items, buttons, check boxes, or column headings. For example:— Click the Start button, point to Programs, point to Accessories, and then

click Command Prompt. — Under Notification Options, select the Warning Alarms check box.

• Text in Courier font indicates a file name, directory path, or command line text. For example:— To return to the root directory from anywhere in the file structure:

Type cd /root and press ENTER. — Enter the following command: sh ./install.bin

• Key names and key strokes are indicated with UPPERCASE:— Press CTRL+P.— Press the UP ARROW key.

• Text in italics indicates terms, emphasis, variables, or document titles. For example:— For a complete listing of license agreements, refer to the Intel Software End

User License Agreement.— What are shortcut keys?— To enter the date type mm/dd/yyyy (where mm is the month, dd is the day,

and yyyy is the year).• Topic titles between quotation marks identify related topics either within this

manual or in the online help throughout this document.



1.6 License AgreementsRefer to the Intel Software End User License Agreement for a complete listing of all license agreements affecting this product.

1.7 Technical SupportTechnical Support for products under warranty is available during local standard working hours excluding Intel Observed Holidays. For customers with extended service, consult your plan for available hours. For Support information, contact your Intel representative.

§ §



2.0 Hardware Installation ChecklistThis section provides an overview of the hardware installation procedures. Detailed steps are found in Chapter 3.0, “Hardware Installation”.

The following steps summarize the basic hardware installation procedure:1. Check that the HFI hardware is appropriate for your platform. See Table 1.2. Check to see that you have the appropriate cables and switches, as described in

“Cabling and Switches” on page 11.3. Following the safety instructions in “Safety with Electricity” on page 11. Unpack the

HFI (“Unpacking Information” on page 12) and verify the package contents. 4. Install the HFI by following the instructions in “Hardware Installation” on page 14.5. Cable the HFI to the switch, as described in “Cabling the HFI to the Switch” on

page 17. Check that all switches are configured.6. Follow the steps in “Completing the Installation” on page 17 to finish the

installation.

§ §



3.0 Hardware InstallationThis section lists the requirements and provides instructions for installing the Intel® HFI. The components are collectively referred to as the HFI and the riser card in the remainder of this document.

3.1 Hardware Installation RequirementsThis section lists hardware and software environment requirements for installing the HFI.

3.1.1 Hardware

The Intel® Omni-Path HFI is for use with UL listed computers. The following statement is true for the HFI:

This device complies with part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) This device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operations.

Table 1 displays the HFI and supported motherboards.

If the installation of the HFI requires the use of a riser card. See Figure 3 for an illustration of a PCI Express (PCIe) slot in a typical motherboard.

The motherboard vendor is the optimal source for information about the layout and use of PCI Express-enabled expansion slots on supported motherboards.

3.1.1.1 Form Factors

The HFI is a standard PCI Express half-height, short-form factor, and can be used with either full-height or low-profile face plates.

3.1.2 Cabling and Switches

The HFI uses standard Quad Small Form-factor Pluggable (QSFP) copper cables. These cables can be passive copper or active copper. Contact Intel Sales or your authorized Reseller for a list of qualified cables.

For cabling instructions, see “Cabling the HFI to the Switch” on page 17.

3.2 Safety with ElectricityObserve these guidelines and safety precautions when working around computer hardware and electrical equipment:

• Locate the power source shutoff for the computer room or lab where you are working. This is where you will turn OFF the power in the event of an emergency or

Table 1. HFI Model and Related Platforms

Intel Model Number Platform Plugs Into

Intel® Omni-Path Host

Fabric Interface

PCI Express systemsx8 PCI Express slotx16 PCI Express slot



accident. Never assume that power has been disconnected for a circuit; always check first.

• Do not wear loose clothing. Fasten your tie or scarf, remove jewelry, and roll up your sleeves. Wear safety glasses when working under any conditions that might be hazardous to your eyes.

• Shut down and disconnect the system’s power supply from AC service before you begin work, to insure that standby power is not active. Power off all attached devices such as monitors, printers, and external components. Note that many motherboards and power supplies maintain standby power at all times. Inserting or removing components while standby is active can damage them.

• Use normal precautions to prevent electrostatic discharge, which can damage integrated circuits.

3.3 Unpacking InformationThis section provides instructions for safely unpacking and handling the Intel® Omni-Path HFI. To avoid damaging the HFI, always take normal precautions to avoid electrostatic discharge.

3.3.1 Verify the Package Contents

The HFI should arrive in good condition. Before unpacking, check for any obvious damage to the packaging. If you find any damage to the packaging or to the contents, please notify your reseller immediately.

3.3.2 List of the Package Contents

The package contents for the HFI are:• Intel® Omni-Path HFI• Additional short bracket• Additional standard bracket• Readme First

Standard PCIe risers can be used. Risers are typically supplied by your system or motherboard vendor.



Figure 1. Intel® Omni-Path Host Fabric Interface x8

Figure 2. Intel® Omni-Path Host Fabric Interface x16



3.3.3 Unpacking the HFI

Follow these steps when unpacking the HFI:1. When unpacking, ground yourself before removing the Intel® HFI from the anti-

static bag. 2. Grasping the Intel® HFI by its face plate, pull the HFI out of the anti-static bag.

Handle the HFI only by its edges or the face plate. Do not allow the HFI or any of its components to touch any metal parts.

3. After checking for visual damage, store the HFI in its anti-static bag until you are ready to install.

3.4 Hardware InstallationThis section contains hardware installation instructions for the HFI.

3.4.1 Hardware Installation for HFI with PCI Express Riser

Most installations will be in 1U and 2U chassis, which requires using a PCIe right angle riser card. This results in an installation of the HFI that is parallel to the motherboard.

A taller riser card can be used if necessary. The HFI can connect to any of the standard compatible PCI Express riser cards.



3.4.1.1 Installation Steps

To install the Intel® HFI with a PCIe riser card: 1. Shut down the power supply to the system into which you will install the Intel®

HFI.2. Take precautions to avoid electrostatic damage (ESD) to the cards by properly

grounding yourself or touching the metal chassis to discharge static electricity before handling the cards.

3. Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system.

4. Locate the PCIe slot on your motherboard. Note that the PCIe slot has two separate sections, with the smaller slot opening located towards the front (see Figure 3). These two sections correspond to the shorter and longer connector edges of the HFI and riser.

5. Determine if a blanking panel is installed in your chassis. If it is, remove it so that the InfiniBand* connector will be accessible. Refer to your system vendor instructions for how to remove the blanking panel.

6. Remove the Intel® HFI from the anti-static bag.7. Locate the face plate on the connector edge of the card.8. Connect the Intel® HFI and PCIe riser card together, forming the assembly that you

will insert into your motherboard. First, visually line up the HFI slot connector edge with the edge connector of the PCIe riser card (see Figure 3).

9. Holding the Intel® HFI by its edges, carefully insert the card slot connector into the PCIe riser card edge connector, as show in Figure 3. The result is a combined L-shaped assembly of the PCIe riser card and Intel® HFI. This assembly is what you will insert into the PCIe slot on the motherboard in the next step.

10. Turn the assembly so that the riser card connector edge is facing the PCIe slot on the motherboard, and the face plate is toward the front of the chassis.

11. Holding this assembly above the motherboard at about a 45 degree angle, slowly lower it so that the connector on the face plate clears the blanking panel opening of the chassis from the inside. Slowly align the connector edge of the riser card with the PCIe slot of the motherboard. The short section of the connector must align with the short section of the slot.

Figure 3. PCIe Slot in a Typical Motherboard



12. Insert the riser assembly into the PCIe slot, ensuring good contact. The Intel® HFI should now be parallel to the motherboard and about one inch above it (see Figure 4).

13. Secure the face plate to the chassis. The Intel® HFI has a screw hole on the side of the face plate that can be attached to the chassis with a retention screw. The securing method may vary depending on the chassis manufacturer. Refer to the system documentation for information about mounting details such as mounting holes, screws to secure the card, or other brackets.

The Intel® HFI with PCIe riser card is now installed. Next, install the cables as described in “Cabling the HFI to the Switch” on page 17. Then test your installation by powering up and verifying link status (see “Completing the Installation” on page 17).

3.4.2 Hardware Installation for HFI Without a PCI Express Riser

Installing the Intel® HFI without a PCI Express riser card requires a 3U or larger chassis.

To install the Intel® HFI without a riser card:1. Shut down the power supply to the system where you will install the Intel® HFI.2. Take precautions to avoid electrostatic discharge (ESD) damage to the cards by

properly grounding yourself or touching the metal chassis to discharge static electricity before handling the cards.

3. If you are installing the Intel® HFI into a covered system, remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system.

4. Locate the PCIe slot on your motherboard.5. Remove the Intel® HFI from the anti-static bag. Hold the card by the top horizontal

section of the bracket, and the top rear corner of the card. Be careful not to touch any of the components on the printed circuit card.

6. Without fully inserting, gently align and rest the gold fingers of the HFI on top of the motherboard’s PCIe slot.

Figure 4. Assembled PCIe Host Fabric Interface with Riser



7. Insert the card by pressing firmly and evenly on the top of the horizontal bracket and the top rear corner of the card simultaneously. The card should insert evenly into the slot. Be careful not to push, grab, or put pressure on any other part of the card, and avoid touching any of the components.

8. Secure the face plate to the chassis. The Intel® HFI has a screw hole on the side of the face plate that can be attached to the chassis with a retention screw. The securing method may vary depending on the chassis manufacturer. Refer to the system documentation for information about mounting details such as mounting holes, and screws to secure the card or other brackets.

Next, install the cables, as described in “Cabling the HFI to the Switch” on page 17. Then test your installation by powering up the system (see “Completing the Installation” on page 17).

3.5 Cabling the HFI to the SwitchFollow the recommendations of your cable vendor for cable management and proper bend radius.

To install the cable(s):1. Check that you have removed the protector plugs (if any) from the cable connector

ends. 2. Different vendor cables might have different latch mechanisms. Determine if your

cable has a spring-loaded latch mechanism.• If your cable is spring-loaded, grasp the metal shell and pull on the plastic latch to

release the cable. To insert, push and the cable snaps into place. You will hear a short “click” sound from the cable connector when it snaps in.

• If your cable latch mechanism is not spring-loaded, push on the metal case, then push the plastic latch to lock the cable in place.

3. Connect the other end of the cable to a switch.

3.6 Completing the InstallationTo complete the hardware installation:1. Complete any other installation steps for other components. 2. Replace the cover plate and back panel.3. Verify that the power cable is properly connected.4. Turn on the power supply and boot the system normally.5. Watch the LED indicator. The LEDs will flash only once, briefly, at power-up. The

LED is functional only after the Omni-Path Fabric software has been installed, the driver has been loaded, and the system is connected to a switch.

3.7 LED Link and Data IndicatorsThe LED functions are as follows:

• Link Down = Off• Link Up Initialized = On solid green• Link Up Active (No Traffic) = On solid green• Link Up: Active (Slow Packet rate <10K/S) = BLINK: 384ms On, 384ms Off• Link Up: Active (Fast Packet rate >10K/S) = BLINK: 128ms On, 128ms Off



§ §

intel omni-path host fabric interface · intel® omni-path host fabric interface november 2015...

Documents