oracle dual port qdr infiniband adapter m3 user’s …iv oracle dual port infiniband adapter m3...

Oracle Dual Port QDR InfiniBand Adapter M3

User’s Guide

Part No.: E40987-01September 2013

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected byintellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate,broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering,disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to usin writing.If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, thefollowing notice is applicable:U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are"commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specificsupplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms setforth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in anyinherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or hardware in dangerousapplications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. OracleCorporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon aretrademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks ofSPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd.This software or hardware and documentation may provide access to or information on content, products, and services from third parties. OracleCorporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, andservices. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-partycontent, products, or services.

Copyright © 2013, Oracle et/ou ses affiliés. Tous droits réservés.Ce logiciel et la documentation qui l’accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à desrestrictions d’utilisation et de divulgation. Sauf disposition de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire,diffuser, modifier, breveter, transmettre, distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme et parquelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté àdes fins d’interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu’ellessoient exemptes d’erreurs et vous invite, le cas échéant, à lui en faire part par écrit.Si ce logiciel, ou la documentation qui l’accompagne, est concédé sous licence au Gouvernement des Etats-Unis, ou à toute entité qui délivre la licence dece logiciel ou l’utilise pour le compte du Gouvernement des Etats-Unis, la notice suivante s’applique :U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customersare "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specificsupplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms setforth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.Ce logiciel ou matériel a été développé pour un usage général dans le cadre d’applications de gestion des informations. Ce logiciel ou matériel n’est pasconçu ni n’est destiné à être utilisé dans des applications à risque, notamment dans des applications pouvant causer des dommages corporels. Si vousutilisez ce logiciel ou matériel dans le cadre d’applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, desauvegarde, de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité. Oracle Corporation et ses affiliésdéclinent toute responsabilité quant aux dommages causés par l’utilisation de ce logiciel ou matériel pour ce type d’applications.Oracle et Java sont des marques déposées d’Oracle Corporation et/ou de ses affiliés.Tout autre nom mentionné peut correspondre à des marquesappartenant à d’autres propriétaires qu’Oracle.AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques déposées d’Advanced Micro Devices. Intel et Intel Xeon sont desmarques ou des marques déposées d’Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des marques ou des marquesdéposées de SPARC International, Inc. UNIX est une marque déposée concédée sous licence par X/Open Company, Ltd.Ce logiciel ou matériel et la documentation qui l’accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits etdes services émanant de tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ouservices émanant de tiers. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûtsoccasionnés ou des dommages causés par l’accès à des contenus, produits ou services tiers, ou à leur utilisation.

PleaseRecycle

Contents

Using This Documentation v

Installing the Adapter 1

Hardware and Software Requirements 1

▼ Install the Adapter 2

Adapter Features 3

Adapter Specifications 4

InfiniBand Interface 5

PCIe Interface 6

LEDs and Ports 6

I2C Compatible Interface 8

Node GUID 9

Installing InfiniBand on Oracle Solaris OS 11

InfiniBand for Oracle Solaris OS 11

▼ Verify the Installation (Oracle Solaris) 12

▼ Update the Firmware (Oracle Solaris) 14

InfiniBand Devices (Oracle Solaris) 15

Installing InfiniBand on Oracle Linux OS 17

InfiniBand for Oracle Linux OS 17

▼ Enable Ethernet Over Oracle IB Gateway 18

▼ Verify the Installation (Oracle Linux) 18

iii

Replacing a Short Bracket With a Tall Bracket 21

▼ Remove the Short Bracket From the Adapter 21

▼ Assemble and Install a Tall Bracket 23

Glossary 25

Index 27

iv Oracle Dual Port InfiniBand Adapter M3 User’s Guide • September 2013

Using This Documentation

This guide provides an overview, installation instructions, and specifications of theOracle Dual Port QDR InfiniBand Adapter M3. The instructions in this guide aredesigned for system administrators with experience installing network hardware andsoftware.

■ “Product Notes” on page v

■ “Related Documentation” on page vi

■ “Feedback” on page vi

■ “Access to Oracle Support” on page vi

Product NotesFor late-breaking information and known issues about this product, refer to theproduct notes at:

http://docs.oracle.com/cd/E40985_01/

v


Related Documentation

FeedbackProvide feedback about this documentation at:

http://www.oracle.com/goto/docfeedback

Access to Oracle SupportOracle customers have access to electronic support through My Oracle Support. Forinformation visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trsif you are hearing impaired.

Documentation Links

All Oracle products http://docs.oracle.com

Oracle Dual Port QDR InfiniBand Adapter M3 http://docs.oracle.com/cd/E40985_01/

Oracle Integrated Lights Out Manager (ILOM) http://www.oracle.com/goto/ILOM/docs

Oracle Solaris 11 OS http://www.oracle.com/goto/Solaris11/docs

Oracle VM Server for SPARC http://www.oracle.com/goto/VM-SPARC/docs

Oracle VTS http://www.oracle.com/goto/VTS/docs

vi Oracle Dual Port InfiniBand Adapter M3 User’s Guide • September 2013

http://www.oracle.com/goto/docfeedback

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs

http://docs.oracle.com


http://www.oracle.com/goto/ILOM/docs

http://www.oracle.com/goto/Solaris11/docs

http://www.oracle.com/goto/VM-SPARC/docs

http://www.oracle.com/goto/VTS/docs

Installing the Adapter

The adapter is a dual-port 4x InfiniBand PCIe Generation 3.0 low-profile adapter.

■ “Hardware and Software Requirements” on page 1

■ “Install the Adapter” on page 2

■ “Adapter Features” on page 3

■ “Adapter Specifications” on page 4

■ “InfiniBand Interface” on page 5

■ “PCIe Interface” on page 6

■ “LEDs and Ports” on page 6

■ “I2C Compatible Interface” on page 8

■ “Node GUID” on page 9

Related Information

■ “Installing InfiniBand on Oracle Solaris OS” on page 11

■ “Installing InfiniBand on Oracle Linux OS” on page 17

Hardware and Software RequirementsFor the latest list of supported platforms and operating systems, refer to the OracleDual Port QDR InfiniBand Adapter M3 Product Notes. This document is available at:http://docs.oracle.com/cd/E40985_01/.

For Solaris OS systems, Oracle Solaris 11.1 SRU9 or newer is required.

Related Information

■ “Install the Adapter” on page 2

1


▼ Install the AdapterRefer to the installation or service manual that applies to your server for detailedinstructions for the following procedure.

Caution – Electronic components on printed circuit boards are extremely sensitiveto static electricity. Ordinary amounts of static electricity generated by your clothingor work environment can damage electronic equipment. When installing the adapterin a system, use antistatic grounding straps and antistatic mats to help preventdamage due to electrostatic discharge.

Note – Hot-swapping is not supported with Oracle Solaris 11.1 SRU9.

1. Check that the bracket on the adapter is the correct size for your system.

An alternative tall bracket is supplied with the adapter. If you need to use adifferent bracket, perform the instructions in “Replacing a Short Bracket With aTall Bracket” on page 21.

2. Power off the server, using the standard shutdown procedures described in thesystem service manual for your server.

Product documentation for Sun servers is available at:http://docs.oracle.com.

3. Remove the cover from the system to access the adapter slots and connectors.

4. Select an available PCIe x8 slot:

■ Remove the blank filler panel for that slot.

■ Or, if you are replacing an existing adapter in that slot, remove the adapter.

5. Attach an antistatic wrist strap.

6. Install the adapter into the slot, pushing the adapter’s edge connector into theconnector on the chassis.

7. Ensure that the front plate on the adapter mounts flush with the chassis panelopening.

8. If applicable, install the screw in the front plate to secure the adapter into thechassis.

9. Replace the cover on the unit.

10. Attach the 4x end of each InfiniBand cable to an IB-HCA port connector.

2 Oracle Dual Port InfiniBand Adapter M3 User’s Guide • September 2013

http://docs.oracle.com

Caution – Avoid putting unnecessary stress on the connection. Do not bend or twistthe cable near the connectors, and avoid cable bends of more than 90 degrees.

11. Ensure that the connectors are properly engaged.

12. If not already connected, connect the other end of the InfiniBand I/O cables tothe appropriate ports on the switch or switches.

The IB-HCA ports can be connected to different ports on the same switch or to aport on different switches.

13. Power on the system and allow the server to reboot.

This step completes the hardware installation.

14. Verify the installation.

See “Verify the Installation (Oracle Solaris)” on page 12 or “Verify the Installation(Oracle Linux)” on page 18.

Related Information■ “Installing InfiniBand on Oracle Solaris OS” on page 11


Adapter FeaturesThe main features of the adapter are as follows:

■ QDR active-active support

■ Delivers full QDR bandwidth on both InfiniBand ports concurrently, inactive-active configuration

■ PCI Express expansion board with an x8 edge connector compatible to the PCIExpress 3.0 specification

■ Two 4x InfiniBand ports (QSFP+) for connecting InfiniBand traffic (4x IBconnectors)

■ 4x IB port speed support: 10 Gbps, 20 Gbps, or 40 Gbps (QDR)

■ Short mounting bracket (an alternative tall bracket is supplied with the adapter)

■ IBTA v1.2.1 compliant

■ Media detect circuit, which supports external InfiniBand fiber solutions

■ RoHS compliant


FIGURE: Adapter With Short Bracket

Related Information

■ “Adapter Specifications” on page 4

Adapter SpecificationsThe adapter receives power from the PCI Express edge connector. All other requiredpower voltages are generated by on-board switch mode regulators.

Physical

SizeConnector

2.71in. x5.6 in. (68.90mm x 142.25 mm)QSFP+ InfiniBand (copper and optical)

Protocol Support

InfiniBand IBTA v1.2.1, autonegotiationSDR, DDR, and QDR:• Links: 1x/4x• Speeds per lane: 2.5 Gb/s, 5 Gb/s, and 10 Gb/s


Related Information


InfiniBand InterfaceThe adapter is compliant with the InfiniBand Architecture Specification, Release 1.2. Theadapter has two compliant 4x InfiniBand ports, 1 and 2. The adapter provides accessto these ports by means of two 4x InfiniBand QSFP+ connectors for externalInfiniBand cables. These cables must be compliant with the InfiniBand ArchitectureSpecification, Release 1.2. Connector 1 connects to Port 1 of the device, while connector2 connects to Port 2.

The adapter is embedded with a media-detect circuit, which supports externalInfiniBand fiber solutions. These external devices are connected to the InfiniBandport connectors using active media converters, such as the Oracle QSFP+ parallelshort wave transceiver (X2124a-N).

QoS 8 InfiniBand virtual lanes for each port

RDMA support All ports

PCIe PCIe 3.0 compliant: SERDES @ 8.0GT/s, 8 lanes (2.0 and 1.1compatible)

Power andEnvironmental

Voltage 12V, 3.3V

Typical power Passive Cables 7.94W

Maximum power Passive Cables 9.35W

Temperature Operational 0 to 55˚CNon-operational 0 to 70˚C

Humidity 90% relative humidity

Regulatory

Safety IEC/EN 60950-1:2006, ETSI EN 300 019-2-2

Environmental IEC 60068-2- 64, 29, 32

RoHS RoHS-R6


Related Information




PCIe InterfaceThe adapter has eight Tx/Rx pairs of SerDes, providing for a PCI Express x8 edgeconnector interface, version 3.0 compliant. The adapter can be either a master,initiating the PCI Express bus operations, or a slave responding to PCI busoperations.

Related Information




LEDs and PortsThe adapter has four LEDs located on the I/O panel. Two LEDs are assigned to each4X port.


FIGURE: I/O Panel With Ports and LEDs

The same port names and LED footprints apply when a tall bracket is installed on theadapter. To install the tall bracket, see “Replacing a Short Bracket With a Tall Bracket”on page 21.

The pair of LEDs for each port have the meanings described in the following table.

Figure Legend

1 InfiniBand Port 1 (QSFP+)

2 Yellow LED for Port 1 (Logical Link)

3 Green LED for Port 1 (Physical Link)

4 InfiniBand Port 2 (QSFP+)

5 Yellow LED for Port 2 (Logical Link)

6 Green LED for Port 2 (Physical Link)


Related Information




I2C Compatible InterfaceA three-pin header, designated with reference name J5 on the adapter, is provided asthe I2C compatible interface.

Related Information




TABLE: LED Meanings

LED Color LED Name LED State Meaning

Green Physical Link Lit The link bringup process hassuccessfully completed and thelink width, link speed, linkpolarity, and link reversal havebeen negotiated with theneighbor port on the other endof the cable.

Unlit A physical connection has notbeen established.

Yellow Logical Link(data activity)

Steady light Infiniband is discovered overthe link, but no data is beingpassed.

Blinking light Data is being passed.

Unlit A physical or logical connection(or both) has not beenestablished.


Node GUIDA label on the back of the adapter displays the 64-bit GUID. This GUID uniquelyidentifies this adapter in the server and on the IB fabric.


Installing InfiniBand on OracleSolaris OS

These topics provide an overview of installing and using the InfiniBand softwarestack for the Oracle Solaris OS.

Refer to the product notes for the adapter and your server for recent informationabout supported operating systems, firmware and software updates, and other issuesnot covered in the main product documentation.

■ “InfiniBand for Oracle Solaris OS” on page 11

■ “Verify the Installation (Oracle Solaris)” on page 12

■ “Update the Firmware (Oracle Solaris)” on page 14

■ “InfiniBand Devices (Oracle Solaris)” on page 15

Related Information


InfiniBand for Oracle Solaris OSInfiniBand is a network architecture for the large-scale interconnection of computingand I/O nodes through a high-speed switched fabric. To operate InfiniBand on anOracle server, you need an InfiniBand HCA (the adapter) and an InfiniBand softwarestack.

InfiniBand software is bundled with the Oracle Solaris OS. For Oracle Solaris 11, Thepackage containing the device driver for the adapter isdriver/infiniband/connectx.

Note – You must use driver/infiniband/ package that is available in the OracleSolaris 11 and newer releases.

11

If you don’t have the package on the system or your system is running an olderrelease of Oracle Solaris, you must use pkg install/update utility to add orupdate the package prior to using the adapter.

For details about InfiniBand software supported in Oracle Solaris OS releases, refer tothe following documents in the Solaris OS Release and Installation Collection athttp://www.oracle.com/goto/Solaris11/docs:

■ Oracle Solaris 11 What’s New

■ Oracle Solaris 11 Release Notes

■ Oracle Solaris 11 Package List

The InfiniBand software stack, consisting of the upper-layer protocols and transportframework, is included in all of the Solaris software groups described in the OracleSolaris Installation Guide.

Related Information




▼ Verify the Installation (Oracle Solaris)1. Install the adapter in the chassis.

See “Install the Adapter” on page 2.

2. Power on the server and cable the server to an operational InfiniBand switch.

Refer to the documentation for your server.

3. Ensure that the cables are connected to the adapter and switches.

4. Verify that the IB Subnet Manager is in operation on the network.

Refer to the documentation for your network hardware for more information.

5. Check that the green LED is illuminated for each port that is connected to theswitch.

If the green LED is not on, check the cable connections at the adapter and theswitch.

6. Check that the yellow LED is illuminated for each port that is connected to theswitch.



7. Verify that the IB-HCA ports are up and the driver is attached.

a. Obtain the state of the device installed.

If more than one IB-HCA device is installed in the server, a row is displayed foreach. Look for the row displaying hca:GUID where GUID is the 64-bit numberfrom the physical label on the adapter. See “Node GUID” on page 9.

The Condition column must display ok to indicate that the driver hasdiscovered the hardware and is bound to it. Refer to the cfgadm_ib(1m) manpages for details about the IB specific extensions.

b. Obtain port GUIDs for each port on the adapter.

If more than one IB-HCA device is installed in the server, a row is displayed foreach device. Look for the row displaying hca:GUID, where GUID is the 64-bitnumber from the physical label on the adapter. See “Node GUID” on page 9.

Use the port number and GUID displayed by this command for your IB-HCAdevice in the following step.

c. Verify that the IB ports and partitions are configured by the Subnet Manager.

The command displays the AP_ID column: where each row has the format ofib:Port GUID, P_Key, ipib. Match the Port GUIDs from the previouscommand with these port GUIDs. There must be one row corresponding to theport and P_Key setup by the Subnet Manager. If an entry is missing, check theSubnet Manager configuration.

# cfgadm -als "cols=ap_id:condition" hcaAp_Id Conditionhca:2C90109763F70 ok

# cfgadm -als "cols=ap_id:info" hcaAp_Id Informationhca:2C90109763F70 VID: 0x15b3, PID: 0x5a44,#ports: 0x2, port1 GUID: 0x2C90109763F71, port2 GUID:0x2C90109763F72

# dladm show-ibLINK HCAGUID PORTGUID PORT STATE PKEYSnet6 2C90300A38360 2C90300A38361 1 up17,18,33,34,800A,800B,8033,8034,FFFFnet7 2C90300A38360 2C90300A38362 2 up 800A,8033,8034,FFFF

# dladm show-part


Related Information■ “Update the Firmware (Oracle Solaris)” on page 14


▼ Update the Firmware (Oracle Solaris)To use this adapter with the Solaris OS, the minimum firmware version must be2.11.1280.

1. Display the revision level of your adapter.

Look for the revision number that appears after the Firmware revision string.If more than one IB-HCA device is displayed, look for the Node Image GUID thatmatches the GUID displayed on the physical GUID label of the adapter beinginstalled. See “Node GUID” on page 9.

If the firmware version is not at 2.11.1280 or higher, you must update thefirmware. Only update the firmware on your adapter with files specificallyapproved for the product.

2. Select and download approved firmware files from My Oracle Support at:

https://support.oracle.com/

3. Use the fwflash command to install the firmware.

4. Reboot the system to enable the new firmware.

Related Information■ “Verify the Installation (Oracle Solaris)” on page 12


LINK PKEY OVER STATE FLAGSibd1 FFFF net9 up ----ibd2 FFFF net10 up ----

# fwflash -l -c IB

# fwflash -d device-path-of-IB-adapter -f firmware-image-file


https://support.oracle.com/

InfiniBand Devices (Oracle Solaris)For details about InfiniBand software stack configurations in an Oracle Solaris OSupdate release, refer to the System Administration Guide: Devices and File Systemsdocument in the Solaris 11 System Administrator Collection athttp://www.oracle.com/goto/Solaris11/docs.

This administrator’s guide includes instructions for setting up upper layer protocolssuch as IPoIB.

When using IPoIB, verify that the broadcast group is configured by the SubnetManager in the partition where the IPoIB link will be used. Refer to thedocumentation for the IB Subnet Manager for more information.

Related Information

■ “InfiniBand for Oracle Solaris OS” on page 11





Installing InfiniBand on OracleLinux OS

These topics provide an overview of enabling and using the InfiniBand softwarestack for Oracle Linux OS.

Refer to the release notes for your server for recent information about supportedoperating systems, firmware and software updates, and other issues not covered inthe main product documentation.

■ “InfiniBand for Oracle Linux OS” on page 17

■ “Verify the Installation (Oracle Linux)” on page 18

Related Information

■ “Installing InfiniBand on Oracle Solaris OS” on page 11

InfiniBand for Oracle Linux OSInfiniBand is a network architecture for the large-scale interconnection of computingand I/O nodes through a high-speed switched fabric. To operate InfiniBand on anOracle server, you need an InfiniBand HCA (the adapter) and an InfiniBand softwarestack.

Note – An InfiniBand software stack is part of the Oracle Linux OS distribution.

Refer to your Linux vendor for software installation recommendations and support.

Related Information

■ “Verify the Installation (Oracle Linux)” on page 18

17

▼ Enable Ethernet Over Oracle IBGatewayDepending upon which version of Oracle Linux OS you are running, perform thefollowing tasks to enable the Sun Network QDR InfiniBand Gateway Switch Ehternetover IB gateway.

1. If you are using Oracle Linux 6.4 or newer, use yum to perform a groupinstallof “Infiniband Support.”

Note – You must use the default Linux kernel (UEK2-400 or newer) whenperforming this task on Oracle Linux OS 6.4 or newer.

Once this task is complete, you will have an IB stack that is usable with the IBGateway Switch.

2. If using earlier versions of Oracle Linux OS 6 or 5:

a. Upgrade the kernel to a supported UEK2-400 version with EoIB support.

b. Add the IB stack.

Related Information■ “Verify the Installation (Oracle Linux)” on page 18

▼ Verify the Installation (Oracle Linux)1. Install the adapter in the chassis.

See “Install the Adapter” on page 2.

2. Power on the server and cable it to an operational InfiniBand switch.

Refer to the documentation for your server.

Note – The InfiniBand switch automatically recognizes the adapter when it isconnected to the fabric, if the IB Subnet Manager is running on the switch or on ahost within the subnet.


3. Ensure that the cables are connected to the adapter and switches.

4. Verify that the IB Subnet Manager is running on the switch or a host within thesubnet.

Refer to the documentation for the IB Subnet Manager for more information.

5. Check that the green LED is illuminated for each port that is connected to theswitch.

If the green LED is not on, check the cable connections at the adapter and theswitch.

6. Check that the yellow LED is illuminated for each port that is connected to theswitch.

7. Verify that the IB-HCA ports are up and the driver is attached:

The output shows system diagnostic messages that have the string mlx4 in themessage (the name of the Linux driver). Included in the output is a message thatindicates whether the port is up or down.

# ibstatCA ’mlx4_0’ CA type: MT4099 Number of ports: 2 Firmware version: 2.11.1280 Hardware version: 0 Node GUID: 0x0002c90300a383c0 System image GUID: 0x0002c90300a383c3 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 1 LMC: 0 SM lid: 14 Capability mask: 0x02514868 Port GUID: 0x0002c90300a383c1 Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 3 LMC: 0 SM lid: 14 Capability mask: 0x02514868 Port GUID: 0x0002c90300a383c2 Link layer: IB

Installing InfiniBand on Oracle Linux OS 19

Related Information■ “InfiniBand for Oracle Linux OS” on page 17


Replacing a Short Bracket With a TallBracket

By default, the Oracle Dual Port QDR InfiniBand Adapter M3 from Oracle is installedon a short bracket. A tall bracket is provided.

■ “Remove the Short Bracket From the Adapter” on page 21

■ “Assemble and Install a Tall Bracket” on page 23

Related Information


▼ Remove the Short Bracket From theAdapter1. Attach an antistatis wriststrap.

2. Remove the bracket screws and washers.

Unscrew both screws from the adapter using a torque screwdriver.

21

3. Detach the bracket.

a. Grip the bracket as shown in the following figure, placing your thumb on theLED component.


b. In a rotating move toward the component side of the adapter, slide thebracket out of the connectors.

c. Gently hold your thumb on the LED component (as shown in the figure)while at the same time extracting the bracket, making sure to protect theLEDs.

d. Make a rotating move to detach the short bracket, as shown in the figure.

Related Information■ “Assemble and Install a Tall Bracket” on page 23

▼ Assemble and Install a Tall Bracket1. Place the tall bracket onto the adapter.

2. Gently fit the connectors through the bracket connector holes.

3. Ensure that the LEDs are aligned with their intended bracket holes.

Replacing a Short Bracket With a Tall Bracket 23

4. Insert a screw with a washer into each of the two holes for holding the bracketas shown in the figure.

5. Use a torque screwdriver to apply up to 2 lbs per-inch torque on each screw.

Related Information■ “Remove the Short Bracket From the Adapter” on page 21


Glossary

BBoIB Boot over InfiniBand.

IIB InfiniBand. A switched fabric communications link primarily used in

high-performance computing.

IB-HCA InfiniBand Host Channel Adapter.

IBTA InfiniBand Trade Association.

IPoIB Internet Protocol over InfiniBand.

QQDR Quad data rate. A communication signaling technique wherein data is

transmitted at four points in the clock cycle.

QSFP+ Quad small form-factor pluggable. An interconnect system for the I/O ports.QSFP supports Ethernet, Fibre Channel, InfiniBand, and SONET/SDHstandards. QSFP+ transceivers support Serial Attached SCSI, 40G Ethernet,20G/40G Infiniband, and other standards. QSFP modules increase theport-density by three to four times, compared to SFP+ modules.

25

RRoHS European Union Restriction of Hazardous Substances.

SSerDes Serializer/Deserializer. A pair of functional blocks used in high-speed

communications to compensate for limited input/output.


Index

Aadapter

features, 3installation, 2LEDs, 6power, 4specifications, 4

Bbracket

height, 3short bracket removal, 21tall bracket assembly, 23tall bracket installation, 23

Ffeatures, 3firmware update for Oracle Solaris, 14

GGUID for node, 9

Hhardware and software requirements, 1

II/O panel, 6I2C compatible interface, 8IB Subnet Manager, 19InfiniBand

interface, 5software for Oracle Solaris 10, 11

InfiniBand devices, 15installing the adapter, 2interfaces

I2C compatible, 8

InfiniBand, 5PCIe, 6

Internet Protocol Over InfiniBand (IPoIB), 15IPoIB, Oracle Solaris, 15

LLED, 6, 19

assignment, 6location, 7

Linuxsoftware installation verification, 18support, 11

Nnode GUID, 9

OOracle Solaris 10

InfiniBand software, 11software installation verification, 12

PPCIe interface, 6

Rrequirements, 1

Ssoftware and hardware requirements, 1software installation

firmware for Oracle Solaris, 14Infiniband for Oracle Solaris, 14Oracle Solaris, 12verification for Linux, 18verification for Oracle Solaris, 12

specifications, 4

27

Subnet Manager, 19


oracle dual port qdr infiniband adapter m3 user’s …iv oracle dual port infiniband adapter m3...

Documents