INFINIBAND

Presented By- Shekhar Kumar 074, CS-’B’Guided By- Latha R. Nair

Contents

IntroductionI/O Architectures- Fabric vs BusTechnical OverviewInfiniBand LayersInfiniBand ArchitectureAdvantages & DrawbacksMarketsConclusionReferences

Introduction

A switch-based serial I/O interconnect architecture

Operates at a base speed of 2.5 Gb/s or 10 Gb/s in each direction (per port)

Provides both QoS (Quality of Service) and RAS (Reliability, Availability and Serviceability)

Used in high performance computing and data centres

I/O Architectures- Fabric vs Bus

Feature Fabric Bus

Topology Switched Shared Bus

Pin Count Low High

Number Of End Points

Many Few

Max Signal Length KMs Inches

Reliability Yes No

Scalable Yes No

Fault Tolerant Yes No

I/O Architectures Contd…Shared Bus Architecture

CPU

System Controller

I/O Controller

I/O Controller

I/O Controller

I/O Controller

I/O Controller

PCI to PCI Bridge

PCI to PCI Bridge

System Memory

I/O Architectures Contd…Switched Fabric Architecture

Technical OverviewSwitch-based point-to-point

interconnect architectureEach individual link is based on a four-

wire 2.5Gb/s bidirectional connection.Architecture defines a layered

hardware protocol (Physical, Link, Network, Transport Layers) as well as a software layer

Each link can support multiple transport services for reliability

InfiniBand Feature Set

Layered Protocol- Physical, Link, Network, Transport, Upper Layers

Packet based communicationQuality Of ServiceThree link speeds1. 1X- 2.5 Gb/s, 4 Wire2. 4X- 10 Gb/s, 16 Wire3. 12X- 30 Gb/s, 48 WirePCB, Copper and Fiber cable

interconnectMulticast and Unicast support

InfiniBand Layers

InfiniBand Layers Contd…1) Physical LayerDefines both electrical and

mechanical characteristics for the system

Includes cables and receptables for fibre and copper media, backplane connectors

Defines three link speeds, 1X,4X,12X

Each individual link is a 4-wire differential connection that provides a full duplex connection at 2.5 Gb/s

InfiniBand Layers Contd…2) Link LayerEncompasses packet layout, point-to-point

operations, and switching within a local subnet

Packets- Two types: Management and Data packets

1. Management Packets- Used for link configuration and maintenance

2. Data Packets- Carry up to 4k bytes of a transaction payload

Switching- Devices within a subnet have a 16-bit Local ID assigned by the subnet manager. This LID is used for addressing.

InfiniBand Layers Contd…

Link LayerQoS- supported through virtual lanes(VL) VLs are separate logical communication

links which share a single physical linkData Integrity Two CRCs per packet, Variant CRC and

Invariant CRC16-bit VCRC includes all fields in the

packet and is recalculated at each hop32-bit ICRC covers only the fields that do

not change from hop to hop

InfiniBand Layers Contd…

Network LayerHandles routing of packets from one subnet

to another(within a subnet a network layer is not required)

Packets contain a global route header(GRH)GRH contains the 128-bit IPv6 addressTransport LayerResponsible for in-order packet delivery,

channel multiplexing and transport servicesAlso handles transaction data segmentation

when sending, and reassembly when receiving

InfiniBand Architecture

InfiniBand Architecture Contd…

Defines multiple devices for system communication: channel adapter, switch, router, and a subnet manager

At least one channel adapter is required for each end node

Subnet Manager sets up and maintains link

Channel adapters and switches contains Subnet Management Agent(SMA) for handling communication with the subnet manager

Channel AdaptersConnects InfiniBand to other devicesTwo types: Host Channel

adapter(HCA) and Target Channel Adapter(TCA)

HCA- Provides interface to a host device and supports all software verbs defined by InfiniBand

TCA- Provides connection to an I/O device from InfiniBand with a subset of HCA features necessary for specific operations of each device

Switches and Routers

Switches and Routers Contd…Switch Contains more than one InfiniBand port and

forwards packets from one of its port to another based on the LID

Can be configured to forward either unicast packets or multicast packets

Can consume or generate management packets

Router Forwards packets from one subnet to another

without consuming or generating packets Reads the Global Route Header to forward the

packet based on IPv6 address

AdvantagesSuperior performanceLow-latencyHigh-efficiencyFabric consolidation and low

energy usageReliable, stable connectionsData IntegrityHighly interoperable environment

DrawbacksComplex in designFew platforms supports it as yetBleeding edge, for now, so users

will need to perform extensive testing

MarketsApplication ClusteringInter-Processor

Communication(IPC)Storage Area Networks

CONCLUSIONGreater processor performanceSupports RASWorks both in-the-box and

enables Bandwidth Out of the Box

Supports scalability

REFERENCES1. InfiniBand Trade Association

www.infinibandta.org 2. InfiniBand Network Architecture,

MindShare, Inc. By Tom Shanley3. InfiniBand Architecture, Development and

Deployment: A strategic guide to server I/O solutions By William T. Futral

4. Qos in InfiniBand Subnetworks: Alfaro,F.J.; Sanchez, J.L.; Duato, J.; Department de Informatica, Univ. de Castilla-La Mancha, Albacete, Spain

THANK YOU

Queries???