division it pdp cern arie van praag igabyte ystem etwork standard g s n potential hep applications...

division IT PDP

CERN

ARIE VAN PRAAG

igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD

G S NG S NPOTENTIAL HEP APPLICATIONS OF A NEW HIGH

PERFORMANCE NETWORKING TECHNOLOGY

Arie Van Praag & Ben SegalCERN 1211 Geneva 23 Switzerland

E-mail [email protected]

http://www.cern.ch/HSI/gsn

1 A SHORT Introduction to The GSN Specifications

HIPPI 6400 PH

ST Scheduled Transfer

SCSI over ST

2 Applications in High Performance Computing

3 Applications in High Energy Physics

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S N

Source Destination

SourceDestination

DATA 16 (8)

Control 4 (2)

Frame 1 (1)Clock 2 (1)

DATA 16 (8)

Control 4 (2)

Frame 1 (1)Clock 2 (1)

HIPPI-6400 PHHIPPI-6400 PHDevelopment Project: HIPPI-6400

Commercial name GSN ( Gigabyte System Network )

Connections: Full Duplex with Copper or Fibre

CableData Lines 16+4 8+2Frame Signal 1 1Clock 2 1

Principles:Data Transfer in Small Micropackets

4 Multiplexed channels called Virtual channel VC0 -VC3

To avoid Congestion

Operating System Bypass with Scheduled Transfer

To avoid OS Latency

Full 48 bit Addressing Mode for Switching and Routing following IEEE 802.nn

To convert easily to other network standards

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NDocument: Description: Status:

HIPPI-6400 PH: Physical Layer 6400 Mbit/s ANSI NCITS 323-1998

or 800 MByte/s network ISO ISO/IEC 11518-10

HIPPI-6400 SC Switch Standard NCITS 324-1999

HIPPI-6400 OP Optical Connection NCITS Submitted

ST Scheduled Transfer NCITS submitted

SCSI over ST SCSI commands over ST NCITS Standard

SCSI T10 R-00

Sub-standards: GSN & ST conversions to Fibre-Channel, HIPPI, Gigabit Ethernet, Sonet, ATM

HIPPI-6400 PHHIPPI-6400 PHHIPPI-6400 StandardsHIPPI-6400 Standards

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NVirtual ChannelsVirtual Channels

Output BufferOutput Buffer Output BufferOutput Buffer

VCO BufferVCO Buffer

VC1 BufferVC1 Buffer







SOURCE DESTINATION

CABLE OR FIBER

VC0 = Maximum Size: Messages of 68 Data Micropackets( 2176 bytes ) + header micropackets Admin Micropackets

VC1 and VC2 = Maximum Size: Messages of 4100 Data Micropackets( 128 KBytes ) + header micropackets

VC3 = Maximum Size: Messages of 4 GBytes Data Micropackets + header micropackets

Virtual Channels

HIPPI-6400 PHHIPPI-6400 PH

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NFlow ControlFlow Control

TSEQ

RSEQ

ACK(seq)

credit ( VCn,amount)

ACKs are generated independent of the VC number, and sent to the Source in the reverse direction micropacket control information.

Credits are generated on a VC basis when data exits from the VC buffer, and sent to the Source in reverse direction microcontrol information.

Credits are consumed as a packet moves from the VCn Buffer to the Output Buffer.









SOURCEDESTINATION

CABLE OR FIBER


Output BufferInput Buffer

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S N

1 Header Information c63-00

2 1 st 32 Bytes of Message data c63-00

3 2 nd 32 Bytes of Message data c63-00

N Last Bytes of Message data c63-00

c00c07c47 c15c63 c55

Micropacket & Message FormatMicropacket & Message FormatDB00

DB30

DB01

DB31

d00.7d00.0

d00.7d00.7

d00.7d00.0

d00.0 d00.0

32 DATA BYTES ( 256 Bits )

64 CONTROL BITS

MESSAGE FORMAT


division IT PDP

CERN

ARIE VAN PRAAG


G S NG S N

c00

c63 c48

c47 c32

c31 c16

c15

VCTYPEE TVCRCR

RSEQTSEQ

ECRC

LCRC

FIELD BITS CONTROL FONCTION

VC 2 VC SelectorTYPE 4 Information TypeT (AIL) 1 Last MicropacketE (ERROR) 1 ERRORVCR 2 Virtual Channel for

credit additionCR 6 Number of CreditsRSEQ 8 ACK. Sequence numberTSEQ 8 Transm. Sequence numberECRC 16 End to End ChecksumLCRC 16 Link Level Checksum

Micropacket StructureMicropacket Structurebit C3 C2 C1 C0

00010203

12131415

08091011

04050607

abcd

D15 D14 D13 D12 D11 D10 D9 D8 D7 D6 D5 D4 D3 D2 D1 D0

00.000.100.200.3

00.400.500.600.7

01.001.101.201.3

02.002.102.202.3

03.003.103.203.3

04.004.104.204.3

05.005.105.205.3

06.006.106.206.3

07.007.107.207.3

01.401.501.601.7

02.402.502.602.7

03.403.503.603.7

04.404.504.604.7

05.405.505.605.7

06.406.506.606.7

07.407.507.607.7

16171819

28293031

24252627

20212223

abcd

08.008.108.208.3

08.408.508.608.7

09.009.109.209.3

10.010.110.210.3

11.011.111.211.3

12.012.112.212.3

13.013.113.213.3

14.014.114.214.3

15.015.115.215.3

09.409.509.609.7

10.410.510.610.7

11.411.511.611.7

12.412.512.612.7

13.413.513.613.7

14.414.514.614.7

15.415.515.615.7

48495051

60616263

56575859

52535455

abcd

00.000.100.200.3

00.400.500.600.7

01.001.101.201.3

02.002.102.202.3

03.003.103.203.3

04.004.104.204.3

05.005.105.205.3

06.006.106.206.3

07.007.107.207.3

01.401.501.601.7

02.402.502.602.7

03.403.503.603.7

04.404.504.604.7

05.405.505.605.7

06.406.506.606.7

07.407.507.607.7


division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NControl Micro PacketsControl Micro Packets


Reset /Initialize

NullCreditOnly

Header Data Admin

Data Byte Contents 0 0 0 32 Byte Header 32 Bytes Data Admin Information

VC 0 0 0 any any

TYPE ( hex ) 2,3,4,5 7 A 9 8 F

Tail 1 0 0 1

ERROR 0 0 0 =1 if Erro =1 if Error =1 if Error

TSEQ xFF xFF Increments Increments Increments Increments

RSEQ 1 ACK ACK ACK ACK ACK

VCR 0 0 any any any any

CR 0 0 any any any any

LCRC single single single single single single

ECRC single single single accumulating accumulating accumulating

=1 on last Micropacket

=1 on last Micropacket

Request on VC1 Request on VC2

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NError CorrectionError Correction

c47 c46 c45 c44 c43 c42 c41 c40 c39 c38 c37 c36 c35 c34 c33 c32

LINK CRC ( LCRC )

End to End CRC ( ECRC )

LCRC covers all of the Data Bytes and Control Bits ( except for itself).

X + X + X + 116 12 5

ECRC includes only Micropacket Data Bytes including the Header Micropacket Data Bytes.

X + X + X + X + 116 12 3

DATA IN

c63 c62 c61 c60 c59 c58 c57 c56 c55 c54 c53 c52 c51 c50 c49 c48

DATA IN

XOR Function


division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NCable and Timing SequencesCable and Timing Sequences50 METER

CLOCK _2

CLOCK

FRAME 16 bit

Dnn or Cn

FRAME 8 bit

d00.0 - d00.3 d08.0 - d08.3 d16.0 - d16.3 d24.0 - d24.3For 16 bit MacropacketFor 8 bit Macropacket d00.0 - d00.3 d00.4 - d00.7 d08.0 - d08.3 d08.4 - d08.7 d16.0 - d160.3 d16.4 - d16.7 d24.0 - d24.3 d24.4 - d24.7

40 nsec

40 nsec

HIPPI 6400 END 2 CONNECTOR

16 DATA

16 DATA

FRAMECLOCK

FRAMECLOCK 2

2

CNTL4

CNTL4

HIPPI 6400 END1 CONNECTOR

16DATA

16DATA

FRAMECLOCK

FRAMECLOCK 2

2

CNTL 4

CNTL 4CONNECTORS

BERG MICROPAX 100

W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X T Y Z


division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NFiber ConnectionFiber Connection

............

............

............

............ 2 CABLES 12 Fibers Each

Connectors: MT Connectors: MT

Transmitter Launch Power -6 to -12 dBmWavelength: 850 nmReceiver Sensitivity: -18 dBm Berr = 10 -12

Distance: Max 1 KmCompliant to: CDRH 1040.1 Class1

IEC 825-1 Class 3a

HIPPI-6400 OPTHIPPI-6400 OPT

Long Distance Connection

Single Mode Fibers 2 Parallel FibersWavelength 1300 NMDistance 10 Km

A working Group is just formed

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NSwitchesSwitches

D S

D S

DS

DS

D S

DS

S

D

S

D

D

S

D

SGSN - 64

SWITCH

GSN - 64

NODEGSN - 64

NODE

HIPPI - 800NODE

TranslationFunction

BRIDGE

HIPPI 6400

HIPPI 800

D HIPPI 6400DESTINATION

S

D

S HIPPI 6400SOURCE

HIPPI 800DESTINATION

HIPPI 800SOURCE

4 x

1 x 1 xFull Crossbar Non Blocking Switch

with Translation boxes for HIPPI-800 and other Data Transporting Media

HIPPI-6400 SCHIPPI-6400 SC

HIPPI 800Fibre Channel

Gigabit EthernetSONET OC48c

8 X

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NScheduled TransferScheduled Transfer

• Specifies small control messages to pre-arrange Data movement

– Virtual Connection Set-up and Tear Down.

– Determine the number of operations the other side can accept

– Determine the Buffer Size of the other side

– Exchange Key, Port, Identifiers, Buffer Size Values

– Determine Maximum Transmission Unit to avoid overrun of the

buffers at the Receiving End

– Acknowledge Partial Transfers

– Easy Transfers over Different type of High Performance Networks

• If used with prestored Tables an extension of a Scatter Gather System

ST ST

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NScheduled TransferScheduled Transfer

ST ST

BuffersBuffer Descriptor TableBlock DescriptorBuff 0Bufx 1Bufx 2

Bufx n

....

local-Portlocal-Keyremote-Port

local-Portlocal-Keyremote-Port

remote End local endPort PortKey KeyMax. Slots Max. SlotsBufsize Bufsize

Max. STU Size Max STU SizeMax. Block Size Max. Block SizeOut_of_order cap.

Ethertype Remote Slots local Slots

local Sync #Op_time

Max_retryremote-id1 local-id1remote-id2 local-id2

remote-idj local-idj

remote End local endPort PortKey KeyMax. Slots Max. SlotsBufsize Bufsize

Max. STU Size Max STU SizeMax. Block Size Max. Block SizeOut_of_order cap.

Ethertype Remote Slots local Slots

local Sync #Op_time

Max_retryremote-id1 local-id1remote-id2 local-id2

remote-idj local-idj

Virtual Connection Descriptors

Transfer Descriptor

Selection and

Validation Criteria

TransferDescriptor

TransferDescriptor

Address 0Address 1Address 2

Address n

....

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NResponde

rRequest Memory Region(RMR)

Put STU Mx=m

Memory Region Available (MRA)

Mx=mlength=loffset=xBufx=n

ResponderBuffer

InitiatorBuffer

Bufx

n

n+1

Get/FetchOpMx=p, length=k

ioffset=y, roffset=ziBufx=o, rBufx=n+1

STU Mx=p

FetchOp Complete

ST ST S T WRITES T WRITE

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NStorage CommandsStorage Commands

Commands are sent in ST’s RTS or RTR optional payload:

Byte Content

0 type = storage command

1 length = 32

2-3 target number

4-31 SCSI Fibre Channel(FCP)Command payload, bytes 0-27

ST ST

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S N

TargetBuffer

Status Put(STU)

STU(data)

STU(data)

Target

CTS

RTSI-id=3opt=cmd

Status PMR

0

1

2

3

4

5

STU(data)

SCSI - ST SCSI - ST Cannot use payload for SCSI status (can be >30 bytes)Use ST’s Persistent Memory Region feature:

Cannot use payload for SCSI status (can be >30 bytes)Use ST’s Persistent Memory Region feature:

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NThe Protocol PictureThe Protocol Picture

Software

STSTIPIP

TCPTCP UDPUDP

FTPFTP NFSNFS

DE/MUXDE/MUX

Other protocols...

...

ST

Hardware

ST

Hardware

General

Delivery

Hardware

General

Delivery

Hardware

Hardware

Media

SCSISCSI

S T S T

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NINTERFACES:

Silicon Graphics Origin Series Available

PCI Interface 64/66 Compaq Alpha, SUN 1 Q 2000

SWITCHES:

ODS - Essential 32 X 32 Available

ODS - Essential 8 X 8 Available

Genroco 8 X 8 Available

PMR 8 X 8 Available

BRIDGES:

ODS-Essential Translation Function HIPPI-800 Available

Genroco Storage Bridge Fibre Channel Available

Genroco Network Bridge HIPPI Available

Fibre Channel Available

Gigabit Ethernet Available

OC48c 3 Q 2000

CABLES:

Berg Copper cables and Connectors Available

COMPONENTS for OPTICAL CONNECTIONS:

Infineon Paroli AC Modules and Fibres Available

Gore Noptical Modules and Fibres 1 Q 2000

Products as of January 2000Products as of January 2000

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S N Standards & Popularity

( 1995 )

Standards & Popularity

( 1995 )

85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2000 01 02 03 04 05

Gigabyte System Network GSN

HIPPI-Serial

HIPPI

Ethernet

T base 100

Gigabit Ethernet

Fibre Channel

ATM

LHCFREEZING

SELECTION

LHCOPERATIONAL

S-LINK

PCI

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NSERVERS

Fibre ChannelHIPPI

Gigabit EthernetGSN

http://www.cern.ch/HSI/hippi/hug/demo99/Demo99.htmlhttp://www.cern.ch/HSI/hippi/hug/demo99/Demo99.html

FC TAPES

SCSI/ST/GSN

SCSI/ST/ETH

FC DISK ARRAYS

SCSI/FC

GSN

F.C.

GSN SWITCHES

GSN

GSN Storage Bridge

SCSI/ST/GigE

ETHER TYPENETWORK

GSN

SCSI/ST /HIPPI

SCSI/FC

GSN Network Bridge

F.C.HIPPI

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S N

Central File Meta Data

on

a central workstation

Proprietary Software, e.g.

DataDirect CDNA

Mercury SANergy, etc..

Central File Meta Data

on

a central workstation

Proprietary Software, e.g.

DataDirect CDNA

Mercury SANergy, etc..

Data Storage Networks

Cluster File SystemsCluster File Systems

Networking

FC DISKARAYSFC TAPES

GFS Minnesota University

Networking

FC DISKARAYSFC TAPES

Symmetrical System with Distributed file system Meta Data

and Locks.

Symmetrical System with Distributed file system Meta Data

and Locks.

LINUXIRIX

NTMcIntoshSolaris

etc.

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NGSN to S-LINK ??GSN to S-LINK ??

FIFO

FIFO

FPGA

TO BRIDGE

PROCESSOR

RJ 12RS232

Register SET-UP

FIFO FIFO FIFOFIFO

FROM BRIDGE

FPGA

reset

Control etc

Glue Logic

Synchronization etc.

16 bit 66 MHz 132 MB/s

64 bit40 MHz320 MB/s

64 bit40 MHz320 MB/s

PROCESSOR BUS

FPGA

FIFO

FPGAVersion withVersion with 6 S-LINK CANNELS 6 S-LINK CANNELSAdded Bandwidth Added Bandwidth 100 Mbyte/s per Interface100 Mbyte/s per InterfaceTotal Added bandwidth Total Added bandwidth 800 Mbyte/s per Bridge800 Mbyte/s per Bridge

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NEvent Building with a SwitchEvent Building with a Switch

VMEbus Read Out Buffers ( ROB )

DETECTOR DATA

BRIDGE BRIDGE

24 GSN Connections

CONNECTIONS768 (4) S-Link

or 1152 (6) S-Linkor 192 HIPPI-800

8 GSN Connections to Workstation Farm

32 X 32 GSN - 64 Switch Fabric

24 GSN Bridges

100- 1000 Bytes/s.

To CentralData Storage

or Data Analyzes

10 - 100 MBytes/s

1 0-100 TByte/s.

FC DISK ARRAYS

Long Distance Connection

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S N

8 Km L.W.

SW Glabs 8X8Giga

Router

Switch 01 Gigalabs 16X16

1 From Detector HIPPI raw data

2 Data Distribution to Processor farm HIPPI

3 processor output in FDDI

4 Gigarouter converts data to FDDI over HIPPI

Giga Router

5 FDDI over HIPPI to computer centre

6 Gigarouter converts

to FDDI for processing

7 Via Gigarouter back to HIPPI for Storage

8 HIPPI to SCSI conversion for Storage

STK Tape Units

NA48 Experiment

CODING &

CONVERTING

and

RECODING &

RECONVERTING

CODING &

CONVERTING

and

RECODING &

RECONVERTING

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NPhysics Data from Experiment GSN = FC / ST / SCSI

Bridge GSN / FC

FC

No Protocol or

Format Change from

DETECTOR to

Storage & Analysis

? ? ?

? ? ?

Central Storage in theComputer Center

GSN / Gigabit EthernetGSN / HIPPI

A FAT ST PIPEA FAT ST PIPE

Long Distance Transfer Connection

( Wavelength Multiplexing ?or Sonet or ?? )

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NPhysics

Data Transportfor LHC

Physics Data

Transportfor LHC

10 Km

LHC Experiments:

Each experiment Transmits up to 100 MBytes/s

How to get this data to the computer center ?

OC 48c does 250 MByte/s

Atlas Alice

LHCB CMS

division IT PDP

CERN

ARIE VAN PRAAG


G S NG S NGSN against EthernetGSN against Ethernet

Sorry Compaq that I mis-used your picture

division it pdp cern arie van praag igabyte ystem etwork standard g s n potential hep applications...

Documents