division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NPOTENTIAL HEP APPLICATIONS OF A NEW HIGH
PERFORMANCE NETWORKING TECHNOLOGY
Arie Van Praag & Ben SegalCERN 1211 Geneva 23 Switzerland
E-mail [email protected]
http://www.cern.ch/HSI/gsn
1 A SHORT Introduction to The GSN Specifications
HIPPI 6400 PH
ST Scheduled Transfer
SCSI over ST
2 Applications in High Performance Computing
3 Applications in High Energy Physics
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S N
Source Destination
SourceDestination
DATA 16 (8)
Control 4 (2)
Frame 1 (1)Clock 2 (1)
DATA 16 (8)
Control 4 (2)
Frame 1 (1)Clock 2 (1)
HIPPI-6400 PHHIPPI-6400 PHDevelopment Project: HIPPI-6400
Commercial name GSN ( Gigabyte System Network )
Connections: Full Duplex with Copper or Fibre
CableData Lines 16+4 8+2Frame Signal 1 1Clock 2 1
Principles:Data Transfer in Small Micropackets
4 Multiplexed channels called Virtual channel VC0 -VC3
To avoid Congestion
Operating System Bypass with Scheduled Transfer
To avoid OS Latency
Full 48 bit Addressing Mode for Switching and Routing following IEEE 802.nn
To convert easily to other network standards
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NDocument: Description: Status:
HIPPI-6400 PH: Physical Layer 6400 Mbit/s ANSI NCITS 323-1998
or 800 MByte/s network ISO ISO/IEC 11518-10
HIPPI-6400 SC Switch Standard NCITS 324-1999
HIPPI-6400 OP Optical Connection NCITS Submitted
ST Scheduled Transfer NCITS submitted
SCSI over ST SCSI commands over ST NCITS Standard
SCSI T10 R-00
Sub-standards: GSN & ST conversions to Fibre-Channel, HIPPI, Gigabit Ethernet, Sonet, ATM
HIPPI-6400 PHHIPPI-6400 PHHIPPI-6400 StandardsHIPPI-6400 Standards
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NVirtual ChannelsVirtual Channels
Output BufferOutput Buffer Output BufferOutput Buffer
VCO BufferVCO Buffer
VC1 BufferVC1 Buffer
VC2 BufferVC2 Buffer
VC3 BufferVC3 Buffer
VCO BufferVCO Buffer
VC1 BufferVC1 Buffer
VC2 BufferVC2 Buffer
VC3 BufferVC3 Buffer
SOURCE DESTINATION
CABLE OR FIBER
VC0 = Maximum Size: Messages of 68 Data Micropackets( 2176 bytes ) + header micropackets Admin Micropackets
VC1 and VC2 = Maximum Size: Messages of 4100 Data Micropackets( 128 KBytes ) + header micropackets
VC3 = Maximum Size: Messages of 4 GBytes Data Micropackets + header micropackets
Virtual Channels
HIPPI-6400 PHHIPPI-6400 PH
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NFlow ControlFlow Control
TSEQ
RSEQ
ACK(seq)
credit ( VCn,amount)
ACKs are generated independent of the VC number, and sent to the Source in the reverse direction micropacket control information.
Credits are generated on a VC basis when data exits from the VC buffer, and sent to the Source in reverse direction microcontrol information.
Credits are consumed as a packet moves from the VCn Buffer to the Output Buffer.
VCO BufferVCO Buffer
VC1 BufferVC1 Buffer
VC2 BufferVC2 Buffer
VC3 BufferVC3 Buffer
VCO BufferVCO Buffer
VC1 BufferVC1 Buffer
VC2 BufferVC2 Buffer
VC3 BufferVC3 Buffer
SOURCEDESTINATION
CABLE OR FIBER
HIPPI-6400 PHHIPPI-6400 PH
Output BufferInput Buffer
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S N
1 Header Information c63-00
2 1 st 32 Bytes of Message data c63-00
3 2 nd 32 Bytes of Message data c63-00
N Last Bytes of Message data c63-00
c00c07c47 c15c63 c55
Micropacket & Message FormatMicropacket & Message FormatDB00
DB30
DB01
DB31
d00.7d00.0
d00.7d00.7
d00.7d00.0
d00.0 d00.0
32 DATA BYTES ( 256 Bits )
64 CONTROL BITS
MESSAGE FORMAT
HIPPI-6400 PHHIPPI-6400 PH
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S N
c00
c63 c48
c47 c32
c31 c16
c15
VCTYPEE TVCRCR
RSEQTSEQ
ECRC
LCRC
FIELD BITS CONTROL FONCTION
VC 2 VC SelectorTYPE 4 Information TypeT (AIL) 1 Last MicropacketE (ERROR) 1 ERRORVCR 2 Virtual Channel for
credit additionCR 6 Number of CreditsRSEQ 8 ACK. Sequence numberTSEQ 8 Transm. Sequence numberECRC 16 End to End ChecksumLCRC 16 Link Level Checksum
Micropacket StructureMicropacket Structurebit C3 C2 C1 C0
00010203
12131415
08091011
04050607
abcd
D15 D14 D13 D12 D11 D10 D9 D8 D7 D6 D5 D4 D3 D2 D1 D0
00.000.100.200.3
00.400.500.600.7
01.001.101.201.3
02.002.102.202.3
03.003.103.203.3
04.004.104.204.3
05.005.105.205.3
06.006.106.206.3
07.007.107.207.3
01.401.501.601.7
02.402.502.602.7
03.403.503.603.7
04.404.504.604.7
05.405.505.605.7
06.406.506.606.7
07.407.507.607.7
16171819
28293031
24252627
20212223
abcd
08.008.108.208.3
08.408.508.608.7
09.009.109.209.3
10.010.110.210.3
11.011.111.211.3
12.012.112.212.3
13.013.113.213.3
14.014.114.214.3
15.015.115.215.3
09.409.509.609.7
10.410.510.610.7
11.411.511.611.7
12.412.512.612.7
13.413.513.613.7
14.414.514.614.7
15.415.515.615.7
48495051
60616263
56575859
52535455
abcd
00.000.100.200.3
00.400.500.600.7
01.001.101.201.3
02.002.102.202.3
03.003.103.203.3
04.004.104.204.3
05.005.105.205.3
06.006.106.206.3
07.007.107.207.3
01.401.501.601.7
02.402.502.602.7
03.403.503.603.7
04.404.504.604.7
05.405.505.605.7
06.406.506.606.7
07.407.507.607.7
HIPPI-6400 PHHIPPI-6400 PH
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NControl Micro PacketsControl Micro Packets
HIPPI-6400 PHHIPPI-6400 PH
Reset /Initialize
NullCreditOnly
Header Data Admin
Data Byte Contents 0 0 0 32 Byte Header 32 Bytes Data Admin Information
VC 0 0 0 any any
TYPE ( hex ) 2,3,4,5 7 A 9 8 F
Tail 1 0 0 1
ERROR 0 0 0 =1 if Erro =1 if Error =1 if Error
TSEQ xFF xFF Increments Increments Increments Increments
RSEQ 1 ACK ACK ACK ACK ACK
VCR 0 0 any any any any
CR 0 0 any any any any
LCRC single single single single single single
ECRC single single single accumulating accumulating accumulating
=1 on last Micropacket
=1 on last Micropacket
Request on VC1 Request on VC2
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NError CorrectionError Correction
c47 c46 c45 c44 c43 c42 c41 c40 c39 c38 c37 c36 c35 c34 c33 c32
LINK CRC ( LCRC )
End to End CRC ( ECRC )
LCRC covers all of the Data Bytes and Control Bits ( except for itself).
X + X + X + 116 12 5
ECRC includes only Micropacket Data Bytes including the Header Micropacket Data Bytes.
X + X + X + X + 116 12 3
DATA IN
c63 c62 c61 c60 c59 c58 c57 c56 c55 c54 c53 c52 c51 c50 c49 c48
DATA IN
XOR Function
HIPPI-6400 PHHIPPI-6400 PH
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NCable and Timing SequencesCable and Timing Sequences50 METER
CLOCK _2
CLOCK
FRAME 16 bit
Dnn or Cn
FRAME 8 bit
d00.0 - d00.3 d08.0 - d08.3 d16.0 - d16.3 d24.0 - d24.3For 16 bit MacropacketFor 8 bit Macropacket d00.0 - d00.3 d00.4 - d00.7 d08.0 - d08.3 d08.4 - d08.7 d16.0 - d160.3 d16.4 - d16.7 d24.0 - d24.3 d24.4 - d24.7
40 nsec
40 nsec
HIPPI 6400 END 2 CONNECTOR
16 DATA
16 DATA
FRAMECLOCK
FRAMECLOCK 2
2
CNTL4
CNTL4
HIPPI 6400 END1 CONNECTOR
16DATA
16DATA
FRAMECLOCK
FRAMECLOCK 2
2
CNTL 4
CNTL 4CONNECTORS
BERG MICROPAX 100
W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X Y ZT W X T Y Z
HIPPI-6400 PHHIPPI-6400 PH
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NFiber ConnectionFiber Connection
............
............
............
............ 2 CABLES 12 Fibers Each
Connectors: MT Connectors: MT
Transmitter Launch Power -6 to -12 dBmWavelength: 850 nmReceiver Sensitivity: -18 dBm Berr = 10 -12
Distance: Max 1 KmCompliant to: CDRH 1040.1 Class1
IEC 825-1 Class 3a
HIPPI-6400 OPTHIPPI-6400 OPT
Long Distance Connection
Single Mode Fibers 2 Parallel FibersWavelength 1300 NMDistance 10 Km
A working Group is just formed
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NSwitchesSwitches
D S
D S
DS
DS
D S
DS
S
D
S
D
D
S
D
SGSN - 64
SWITCH
GSN - 64
NODEGSN - 64
NODE
HIPPI - 800NODE
TranslationFunction
BRIDGE
HIPPI 6400
HIPPI 800
D HIPPI 6400DESTINATION
S
D
S HIPPI 6400SOURCE
HIPPI 800DESTINATION
HIPPI 800SOURCE
4 x
1 x 1 xFull Crossbar Non Blocking Switch
with Translation boxes for HIPPI-800 and other Data Transporting Media
HIPPI-6400 SCHIPPI-6400 SC
HIPPI 800Fibre Channel
Gigabit EthernetSONET OC48c
8 X
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NScheduled TransferScheduled Transfer
• Specifies small control messages to pre-arrange Data movement
– Virtual Connection Set-up and Tear Down.
– Determine the number of operations the other side can accept
– Determine the Buffer Size of the other side
– Exchange Key, Port, Identifiers, Buffer Size Values
– Determine Maximum Transmission Unit to avoid overrun of the
buffers at the Receiving End
– Acknowledge Partial Transfers
– Easy Transfers over Different type of High Performance Networks
• If used with prestored Tables an extension of a Scatter Gather System
ST ST
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NScheduled TransferScheduled Transfer
ST ST
BuffersBuffer Descriptor TableBlock DescriptorBuff 0Bufx 1Bufx 2
Bufx n
....
local-Portlocal-Keyremote-Port
local-Portlocal-Keyremote-Port
remote End local endPort PortKey KeyMax. Slots Max. SlotsBufsize Bufsize
Max. STU Size Max STU SizeMax. Block Size Max. Block SizeOut_of_order cap.
Ethertype Remote Slots local Slots
local Sync #Op_time
Max_retryremote-id1 local-id1remote-id2 local-id2
remote-idj local-idj
remote End local endPort PortKey KeyMax. Slots Max. SlotsBufsize Bufsize
Max. STU Size Max STU SizeMax. Block Size Max. Block SizeOut_of_order cap.
Ethertype Remote Slots local Slots
local Sync #Op_time
Max_retryremote-id1 local-id1remote-id2 local-id2
remote-idj local-idj
Virtual Connection Descriptors
Transfer Descriptor
Selection and
Validation Criteria
TransferDescriptor
TransferDescriptor
Address 0Address 1Address 2
Address n
....
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NResponde
rRequest Memory Region(RMR)
Put STU Mx=m
Memory Region Available (MRA)
Mx=mlength=loffset=xBufx=n
ResponderBuffer
InitiatorBuffer
Bufx
n
n+1
Get/FetchOpMx=p, length=k
ioffset=y, roffset=ziBufx=o, rBufx=n+1
STU Mx=p
FetchOp Complete
ST ST S T WRITES T WRITE
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NStorage CommandsStorage Commands
Commands are sent in ST’s RTS or RTR optional payload:
Byte Content
0 type = storage command
1 length = 32
2-3 target number
4-31 SCSI Fibre Channel(FCP)Command payload, bytes 0-27
ST ST
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S N
TargetBuffer
Status Put(STU)
STU(data)
STU(data)
Target
CTS
RTSI-id=3opt=cmd
Status PMR
0
1
2
3
4
5
STU(data)
SCSI - ST SCSI - ST Cannot use payload for SCSI status (can be >30 bytes)Use ST’s Persistent Memory Region feature:
Cannot use payload for SCSI status (can be >30 bytes)Use ST’s Persistent Memory Region feature:
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NThe Protocol PictureThe Protocol Picture
Software
STSTIPIP
TCPTCP UDPUDP
FTPFTP NFSNFS
DE/MUXDE/MUX
Other protocols...
...
ST
Hardware
ST
Hardware
General
Delivery
Hardware
General
Delivery
Hardware
Hardware
Media
SCSISCSI
S T S T
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NINTERFACES:
Silicon Graphics Origin Series Available
PCI Interface 64/66 Compaq Alpha, SUN 1 Q 2000
SWITCHES:
ODS - Essential 32 X 32 Available
ODS - Essential 8 X 8 Available
Genroco 8 X 8 Available
PMR 8 X 8 Available
BRIDGES:
ODS-Essential Translation Function HIPPI-800 Available
Genroco Storage Bridge Fibre Channel Available
Genroco Network Bridge HIPPI Available
Fibre Channel Available
Gigabit Ethernet Available
OC48c 3 Q 2000
CABLES:
Berg Copper cables and Connectors Available
COMPONENTS for OPTICAL CONNECTIONS:
Infineon Paroli AC Modules and Fibres Available
Gore Noptical Modules and Fibres 1 Q 2000
Products as of January 2000Products as of January 2000
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S N Standards & Popularity
( 1995 )
Standards & Popularity
( 1995 )
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2000 01 02 03 04 05
Gigabyte System Network GSN
HIPPI-Serial
HIPPI
Ethernet
T base 100
Gigabit Ethernet
Fibre Channel
ATM
LHCFREEZING
SELECTION
LHCOPERATIONAL
S-LINK
PCI
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NSERVERS
Fibre ChannelHIPPI
Gigabit EthernetGSN
http://www.cern.ch/HSI/hippi/hug/demo99/Demo99.htmlhttp://www.cern.ch/HSI/hippi/hug/demo99/Demo99.html
FC TAPES
SCSI/ST/GSN
SCSI/ST/ETH
FC DISK ARRAYS
SCSI/FC
GSN
F.C.
GSN SWITCHES
GSN
GSN Storage Bridge
SCSI/ST/GigE
ETHER TYPENETWORK
GSN
SCSI/ST /HIPPI
SCSI/FC
GSN Network Bridge
F.C.HIPPI
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S N
Central File Meta Data
on
a central workstation
Proprietary Software, e.g.
DataDirect CDNA
Mercury SANergy, etc..
Central File Meta Data
on
a central workstation
Proprietary Software, e.g.
DataDirect CDNA
Mercury SANergy, etc..
Data Storage Networks
Cluster File SystemsCluster File Systems
Networking
FC DISKARAYSFC TAPES
GFS Minnesota University
Networking
FC DISKARAYSFC TAPES
Symmetrical System with Distributed file system Meta Data
and Locks.
Symmetrical System with Distributed file system Meta Data
and Locks.
LINUXIRIX
NTMcIntoshSolaris
etc.
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NGSN to S-LINK ??GSN to S-LINK ??
FIFO
FIFO
FPGA
TO BRIDGE
PROCESSOR
RJ 12RS232
Register SET-UP
FIFO FIFO FIFOFIFO
FROM BRIDGE
FPGA
reset
Control etc
Glue Logic
Synchronization etc.
16 bit 66 MHz 132 MB/s
64 bit40 MHz320 MB/s
64 bit40 MHz320 MB/s
PROCESSOR BUS
FPGA
FIFO
FPGAVersion withVersion with 6 S-LINK CANNELS 6 S-LINK CANNELSAdded Bandwidth Added Bandwidth 100 Mbyte/s per Interface100 Mbyte/s per InterfaceTotal Added bandwidth Total Added bandwidth 800 Mbyte/s per Bridge800 Mbyte/s per Bridge
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NEvent Building with a SwitchEvent Building with a Switch
VMEbus Read Out Buffers ( ROB )
DETECTOR DATA
BRIDGE BRIDGE
24 GSN Connections
CONNECTIONS768 (4) S-Link
or 1152 (6) S-Linkor 192 HIPPI-800
8 GSN Connections to Workstation Farm
32 X 32 GSN - 64 Switch Fabric
24 GSN Bridges
100- 1000 Bytes/s.
To CentralData Storage
or Data Analyzes
10 - 100 MBytes/s
1 0-100 TByte/s.
FC DISK ARRAYS
Long Distance Connection
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S N
8 Km L.W.
SW Glabs 8X8Giga
Router
Switch 01 Gigalabs 16X16
1 From Detector HIPPI raw data
2 Data Distribution to Processor farm HIPPI
3 processor output in FDDI
4 Gigarouter converts data to FDDI over HIPPI
Giga Router
5 FDDI over HIPPI to computer centre
6 Gigarouter converts
to FDDI for processing
7 Via Gigarouter back to HIPPI for Storage
8 HIPPI to SCSI conversion for Storage
STK Tape Units
NA48 Experiment
CODING &
CONVERTING
and
RECODING &
RECONVERTING
CODING &
CONVERTING
and
RECODING &
RECONVERTING
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NPhysics Data from Experiment GSN = FC / ST / SCSI
Bridge GSN / FC
FC
No Protocol or
Format Change from
DETECTOR to
Storage & Analysis
? ? ?
? ? ?
Central Storage in theComputer Center
GSN / Gigabit EthernetGSN / HIPPI
A FAT ST PIPEA FAT ST PIPE
Long Distance Transfer Connection
( Wavelength Multiplexing ?or Sonet or ?? )
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NPhysics
Data Transportfor LHC
Physics Data
Transportfor LHC
10 Km
LHC Experiments:
Each experiment Transmits up to 100 MBytes/s
How to get this data to the computer center ?
OC 48c does 250 MByte/s
Atlas Alice
LHCB CMS
division IT PDP
CERN
ARIE VAN PRAAG
igabyte ystem etworkigabyte ystem etworkSTANDARDSTANDARD
G S NG S NGSN against EthernetGSN against Ethernet
Sorry Compaq that I mis-used your picture