F. Hemmer

CERN

HepIx Fall 94

Experiences & directions

of SHIFT/COREat CERN

October 13, 1994

F. Hemmer

CERN

HepIx Fall 94

SHIFT

Designed in 1990

• fast access to large amounts of data

• good tape support

• cheap & easy to expand

• vendor independent

First implementation in operation in Jan 1991

F. Hemmer

CERN

HepIx Fall 94

SHIFT Model

CPUServerCPU

ServerCPUServerCPU

ServerCPUServerCPU

ServerDisk

ServerDiskServer

TapeServerTape

ServerTapeServer

High Speed network

F. Hemmer

CERN

HepIx Fall 94

CORE

• Centrally Operated RISC environment

• Grouped CSF, SHIFT, HOPE,PIAF, TAPES, Network, Racks ...

• CERNVM vs. CORE

F. Hemmer

CERN

HepIx Fall 94

1993-1994 Strategy

• Decision to run down the CRAY X/MP

• Decision to run down the IBM 9000

Migrate batch by end of 94

Migrate interactive by end of 95

• Decision to acquire a 64 node SP2

F. Hemmer

CERN

HepIx Fall 94

SGI 340 SGI 340SGI 340 SGI 340

SUN

4/330

SUN

4/330

STK 4280

STK 4280

EXABYTE

UltraNet

EXABYTE

JUKEBOX

cisco AGS+IP router

CHEOPS

~250 GBytesSCSI disk

10 GBytesSCSI disk

25 H-P 9000-720s

H-P Apollo DN10040

HOPE

CSF

shift

Simulation

Facilities

Data AnalysisFacility

CrayX-MP/48

cassette tapesand robot

ethernet

CERN site LAN

FDDI

SUN

4/630

STK 4280

STK 4280

STK 4280

STK 4280

CrimsonCrimson

1992

F. Hemmer

CERN

HepIx Fall 94

CORE in 1994

Put les core picture

F. Hemmer

CERN

HepIx Fall 94

CORE in 1994• CPU server

HP 9000-735/755

SGI Challenge

SGI Power Series

DEC 3000/400,500

IBM RS/6000-370

• Disk servers

SGI Power series

SGI Crimson

DEC 3000-600

IBM RS/6000 970

SUN Sparc 10

• Tape servers

IBM RS/6000-370

Sun 4/330

• Monitors,consoles,YP,...

Sun SPARCstations

TOTAL

# Systems

43

4

1

11

3

1

2

3

1

1

8

3

8

90

# CPUs

43

44

4

11

10

4

2

3

1

1

8

3

8

143

F. Hemmer

CERN

HepIx Fall 94

CORE operation

issues• Reliability

• Manpower/machine decreases

• Automation

• Monitoring

• "Standard" tools

• Centralize management

F. Hemmer

CERN

HepIx Fall 94

CORE 1994

Here MTBI's per manuf

F. Hemmer

CERN

HepIx Fall 94

CORE 1995

• CORE is the infrastructure

operation, network, racks, cables, ...

• Public services

CSF,PIAF,SP2,CS/2

• Public staging pool

• Tape servers

F. Hemmer

CERN

HepIx Fall 94

SHIFT CPU 1992

IBM

9000

CSF Cray SHIFT VAX

9000

0

200000

400000

600000

800000

1000000

F. Hemmer

CERN

HepIx Fall 94

CORE CPU 1992-94

CERNVM VXCERN CSF SHIFT0

1000000

2000000

3000000

4000000

5000000

F. Hemmer

CERN

HepIx Fall 94

SHIFT 1995

here LEP Z0 + Req for 95 + NA 48/49 + LHC

Plus data flows LEP LHC

F. Hemmer

CERN

HepIx Fall 94

DISKS 07/94Vendor Model Formatted capacity Number Total size (MB)

IBM OEM 0664CSH IBM OEM 0664M1H 1920 12 23040IBM OEM 0664N1D 1920 228 437760SGI 0664N1D 1920 1 1920MICROP 2112 1001 1 1001HP C2247 1001 10 10010DEC DSP5350 3406 31 105586DEC RZ26-VA 1075 10 10750DEC RZ28-VA 2150 6 12900DEC RZ74 3406 63 214578SEAGATE ST12400 2048 52 106496SEAGATE ST2383N 317 2 634SEAGATE ST41200 990 44 43560SEAGATE ST41600 1331 1 1331SEAGATE ST41650 1351 162 218862SEAGATE ST41651 1350 6 8100SEAGATE ST42100 1812 12 21744SEAGATE ST43400 2778 61 169458SEAGATE ST4766N 669 1 669 Total 704 1392239

F. Hemmer

CERN

HepIx Fall 94

Tape mounts week 1-43

1992

4% 4%

75%

17% SHIFT/CRAY

SHIFT/SUN

VM

Others

F. Hemmer

CERN

HepIx Fall 94

TAPES in 1994

46%54%

SHIFT

CERNVM

F. Hemmer

CERN

HepIx Fall 94

Tapes in 1994

1 7

13

19

25

31

370

50001000015000200002500030000

SHIFT

CERNVM

F. Hemmer

CERN

HepIx Fall 94

TAPES 1995

• NTP robots

• 3494 & central data recording

• DLT robot

• Any "reasonable" media support

F. Hemmer

CERN

HepIx Fall 94

Network

• UltraNet still the most used

• FDDI used more and more

• 1995 : HIPPi, FCS ...

• Spectrum

F. Hemmer

CERN

HepIx Fall 94

UltraNet Bandwith

Oct 92

MB/S

293 294 295 296 297 298 299 300 301 DAY No

0

0.5

1

1.5

2

2.5

3

3.5

F. Hemmer

CERN

HepIx Fall 94

Network

Jacek Foils

F. Hemmer

CERN

HepIx Fall 94

UltraNet SoftwareUser application Software

Socket Compatibility Library

NFS Sockets

UDP TCPIP

Data LinkDriver

EtherNet UltraNetFDDI

UltraNet Driver

TransportNetwork

Data LinkPhysical Link

Hardware AssistedProtocol Engine

F. Hemmer

CERN

HepIx Fall 94

SHIFT Software

• Unix Tape subsystem

multi-user, labels, multi-file, operation

• Fast Remote File Access system

• Remote Tape Copy System

• Disk Pool manager

• Tape Stager

• Clustered NQS Batch System

• Integration with standard I/O packages

Fatmen, Zebra, EPIO

• Network Operation

• Monitoring

F. Hemmer

CERN

HepIx Fall 94

Remote Tape Copy System

• select a suitable tape server

• initiates the tape-disk copy

tpread -v I29127 -g SMCF -q 4,6 pathname

tpread -v CUT322 ‘sfget -p opaldst filename‘

F. Hemmer

CERN

HepIx Fall 94

RFIO

High performance, reliability (improve on NFS)

• C I/O compatibility

Fortran subroutine interface

• rfio daemon started by open on remote machine

• optimized for specific networks

• asynchronous operation (read ahead)

• optional vector preseek

F. Hemmer

CERN

HepIx Fall 94

SW

show what has changed since 1992AFS

F. Hemmer

CERN

HepIx Fall 94

Stager

• up to now : dpm + tpread

• lack of robustness faced to system errors

• concurrency not handled

• staging space not handled

F. Hemmer

CERN

HepIx Fall 94

Stager architecture

CPUServer

TapeServerDisk

Server

CPUServer

CPUServer

DiskServer

stgcat

TapeServer

TapeServer

F. Hemmer

CERN

HepIx Fall 94

Stager functions

• process stage requests

• manage space

- By default purge oldest file when space needed

- if (size < 1024) w=max(atime, mtime)

else w=max(atime,mtime)-(86400*log(size/1024))

• a different algorithm may be provided

F. Hemmer

CERN

HepIx Fall 94

Stager commands

• stagein

• stageout/stageput

• stagewrt

• stageclr

• stageqry

F. Hemmer

CERN

HepIx Fall 94

Stager future extensions

• Staging of disk files

• Transfer of files already staged on CERNVM

• Access control lists

• Prioritization of requests

• Tape to Tape copy

• Automatic file migration to robots

F. Hemmer

CERN

HepIx Fall 94

SHIFT SW Futures

• SHIFT library in CERNLIB

• RFIO review for high speed protocols

• RFIO wide area

• RFIO "gateway" for SP/2

• RFIO per platform tuning

• VMS final port

• RFIO Checkpoint/Restart

• Ports to Irix 6, AIX 4, HP/UX 10