hades eb status€¦ · daq storage for hades (dabc plug-in) ltsm gateway for mbs systems summary...

Jörn Adamczewski-Musch

GSI / EEL 5 December 2017

DAQ storage for HADES (dabc plug-in)

LTSM gateway for MBS systems

Summary

5th December 2017 Jörn Adamczewski-Musch 2

DABC


TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

Combiner

output

stream

http

DABC hld

Combiner

output

stream

http

DABC hld

Combiner

output

stream

http

DABC hld

Combiner

output

stream

http

EPICS run sync, GUI

/shm

/shm

/shm

/shm

QA monitoring HYDRA2

hld RFIO

/data0i

/lustre

storage interface

DABC


TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

TRB

Combiner

output

stream

http

DABC hld

Combiner

output

stream

http

DABC hld

Combiner

output

stream

http

DABC hld

Combiner

output

stream

http

EPICS run sync, GUI

/shm

/shm

/shm

/shm

QA monitoring HYDRA2

hld LTSM

/data0i

/lustre

storage interface

Data Acquisition Backbone Core (http://dabc.gsi.de) Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ

DABC LTSM plugin :

Implements dabc::FileInterface with fopen, fwrite, etc. methods (POSIX like)

Wraps and uses LTSM library by T.Stibor and tsm client by IBM

May write any file format (HADES .hld, MBS .lmd, ...)

generally configurable in dabc application by xml files

for HADES configurable by existing DAQ/EB scripts and setup files


http://dabc.gsi.de/


mbs> con rfio lxg0546:2345 –disk

X86L-10:transport :Connected RFIO

server!

mbs> open file

/data.local1/myaccount/test_files_

first=10 size=1000 -auto -rfiofile

test.lmd

rfio client

rawDispRFIO64

rfio dispach

rawServRFIO64

rfio server

fork

write data

local harddisk

widely used at standalone mbs „traveling systems“

with local disk servers for several years.


mbs> con rfio x86l-59:2345 –ltsm

X86L-10:transport :Connected RFIO

server!

mbs> open file /lustre/hebe/mbs

first=10 size=1000 -auto –rfiofile

test.lmd

rfio client

rawDispLTSM64

rfio dispach

fork

write data

tape

rawServLTSM64

rfio server

TSM server

ltsm01_test

ltsm API

lustre

ltsmsync + copytool

benefit: MBS host does not

need TSM client library

(<- not available for LynxOS,

PPC Linux etc.)

new development

for MBS v6.3

Wrote data in parallel with HADES/dabc in September 2017:

(12 files x 3 MB/s), (8 files x 5 MB/s) from HADES DAQ pulser trigger

(1 x 75 MB/s) , (2 x 40 MB/s) (4 x 18 MB/s) self-generated data

10 GbE at HADES servers, but 1 TSM server only!

MBS with via RFIO/LTSM gateway (both on x86l-59) in August 2017: single file 40 MB/s at 5.5 kHz (without file: 70MB/s at 7 kHz)

1 GbE at MBS node, single TSM server



Data writing with LTSM has been implemented for GSI major DAQ systems (MBS, HADES/dabc)

Data access from batch farm possible via transparent /lustre file Data reading/copying possible via ltsmc (or easy wrapper script?) To do:

HADES needs 4 TSM servers at 10 GbE, optimization for 2018 beamtime

Test of MBS LTSM interface software rawDispLTSM64 / rawServLTSM64

Deployment and test of LTSM gateways for MBS ? dedicated gateway machine for each experiment ? ? common gateway servers provided by GSI IT department ?


The old gstore/RFIO system will no longer be supported by GSI

Writing to RFIO from HADES DAQ has failed already since 2016

New System: Ligtweight Tivoli Storage Manager (LTSM)

Development by T.Stibor, GSI-IT (https:/github.com/tstibor/ltsm)

Write to tape archive via sockets, like RFIO (but use IBM client directly)

Transparent synchronization between tape and lustre file system

„archive“ file name is lustre file path, file is retrieved on first lustre read access

DABC plug-in for LTSM has been developed in 2017

Successfully wrote data with HADES EB in September 2017:

(12 files x 3 MB/s), (8 files x 5 MB/s) from HADES DAQ pulser trigger

(1 x 75 MB/s) , (2 x 40 MB/s) (4 x 18 MB/s) self-generated data, 1 TSM server only!

https://github.com/tstibor/ltsm





Data Acquisition Backbone Core (http://dabc.gsi.de) Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ

DABC hadaq plugin : UDP subevent receiver, event builder functionality

hld file output to RFIO and disk

EPICS interface via intermediate shared memory

writes same run info for ORACLE logbook

configurable by existing DAQ/EB scripts and setup files

QA monitoring events provided by „streamserver“ socket

HTTP server for EB monitoring and control

May optionally run pre-analysis processing (TDC calibration etc)


http://dabc.gsi.de/

DABC performance (August 2014)

29 subsystems, 25 kHz pulser, 3 EB, RFIO and local disk file

=> 38 Mbyte/s per EB process, max CPU load 78%, no event loss

=> would achieve 608 Mbyte/s with all 16 EB processes (?)

typical pion beam data rate 2014: 9 Mbyte/s at <2kHz trigger rate)

netmem/evtbuild performance (August 2011)

28 subsystems, Au-beam trigger, 8 EB, RFIO and local disk file

=> typical 17 Mbyte/s per EB process without event loss

=> overall 135 Mbyte/s


hades eb status€¦ · daq storage for hades (dabc plug-in) ltsm gateway for mbs systems summary...

Documents