lincoln bryant pascal paschos judith stephen robert
TRANSCRIPT
Data management with Rucio for XENONnT
3rd Rucio WorkshopFermilab March 10-12 2020
University of ChicagoLuca Grandi Evan ShockleyRobert GardnerJudith StephenPascal PaschosLIncoln Bryant
University Wisconsin MadisonBenedikt Riedel
Stockholm UniversityBoris BauermeisterJan Conrad
USC-ISIMats Rynge
Rice University Chris Tunnell
XENON updates: nT● New phase for dark marker search experiment at LNGS Facility (Italy) ● Expands detector sensitivity with 8 tonnes of liquid Xe.● Notable changes:
○ TPC (time projection chamber)+neutron veto+muon veto○ Increased raw data volume - up 2 times from XENON1T○ Upgraded processing and analysis pipelines:
■ Base_environment: base computing framework for nT ■ Strax: Analysis framework for nT - replaces pax+hax■ Outsource: Job submission pipeline for nT (Pegasus) - replaces portions of CAX
(Mats Rynge)■ Admix: python wrapper for Rucio uploads - replaces Ruciax
● Updates in infrastructure at UChicago and LNGS
Lessons learned from XENON1T● XENON1T operated from 2016-2018 at the Gran Sasso National
Laboratory (LNGS)○ Collected over 800 TB of raw data ○ 10 peer-reviewed publications so far by the collaboration in the search for Dark Matter
● For 1T, Rucio and job submission to OSG was not in place at the start of the experiment. Collaboration transferred data from LNGS to Uchicago for processing on local HPC (Midway)○ UChicago team (Judith Stephen, Benedikt Riedel) in collaboration with Boris
Bauermeister (University of Stockholm) deployed, operationalized and integrated Rucio Services for the XENON data management
○ In collaboration with USC-ISI (Mats Rynge) integrated job submission to OSG for processing
● Lesson Learned: Be better prepared at the start of the experiment to meet scale of compute and storage requirements
Lessons learned (continued)● For 1T, there was no local storage for the event builder. If the latter
crashed, it could take down the DAQ and stop the flow of data○ Introduced the use of a local buffer to prevent this from happening for nT
● Processing pipeline for 1T (CAX) was too ‘monolithic’ as it was in charge for all pipeline components (updating DB, processing, creating minitrees, checking transfers etc.). Data re-processing was slow and inefficient ○ Resolved by modularizing the pipeline using existing tools (e.g. Strax) repurposed for
XENON (straxen)
Rucio XENON
● Data live on disk/tape on our various Rucio Storage Endpoints (RSEs) around the world - all managed by Rucio
● Supported distribution of data during previous XENON phases● Major Updates between 1T to nT
○ store raw, reduced and processed data in Rucio■ Optional: store MC simulations for each run
○ Updated Rucio to 1.20.6, migrated mariaDB to postgresql and updated new DB to the 1.20.6 schema
○ Added an RSE for “DaLI” at UChicago Research Computing Center to augment storage for analysis
Data Pipeline XENONnT
DAQ Event Builder xent-datamanager (LNGS_USERDISK)
NIKHEF_USERDISK
CCIN2P3_USERDISK
WEIZMANN_USERDISK
CNAF_USERDISK
SURFSARA_USERDISK
UC_OSG_USERDISK
CNAF_TAPE_USERDISK
UC_DALI_USERDISK
Midway Cluster (UChicago)
Open Science Grid (OSG)+
European Grid Infrastructure (EGI)
raw_records,records, peaks, ...
● Computing Scheme for XENONnT - presented during the 2nd Rucio Workshop in Stockholm by Boris Bauermeister et al
Data processing-analysis pipeline
● Strax/straxen is the new, fast processing and analysis framework for XENONnT
● Replaces previous scheme (pax): raw data, processed data and minitrees● Strax allows for checkpointing the pipeline (state keeping)
○ Critical data objects are stored during processing stages ○ Other data objects can be recreated from the nearest stored one it depends on if a
parameter changes. This accelerates reprocessing campaigns.○ Stored data objects, e.g. raw_records, records (reduced raw) and peaks ( pulses in the
PMT - photomultiplier tubes) are ingested in and distributed by Rucio to RSE end points● Online: processing directly out of DAQ before Rucio ingestion ● Offline: Grid computing for processing ingested files in Rucio
Evan Shockley, Mats Rynge, Benedikt Riedel
Expanded scopes in catalogue for nT ● XENON1T →x1t_SR001_181129_1730_tpc:raw (both to Disk and Tape)
● XENONnT → ○ Xnt_170321_0520_tpc:corrected_areas○ Xnt_170321_0520_tpc:energy_estimates○ Xnt_170321_0520_tpc:event_basics○ xnt_170321_0520_tpc:event_positions○ Xnt_170321_0520_tpc:events○ xnt_170321_0520_tpc:peak_basics○ Xnt_170321_0520_tpc:peak_classification○ Xnt_170321_0520_tpc:peak_positions○ Xnt_170321_0520_tpc:peaks # first level of processed data of TPC○ Xnt_170321_0520_tpc:records #data-reduced raw data of TPC ○ Xnt_170321_0520_tpc:raw_records #raw data of TPC (tape)○ Xnt_170321_0520_mv:raw_records #raw data of muon veto○ Xnt_170321_0520_nv:raw_records #raw data of neutron veto
Aggressive ingestion into Rucio
Minimal ingestion into Rucio
Gateway to OSG for XENON● Grid Processing:
○ Likely needed during commissioning when gains, maps, etc. not ready
○ reprocess low-level data
○ Backup to online processing
○ Handled by Outsource, a wrapper around the workflow manager Pegasus
OSG login node
raw_records-000001
raw_records
raw_records-000000
raw_records-000002
raw_records-000003
Mats Rynge, Evan Shockley Grid Processing with Outsource
OSG login node
raw_records-000001
raw_records-000000
raw_records-000002
raw_records-000003
records-000001
records-000000
records-000002
records-000003
Rucio
raw_records records
Grid Processing with Outsource Mats Rynge, Evan Shockley
OSG login node
raw_records-000001
raw_records-000000
raw_records-000002
raw_records-000003
records-000001
records-000000
records-000002
records-000003
Rucio
raw_records records
recordspeaks
peaks Midway
Rucio
Grid Processing with Outsource Mats Rynge, Evan Shockley
aDMiX Boris Bauermeister https://advance-data-management-in-xenon.readthedocs.io/en/latest/usage.html
● advanced Data Management in XENON (aDMiX) handles Rucio API calls It allows: ○ Uploads/Downloads ○ Init Rucio transfer rules (runDB supported) ○ Update run database with RSE locations
● ADMIX will (mostly) run on xent-datamanager to upload data into Rucio and move to respective sites○ raw_records to tape storage (CNAF_TAPE_USERDISK), once we’re sure we don’t
need them anymore○ records to grid sites, e.g. UC_OSG_USERDISK, NIKHEF_USERDISK, etc. ○ High level data to analysis sites (UC_DALI_USERDISK, /dali/path/to/data)
● Middleware between Rucio and XENON ‘runs database’ that tracks information for each dataset.
Rucio Infrastructure Upgrades Judith Stephen UChicago
● Upgraded Rucio on the frontend server from 1.8.5 to 1.20.6○ Required the usual schema changes (easy via SQLAlchemy, no issues)○ Needed to move back to using the rucio-conveyor-submitter daemon instead of
the rucio-conveyor-transfer-submitter; otherwise the Rucio upgrade itself was smooth
● Migrated backend database from MariaDB 10.3 to PostgreSQL 11○ Used pgloader to perform the actual database migration
■ Due to the size of some of the tables, pgloader was very slow■ Ended up dropping some of the *_history tables in the original conversion
and eventually migrating later○ Problems with converting the various UUID columns, resolved by manually
pivoting the affected columns○ Mario helped verify the final schema (thank you!)
XENON1T data
● Raw data: keep at least a tape copy of the 1T catalogue for the extended future. Still have some time before disk space becomes an issue
● Processed data: Due to limits in project allocation on Midway, the start of nT operations will necessitate moving 1T processed data out
Summary
● Hardware and software setup is largely completed● XENON detector assembly ongoing, science data expected this year● Still to do
○ Build all services at LNGS (xent-datamanager) including the new RSE for that endpoint
○ Continue testing the full analysis software chain for any last minute bug fixes
● Questions?
strax/straxen - Data Chunking● As the data moves through the chain and various
algorithms refines the “Data chunk”● What does this mean in practice?
○ raw_records - Portions of time separated but detector inactivity for X seconds
○ records - First refinement step○ peaks - Finding PMT (photomultiplier tubes)
pulses in data stream○ peak_* - Peak information○ events - Creating a grouping peaks that are
clustered
strax/straxen - Keeping State● Various algorithms require various levels of input
data and data fidelity - Peaks can be processed on a “chunk” level, while “events” need to processed on a per-run level
● Keeps state○ e.g. when the energy_estimates algorithm
changes only produce new event_info objects○ Great for reprocessing - Less churn through
data● Comes at the cost of workflow complexity
and many tiny files
base_environment GitHub repo
CVMFS (Singularity image and source-able environment)CVMFS is a globally distributed read-only filesystem, available across Open Science Grid, EGI and Midway.
Interactive login and worker nodesSingularity images: /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-xenon:{version}Source-able environment: /cvmfs/xenon.opensciencegrid.org/releases/nT/{version}
Docker ImageDeployHQ first builds a new version of the Docker Image using the Dockerfile and create-env files.
CVMFS Tarball (source-able)The newly build Docker image and again the create-env scripts are used to also build a base_environment under /cvmfs - this can be used as a “source-able” environment just like in XENON1T
Checkins to the GitHub repo triggers DeployHQ builds
docker push ...
Container image is hosted in DockerHub (opensciencegrid/osgvo-xenon) The resulting tarball is deployed into the CVMFS server and published.
Synchronized automatically by OSGContainer image is hosted in Sylabs Cloud
(rynge/default/xenonnt)
Singularity ImageDeployHQ builds a Singularity image from the Docker image