the atlas software distribution christian arnaultalessandro de salvosimon georgegrigori rybkine...
TRANSCRIPT
![Page 1: The Atlas Software Distribution Christian ArnaultAlessandro De SalvoSimon GeorgeGrigori Rybkine arnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.itarnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.it](https://reader036.vdocument.in/reader036/viewer/2022083004/56649e025503460f94aec0bd/html5/thumbnails/1.jpg)
The Atlas Software Distribution
Christian Arnault Alessandro De Salvo Simon George Grigori [email protected] [email protected] [email protected] [email protected]
CVS
CMT
Work area
Nightly build
PacmancacheIntegration
tests
Grid CE Laptop
cmt co
cmt co
cmt co
cvs cicvs tag
Kit construction
pacman -getpacman -get
Tagcollector
Prod releaseProd releaseProd releaseProd release
Prod releaseProd releaseProd release
Prod release
Kit validation DC job Work area
cmt co
•Packages are first created in a work area, registered to the TagCollector and imported to the CVS repository•Packages are checked out to work area(s)•Tags are declared to the TagCollector, registered for the next release•Nightly builds take all registered tags for the next release•Quality checks and Integration tests results are reported to developers (through Web pages)•Production releases take validated tags•Distribution kit is automatically constructed from production releases•Release kits are installed on Grid CE and laptops•KitValidation jobs are run after installation•Users may develop packages against installed kits
Register packageregister tag
The software flow
• Individual packages are tagged randomly•Tags can be submitted to the Tag Collector for the next release until it is locked•Every day the last registered tags are selected for the nightly build•Every three weeks the tag collection is locked during the release build (between the last nightly and the end of the release validation)•Tags for core packages are locked one week before the release build•Twice a year a major release is produced (global tag M.0.0)•Patch releases may be created after any minor or major release (global tag M.m.p)
8.2.0
days
weeks
Any scale
Development:tagging
Tagcollector
Tag registering
Nightly buildTag selection
Core tags lock All tags lockMinor release cycle
months
Major release cycle
A-00-01-07 A-01-00-03
8.1.08.3.0
8.0.0 9.0.0 10.0.0
The software life cycle
Atlas packages
Gaudi packages
LCGCMT interfaces CMT packages
Software
Interfaces
Policy
Containers Atlas External software•Generators•Local installations of other software
•Geant4•Java suites
LCG software•SEAL•POOL•PI•…
HEP or Free software•Root•MySQL•…
Atlas generic data files
•Managed by CMT•Three sub-projects Atlas, Gaudi and LCGCMT•Each sub-project has its own global version (the release tag)
•Atlas selects the release of Gaudi•Gaudi selects the release of LCGCMT
•Each release of each sub-project selects the set of versions of individual packages•Both Atlas and Gaudi define their specific centralized installation area
•Binaries•Include files•jobOptions•Scripts (shell, python)•Java packages
The software structure
Container packages:•Purely managerial packages•Not involved when using the software•Only provide a CMT requirements file with a set of explicit use statements (no wild card)•Purpose:
•Specifies the set of exact version for a release of the software group•Drive the cmt broadcast commands over the complete software group
•The top container is AtlasRelease and describes the complete Atlas software release•Every new release of the Atlas software yields a new release of the AtlasRelease container.
Policy packages:•Provide policy statements for a software domain
•Include dir strategy•CMT strategies•CMT patterns•Generic symbols (macros, sets, …)
•AtlasPolicy is the main policy package, mandatory for all Atlas packages.•Examples: ExternalPolicy, AtlasCxxPolicy, AtlasFortranPolicy
Interface packages:•Provide an interface between a software project managed by CMT and a so-called external product (not managed by CMT)•Specify the native version tag defined by the product (as opposed with the CMT version tag)
•Macro <package>_native_version•Adapt the policies and conventions to the project’s ones
•Eg Include directory•Construct the compiler and linker options for this product•Set the runtime options (PATH, LD_LIBRARY_PATH, etc…)•Specify the material to export when constructing a distribution kit•Examples
•MySQL•ROOT•SEAL•Geant4•Java kits
•The LCGCMT project is entirely made of such interface packages•Connect to LCG internal (SEAL, POOL, PI, …) or external products
Package semantics
•http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/ The atlas software page•http://isnpx0168.in2p3.fr/athena/ The tag collector•http://www.cmtsite.org CMT
Links•Run quality checks•Run unit and integration tests•Generate documentation•Operate on fast local disks
•Then transferred to AFS
Alex Undrus
NICOS: the build driver
Jérôme Fulachier – Johann Collot The Tag Collector
•Control the consistency of package versions assigned to any release•Can be locked to prepare the release•Support branch release management
•Bug fixes•Late patches
•Directly connected to CVS•Perform cross checks with CMT settings•Handle the container hierarchy
![Page 2: The Atlas Software Distribution Christian ArnaultAlessandro De SalvoSimon GeorgeGrigori Rybkine arnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.itarnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.it](https://reader036.vdocument.in/reader036/viewer/2022083004/56649e025503460f94aec0bd/html5/thumbnails/2.jpg)
The Atlas Software Distribution
Christian Arnault Alessandro De Salvo Simon George Grigori [email protected] [email protected] [email protected] [email protected]
CMT statement Pacman statement Location in the tar ball
Use depends
application InstallArea/$(bin)/lib
Library InstallArea/$(bin)/bin
Jar InstallArea/share/lib
Special macros
<package>_runtime InstallArea/share
<package>_joboptions
InstallArea/jobOptions/<package>
<package>_xmls InstallArea/XML/<package>
<package>_scripts InstallArea/share/bin
include_dirs InstallArea/include/<package>
Whole cmt directory <o>/<p>/<v>/cmt/...•All specifications to create the kit for one package are uniquely extracted from the requirements file of this package•CMT Use statements produce Pacman depends statements
•The entire dependency graph is preserved•Dependency cycles are filtered out
•Constituent definitions populate the tar ball from the binary installation area•Conventional macros describe special files to bring to the tar ball (relocated to the installation area)
•Runtime files•Gaudi jobOptions•Scripts•XML files•Header files
•The cmt directory and its supporting directory tree is copied into the tar ball to permit configuration activity against the installed kit•The generated tar ball is relocatable
description='Package IdDictParser (v1)'
source = '../kits/8.7.0'download = { '*':'IdDictParser-opt.tar.gz' }
depends = [ 'dist-8.7.0-opt', '8.7.0/AtlasPolicy-opt' , '8.7.0/XMLCoreParser-opt' , '8.7.0/IdDict-opt' , '8.7.0/Identifier-opt' ]
kits/<project>/<release>/<package>-<platform>.tar.gz
cache/<project>/<release>/<package>-<platform>.pacman
cache/dist/8.7.0/IdDictParser-opt.pacman
kits/dist/8.7.0/IdDictParser-opt.tar.gz
description='Package IdDictParser (v1)'
source = '../kits/8.7.0'download = { '*':'IdDictParser-opt.tar.gz' }
depends = [ 'dist-8.7.0-opt', '8.7.0/AtlasPolicy-opt' , '8.7.0/XMLCoreParser-opt' , '8.7.0/IdDict-opt' , '8.7.0/Identifier-opt' ]
cache
•Manifest files describe packages•Version•Link to the contents tar ball•Describe dependencies•Specific installation procedure•Contribution to the global post-install setup procedure•Specifies and checks the site configuration
•A cache is a web-visible disk space•Packages may be found in different caches•Caches may be virtual or mirrored (snapshots)
•Installation transitively and recursively fetches packages from caches
•Apply the installation procedures•Generate setup scripts
Saul [email protected]
http://physics.bu.edu/pacman/
cache
cache
Pacman –get ATLAS:8.7.0/IdDictParser-opt
Saul Youssef
Pacman From the requirements to the pacman kit
package Expat
use LCG_Settings v*
macro Expat_home "$(LCG_external)/expat/$(Expat_native_version)/$(LCG_system)"
macro Expat_native_version "1.95.5"macro Expat_export_paths "$(Expat_home)/lib/libexpat.so $(Expat_home)/include/expat.h“
include_path noneinclude_dirs $(Expat_home)/include
macro Expat_linkopts "-L$(Expat_home)/Libs -lexpat" \ WIN32 "/LIBPATH:$(Expat_home)/Libs libexpat.lib"
path_remove LD_LIBRARY_PATH "/expat/" WIN32 ""path_prepend LD_LIBRARY_PATH "$(Expat_home)/Libs" WIN32 ""path_remove PATH "" WIN32 "\expat\"path_prepend PATH "" WIN32 "$(Expat_home)/Libs"
•An external package is described in a CMT Interface package•is characterized by two special macros
<package>_native_version<package>_export_paths
description='Package Expat v1 opt in project=LCGCMT/LCGCMT_26_2c (v1)'
url = 'http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/Development/'source = '../kits/LCGCMT_26_2c'download = { '*':'Expat-opt.tar.gz' }
depends = [ 'LCGCMT-LCGCMT_26_2c-opt', 'LCGCMT_26_2c/LCG_Settings-opt' , 'Expat-1.95.5-opt' ]install = { '*': ['. $PACMAN_INSTALLATION/CMT/*/mgr/setup.sh; cd $PACMAN_INSTALLATION/LCGCMT/LCGCMT_26_2c/LCG_Interfaces/Expat/cmt; cmt -quiet -no_cleanup config; a=1'] }
•The tar ball only contains the CMT directorykits/<project>/<package>-<platform>.tar.gzkits/LCGCMT_26_2c/Expat-opt.tar.gz
Cache/<project>/<package>-<platform>.pacmanCache/LCGCMT_26_2c/Expat-opt.pacman
•The tar ball contains the real package contentskits/<package>-<native_version>-<platform>.tar.gzkits/Expat-1.95.5-opt.tar.gz
description='External Package Expat v1 opt (v2)'
url = 'http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/Development/'source = '../kits'download = { '*':'Expat-1.95.5-opt.tar.gz' }
Cache/<package>-<native_version>-<platform>.pacmanCache/Expat-1.95.5-opt.pacman
•The pacman kit for the physical contents•Independent of CMT•Fully standalone kit
•The pacman kit for the interface package•Related with CMT, •and to the top project (e.g. LCGCMT)
The special case of external packages
Creating the kit:•Define CMTPATH for the subproject•Select a cache directory•Select a platform tag•Launch a cmt broadcast loop over the release package
•Will reach all packages of the sub-project•Run the cmtpacman/scripts/create_kit.sh shell script
Installing the kit:•Select an installation location•Select a release•Run pacman
Setup the runtime environment:•Source the setup script generated by pacman•Source the setup script for the specific application•Run athena straight from the installed kit
Develop against an installed kit:•Setup the runtime environment•Select a work area•Checkout a package using client CVS•Build it and run it
> cmt broadcast $CMTROOT/../cmtpacman/scripts/create_kit.sh -platform opt /afs/cern.ch/…/pacman
> cd /myinstall> pacman –get ATLAS:8.7.0/AtlasRelease-opt
> source setup.sh> source dist/8.7.0/Control/AthenaRunTime/*/cmt/setup.sh
> cd /myworkarea> export CMTPATH=`pwd`:$CMTPATH> cmt co DetectorDescription/IdDictParser> cd DetectorDescription/IdDictParser/*/cmt> gmake> …
•Currently we only produce development kits•Source kits can also be produced
•Development against the installed kit implies getting the source through CVS
•Pacman 3 permits to install several releases in a common location•External packages are shared between releases
Operations
•One of the Atlas packages•Available from pacman too
•Provide scripts to automate the post-install kit validation•Structured as a set of transformation scripts
Transform data from a state to another state (generation, simulation, etc…)•Launch athena.py over existing job options (picked up from the installed release)•Run various configuration tests
•Checking of DLLs•Test the development scenario (building against the installed kit)
•Can generate dedicated data files or job options•Construct job parameters
•IO filenames•Random seeds•Number of events
•Provide a more general framework of batch job submission•Data Challenges productions•Grid job submissions
•Format and capture the job output•Can describe the PASS/FAIL conditions in terms of patterns (regular expressions)•Filter PASS/FAILURE conditions according to the patterns•Generate a consolidated report
•Can be formatted as a web page (or even Wiki page)
ALL:SimpleChecks.kvt:Release Simple ChecksALL:AthExHelloWorld.kvt:Athena Hello WorldALL:AthenaPoolTest.kvt:Athena POOL TestALL:InDetDetDescrExample.kvt:InDetDetDescr ExampleALL:MooEventCompile.kvt:MooEvent compilationALL:EvgenMuon.kvt:Athena Muon GenerationALL:DC2EvgenZee.kvt:DC2 Z -> e e Event GenerationALL:DC2EvgenZmm.kvt:DC2 Z -> mu mu Event GenerationALL:DC2EvgenZtt.kvt:DC2 Z -> tau tau Event GenerationALL:DC2EvgenJJ.kvt:DC2 DiJet Event GenerationALL:DC2EvgenMinBias.kvt:DC2 Minimum Bias Event GenerationALL:DC2EvgenSusy.kvt:DC2 Susy Event GenerationALL:DC2EvgenTop.kvt:DC2 Top Event GenerationALL:DC2EvgenJetCalib.kvt:DC2 Jet Calibration Event GenerationALL:DC2EvgenJetFilter.kvt:DC2 Jet (pt>17) Event GenerationALL:DC2AtlfastZee.kvt:DC2 atlfast Z -> e e Ntuple/Histo productionALL:DC2G4simZee.kvt:DC2 Z -> e e G4 SimulationALL:DC2G4digitZee.kvt:DC2 Z -> e e DigitizationALL:DC2reconZee.kvt:DC2 Z -> e e Reconstruction
Kit validation