Production of simulated Extensive Air Showers for the Pierre Auger Collaboration using Grid Technology

Julio Lozano Bahilo Dpto. de Física Teórica y del Cosmos

Universidad de Granada Granada, Spain

e-mail: julio@ugr.es

Pierre Auger Collaboration (Full author list:

http://www.auger.org/archive/authors_2012_03.html) Pierre Auger Observatory

Malargüe, Mendoza, Argentina

Abstract— The Pierre Auger Observatory detects the secondary particles generated when an Ultra High Energy Cosmic Ray (UHECR) interacts at the top of the atmosphere. Our collaboration studies the energy spectrum, chemical composition and origin of those extremely high energetic objects. The secondary particles multiply within the atmosphere creating an Extensive Air Shower (EAS). The number of particles involved in an EAS at these energies (1016-1021 eV) is of the order of billions and the generation of a single simulated EAS requires many hours of computing time with current processors. In addition, the storage space consumed by the output of one simulated EAS is, in average, of the order of hundreds of MB. Therefore we have to make use of Grid resources to be able to generate sufficient quantities of showers for our physics studies in reasonable time periods. We have developed a set of highly automated scripts written in common software scripting languages in order to deal with the high number of jobs which we have to submit regularly to the Grid. In spite of the low number of sites supporting our Virtual Organization (VO) we have reached the top spot on CPU consumption among non LHC (Large Hadron Collider) VOs within EGI (European Grid Infrastructure).

Astroparticle; PAO; Grid

I. GENERATION OF EXTENSIVE AIR SHOWERS (EAS) FOR THE PIERRE AUGER OBSERVATORY

A. The Pierre Auger Observatory The Pierre Auger Observatory [1] combines two

different techniques to infer information from Extensive Air Showers (EAS) produced by ultra-high energy cosmic rays (UHECR) entering the earth’s atmosphere. Fluorescence telescopes collect fluorescence light produced as the cascade of secondary particles composing the EAS travels through the atmosphere, thus providing information on the longitudinal development of the shower. The Observatory uses a vast array of water tanks to detect samples of particles remaining at ground level after the partial absorption of the EAS in the air. The passage of those highly energetic particles through the tanks produces the emission of Cerenkov light. Both detectors digitize light using photomultiplier tubes (PMT) and their associated electronics and transmit analog-to-digital converter (ADC) traces to the data acquisition system (DAQ).

The simulation of the events collected by the Pierre Auger detectors proceeds in two separated stages making use of very different software packages. First, we generate the EAS with a given set of parameters for the primary cosmic ray and then we simulate the detector response and perform the reconstruction of the physics information of the shower. The computing demands of each step will be discussed in the next sections. In the following one we will describe briefly an EAS and the requirements of the software programs used in order to generate them.

B. Simulation of Extensive Air Showers The energies of the primary cosmic rays we are

concerned with range roughly from 1016 to 1021 eV, thus being the most energetic particles studied by particle physics scientists. A single cosmic ray at those energies interacts soon after entering the earth’s atmosphere producing an enormous cascade of secondary particles. It has a large electromagnetic component and a smaller hadronic component that must be followed in order to characterize completely the development of the EAS. This means that the program to generate the shower must track billions of particles over tens of kilometers in the atmosphere until the particles reach ground level. An example of such an EAS generated by a UHECR is depicted in figure 1 [2].

There are several simulation packages widely used in our field in order to generate EAS: AIRES (AIRshower Extended Simulations) [3], CORSIKA (Cosmic Ray SImulations for KAscade) [4], CONEX [5] and others. Though our production framework would be easily adapted to any other shower simulation package, our collaboration has only requested the use of CORSIKA in Grid productions. These packages adopt high energy (epos [6], QGSjetII [7], …) and low energy interaction models (Fluka [8][9], Gheisha [10], …) during the tracking of the secondary shower particles. The interaction models can be selected by the user and the executables are statically linked to the corresponding libraries when they are compiled. Their execution is driven through input data files which contain parameters relevant to the set of EAS which we want to simulate: primary particle species, energy, zenith and azimuth ranges and other values changing the simulation conditions like, e.g., atmospheric related quantities. We group these sets of events into libraries. By library we mean the set of showers simulated with the same CORSIKA

2012 10th IEEE International Symposium on Parallel and Distributed Processing with Applications

978-0-7695-4701-5/12 $26.00 © 2012 IEEE

DOI 10.1109/ISPA.2012.81

545

compilation options, high energy and low energy models in particular, and the same kind of primary cosmic ray.

Figure 1. Image showing the development of a single EAS produced by a proton with an energy of 1019 eV. Most prominent particle types are shown

in different colours

The full simulation of a single EAS at the highest energies demands extremely high computing requirements. Figures 2 and 3 give a quantitative picture of the CPU time spent on a 2GHz processor and the amount of storage space needed for the output files [11].

Figure 2. Mean measured CPU times of whole shower simulations on a 2 GHz processor as a function of the primary energy. The green line shows

the result of a power law fit

It would take 1 year of CPU time and 1 TB of disk space to fully simulate a cosmic primary at energies close to 1019eV and it would be almost impossible to complete a shower at our highest energies (1021eV).

Figure 3. Mean storage space of the output files of full shower

simulations vs. the energy of the cosmic ray. The dashed line corresponds to a power law fit to the data

To alleviate both problems, we use statistical thinning techniques [12]: below a certain energy, only a subsample of secondary particles are followed, with a statistical weight equal to the inverse of the probability to survive down to ground level.

Figure 4. Top: CORSIKA average execution time as a function of the logarithm of the energy for iron and proton primary cosmic rays using epos

and QgsjetII high interaction models

Usually, the applied thinning parameter depends directly on the energy of the cosmic ray. Even if the thinning is stronger for higher energies, the CPU consumption times, as can be seen in figure 4, increase almost linearly with the logarithm of the energy of the primary cosmic ray. There is also a dependence on the type of primary so that lighter nuclei primaries consume lower amounts of time. Simulations with the epos high energy interaction model are also more time consuming. On average it takes several hours to complete a single simulation, although the spread is high.

546

Figure 5. Size in MBs of the output files as a function of the zenith angle for iron primaries (epos interaction model) at diverse energies (expressed

as log(E))

The size of the output files changes dramatically according to the primary energy, but it also has a strong dependence on the zenith angle as seen in figure 5. Sizes also depend on models and parameters so we present just the ones obtained for the primary iron and epos interaction model. The size of the output files ranges from tens to hundreds of MBs.

II. DETECTOR SIMULATION AND RECONSTRUCTION The detector simulation package, referred to as Offline

[13], is a software framework composed of independent modules which allow the user to perform any step or set of steps within the simulation and reconstruction chain. XML files contain configurable parameters to modify the behavior of the modules. The tracking of the particles traversing the water tanks, which is done with the help of the GEANT4 [14] package, is the most time demanding process.

Figure 6. CPU time of an Offline execution as a function of the logarithm

of the energy for various zenith angles

The execution times of the Offline are smaller in general than those of the shower generation program. Whereas the generation of a shower requires several hours, the processing of one of those showers takes normally less than or about an hour except for the highest energies (see figure 6). The plot shows an approximately exponential rise.

Figure 7. Total size of Offline output files as a function of the energy for

various zenith angles

The sizes of the resulting output files are also much smaller (figure 7) than the files produced by CORSIKA. The former files include information of an enormous amount of particles at ground level whereas the Offline files just contain the information collected by the detectors, some processed information, and physical shower reconstructed parameters. They increase with the energy and do not depend very much on the zenith angle of the cosmic ray.

III. GRID COMPUTING MODEL FOR AUGER OFFICIAL SIMULATIONS

The submission of jobs to Grid proceeds via a set of scripts written in Bash and Python that are running continuously at a User Interface (UI). Several instances of the scripts can be executed in parallel without interference. The main script looks for certain input files needed by our programs: CORSIKA needs input data cards and the Offline requires a list of Logical File Names (LFN) associated to the input showers. The input files are represented by the squares within the light shaded areas in figure 8. In addition to the files mentioned previously, an execution script of the corresponding software package is also sent to the Grid within each individual job. This script is automatically generated from a common template. The Job Description Language (JDL) file required by the submission command is also created with the help of a template file. The presence of those files in specific directories is checked by the management scripts (see again figure 8) and it triggers the submission of jobs to Grid Computing Elements (CE) as collections of jobs handled by the Workload Management System (WMS) services.

A set of parameters determines how the scripts behave: • frequency of job submission depending on the ratio

of running to queued jobs • number of jobs submitted within a collection • maximum number of jobs to be submitted in a

submission cycle • allowed number of aborted and failed jobs before

interrupting the main script execution

547

• etc …

Figure 8. Workflow of the Auger simulation productions

The individual jobs are submitted together as a collection of jobs until the requested number of total jobs has been reached. At this time, aborted and/or failed jobs may be resubmitted. The scripts react according to the status of the previously submitted jobs and determine whether more jobs must be sent. The status is flagged and transmitted, along with some additional information, to a database that is coupled to a PHP-enabled web server. Thus we have appropriate means to monitor constantly the progress of the production and we can also set-up public web pages with updated information. Detailed information on executed jobs is obtained from the output logs and stored in the database in order to monitor accurately consumed CPU times, output files sizes, CE and SE (Storage Element) involved, reason of the failure, if applicable, and other aspects of the job execution. Whenever possible the running job sends itself the information of the execution status and the other aforementioned parameters directly to the database.

To reduce the load of the jobs on the network we copy our software packages in the Sofware Areas (SA) of the sites, though we can also fetch the CORSIKA package from

SE where it has been previously placed. CORSIKA can be compiled statically and be copied in a single operational package. On the other hand the Offline software has many dependencies and needs a complicated and time consuming installation procedure; it has also a much bigger size. In both cases, installation jobs are submitted before any new library is started.

The experience accumulated so far has pushed us to develop mechanisms to handle the many diverse problems that a Grid user is constantly facing. To reduce the number of failed and aborted jobs we have implemented different procedures like, for example, that of avoiding the submission of jobs to CEs where the ratio of aborted versus successfully completed jobs is high. Another example is the case when a SE suffers from a downtime which affects the execution of the Offline jobs that require the retrieval of the input shower files. We are continuously monitoring their status and postpone the submission of jobs needing files on unavailable Grid SEs.

IV. PERFORMANCE OF OUR GRID PRODUCTION MODEL The Auger Virtual Organization is a relatively young VO

with access to a limited number of EGI (European Grid Infrastructure) [15] sites, including some in America, and a low number of prioritized CPUs: just about a few hundred. In spite of the continuous and diverse problems that we have encountered, the average production rate has been very good. As seen in figure 9, the daily CPU consumption presents remarkable fluctuations which are probably related to site availability, global job submission activity and various temporary problems concerning Grid services. The dip at the beginning of august of 2011 was due to construction works that affected the operation of our UI and our monitoring services. There was also a period of time when our VO Management System was unstable and it resulted in a lower average production rate. The tendency of the cumulative mean (light shaded area) indicates a stabilization of the CPU consumption since the beginning of the year 2011 reaching a current average at about 850 CPU*days. Anyway the average value for long periods without major disturbances give us a CPU consumption rate closer to 1200 daily CPUs which we hope to maintain.

Figure 9. Daily CPU and cumulative mean CPU consumption of the

Auger VO since end of 2010

548

The storage space occupied on Grid Storage Elements (SE) has grown steadily though the rate of increase depends strongly on the characteristics of the libraries produced at a given time (see figure 10). The sizes of output files for primaries other than Fe or p, usually neutrinos and photons, are smaller than those shown in figure 5. In addition, there are options that can be turned within the input cards that increase the amount of output contained in the final files. Those factors explain the variability in the slope of the consumed disk space rate.

Figure 10. Amount of disk space occupied by files corresponding to Auger

simulation productions

The average amount of disk space filled per day is about 600 GB and we had a period of a couple of months where that number reached 1.4 TB.

The continuous efforts to improve our set of automated scripts have lead us to one of the top spots in the ranking of CPU consumers of the EGI Grid. We are consistently ranked fifth only surpassed by the Large Hadron Collider (LHC) collaborations which dominate this statistic as shown in figure 11. The average percentage of CPU used by the Auger VO since the beginning of 2010 is 2%.

Figure 11. Percentage as pie chart of the top CPU consumers at EGI Grid. Auger ranks 5th

Given the numbers presented for our consumed mean cumulative CPU time (figure 8) and the mean CPU time required for a shower with energy 1019 eV and zenith angle 380 (figure 5) we can calculate that the average daily number of iron showers with these characteristics we can produce is

about 2500 though without any major problems the average rate would be of more then 3500 showers.

V. SUMMARY AND CONCLUSIONS The Pierre Auger Observatory requires high numbers of

simulated EAS which demand a very high amount of computing time and storage space. Grid is an ideal technology for accessing the needed computing resources, although it suffers from the lack of stability. This can be specially the case for small Virtual Organizations where resources are concentrated on few sites.

We have developed a simple computing model based on scripts coupled to a web server and a database which make use of Grid middleware. The automation of all job-related tasks and the development of monitoring tools have allowed us to produce hundreds of thousands of EAS to fulfill the needs of the Auger scientists. In spite of being a small team we have reached the 5th place in the CPU consumption ranking within the EGI Grid.

VI. ACKNOWLEDGMENTS We gratefully acknowledge the contribution of the sites

supporting the Auger Virtual Organization and the efforts of their administrators to maintain stable operation of their centers. We also want to commend the remarkable effort put into this initiative by the members of the production teams at the University of Granada and at the FZU Institute in Prague.

REFERENCES

[1] J. Abrahams et al. (Pierre Auger Collanoration). Properties and performance of the prototype instrument for the Pierre Auger Observatory. Nucl. Instr. and Meth. A 523, 50-95, 2004.

[2] C. Pryke. The Highest Energy Cosmic Rays and the Auger Project. Phys. Dpt. Colloquium, Univ. of Wisconsin (Madison), 1998. http://find.spa.umn.edu/~pryke/publ.html

[3] S. J. Sciutto. The AIRES system for air shower simulations. An update. (astro-ph/0106044, 27th ICRC Hamburg, August 7-15, 2001).

[4] D. Heck, J. Knapp, J.N. Capdevielle, G. Schatz, T. Thouw. CORSIKA: A Monte Carlo Code to Simulate Extensive Air Showers. Forschungszentrum Karlsruhe Report FZKA 6019, 1998.

[5] T. Bergmann et al. . One-dimensional hybrid approach to exensive air shower simulation. Astroparticle Physics 26 420–432, 2007.

[6] K. Werner, F. M. Liu and T. Pierog, Phys. Rev. C 74 (2006) 044902 [7] S.S. Ostapchenko, Nucl. Phys. B (Proc. Suppl.) 151 (2006) 143- 147;

Phys. Rev. D 74 (2006) 014026 [8] G. Battistoni, S. Muraro, P.R. Sala, F. Cerutti, A. Ferrari, S. Roesler,

A. Fassó, J. Ranft, The FLUKA code: Description and benchmarking. Proceedings of the Hadronic Shower Simulation Workshop 2006, Fermilab 6--8 September 2006, M. Albrow, R. Raja eds., AIP Conference Proceeding 896, 31-49, (2007)

[9] A. Ferrari, P.R. Sala, A. Fassó, and J. Ranft. FLUKA: a multi-particle transport code. CERN-2005-10 (2005), INFN/TC_05/11, SLAC-R-773

[10] H. Fesefeldt, Report PITHA-85/02 (1985), RWTH Aachen

549

[11] F. Schmidt, J. Knapp and M. Zha. Results from the Simulation of an Unthinned Proton Shower at 5×1018 eV. Auger GAP note 2005-095

[12] A.M. Hillas, Proc. 19th ICRC, La Jolla, USA, 1(1985)155

[13] Argiro, S. et al. . The offline software framework of the Pierre Auger Observatory. Nucl. Instr. Meth., A 580, 1485–1496, 2007.

[14] S. Agostinelli et al. . Geant4 - A Simulation Toolkit. Nucl. Instr. Meth. A 506 250-303, 2003.

[15] EGI, European Grid Infrastructure. http://www.egi.eu/

550