high energy and nuclear physics white paper for the … · high energy and nuclear physics white...

High Energy and Nuclear Physics White Paperfor the

DOEÕs Science Simulation Initiative

Executive Summary 2

1. Introduction 4

2. Accelerator Science 6

(A) HIGH PERFORMANCE COMPUTING AND ACCELERATORS 6

(B) SCIENTIFIC AND TECHNICAL MERIT 6

(C) READINESS FOR TERA-SCALE COMPUTING 8

(D) BENEFITS TO AGENCY MISSIONS 12

(E) IMPACT ON OTHER SCIENTIFIC DISCIPLINES 13

3. Theoretical Physics 15

(A) HIGH PERFORMANCE COMPUTING AND THEORETICAL PHYSICS 15

(B) HIGH ENERGY AND NUCLEAR PHYSICS 15

(C) COMPUTATIONAL COSMOLOGY 17

(D) GRAVITATIONAL PHYSICS 18

(E) CONNECTIONS 18

(F) READINESS 19

4. Experimental High Energy and Nuclear Physics 20

(A) DATA MANAGEMENT IN HIGH ENERGY AND NUCLEAR PHYSICS 20

(B) BRIEF HISTORY OF DATA MANAGEMENT IN HENP 21

(C) CURRENT AND FUTURE CHALLENGES IN DATA MANAGEMENT 22

(D) REQUIRED RESEARCH AND DEVELOPMENT IN HENP DATA MANAGEMENT 25

(E) BENEFITS OF THE PROPOSED INITIATIVE FOR HENP AND OTHER SCIENCE 27

5. Conclusions 28

High Energy and Nuclear Physics White Paper for the DOE's Science Simulation Initiative

2

High Energy and Nuclear Physics White Paperfor the

DOEÕs Science Simulation Initiative

Executive Summary

The Vice-President has recently announced a major new initiativeÑInformation Technology for theTwenty-First Century (IT2). The DOE Science Simulation Initiative (SSI) is to be an important partof that initiative. It offers significant opportunities to advance science and technology throughselective access to tera-scale computing and an associated national infrastructure. This paperdescribes the advances that would be possible in three central elements of the High Energy andNuclear Physics (HENP) programs: accelerator science, theoretical physics, and large-scale datamanagement. It also describes the contributions that the HENP program could make to SSI.

Quantum mechanics and elementary particle physics describe the most fundamental aspects of thephysical world. They pose questions that can often only be studied by experimental measurementsand computations, involving massive amounts of simulated and measured data. The field has anenviable record in terms of fundamental discoveries and award of Nobel prizes; it has remainedattractive in recruiting the best young research minds. The field has learned how to work effectivelyin large international collaborations that have succeeded in building and operating the sophisticateddetector systems and in the analysis of the data using high-end information technology. Theunderstanding of accelerators at their most basic level and the design of new accelerator systemshave also pushed the boundaries of computational science through the Grand Challenge process.

Opportunities exist today to build on the proven scientific and technical record of high energy andnuclear physics. Tera-scale computing, together with the associated national computinginfrastructure, must be accessible and effectively coupled to the basic research and developmententerprise in general, but for our field, in particular, we focus on:

• Particle accelerators,

• Theoretical computation, and

• Data management and visualization for the massive volumes of complex data fromcurrent and future experimental facilities.

Accelerator science and charged-particle accelerators are now fundamental tools, and targets forfuture investment, in all four of the DOEÕs mission areas: science, energy, national security, andenvironmental restoration. The SSI offers dramatically enhanced levels of performance andimproved capability for the accelerator modeling and simulation that is critical for the design andunderstanding of modern accelerators. The promise is that of ÒASCI-likeÓ simulations based onexperience and experiments. The computational accelerator physics community is already involvedin large-scale parallel computing through the DOE Grand Challenge in Computational AcceleratorPhysics, and would be able to make immediate use of the tera-scale resources of the SSI. Two setsof accelerator applications, designed and optimized for the tera-scale computing environment,would be developed. These programs would have features that would facilitate easy use; they wouldbe benchmarked against other codes, and they would be verified by experiments. One set, thoseworking on a microscopic scale, would focus on the performance of individual components.Electromagnetic fields and particle motion would be calculated at the level of detail needed to


3

optimize designs and to understand limiting phenomena, such as electrical breakdown. The otherset, working on the accelerator system scale, would bring together the complex nonlinear andintensity-dependent effects that determine the overall performance of an accelerator. Thesesimulations are not possible without the tera-scale computing offered by SSI. The potential benefitspromise to be immense; from the design and cost impact on future high-energy and nuclear physicsaccelerators, synchrotron light sources, and neutron sources, to future x-ray sources, lasers andplasma-based accelerators, to speculative areas of new accelerator applicationsÑmany withpotential economic, medical, environmental and national security impact, in addition to the basicscience arena.

The drive to make progress in quantum chromo-dynamics (QCD), the theory of the structure ofmatter, has already motivated the design, construction and use of tera-scale machines. Techniqueshave already been developed to bring QCD simulations to a new level of sophistication and to thepoint where it is clear that tera-scale computers will have decisive impact. It is now possible tocalculate a number of important quantities in HENP theoretical physics to an accuracy comparablewith their expected experimental determination. This area of computation is already active in high-end computing and would make immediate use of access to SSI-scale resources. To fully exploit thelarge investment in experiments exploring QCD, the quark-gluon plasma, and the precision tests ofthe Standard Model, it is important to also take the (modest) step to push the theoreticalcomputations to the SSI tera-scale level! The extensive experience in supercomputing of the QCDcommunity could be an important resource for SSI, together with the potentially close couplingsbetween many of the theoretical calculations and the simulations of strongly correlated electronsystems (high temperature superconductors, itinerant magnetism and condensed-matter physics).

Finally, the management and analysis of HENP experimental data is not usually regarded as asupercomputing area, but it does require a substantial advance in the computer science of datamanagement to meet future needs. We must exploit tera-scale computational tools and manage dataon an unprecedented scale to meet the challenges posed by the experimental programs to which ourfield is committed. Data management is so integral to progress in experimental high energy andnuclear physics that it is quite proper to treat it as part of the basic science. Nevertheless, as datavolumes grow in other sciences, they too will be driven to compress data into vast complexdatabases now typical in high energy and nuclear physics. It is already clear that within SSI, datamanagement will become a major component of CSET, the associated infrastructure required fortera-scale computing. The high energy and nuclear physics data management effort will contributeto, and benefit from, integration within the CSET infrastructure of SSI.

The Strategic Simulation Initiative, SSI, presents an exciting opportunity to join with a widercommunity, working at the limits of information science and technology, to leverage the particlephysics investment and achieve revolutionary advances. The HENP community has become adept atworking within very large international collaborations to build and to analyze their data, and tocollegially interact at a distance using advanced information technology. Indeed, this is thecommunity that originated the World Wide Web. We look forward to playing a similarly importantrole within SSI. In the coming years SSI may reshape how our accelerators are designed andcommissioned, new experiments are designed and carried out, and expand the applicable boundariesof our theoretical explorationsÑmaking high-end computational simulation a new tool fordiscovery in the 21st century!


4

1. Introduction

The High Energy and Nuclear Physics science programs are world leaders in fundamentalscience. These programs have a history of important discoveries, and the number of NobelPrizes measures their recognition. Computer science, computer technology, and computerusage are critical components of these programs. Accelerators and detectors could not bedesigned and operated without extensive simulation and computer control, and manytheoretical studies rely on simulations using high performance computing. In addition, HighEnergy and Nuclear Physics (HENP) are the scientific leaders today in large-scale scientificdata management, and there is a well-established record of large internationalcollaborations effectively working together via Information Technology. The World WideWeb was developed by high energy physicists in the 1990s to facilitate such collaboration.

The HENP programs of DOE/SC are proposing to bring their expertise to the new DOEStrategic Simulation Initiative (SSI) in two complementary modes. The first mode would befocussed on the broad, crosscutting computing infrastructure needed to support the overallSSI. The HENP programs would be investing heavily, for their own science activities, indeveloping large-scale data management and networking tools to support large, effective,worldwide collaborations of scientists. With these investments they will have appropriatecredentials and will be in a strong position to be accepted as partners in the infrastructureaspect of SSI (i.e. the CSET program).

The second mode for participating in SSI is the use of high-end computational facilities tofurther the HENP science program itself. This paper concentrates on the science programwhere the emphasis is threefold: (1) a new venture to bring high-end computation toaccelerator simulation, (2) new opportunities in theory, and (3) support of major newexperiments in handling multi-terabyte-scale data samples.

Particle accelerators are fundamental tools in all four of the DOEÕs mission areas. A majorDOE role in basic science is the building and operating of large accelerator facilities for thenationÕs science community in universities, national laboratories, and industry. Thecomputational capability of SSI offers the opportunity to revolutionize our approach toaccelerator design by allowing simulations that approach the level of a complete acceleratorsystem. This promises significant cost reduction for the design and construction of futureaccelerators, and it promises the opening of new applications in material science, biology,and medicine.

Quantum chromodynamics, precision tests of the Standard model, the quark-gluon plasma,and string theory are important areas of theoretical physics research where significantadvances can be made with the computational power offered by SSI. These studies will alsoprovide important benefits to other aspects of SSI. Members of the HENP theorycommunity have been at the forefront of supercomputing. The algorithm and softwareproblems that they encounter have considerable overlap with those faced in simulations ofclimate, combustion, fusion, and materials science. There are opportunities for criticalcross-fertilization among many fields of science.

Answering fundamental questions at the frontiers of science is a key component of theDOEÕs strategic mission. New experimental facilities that are at the very heart of researchin high energy and nuclear physics are coming on line in the next two years. These include


5

BABAR at the SLAC B Factory, the four new experiments at BNLÕs Relativistic Heavy IonCollider (RHIC), and the upgraded CDF and D0 experiments at FNAL. The new detectorswill produce unprecedented amounts of data. There will be terabytes of data per year in theshort term and petabytes per year when the LHC experiments begin their science programs.This program requires development of new computer science tools to manage the hugeamount of data. Data management, as a science and as a tool, is critical for the next stage ofHENP! The goal is to put in place a system that provides data management intelligence tonetworked computers so they can efficiently and effectively manage this volume of dataand allow scientists to do in hours what currently takes months or years to accomplish.Such advanced new computing tools will be broadly useful to the whole of the SSIcommunity.

Beyond the direct needs of the individual experiments, the success of high priority DOEfacilities also depends critically on the timely availability of enhanced computation,networking and simulation capabilities. Similar capabilities are required in other fields, andprogress in addressing their implementation within HENP facilities will have valuablebenefits across the agencyÕs science portfolio. In that sense, the participation of HENP,broadly, within SSI will be very beneficial.

The following sections will describe the vision for computational opportunities in the threefocus areas of the HENP program: accelerator science, theoretical physics, and datamanagement.


6

2. Accelerator Science

(A) HIGH PERFORMANCE COMPUTING AND ACCELERATORS

Charged-particle accelerators are fundamental tools used for a broad spectrum of importantresearch and development in all four of the DOEÕs mission areas: science, energy, nationalsecurity, and environmental restoration. In the science programs they are central to a largefraction of the work that includes the giant accelerators of the High Energy and NuclearPhysics Program; the synchrotron light sources and the spallation neutron sources of BasicEnergy Sciences, and Biological and Environmental Sciences; and the Fusion Sciences. Inthe energy area there are the neutral beam injectors of the plasma fusion program and theheavy ion drivers of the inertial confinement fusion program. In the national security areathere are the neutron sources, flash radiographic systems and pulsed-power systems. In theenvironmental restoration area, increasing use is being made of the synchrotron lightsources, and accelerators for transmutation of nuclear waste are under consideration. Thereare many other applications of accelerators and accelerator science that extend far beyondthe broad mission of the DOE. These include such things as electron microscopy, protonmicroprobes, charged-particle beam lithography, ion implantation, medical isotopeproduction, radiation therapy, x-ray lithography, free-electron lasers, etc.

Computer modeling and simulation have been a key element in the design andunderstanding of all modern accelerators. Computer programs have been written and usedto study a wide variety of problems ranging from microwave component design to the long-term stability of particles in nonlinear magnetic fields. This modeling has been assophisticated and extensive as the available computers would support, and has allowed thedevelopment of far more complex and innovative accelerators of all energies as computercapabilities have increased. The interplay between simulation, experiment, and theory hasbeen of great importance. Together they have provided a framework for calculation, design,verification, discovery, and understanding.

The Strategic Simulation Initiative (SSI) offers dramatically new levels of performance andcapability for accelerator modeling and simulation which could qualitatively change thecomplexity of the physics incorporated and greatly extend the scale of the problems studied.Components could be designed with significantly improved performance and costeffectiveness, and accelerator simulations could approach the level of a completeaccelerator system. SSI could result in significant cost reduction for present and futureparticle accelerators for high energy and nuclear physics and could also open up newapplications in material science, biology and medicine. The following sections provide adescription of the new approaches and ideas for accelerator simulation that would bepossible with SSI.

(B) SCIENTIFIC AND TECHNICAL MERIT

Particle accelerators are complex systems with performance-determining phenomenacovering a wide range of scales. One extreme is the microscopic scale where particle beamsand self-consistent electromagnetic fields propagate and interact in complex three-dimensional structures. The other extreme is the accelerator system scale. In this case oneis interested in beam dynamics and performance in an accelerator system that combines


7

particle beams, beam-induced and externally-applied electromagnetic fields, and feedbacksystems.

The design and understanding of accelerators would enter a new era with SSI resources andwith the development of two sets of program modules, one for the microscopic scale andone for the system scale. These simulation tools would be designed and optimized for thetera-scale-computing environment. Programs would allow user selection of physics, andthere would be a growth path and scripting language to facilitate use. They would bebenchmarked against other codes and would be verified with experiments at existingfacilities. This library of modules would be available to accelerator scientists and engineerswith problems that require tera-scale computing and would be supported by documentationand a suite of solved problems for effective and efficient learning by users. Once available,such a proven, documented simulation library would allow the study of critical acceleratorproblems that must be solved and high performance, cost effective accelerators could bedesigned and built for a wide variety of sciences.

On the microscopic level, electromagnetic fields and particle motion could be calculatedwith such precision and detail that large components with demanding requirements could bedesigned with confidence. Ongoing work that has been funded by the DOE GrandChallenge is described below in the section on ÒReadiness for Tera-Scale Computing.Ó

At the present time the development of high power RF sources, which are majorcomponents and costs of future linear accelerators, involves iteration between computermodeling and experiments with prototypes. With improved modeling, new concepts and theimpact of cost saving manufacturing steps could be considered without extensiveprototyping. The empirical understanding of accelerating-gradient limitations from vacuumelectronic phenomena, such as breakdown and field emission, could advance tofundamental understanding of the underlying science. The simulation of plasma-basedaccelerators could open up the possibility of compact devices providing GeV-energyelectrons in ultrashort bursts that might find wide application in materials studies. Suchmodeling and simulations on the microscopic scale would open up new opportunities andbring significant advances in accelerator performance and cost effectiveness.

On the accelerator system scale a broad range of physical effects could be combined into asingle simulation. We know from operating accelerators that performance limits come fromthe combination of nonlinear single-particle dynamics, beam-intensity dependent fields anddynamics, stochastic effects, and external systems such as feedback. Simulations have beenwritten to study all of these effects independent of one another, but the simulation of thecombination over the full range of interesting parameters is beyond the capability of presentday computers.

The detailed design of a large electron-positron linear collider (NLC) for high-energyphysics is now underway. It is the most complex accelerator ever planned. Micron-sizedbeams need to be tuned in the presence of very strong wakefields, beam jitter from pulse-to-pulse and coupled feedback systems stabilizing the entire system. The design andimplementation of this complete system would benefit from a code which reflects the fullcomplexity of the system.

The heart of a possible muon collider, also for high energy physics, would contain acooling system that must reduce the phase space volume of a muon beam produced by pion


8

decay. This reduction must be a factor of 105Ð106, and it must be accomplished in a timethat is short compared to the muon lifetime. So far the only possible method foreseen isionization cooling. The energy lost to ionization when passing through a material is restoredby acceleration; however, muons passing through material are also ÒheatedÓ by multiplescattering. Study of ionization cooling requires computationally demanding simulationsbecause nonlinear and stochastic effects dominate. Designing a muon cooling systemrequires tera-scale computers and the simulation library described above.

Linac-based free electron lasers are being developed for high power production of infraredradiation and as a source of coherent x-rays for material science and biology. Theirperformance depends critically on beam quality, and they share many accelerator physicsand systems issues with linear colliders. One outstanding issue is the effect of coherentsynchrotron radiation in the bunch compressors where the bunch length is shortened.Calculation of coherent synchrotron radiation requires careful attention to retardationeffects and relativistic dynamics. This problem has been studied with NERSC resources,and the scaling to tera-scale computers is discussed below.

In high intensity proton linacs the motion of the beam core together with nonlinear space-charge forces generate halos that must be controlled to avoid irradiating the acceleratorstructures. Similarly, halos and losses must be minimized in the accumulator ring of theSpallation Neutron Source where particles are injected continuously for roughly 1 msec andthen extracted in less than 1 µsec. The ring must have a large aperture (with nonlinear fieldsnear the edges of the beam) for continuous injection, feedback systems must track thecontinuously changing current, and, at the end of the injection cycle, the ring must handle avery high current. This combined system can only be studied with the high performancecomputing provided by SSI and a proven set of simulation tools.

(C) READINESS FOR TERA-SCALE COMPUTING

The computational accelerator physics community is already involved in large-scaleparallel computing through the DOE Grand Challenge in Computational AcceleratorPhysics, and will be able to make immediate use of the tera-scale resources provided bySSI. This Grand Challenge, initiated in 1997, is a collaborative effort involving LANL,SLAC, UCLA, and Stanford, together with two High Performance Computing (HPC)centers, the National Energy Research Scientific Computing Center (NERSC) and theAdvanced Computing Laboratory (ACL). The primary goal of this project is to develop anew generation of accelerator modeling tools for HPC platforms, and to apply them tocomplex problems of importance to future accelerators. This experience can now be builtupon to launch accelerator simulations and thereby provide the accelerator community witha badly needed resource to confront a diverse range of challenging problems and projects.

Accelerator System SimulationsAn accelerator system simulation library must be built upon a sophisticated, industrial-strength, flexible and expandable, single-particle dynamics code. An internationalcollaboration addressing the problem of building a class library for accelerator systemsimulation and control agreed on the features for such a single-particle code and in the lastyears two such codes have been created, one in the US and one in Europe. Each has beenbenchmarked against previous codes and has already been used and proven in major latticedesign efforts. These object-oriented codes separate the physical description of elements


9

from their effect on particle trajectories, offer a wide choice of integrators, contain a fullimplementation of the 3-D Euclidean group for element placement (and misplacement),support families and strings (e.g. elements all on one power supply) and model errors of alltypes. They calculate maps and can replace an element or set of elements by those maps,have weak-strong head-on and parasitic beam-beam elements, and include effects ofsynchrotron radiation and quantum excitation. These codes could be easily enhanced toinclude additional single-particle effects including, for example, the passage of particlesthrough materials as required for the ionization cooling of muons for a muon collider.

By assigning different particles to different processors, such codes can immediately beready to study long-term dynamic apertures of large hadron machines and to search forgood working points in electron-positron colliders. The latter application requirescomputation of a self-consistent beam-beam shape, but remains loosely coupled. Theseapplications could profitably use tera-flop power and would immediately break importantnew simulation ground.

The first major programming effort would create modules to implement two general typesof collective effects: interparticle forces such as space-charge forces, and forces arisingfrom the interaction of the beam with the environment. In the latter the beam drives currentsin the beam environment, which in turn create fields acting back on the particles in thesame bunch and/or subsequent bunches.

Space-charge forces have been the subject of the current accelerator physics DOE GrandChallenge and are ready for tera-scale computing. Split-operator techniques are used toseparately include the space-charge self-field and externally applied fields. As appropriate,2-D or 3-D particle-in-cell codes with area weighting and open boundary conditions areemployed. Charge deposition and field interpolation are parallelized, and a particle manageris used to arrange the data needed by processors prior to charge deposition and fieldinterpolation. Stochastic corrections arising from noise and intra-beam scattering have beenadded.

Simulations of strong-strong beam-beam effects are based on similar concepts and providea fascinating range of dynamic behaviors. There are two distinct types of strong-strongbeam-beam situations: the large-disruption regime of linear colliders and the small-disruption regime of circular colliders. In each the beam is sliced and slices advancethrough one another. At each time step in the strong-strong simulation, the fields must becalculated based on the charge distribution at that time, again by placing particles on a gridand using parallelized Poisson solvers.

In most even-slightly-relativistic situations the trajectory of the particle through manybeamline components is well approximated by a straight line, and the dynamic effects canbe found by calculating a wake function. These wake functions are available for manysituations, and in complicated cases can be calculated by electromagnetic (EM) codes.Tera-ready EM codes have also been developed under the DOE Grand Challenge programand are described below under micro-scale efforts. The wake function is an example of abridge between systems-scale and micro-scale simulations. Each processor can propagateparticles in one or more time-slices of the beam. The particles in a slice are affected by thestrength of the wake and also change its strength. The resultant strengths must be passed tothe processor containing the subsequent slices. Processors are simply ordered according tothe time ordering of the slice. There are two distinct types of simulation problems according


10

to whether particles remain in the same slice or not. In relativistic linacs, time ordering isfrozen and in rings, the order of distinct bunches does not change. Hence in both these casesno particle passing is required. In the most general situation, time ordering is changing, butparticles need only be passed to nearest neighbors. Important applications are high currentrings, damping rings, and bunch compressors.

Since wake effects require only nearest-neighbor message passing, simulations includingwakefields are scaleable to an arbitrary numbers of processors. Since wakefield effects havebeen included in several existing codes, the characterization of wake functions has beenthoroughly studied. These features can be added to the single-particle code in a matter ofmonths. One can estimate that the requirements to simulate the behavior of existing storagerings are in the tera-flop regime, and the application of the code to these situations wouldserve to test the code and simultaneously provide important insight into the behavior ofthese machines. Since we know that workstations with compute power of 108 flops requireone hour to propagate 105 particle-turns, we can deduce that a storage ring simulationcontaining 1000 macro-particles per bunch and 200 bunches tracked for several dampingtimes (perhaps 5×104 turns), would require a total of 1010 particle turns. Thus, a one-hoursimulation for one choice of system parameters would require a compute-power of 1013

flops. A high fidelity simulation might have 104 particles per bunch, and as many as 2000bunches, profitably utilizing 1015 flops.

After augmenting a single-particle code with wake-function and space-charge effects, asystems simulation effort would branch into parallel efforts. We divide these efforts broadlyinto five areas:

(i) beam environment changes and control systems,

(ii) halo generation and emittance growth,

(iii) ion, macroparticle, and photoelectron effects,

(iv) radiative effects, and

(v) emittance-cooling systems.

Item (i) is conceptually simple but can give rise to important new and complex behavior inlarge systems arising from the effects of wakefields and single-particle dynamics. Forexample, in the SLC at SLAC, a transverse adaptive feedback system was not able toachieve its design objectives due to the complex interaction of wakefields with the feedbacksystem design. Item (i) includes the modeling of complex RF systems containing feedbackloops, transverse and longitudinal feedback systems, movers, diagnostic devices, injectionand extraction devices, power supply ripple, ground motion and vibration, klystron phaseand amplitude errors in linacs, and aberration tuning systems. As an example of animportant application, future linear collider costs could be reduced by understandingreliability issues associated with the redundancy of movers, and by establishing looserstructure-alignment tolerances based on an optimally designed transverse feedback system.

Halo generation and emittance growth in item (ii) arise from particle diffusion due tononlinearities, scattering from gas and thermal photons, intra-beam scattering, noisydevices, surface breakdown, multipacting and dark current, as well as space-charge andwakefield forces. These effects are very important in linear and circular colliders as theygive rise to backgrounds and affect lifetimes. Several of these effects require extensive


11

microscale simulations and experimental verification. These simulations require tera-floppower because the entire beam must be simulated, but only a tiny fraction of the beam is inthe halo. The present Grand Challenge effort on space charge has been studying halogeneration in intense proton beams. As noted above, the field calculations are alreadyparallelized, and take full advantage of the communication architecture.

Items in (iii) give rise to important instabilities in rings, and create new systems behaviorthat is important to quantify. Spontaneous amplification by stimulated emission (SASE)which is included in item (iv) is an important phenomenon in fourth-generation lightsources. Stochastic and ionization cooling included in item (v) are crucial in anti-protoncollection and in cooling for muon colliders. The fast-ion instability and SASE simulationsare tera-flop scale problems in their own right.

Microsystems SimulationThere is a wide array of microsystems problems to be studied that require tera-scaleresources. These include electromagnetic field determination in complex three-dimensionalstructures; high voltage breakdown and dark current emission in acceleration cavities; avariety of problems in which low-energy charged particles pass through structures andinteract with them, such as klystrons and particle sources; acceleration of particles byplasmas and lasers; focusing of particles by plasma and beams; and coherent synchrotronradiation. Here we pick as examples electromagnetic field modeling and coherentsynchrotron radiation modeling.

The development of new electromagnetic tools for the Grand Challenge project originatedin advanced accelerator-structure research for the NLC. The main thrust of the effort isaimed towards large-scale simulations of realistic 3-D structures. Such a capability can beapplied to system-scale analysis such as finding the wakefields in an entire acceleratorsection, or to individual component design such as optimizing a single accelerator cavity toan accuracy approaching fabrication tolerance level. The new set of tools incorporates thefollowing features: (i) unstructured grids to capture realistic geometries, (ii) advancedsolvers to improve accuracy and convergence, (iii) refinement strategies to optimizecomputing efficiency, and (iv) scaleable algorithms to take advantage of the latest in HPCresources.

Presently there are two types of solvers that have been developed for the tool set. One isformulated in the frequency domain using linear and quadratic finite elements on anirregular grid, and includes eigenmode solvers in two and three dimensions. These parallelsolvers named ω2P and ω3P respectively, use MPI message passing and currently run onthe SGI/Cray T3E at NERSC using available parallel libraries. The other type of parallelsolvers is a three-dimensional time-domain code, called τ3P, which uses a generalized Yee

algorithm on an unstructured grid. A leapfrog time-advancement scheme with filtering hasbeen implemented as well as broadband termination at waveguide input/output ports. Thelatter allows for pulse propagation through an RF component so that the S-parameters canbe evaluated over a wide frequency range in a single run. Dipole excitations also possible tocalculate external Q's of loaded cavities. τ3P currently runs on the SGI/Cray T3E usingMPI, and also on shared-memory machines using threads.

ω3P is being used to model the complex 3-D accelerating cavity of the NLC, called theDamped Detuned Structure (DDS). To obtain an accuracy in frequency of 1 part in 10,000,


12

simulations for a DDS single cell require several million degrees of freedom. ω3P has been

run on 256 processors on the T3E and obtained close to linear speedup. The calculation ofthe wakefields in the DDS would require modeling an entire accelerating section of 206cells. The computing resources needed for this simulation are well beyond those that theT3E can provide, requiring terabyte memory and tera-flop computing power. ω3P stands

ready to take advantage of the SSI next-generation supercomputers to solve challenging yetimportant problems in present and future accelerators. The same holds true for τ3P whereefforts are underway to improve scalability. τ3P will be used to simulate dark current in the

DDS section, which is another tera-scale computation. Both the wakefield calculation andthe dark current simulation have been primary applications targeted by the electromagneticcomponent of the DOE Grand Challenge project.

The effect of coherent synchrotron radiation (CSR) on beam quality in bunch compressorsfor linear colliders and for FEL drivers is a major outstanding problem in acceleratorphysics. During the past year, a self-consistent computer simulation program of coherentsynchrotron radiation has been developed and used for preliminary studies of its impact onbeam quality. This work has been carried out in conjunction with plans for measurements inthe Jefferson Lab IR FEL driver, and will provide experimental confirmation andbenchmarking of this effect.

Calculation of CSR requires careful attention to retardation effects and relativisticdynamics. The force calculations are intrinsically suitable for parallel computation. This hasbeen demonstrated on the Cray PVP machine, and the first stage of porting of a CSRsimulation code to the T3E using MPI has recently been finished. Scaling up the applicationto a large number of particles per bunch and time steps, plus doing parametric studies,would result in an application which could effectively use a tera-scale computing facility.

(D) BENEFITS TO AGENCY MISSIONS

The missions of the DOE include improving basic understanding of matter and energy,understanding existing and creating new materials, creating a cleaner environment andfuture energy resources, improving the lives of citizens through education and medicine,and maintaining the safety and reliability of the nuclear weapons stockpile. Acceleratorshave a role in each of those missions, and tera-scale computing is critical for many of them.

Future facilities for high energy physics will be large and expensive. They must bedesigned to be cost effective and to have high performance. Both the micro-scale andsystems-scale simulations could reduce the design costs, simplify the development cycleand improve operations.

There would be comparable impact on existing high energy/nuclear physics acceleratorsand synchrotron light sources. Detailed modeling could explain performance limitations,and, with that understanding, new operating conditions could be chosen for betterperformance. This comparison between simulations and performance is also critical forproving the validity of the programs being used to design future accelerators.

X-ray and neutron beams are vital for understanding and advancing material, chemical,biological and pharmaceutical sciences. The highest priority of the DOE Science Programis the Spallation Neutron Source (SNS). This large collaborative project, involving severalDOE laboratories, depends upon accelerator physics supported by modeling and simulation


13

of the components and the entire facility. End-to-end modeling of the SNS would beinvaluable in developing cost-effective operating scenarios, thereby maximizing acceleratoravailability and minimizing system downtime.

Future x-ray sources could be linac-driven Free Electron Lasers or compact, laser-driven,plasma-based accelerators. The understanding of the accelerator physics and design of thesesources would be greatly enhanced by modeling at the level that would be possible withSSI.

Other accelerator applications that could be important to the DOE Science Office missionare: accelerator transmutation of radioactive waste for a cleaner environment; particle-beam-based medical therapy such as boron neutron capture therapy; cancer treatment usingheavy ions; and heavy-ion accelerators as drivers of inertial fusion reactors. While many ofthese are speculative, their realistic potential could be evaluated by computer simulationmade possible by SSI.

The DOE Defense Program is charged with maintaining the safety and reliability of thenuclear weapons stockpile. Part of the Science Based Stockpile Stewardship Program is theconstruction of the Dual Axis Radiographic Hydrotest facility. This facility includes twolinear accelerators delivering intense relativistic electron beams of very high beam quality.The simulation needed for these facilities exceeds the most advanced computationcapabilities available at the present time. Computer simulation with SSI scale resources andproven accelerator simulation tools would also be an important benefit to this aspect of theDOE mission.

(E) IMPACT ON OTHER SCIENTIFIC DISCIPLINES

One critical aspect of the study of particle accelerators is the understanding of the stabilityof dynamical systems subjected to strong nonlinear forces. For hadron colliding beamfacilities this requires particles to orbit an accelerator for about a billion turns. Thesimulation of beams of particles stored in storage rings for this long is far beyond presentcapabilities. In spite of this, long-term beam ÔtrackingÕ has been under development formany years. This has led to algorithms that are applicable to other fields of science.

Because of the nature of particle dynamics the trajectories of non-radiating particles areconstrained in a special way. The maps must be symplectic. A typical numerical integrationalgorithm does not necessarily guarantee this special property, but it is especially importantfor particle accelerators. This led to the development of symplectic integration techniquesand codes based on these techniques that automatically preserve this key property.

The most obvious application of these techniques is to solar-system dynamics where therehas already been cross-fertilization. A somewhat less obvious application is in moleculardynamics simulations. In the classical molecular dynamics of macromolecules, such asnucleic acids, proteins and polymers, the maps of the atomic trajectories must besymplectic. It is necessary to simulate large systems with very nonlinear fields, and largetime steps are desirable to extend the length of the simulation. Thus, this problem canbenefit from the use of symplectic integration tools, and it has many features in commonwith problems encountered in the study of particle accelerators.

In the area of electromagnetics, eigensolvers are central to the calculation of normal modesin an RF cavity. Many other disciplines routinely employ eigensolvers in their respective


14

applications; for example, in material science, one uses eigensolvers in determining theelectronic structure of matter. For the DOE Grand Challenge project, a new eigensolver,comprised of three methods integrated in a hybrid scheme, has been developed for theparallel electromagnetic code ω3P and found to deliver accuracy, convergence andscaleability far superior to any existing algorithms. This software tool will be madeavailable to other disciplines and will have a significant impact on their simulations as theyprogress into the tera-scale regime.

Based on past experience, it is highly likely that the codes and techniques used foraccelerator science will have a significant impact on related fields.


15

3. Theoretical Physics

(A) HIGH PERFORMANCE COMPUTING AND THEORETICAL PHYSICS

In the following, a few areas of theoretical physics will be discussed for which a largeincrease in computational power can have a major impact. These examples cover a broadspectrum of phenomena. The work described in high energy and nuclear physics focuses onthe smallest building blocks of matter, while that in astrophysics and numerical relativityaims at understanding the largest structures in the universe. Very large investments inexperiment are being made in all of these fields. In order to obtain the full return on theseinvestments, however, it is necessary to make corresponding, but much smaller, ones in thecomputational infrastructure needed for theoretical analysis.

(B) HIGH ENERGY AND NUCLEAR PHYSICS

Large-scale numerical simulations within the framework of lattice gauge theory currentlyprovide the only means of obtaining the non-perturbative predictions of quantumchromodyanics (QCD) from first principles with controlled systematic errors. It is essentialto extract these predictions in order to make precise determinations of parameters of theStandard Model and search for new physics beyond it, to understand the structure ofnucleons and other hadrons, and to discover the properties of hadronic matter underextreme conditions in the laboratory and in the cosmos. Lattice QCD simulations are thusof fundamental importance in high energy and nuclear physics.

Lattice gauge theory has been the subfield within theoretical high energy and nuclearphysics which has made the most extensive use of high performance computers, andtechniques have now been developed to the point that it is clear that tera-scale computerswill have decisive impact. The immediate availability of tera-scale computers for latticeQCD simulations is particularly important to realize the full scientific potential of the threenewest national experimental facilities in high energy and nuclear physics: the StanfordLinear Accelerator Center (SLAC) B-factory, the Brookhaven National LaboratoryRelativistic Heavy Ion Collider (RHIC) and the Jefferson National Laboratory ContinuousElectron Beam Accelerator Facility (CEBAF).

Over the past several years, improvements in algorithms and computational techniques,coupled with major increases in the capabilities of massively parallel computers havebrought QCD simulations to a new level. It is now possible to calculate a limited number ofquantities to an accuracy comparable with their experimental determination. The evaluationof the strong coupling constant at the mass of the Z meson with an accuracy of 1% to 2% isa notable example. The masses of the b and c quarks have been computed with similarprecision. Computers in the tera-flops range would allow a determination of the u, d, and squark masses to approximately 5%. The same simulations would allow calculations of themasses of some of the more unusual states predicted by QCD, such as glueballs, gluonicexcitations, the H particle and other particles with exotic quantum numbers. These particleshave proven difficult to observe experimentally, and accurate calculations of their massesand branching ratios would aid significantly in their detection. Experiments to search forthese particles are in progress at Brookhaven National Laboratory and are planned forCEBAF.


16

Among the major goals of experiments planned for the CESR storage ring at Cornell, theSLAC B-factory, and the Fermi National Accelerator CenterÕs Tevatron, are to performprecision tests of the Standard Model, and to look for new physics within and beyond it.These tests require accurate calculations of the effects of the strong interactions on weakinteraction decays, calculations that can only be carried out through large-scale numericalsimulations. Access to the computing facilities envisioned for SSI would enable theorists tomake controlled calculations of a number of crucial quantities, such as the decay constantsand semileptonic form factors of B mesons. When coupled with the results of plannedexperiments, such calculations would allow precise tests of the theory, and enable accuratedetermination of those elements of the quark mixing matrix which are among the least welldetermined parameters of the Standard Model. This would make it much more likely thatthe physics beyond the Standard Model is discovered.

Under normal conditions quarks and gluons are always confined inside hadrons. However,at sufficiently high temperatures or pressures one expects to find a new state of matterconsisting of a plasma of quarks and gluons. The quark-gluon plasma existed in the firstmoments after the big bang, and may exist today in the cores of neutron stars. Itsobservation is the primary goal of RHIC. QCD simulations have provided an estimate ofthe temperature at which the plasma is produced, but computing power in the tera-flopsregime is needed to determine the precise nature of the transition and to accurately calculatethe equation of state, which are essential to interpret the complex data that will come fromthe RHIC experiments.

Although experimentalists have made major progress in measuring the quark/gluonstructure of hadrons during the last decade and are poised to make further advances, thesemeasurements have yet to be confronted by precision theoretical calculations because ofinadequate computational resources. Deep inelastic lepton scattering at SLAC, CERN, andHERA has provided a detailed knowledge of the structure functions characterizing thedistribution of quarks and gluons in the nucleon. Drell-Yan processes at Fermilab haveprobed the flavor dependence of sea quarks, and the RHIC spin program will providefurther details. Parity-violating electron scattering at CEBAF and Bates is revealing theform factor of strange quarks in the nucleon, and CEBAF will explore the transition formfactors to excited states. With multi-tera-flop computers, lattice calculations of form factors,moments of structure functions, and other aspects of hadron structure would confront theseexperiments with the predictions of nonperturbative QCD and provide fundamentalunderstanding of the basic structure of matter.

Estimating the computational resources required to perform QCD calculations of the typediscussed above is straightforward. In any QCD calculation it is necessary to carry outsimulations for a range of lattice spacings and quark masses in order to extrapolate to thecontinuum and chiral limits. A major source of uncertainty in many current calculations isthe neglect of quark-antiquark creation and annihilation. To correctly take into account sucheffects requires one to two orders of magnitude of extra cpu time, depending on the quarkmass being studied. Current large calculations require one to ten gigaflop-years of cpu time,0.5 to 5 gigabytes of main memory, and a few terabytes of mass storage. The largest latticesize for calculations that take into account quark-antiquark annihilation has been 243 × 64.By contrast, the studies outlined above would require one to ten tera-flop-years of cpu time,20 to 200 gigabytes of main memory, and tens of terabytes of mass storage. To be specific,recent estimates of the resources needed to generate independent lattices fully taking into


17

account quark-antiquark annihilation are shown in the table below. Estimates are shown forthree different lattice spacings, a (measured in fermis), and three different u and d quarkmasses, which are parameterized in terms of the ratio of the pion to ρ mass. For each set of

parameters, the lattice size is chosen to be the minimum needed to avoid finite size effects.The memory required to store the lattice is given in gigabytes. Estimates are shown for thenumber of flops and the time required to generate an independent lattice with a computersustaining two tera-flops. These estimates are for the Kogut-Susskind formulation of latticequarks. Generating lattices for the Wilson formulation of quarks takes three to five timeslonger. Typically, 200 independent lattices are required to reduce statistical errors belowsystematic ones. Once such lattices are in hand, they can be used to study a wide range ofphysical phenomena. A research program based on such a lattice library would lead tomajor advances in our understanding of the fundamental forces of nature.

Table of resources required to generate independent QCD lattices on a two-Tera-flop computer for avariety of input parameters.

mπ/mρ a = 0.1 fm a = 0.066 fm a = 0.05 fm

0.5 Size

Flops

Time

283 × 56

1.2 × 1015

10 mins

423 x 84

1.7 × 1016

2 hrs

563 × 112

1.0 × 1017

15 hrs

0.4 Size

Flops

Time

283 × 56

5.0 × 1016

40 mins

423 × 84

7.0 × 1016

10 hrs

563 × 112

4.5 × 1017

65 hrs

0.3 Size

Flops

Time

363 × 72

7.0 × 1016

10 hrs

523 × 104

1.0 × 1018

6 days

723 x 144

6.0 × 1018

38 days

On a more speculative note, there is growing expectation that work in string theory willlead to a unified theory of all of the fundamental interactions. Non-perturbative calculationsin such a theory may well require numerical studies even more challenging than thoserequired for the study of QCD. A hint of this arises from the D0-brane sector of Matrixtheory. Standard numerical approaches appear to be directly applicable to the study of thisvery interesting system; however, preliminary analysis indicates that the numericalsimulation of even this relatively simple problem would require computational resourcescomparable to those needed for QCD.

(C) COMPUTATIONAL COSMOLOGY

A revolution in detailed observations of the early life of our universe has been broughtabout by a new generation of powerful scientific instruments, which are constructing anincreasingly accurate and unprecedented picture of the structure and evolution of the earlyuniverse. The goal of physical cosmology is to understand the mechanisms that created thisstructure, and to discover the principles that established the initial conditions. Computersimulations play a major role in this process by enabling cosmologists to simulate model


18

universes for comparison with observations. The development of structure in the universecan be studied in its full gravitational nonlinearity only through numerical simulations.Because of the size and complexity of cosmological simulations, computational cosmologyranks as one of the most demanding and most important supercomputer applications.

(D) GRAVITATIONAL PHYSICS

The application of the Einstein theory of gravity to realistic astrophysical processes requiresaccess to multi-tera-flop/terabyte computers. Two major directions of astronomy in the nextcentury which will rely on such large-scale numerical calculations are high energy (x-ray,γ-ray) astronomy, and gravitational wave astronomy. The former is driven by observationsby x- and γ-ray satellites and ground-based detector arrays. The latter involves directly thedynamical nature of space-time in EinsteinÕs theory of gravity. The tremendous recentinterest in this frontier is driven by the gravitational wave observatories presently beingbuilt or planned in the US, Europe, and outer space. These projects will make gravitationalwave astronomy a reality, opening a new window which will provide information about ouruniverse that is either difficult or impossible to obtain by traditional observations of theelectromagnetic spectrum. However, theoretical informations that can only be provided byfull-scale general relativistic simulations is essential in order to extract physical informationfrom these observations.

The computational challenges of simulating EinsteinÕs theory of gravity are due to thecomplexity of the physics involved and to the fact that the partial differential equationswhich must be solved are probably the most complex in all of physics. Astrophysics ofstrongly gravitating systems inherently involves many length and time scales. The propertreatment of these different scales requires sophisticated numerical methods andvisualization as well as raw computing power.

(E) CONNECTIONS

There are a number of possible connections between theoretical high energy and nuclearphysicists and SSI participants working in other areas, such as climate modeling,combustion, and materials science. For example, lattice gauge theorists have had extensive,long-term interactions with materials scientists engaged in numerical simulations ofstrongly correlated electron systems (high temperature superconductors, heavy fermions,itinerant magnetism, etc). Scientists in both fields are interested in simulating stronglycoupled fermions (quarks in the case of high energy physics, electrons in the case ofcondensed matter physics) on regular lattices. Algorithms developed for lattice QCD havebeen used to study strongly correlated electron systems and lattice gauge theorists havebeen involved in large-scale simulations of these systems. A major problem in both fields isto develop stable algorithms for the simulation of fermion systems with non-zero chemicalpotential. This problem is important in high energy physics in order to study hadronicmatter at finite baryon density, and in condensed matter physics in order to study a widevariety of problems, such as high temperature superconductivity. So, there is ampleopportunity and motivation for future interactions.

The most difficult problem in obtaining good performance on current computers for latticeQCD and many other, if not all, applications is to have data in cache when the processorsare ready for it. This problem will grow in complexity as cache structures do. Addressing it


19

is an area in which SSI users could fruitfully interact, and in which the experience of latticegauge theorists might well prove useful. Because their calculations are carried out onregular grids, lattice gauge theorists are not faced with interprocessor communicationsproblems as severe as those working with irregular and/or adaptive grids. Nevertheless,they have developed data movement software for parallel machines, such as gather/scatterroutines, which may prove useful in other applications. The computational gravity andcosmology applications have data movement problems comparable in complexity to thoseencountered in climate modeling and combustion. There is also room for profitableinteraction on the optimization of algorithms and code for linear algebra operations (e.g.,sparse matrix inversion) common to many fields. Computer scientists have on occasionused lattice gauge theory codes to test their software and performance tools. More extensiveinteractions between computer and applications scientists has been cited as an importantcomponent of SSI, so these interactions could and should be built upon.

Most large lattice gauge theory groups are geographically distributed. They have thereforebecome experienced in computing at a distance, in group software development, in thehandling of large data repositories, and in the coordination of large simulation efforts.These are all activities which will be engaged in by most SSI participants.

(F) READINESS

Lattice gauge theorists are very experienced users of supercomputers, and their problemsare particularly well suited for parallel machines. They have been among the first to havecode running on all major parallel platforms, and their performance achievements rankamong the best on each of them. Lattice gauge theorists have extensive experience asfriendly users of new machines, and have on a number of occasions found hardware andsoftware problems at very early stages in their development. They have highly portablecode which can ordinarily be brought up on new platforms platforms with a few hours ofwork. As a result, lattice gauge theorists would be prepared to assist in debugging SSImachines, and would be ready to begin production runs as soon as the SSI facilities becomeavailable for that purpose. Cosmologists and numerical relativists have also hadconsiderable experience with massively parallel computers. Workers in both of these fieldshave recently undertaken major code development projects under NSF Grand Challengegrants and would be ready to make use of SSI facilities at early stages in the program.


20

4. Experimental High Energy and Nuclear Physics

(A) DATA MANAGEMENT IN HIGH ENERGY AND NUCLEAR PHYSICS

High energy and nuclear physics (HENP) have become drivers of the science andtechnology of data management. The large and growing importance of data managementwithin HENP has striking similarities to the fundamental role of accelerator science andtechnology in physics. Both accelerators and data management are vital for progress inexperimental HENP, and while both technologies have major applications outside HENP,HENP needs have been consistently ahead of those in other sciences or commerce.Management of HENP data is especially challenging since it combines large data volumes,a complex logical structure and fine granularity.

Data management in high energy and nuclear physics is uniqe in its combination of scaleand complexity. Each HENP experiment requires the intellect and labor of hundreds orthousands of physicists at universities and laboratories all over the nation or the world. Ourdata analysis capability limits the number of collisions that can be studied and hence thefull scientific benefit that can be realized. TodayÕs limit for the complex and granularHENP data is around one petabyte. At this limit, it can take many months for a student totry a simple new analysis idea.

HENP must also increase the tractable number of collisions in HENP experiments byfactors of tens or hundreds to realize the potential physics gains. The need to makedangerous guesses about which collisions are worth recording can then be reduced. Theprecision of physics measurements will increase and the sensitivity to the unexpected willbe vastly improved.

A revolutionary advance in the science of distributed data management and analysis isneeded in high energy and nuclear physics. The goal is to give data-managementintelligence to networked computers and storage, such that queries that might have takenmonths or years complete in minutes or hours. An increasingly rigorous approach to thedesign and exploitation of HENP data-management systems is also needed. Systems arebuilt from mass storage (tape), caches (disk), wide-area networks, computers and HENP-specific or commercial software components. A rigorous approach requires that existingsystems be instrumented and that models be developed that can predict the behavior of newor improved systems. These models will facilitate rapid evaluation and improvement ofnew approaches in the science of HENP data management. Dedicated testbeds must also becreated so that adventurous new ideas can be explored. The potential benefits of thisHENP-motivated data-management science for other data-intensive sciences and forindustry will be vigorously pursued.

New experimental insight into fundamental physics requires detectors and accelerators thatare too complex and costly to be replicated. National and international collaboration onHENP experiments and data analysis is therefore normal and necessary. The combinationof vast data volumes and geographic distribution of the scientists who need access to thedata presents an unparalleled challenge in distributed data analysis.


21

(B) BRIEF HISTORY OF DATA MANAGEMENT IN HENPThirty years ago the terabytes of high energy physics data resided mainly on photographicfilm and were accessed very slowly by human pattern recognition and feature extraction.Typical experiments acquired data in a few weeks and then took years to perform thelaborious analysis. As the era of photographic film in high energy physics ended, so did thestorage of high energy physics data in image form. Modern experiments store only datacorresponding to the extracted features of the images of collisions.

Fifteen to twenty years ago the major discoveries were being made by entirely electronicexperiments filtering up to millions of collisions per second and recording and analyzing afew per second for months or years. The recorded data sample from one year could occupytens of thousands of magnetic tapes and several years of data might need to be repeatedlyanalyzed to make new discoveries or key measurements. Commercial database technologywas typically comfortable with the quantity of data that would fit on to one tape but wastotally inapplicable to tens of thousands. high energy physics experiments developed datamanagement schemes that anticipated access patterns and wrote out data on tape such thattape mounts were likely to be minimized. The consequence of inadequate foresight could bemonths of tape mounting to perform a simple query on the data. Data distribution requiredair freight to transport cartons of tapes to remote computing centers. Physicists analyzingdata at remote centers were severely disadvantaged by the delay in access to new datasetsoptimized for the study of the hottest physics topics.

In the last fifteen years, wide area networks have tantalized physicists by showing theirrevolutionary potential when applied to small-scale data analysis while totally failing tosupport access to large fractions of a typical experimentÕs data. As available networkbandwidths have risen, so have the sizes of experimental datasets. As recording densitiesincreased, the manageable limit of a few tens of thousands of tapes has increased in datacapacity from terabytes to petabytes. The larger HENP experiments consistently pushagainst the limits of manageable data volumes; the data management capability is a keylimiting factor on the possible scale of experiments. However, network bandwidth capableof supporting distributed data analysis is beginning to appear. The World Wide Web,invented and developed within HENP to address its information management problems, hasincreased the worldwide demand for bandwidth and is pushing down unit costs. The focusnow is on the computer science and software technology to support distributed datamanagement and analysis.

The advance of magnetic disk technology within the last ten years has made it realistic tostrive for a data management system which would give Ôinstant accessÕ to any byte withinthe tens of thousands of magnetic tapes holding the raw experimental data. It is nowfinancially responsible to cache on disk a few percent of the data recorded on the tens ofthousands of tapes. Once on disk, ingenious commercial or HEP-written databasetechnologies can be used to access the data with potentially high speed. The interchangebetween tape and disk can be managed by equally ingenious technology making use ofpredicted access patterns to cluster data appropriately on tapes and load the appropriate datainto the disk cache. Current projects in HENP data management1 are achieving significant

1 Examples include the BABAR data-management system using a commercial object database management

system interfaced to the HPSS mass storage system, the RHIC data management system using both


22

practical success while at the same time exposing the complexity and richness of thescience of data management that exploits todayÕs storage and networking technologies.

(C) CURRENT AND FUTURE CHALLENGES IN DATA MANAGEMENT

The management and distribution of the primary and derived data is in most cases going tobe the limiting factor in the extraction of timely results from the current and next generationof high energy and nuclear physics detectors.

Some of the principal data management challenges facing HENP are summarized in thefollowing table. The principal goals of the experimental programs in the table are:

• The SLAC B-FactoryÑthe asymmetry between matter and antimatter (probablyresponsible for our existence) as revealed by the differing behavior of B and anti-B mesons;

• The Relativistic Heavy Ion Collider at BrookhavenÑre-creation of the highlyenergetic particle soup (quark-gluon plasma) that we believe existed for the firstmicrosecond or so after the Ôbig bangÕ that created the universe;

• The Fermilab Tevatron Run 2Ñfollowing on the discovery of the top quark (thesixth and last quark?) at Fermilab with detailed measurements of top productionand decay.

• The Large Hadron Collider at CERNÑprobing the energy range where themysterious Higgs boson or some even more exotic physics must lie.

Each program should reach the data rates given in the table within two years of startingoperation and each will almost certainly become capable of producing much more data laterin its 10Ð20 year life.

commercial and HENP-written object management software also together with HPSS, and the FermilabRun 2 system based on an Oracle meta-data catalog and other software written within HENP.


23

Some Current and Future HENP Data-Management Challenges

HENP Laboratory SLACStanford, CA

USA

BNLUpton, NY

USA

FermilabBatavia, IL

USA

CERNGeneva

Switzerland

Accelerator PEP-IIB Factory

RHIC Tevatron(Run 2)

LHC

Experiment(s) BaBar STARPHENIXPHOBOS

CDFD0

ATLASCMS

In Operation 1999 on 1999 on 2000 on 2005 on

Collisions per Second 20 10,000,000 7,500,000 100,000,000

Planned to StoreCollisions per Second(Incl. background)

≤ 100 1 < 75 ≤ 100

Stored Data Volumeper Year (raw andprocessed)

0.3 petabytes > 1 petabyte > 1 petabyte 3Ð100petabytes

Aggregate Disk DataRate during Analysis

200megabytes/s

> 1 gigabyte/s >1 gigabyte/s ~several to100 Gbytes/s

Aggregate Tape DataRate during Analysis

40megabytes/s

> 250megabytes/s

>300megabytes/s

1 gigabyte/sand up

Processing Power(SPECint95)

5,000 35,000 60,000 250,000 andup

Number of Physicists 800 1000 1000 5000 Ð 6000

Number of Universitiesand Laboratories

87 80 90 300+

Number of RegionalData Centers

3 ∼ 3 ∼ 4 ~ 20

Number of Countries 9 25 20 ~ 50


24

The scope, scale, and worldwide geographical spread of the data-access, data-analysis anddata-management problems to be faced by these experiments are unprecedented in thehistory of scientific research. The major collider and fixed-target experiments at JeffersonLaboratory, SLAC, BNL, Fermilab, and CERN will acquire datasets in the petabyte rangeby 2002, rising to an estimated 10 PB by 2006 and 100 PB by 2010. To reduce the dataeven to these levels, risky real-time decisions must be taken to record, for example, onlyone collision in every million based on best guesses as to which collisions will reveal newphysics. The datasets, consisting of pre-filtered, compacted numerical data, have aninformation content far beyond that of existing or foreseen digital text or image libraries.The need to access, analyze and successively improve the quality of the stored data byhundreds to thousands of physicists distributed at hundreds of institutions throughout theworld, using national and international networks at speeds of hundreds of Mbps (by 2002)to Gbps (by 2005) also presents unique challenges.

The solutions to these challenges are mission-critical to the experiments and will be of greatimportance to the broader academic community and in commerce. Petabyte-scale dataaccess and processing systems distributed across international networks are likely tobecome increasingly common by the time the LHC starts in 2005.

The software systems that will address these challenges must be robustly engineered tohave the reliability, maintainability, and adaptability required for a lifetime of two decadesor more. Mainstream commercial components and other reusable components must beexploited wherever possible.

Key technologies favored for handling petabytes of HENP data are large Object DatabaseManagement Systems (ODBMSs) and scalable mass-storage management systems such asHPSS2. However, existing data-management prototypes based, for example, on availableODBMS technology and the current HPSS are very limited in function. Intelligent caching,automated data re-clustering, and almost all network awareness are still research topics.With continuous human intervention to guide data placement, cache management andnetwork transfers, the systems will outperform their antecedents but will still heavilyrestrict access to data.

Geographically the computing systems of each experiment will consist of a principal centerat the HENP laboratory where the experiment is situated, complemented by a set ofregional centers, built using state-of-the-art data-handling facilities. It is imperative, by thenature of these distributed physics collaborations, that all are able to participate fully in thephysics analysis, and hence in any discoveries that may occur, irrespective of location. Thisimplies the need for excellent network connectivity between the various sites. For the LHCera, this presents a particular challenge for US groups because of the higher costs oftransoceanic networks. The highly distributed nature of the collaborations also necessitatesthe development and use of network-based collaborative working tools, such asvideoconferencing today and immersive shared virtual environments in the future.

2 High Performance Storage System developed by a consortium of DoE Laboratories in collaboration with

IBM.


25

(D) REQUIRED RESEARCH AND DEVELOPMENT IN HENP DATA MANAGEMENT

Revolutionary progress can be made in HENP distributed data management through anaggressive, ongoing, research and development program involving:

• Instrumentation of existing and future data-management systems to capture theevolving requirements of users and reveal the internal behavior of alreadycomplex systems;

• Modeling tools with true predictive power used to optimize existing systemsand to estimate the effects of new approaches;

• Enhancing commercial or widely used software components to meet the needsof HENP;

• Research into new approaches;

• Creation of testbeds to test the validity of new approaches and thecorresponding models.

InstrumentationHENP data management must remove damaging restrictions on data access patternswithout knowing in advance what these patterns will be. Access patterns develop inresponse to the physics being uncovered and the ingenuity of young physicists finding newways to exploit the data-access technology. An integral component of a data managementstrategy with medium to long-term validity must be instrumentation to measure evolvingaccess patterns so that the data-management software and its designers can respondappropriately.

In addition, prototype or production data-management software that is in use by HENPexperiments, must be internally instrumented to aid the optimization of hardware andnetwork configurations and to provide detailed measurements against which data-management models can be validated.

Modeling and SimulationModeling and simulation of HENP data-management systems are essential to achieve therevolutionary improvements that are the goal of this initiative. Individual hardware (e.g. adisk cache) or software (e.g., a database management system) components of systems in usefor todayÕs experiments cost of the order of $1M each and the rising complexity of systemsmakes it quite impossible to guess how each component is contributing to the overallsystem performance. The creation and continuous development of modeling and simulationtools with real predictive power is increasingly desirable for the effective exploitation ofexisting systems. Models are even more essential in the development of new approaches.No testbed can ever approach the scale of activity of production systems and model-basedsimulation must be used to extrapolate from testbed measurements to the likely impact of afull-scale system.

Enhancements to Software ComponentsWhen seeking to make revolutionary progress in any area of experimental HENP, it isalways vital to exploit available technologies to the full and thus to allow the maximumintellectual energy to be applied to the needs that are unique to the field. Government andindustry are making substantial investments in systems such as HPSS and Object Database


26

Management Systems to meet a wide range of scientific and commercial needs. Whenexisting systems are confronted with the requirements of HENP they invariably are foundto lack key features. For example, HPSS lacks features such as continuous availability andguaranteed minimum throughput for priority traffic that are essential if its is to provide datastorage for HENPÕs real-time data acquisition systems.

We propose an ongoing program to identify where HENP-specific enhancements arerequired to widely applicable software components. In general, work on the enhancementsshould be undertaken as cooperative projects involving universities or laboratories engagedin HENP and the developers of the software to be enhanced. Experience shows that manyHENP-motivated enhancements become valuable to the wider user community within twoyears.

Research into New ApproachesCollaboration between HENP programs and computer scientists are known to be veryproductive. We propose to identify a limited number of university- or laboratory-basedcomputer science groups with proven strength in data management that are interested inaddressing HENP data-management problems. Such groups will receive partial supportfrom the HENP data-management initiative.

In parallel, the small teams of engineers and computer scientists working within HENP ondata-management problems will be strengthened to allow collaborative work on longer termdevelopments of value to the whole of HENP.

Research topics to be addressed include cache management in systems with several multi-level storage hierarchies linked by networks offering differentiated services, automatedreclustering in response to measured access patterns, estimation of query cost and durationand resource management. The greatest improvements in responsiveness will come from anexploitation of Òagent technologyÓ where each query or calculation is executed at thelocation where the work can be done most efficiently.

TestbedsTestbeds will be vital to measure the behavior of candidate new components of HENP data-management systems and to try out the more adventurous results of research beforeattempting to move them into a production environment. Testbeds will be composed ofseveral levels of storage hierarchy at least two sites linked by high speeds networks offeringdifferentiated services. The data-management testbeds, and later production systems, willrely heavily on the development of network services that forms part of the Next GenerationInternet program. The success of this program will remove the need to develop HENP-focussed network services.

Coordination of the InitiativeThe work on instrumentation, modeling, computer science and testbeds will requireeffective and motivating coordination. HENP has a long history of successful collaborationon the construction and operation of detector systems and a shorter, but very encouraginghistory of collaboration on software projects. Multi-year collaborations on software systemsinvolving many universities and laboratories have already brought major advances in, forexample, object-oriented systems that simulate the interaction of particles with complexdetection apparatus.


27

We believe that, independent of the precise funding processes, effective coordination of theoverall initiative can be set up using the approach to collaboration management that hasalready proved itself in HENP.

(E) BENEFITS OF THE PROPOSED INITIATIVE FOR HENP AND OTHER SCIENCE

Within HENP, this new thrust will increase the tractable numbers of collisions that anexperiment can handle by factors of tens or hundreds. This will allow physics processes tobe studied that would otherwise be declared Ôless interestingÕ and removed from theselection process. It will lead to an increase in the precision of measurements and thereforeincrease the probability of making fundamental discoveries. Above all, it will greatly lessenthe chance that we discard measurements of unexpected and revolutionary new physicsbecause we could only afford to retain the collisions that conformed to conventionalwisdom on the likely signatures for new physics. If the new tools perform as hoped, theywill reduce the time required for detailed analysis of these large data sets and revolutionizethe way we ask questions of nature.

We believe that the management of challenging quantities of information will become adecisive factor for computation in most areas of science and even of commerce. We areconvinced that the development of these new tools will be of wide, long-term benefit andare committed to a vigorous pursuit of the satisfaction of seeing our work widely applied.More locally, the application work within SSI on Climate, Combustion, FusionÑevenGenomics researchÑwill require such tools. The early need of HENP to develop and learnhow to effectively use these new computer-science tools should prove to be the platform fora valuable partnership within SSI.


28

5. Conclusions

We have presented a vision for involvement of the HENP programs in the new DOE SSIactivity on high-end computing that promises to change science in the 21st century. Thisvision is broad and includes three specific areas for high-end computing activity:accelerator simulation, theory, and data management. In addition, HENP proposes aninvolvement in the crosscutting CSET infrastructure aspect of SSI.

For accelerators there is a promise of ÒASCI-likeÓ simulations that would be based onexperience and experiments. These simulations would bring cost savings and efficiency tothe new projects on the agency's horizon, and they would be critical for developing newideas for future accelerators. The HENP programs have consistently been the fiscal andintellectual center of the accelerator R&D that has benefited science programs throughoutthe entire DOE. Accelerators for nuclear and high energy physics, synchrotron lightsources, and neutron sources are the fruits of the past investments. It is natural, andpotentially widely beneficial, to apply the very high-end SSI computational capabilities tothis ubiquitous applied science discipline.

The lattice gauge theory calculations described in the theory section are perhaps one of theoldest super-computing activities in the DOE. This tradition of high-end theoretical nuclearand high energy physics computing should continue; there are areas of intellectual inquirythat are receiving strong support for experimental study and would be effectively advancedby access to SSI-scale computers. We stress that there are rich interconnections betweenmany of these activities, other areas of physics such as high temperature superconductors,and the Ònearest neighborÓ calculations encountered throughout the SSI portfolio.

The third thrust of the proposal is data management. The detectors coming on line over thenext few years will bring thousands of times more data than ever before. A few yearsbeyond that, by 2005, the Large Hadron Collider will bring yet another large increase in thevolume of data. The sheer quantities of data demand that new methods of data managementbe developed by HENP to pursue its science goals. Computer science developments are ofthe same fundamental importance as the new accelerators and detectors themselves! Wepropose that this be the third of our three-pronged approach to the SSI.

The investment in networking, collaboration-at-a-distance, data management andvisualization leads to the other aspect of the vision presented in this White Paper. Driven bythe needs of BABAR, CDF, D0, the RHIC experiments, and the LHC experiments, this fieldmust develop new tools to handle terabytes and then the petabytes of data. This same fieldhas been a world leader in supporting large international collaborations of scientists doingfundamental basic science with strong emphasis on computational tools. This is, after all,the community that brought us the World Wide Web! We propose that HENP be includedin the team, CSET, which will develop the new tools that SSI tera-scale computationalpower will demand.

high energy and nuclear physics white paper for the … · high energy and nuclear physics white...

Documents