maria grazia pia, infn genova methods and techniques for monte carlo physics validation mc 2015...
TRANSCRIPT
Maria Grazia Pia, INFN Genova
Methods and techniques for Monte Carlo physics
validation
MC 201519-23 April 2015, Nashville, TN, USA
C. Choi, M. C. Han, S. Hauf, G. Hoff, C. H. Kim, H. S. Kim, S. H. Kim, M. Kuster, M. G. Pia, P. Saracco, G. Weidenspointner
Hanyang University, Seoul, KoreaEU XFEL GmbH, Hamburg, Germany
CAPES, Brasilia, BrazilMPE, Garching, Germany
ForewordDue to limited time allocation, there is room to highlight concepts only
Details are documented and discussed in dedicated journal publications
Maria Grazia Pia, INFN Genova
In the literature…
Limited documentation of simulation validation‒ Mostly in the form of specific use cases compared to
measurements in the same experimental scenario▻ Do they apply to similar/different use cases?▻ How to extrapolate the results to different scenarios?
Hardly any validation of the basic physics models implemented in Monte Carlo codes‒ Why?
Ongoing projects on uncertainty quantification‒ Methods to predict the uncertainty of simulation observables
based on knowledge of the uncertainties of simulation “ingredients”
(quantitative)
2
Maria Grazia Pia, INFN Genova
What is what Verification
Validation
Calibration
3
IEEE Standard 1012Conforms to ISO/IEC 15288 (IEEE Std 15288)
Systems and Software Engineering – System Life Cycle Processes
ISO/IEC 12207 (IEEE Std 12207)Systems and Software Engineering – Software Life Cycle Processes
IEEE Std 1074IEEE Standard for Developing a Software Project Life Cycle Process
Maria Grazia Pia, INFN Genova
VerificationA. The process of evaluating a system or component to determine whether the products
of a given development phase satisfy the conditions imposed at the start of that phase.
B. The process of providing objective evidence that the system, software, or hardware and its associated products conform to requirements (e.g., for correctness, completeness, consistency, and accuracy) for all life cycle activities during each life cycle process (acquisition, supply, development, operation, and maintenance); satisfy standards, practices, and conventions during life cycle processes; and successfully complete each life cycle activity and satisfy all the criteria for initiating succeeding life cycle activities.
4
e.g. in the context of Monte Carlo simulation
Requirement: Compton scattering cross section shall be described by the Klein-Nishina formula
Verification: the software calculates
consistently, correctly, with adequate numerical precision…
Maria Grazia Pia, INFN Genova
ValidationA. The process of evaluating a system or component during or at the end of the
development process to determine whether it satisfies specified requirements.
B. The process of providing evidence that the system, software, or hardware and its associated products satisfy requirements allocated to it at the end of each life cycle activity, solve the right problem (e.g., correctly model physical laws, implement business rules, and use the proper system assumptions), and satisfy intended use and user needs.
5
In the context of Monte Carlo simulation
validationconsistency with
experimental measurements
e.g. does the Klein-Nishina formula reproduce measured differential cross sections of photon inelastic scattering?
Maria Grazia Pia, INFN Genova
CalibrationThe process of improving the agreement of a code calculation with respect to a chosen set of benchmarks through the adjustment of parameters implemented in the code
Calibration is not validation‒ Validation is the process of confirming that the predictions of
a code adequately represent measured physical phenomena
6
T. G. Trucano et al., Calibration, validation, and sensitivity analysis: What's what, Reliability Eng. & System Safety, vol. 91, no. 10-11, pp. 1331-1357, 2006
M. G. Pia et al, Physics-related epistemic uncertainties of proton depth dose simulation, IEEE Trans. Nucl. Sci., vol. 57, no. 5, pp. 2805-2830, 2010
AKA “tuning”
Maria Grazia Pia, INFN Genova
What is NOT validationComparison of simulations using different Monte Carlo codes‒Or comparison of different simulation models
Comparison of simulation with theory ‒ Or so-called “analytical calculations”
Comparison of simulation with non-pertinent experimental data
Calibration
Oenology
Mozart opera
7
Maria Grazia Pia, INFN Genova8
Establishing validity
Comparison of simulation results and experimental data in the literature mainly rests on
qualitative visual appraisal of figures
indicators (%) deprived of any statistical meaning
AgreementGood agreement
Excellent agreementSatisfactory agreement
…
Maria Grazia Pia, INFN Genova
Statistics Mathematical foundation of Monte Carlo physics validation
Rigorous statistical methods assess
‒ Whether a simulation model is consistent with nature▻ well, whether a simulation model is not inconsistent with nature…
‒ Whether different simulation models produce (or do not produce) equivalent results in terms of compatibility with experiment
Hypothesis testing‒ Well established methods
9
c2
Kolmogorov-Smirnov Anderson-Darling Cramer-von Mises etc.
Fisher exact text Barnard test c2
etc.
Goodness-of-fit tests Categorical data analysis
Mainly applied to contingency tables
Maria Grazia Pia, INFN Genova
What is validated
Validation of the “ingredients” of Monte Carlo codes‒ The foundation of physics models used in the code‒ Cross sections (total, partial, differential)‒ Secondary particle production‒ Atomic and nuclear parameters (e.g. binding energies,
transition probabilities etc.)
Validation of simulated observables produced by Monte Carlo codes in use cases‒ Largely represented in the literature‒ Often qualitative only‒ Seldom related to “physics ingredients”
10
Maria Grazia Pia, INFN Genova11
How is validation performed?
Validation of basic physics “ingredients”
Unit tests
Validation of simulated observables
Simulation applications
Testability must be embedded in the software design to enable physics unit tests
Amending the software design of a mature Monte Carlo system that did not account for testability is expensive
com
ple
men
tary
Maria Grazia Pia, INFN Genova
Post-RD44 Geant4 electromagnetic software design
12
Hiddendependencies
on other parts of the software
One needs a geometry (and a full scale application)
to test any photon cross section
Difficult to test no testing often
Reverse engineered
G4VEmProcess G4VEnergyLossProcess
G4VMultipleScattering
G4VEmModel
Attributes
abstract class
Operations
Maria Grazia Pia, INFN Genova
Discipline of software engineering
Most of the problems with physics tests can be easily solved if we simply write tests as we develop our code
‒ …and we maintain the tests‒ …and we regularly execute them‒ …and we investigate the reasons for failure
Software design reviews: care about testability13
If a test is hard to write, that means that we have to find a different design
which is testable
Maria Grazia Pia, INFN Genova14
Ongoing activity
Extensive R&D on Geant4 physics validation
Software design‒ Enables testability‒ Facilitates the validation of a wide set of modeling options, including
some that have not yet been used in Monte Carlo codes
Validation of basic physics models and parameters‒ Electron-photon interaction cross sections, atomic binding energies,
radiative transition probabilities etc.
Validation of simple observables of general interest‒ Recent project on electron backscattering validation
Uncertainty quantification‒ Original method, further R&D in progress
Maria Grazia Pia, INFN Genova15
Detangling
TestableOpen - closed
Photoionisation
New modelsHandles any tabulated
cross section
Can be validated in a unit test
Cross section models can be compared with
statistical categorical tests
Maria Grazia Pia, INFN Genova16
Tools for statistical analysis
The Statistical Toolkit
Large collection of algorithms for goodness-of-fit testing
Two-sample problem: comparing two distributions
aidaRBridge between iAIDA
and R
Our team developed software tools for statistical data analysis specifically to support simulation validation
Maria Grazia Pia, INFN Genova17
A sample of validation results
More extensive information in journal publications
Physics processes: photon interaction cross sections
An example of complex observable: electron backscattering fraction
Maria Grazia Pia, INFN Genova
Photoionisation cross section sources
18
Year Compilation Energy Z (sub)Shell Method1967-1988 Biggs-Lighthill 10 eV – 100 GeV 1-100 - parameterised
1992 Brennan-Cowan 30 eV – 700 keV 3-92 - tabulated
2000 Chantler 10 eV – 433 keV 1-92 K tabulated
2003 Ebel 1 keV – 300 keV 1-92 all parameterised
2002 Elam 100 eV – 1 MeV 1-98 - tabulated
1997 EPDL97 (Scofield) 10 eV – 100 GeV 1-100 all tabulated
1982-1993 Henke 10 eV – 30 keV 1-92 - tabulated
1970-2006 McMaster/Shaltout 1 keV – 700 keV 1-94 - tabulated
1989 PHOTX (Scofield) 1 keV – 100 MeV 1-100 tabulated
2001 RTAB 10 eV – 30 keV 1-99 all tabulated
1973 Scofield 1 keV – 1.5 MeV 1-100 all tabulated
1970 Storm-Israel 1 keV – 100 GeV 1-100 - tabulated
1973 Veigele 100 eV – 100 MeV 1-94 - tabulated
1987-2010 XCOM (Scofield) 1 keV – 100 GeV 1-100 - tabulated
e.g. Chantler’s exchange potential in his DHF calculation is different from Scofield’s
Different methods and calculations
Maria Grazia Pia, INFN Genova
Total photoionisation cross sections
Most calculation methods exhibit similar compatibility with experiment for E >250 eV‒ Chantler, Brennan-Cowan look worse
Degraded accuracy below 250 eV
19
preliminary
Analysis of contingency tablesEPDL
ChantlerEPDL
Brennan-Cowan
Fisher 0.044 0.011Pearson c2 0.033 0.007Barnard 0.035 0.007
H O Fe
Maria Grazia Pia, INFN Genova
Shell cross sections
20
shell EPDL Chantler RTAB scRTAB EbelK 0.209 0.350 <0.001 0.315 <0.001L1 0.075 <0.001 0.069 0.964L2 0.339 <0.001 0.299 0.154L3 1 <0.001 1 1M1 <0.001 <0.001 <0.001M4 0.031 <0.001 <0.001M5 <0.001 <0.001 <0.001N1 <0.001 <0.001 <0.001N6 <0.001 <0.001 <0.001 <0.001N7 <0.001 <0.001 <0.001 <0.001O1 <0.001 <0.001 <0.001 <0.001O2 <0.001 <0.001 <0.001 <0.001O3 <0.001 <0.001 <0.001 <0.001P1 <0.001 <0.001 <0.001 <0.001
p-value c2 test
Systematic effect observed with RTAB shell cross sections
(presumably a missing factor in the calculation)
Calculated inner shell cross sections compatible with experiment
Outer shell cross sections inconsistent with experimental data
Beware: small data sample, limited experimental sources
K
L3
M4
O1
Maria Grazia Pia, INFN Genova
Angular distribution
21
Qualitative appraisal Limited experimental sample
Experimental systematic effects(corrected/uncorrected data)
Option à la GEANT 3 (Sauter) evaluated along with other Geant4 options
Maria Grazia Pia, INFN Genova
Photon elastic scattering
Penelope Penelope EPDL Relativ. Non-Rel. Modified MFF RFF SM2001 2008 FF FF FF ASF ASF NT
e 0.27 0.38 0.38 0.25 0.35 0.49 0.52 0.48 0.77 error ±0.05 ±0.06 ±0.06 ±0.05 ±0.06 ±0.06 ±0.06 ±0.06 ±0.05
Form factor approximation: non relativistic, relativistic, modified + anomalous scattering factors
2nd order S-matrix calculationsrecent calculations, not yet used in Monte Carlo codes
e = fraction of test cases compatible with experiment, 0.01 significance
Differential cross sections
State of the art
Quantification Statistical analysis, GoF + categorical
Maria Grazia Pia, INFN Genova
Differential Compton scattering cross section
model efficiency error
EPDL 0.82 0.02
Penelope 0.82 0.02
Klein-Nishina 0.54 0.03
Brusa 0.84 0.02
BrusaF 0.84 0.02
PenBrusa 0.84 0.02
PenBrusaF 0.84 0.02
Biggs 0.84 0.02
BiggsF 0.85 0.02
Hubbell 0.82 0.02
Work in progress!
Various scattering functions are evaluated w.r.t. experimental data
>2300 experimental data
Geant4 standard
Geant4 lowenergy
Maria Grazia Pia, INFN Genova
e+e- pair productionTotal cross section: Bethe-Heitler with corrections (Hubbell, Gimm, Overbo)
Near threshold
24
Geant4 standard EPDL XCOM
p-value <0.001 0.982 <0.001E>1.119 MeV
Validation at high energy in progress
Maria Grazia Pia, INFN Genova25
Electron backscattering
• Goudsmit-Saunderson• Urban• WentzelVI• Single Coulomb scattering+ Various Geant4 PhysicsLists
with various configuration options
Interplay of geometry and physics
Urban model, “DistanceToBoundary step limitation option
S. H. Kim, M. G. Pia, T. Basaglia, M. C. Han, G. Hoff, C. H. Kim, P. Saracco, Validation Test of Geant4 Simulation of Electron Backscattering, IEEE Trans. Nucl. Sci., vol. 62, no. 2, pp. 451-479, Apr. 2015.
T. Basaglia, M. C. Han, G. Hoff, C. H. Kim, S. H. Kim, M. G. Pia, P. Saracco, Investigation of Geant4 Simulation of Electron Backscattering, IEEE Trans. Nucl. Sci., submitted March 2015.
Further work in progress
Maria Grazia Pia, INFN Genova
Uncertainty quantification
26
Input
observablewith uncertainties
Monte Carlo methodStatistical uncertainty
Uncertainty quantification is the ground for predictive Monte Carlo simulation
Beware: input uncertainties can be hidden in the code(in models and algorithms)
Validation of MC modeling ingredients
Parameter uncertainties
N18-5 Progress with Uncertainty Quantification in Generic Monte Carlo Simulations
cross sections,branching ratios,physics models,physics parameters..
Maria Grazia Pia, INFN Genova
ConclusionDetector design, experimental strategies, physics results depend critically on software
Monte Carlo simulation plays a crucial role in many experimental domains
Methods of simulation validation‒ Basic physics and complex experimental observables‒ Unit tests and full simulation applications‒ Quantification through statistical methods
Testability embedded in the software design‒ Since the early stages of the software development
Ongoing effort to make Geant4 physics testablehttp://www.ge.infn.it/geant4/papers and to test it