verification and validation in computational geomechanicssokocalo.engr.ucdavis.edu/~jeremic/ ·...
TRANSCRIPT
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification and Validation in ComputationalGeomechanics
Boris JeremicMahdi Taiebat (UBC), Nima Tafazzoli, Panagiota
Tasiopoulou
Department of Civil and Environmental EngineeringUniversity of California, Davis
GheoMatMasseria Salamina
Italy, June 2009
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Outline
Before We Start
Why Do You Trust a Simulation?Verification, ValidationPrediction
Verification and ValidationThe T ExperimentsSelect V and V Examples
Summary
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Before We Start
Important SourcesI W. L. OBERKAMPF, T. G. TRUCANO, AND C. HIRSCH.
Verification, validation and predictive capability incomputational engineering and physics. In Proceedings ofthe Foundations for Verification and Validation on the 21stCentury Workshop, pages 1–74, Laurel, Maryland,October 22-23 2002. Johns Hopkins University / AppliedPhysics Laboratory.
I P. J. ROACHE. Verification and Validation in ComputationalScience and Engineering. Hermosa publishers, 1998.ISBN 0-913478-08-3.
I Material from Verification and Validation in ComputationalMechanics web site http://www.usacm.org/vnvcsm/at the USACM.
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Before We Start
Motivation
I How much can (should) we trust model implementations(verification)?
I How much can (should) we trust numerical simulations(validation)?
I How good are our numerical predictions?
I Can simulation tools (SimTool) be used for assessingpublic safety?
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Before We Start
Verification
I Verification: the process of determining that a modelimplementation accurately represents the developer’sconceptual description and specification. Mathematicsissue. Verification provides evidence that the model issolved correctly.
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Before We Start
Validation
I Validation: The process of determining the degree to whicha model is accurate representation of the real world fromthe perspective of the intended uses of the model. Physicsissue. Validation provides evidence that the correct modelis solved.
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Before We Start
Prediction
I Prediction: use of computational model to foretell the stateof a physical system under consideration under conditionsfor which the computational model has not been validated
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Before We Start
Role of Verification and Validation
MathematicalModel
ComputerImplementationDiscrete Mathematics
Continuum Mathematics
Programming
Analysis
Code Verification
SimulationComputer
ValidationModel
Reality Model Discovery
and Building
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Outline
Before We Start
Why Do You Trust a Simulation?Verification, ValidationPrediction
Verification and ValidationThe T ExperimentsSelect V and V Examples
Summary
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Importance of V & V
I V & V procedures are the primary means of assessingaccuracy in modeling and computational simulations
I V & V procedures are the tools with which we buildconfidence and credibility in modeling and computationalsimulations
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Maturity of Computational SimulationsNational Research Council committee (1986) identified fivestages of maturity in computational fluid dynamics
I Stage 1: Developing enabling technologies (scientificpapers published)
I Stage 2: Demonstration of and Confidence in technologiesand tools (capabilities and limitations of technologyunderstood)
I Stage 3: Compilation of technologies and tools(capabilities and limitations of technology understood)
I Stage 4: Spreading of the effective use (changes theengineering process, value exceeds expectations)
I Stage 5: Mature capabilities (fully dependable, costeffective design applications)
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Alternative V & V DefinitionsIEEE V & V definitions (1984):
I Verification: The process of determining whether theproducts of a given phase of the software developmentcycle fulfill the requirements established during theprevious phase
I Validation: The process of evaluating software at the endof the software development process to ensure compliancewith software requirements.
I Other organization have similar definitions:I Software quality assurance communityI American Nuclear Society (safety analysis of commercial
nuclear reactors)I International Organization for Standardization (ISO)
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Certification and AccreditationI Certification: A written guarantee that a system or
component complies with its specified requirements and isacceptable for operational use (IEEE (1990)).
I Written guarantee can be issued by anyone (codedeveloper, code user, independent code evaluator)
I Code certification is more formal than validationdocumentation
I Accreditation: The official certification that a model orsimulation is acceptable for use for a specific purpose(DOD/DMSO (1994))
I Only officially designated entities can accreditI Normally appointed by the customers/users of the code or
legal authorityI Appropriate for major liability or public safety applications
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Independence of Computational ConfidenceAssessment
1. V&V conducted by the computational tool developer, NoIndependence
2. V&V conducted by a user from same organization
3. V&V conducted by a computational tool evaluatorcontracted by developer’s organization
4. V&V conducted by a computational tool evaluatorcontracted by the customer
5. V&V conducted by a computational tool evaluatorcontracted by the a legal authority High Independence
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Fundamentals of Verification and Validation
Real World
Benchmark PDE solutionBenchmark ODE solutionAnalytical solution
Complete System
Subsystem Cases
Benchmark Cases
Unit ProblemsHighly accurate solution
Experimental Data
Conceptual Model
Computational Model
Computational Solution
ValidationVerification
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
VerificationThe process of determining that a model implementationaccurately represents the developer’s conceptual descriptionand specification.
Benchmark PDE solutionBenchmark ODE solutionAnalytical solution
Highly accurate solution
Conceptual Model
Computational Model
Computational Solution
Verification
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Verification
Mathematics issue. Verification provides evidence that themodel is solved correctly.
I Identify and remove errors in computer codingI Numerical algorithm verificationI Software quality assurance practice
I Quantification of the numerical errors in computed solution
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Verification
I Improving software quality through development process
I Application Programming Interface (API) defines softwareservices developed by Developers that are used by Users
I Public API is forever (asset, liability)I Realistically, API evolvesI API design is tough (perfection unachievable, try anyway)I Literate programming for API
I Two sided software platform connect Developers ANDusers through positive feedback (through API)
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
ValidationThe process of determining the degree to which a model isaccurate representation of the real world from the perspectiveof the intended uses of the model.
Real World
Complete System
Subsystem Cases
Benchmark Cases
Unit Problems
Experimental Data
Conceptual Model
Computational Model
Computational Solution
Validation
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Validation
Quantification of uncertainties and errors in the computationalmodel and the experimental measurements
I Goals on validationI Tactical goal: Identification and minimization of
uncertainties and errors in the computational modelI Strategic goal: Increase confidence in the quantitative
predictive capability of the computational modelI Strategy is to reduce as much as possible the following:
I Computational model uncertainties and errorsI Random (precision) errors and bias (systematic) errors in
the experimentsI Incomplete physical characterization of the experiment
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Validation Procedure UncertaintyI Aleatory uncertainty→ inherent variation associated with
the physical system (variation in external excitation,material properties...). AKA irreducible uncertainty,variability and stochastic uncertainty.
I Epistemic uncertainty→ potential deficiency in any phaseof the modeling process that is due to lack of knowledge(poor understanding of mechanics...). AKA reducibleuncertainty, model form uncertainty and subjectiveuncertainty.
Hei
sen
ber
gp
rin
cip
le
un
cert
ain
tyA
leat
ory
un
cert
ain
tyE
pis
tem
ic
Det
erm
inis
tic
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Use of Physical Experiments for Validation
I Traditional ExperimentsI Improve the fundamental understanding of physics involvedI Improve the mathematical models for physical phenomenaI Assess component performance
I Validation ExperimentsI Model validation experimentsI Designed and executed to quantitatively estimate
mathematical model’s ability to simulate well definedphysical behavior
I The simulation tool (SimTool) (conceptual model,computational model, computational solution) is thecustomer
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Validation Experiments
I A validation experiment should be jointly designed andexecuted by experimentalist and computationalist
I Need for close working relationship from inception todocumentation
I Elimination of typical competition between eachI Complete honesty concerning strengths and weaknesses
of both experimental and computational simulations
I A validation experiment should be designed to capture therelevant physics
I Measure all important modeling data in the experimentI Characteristics and imperfections of the experimental
facility should be included in the model
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Validation Experiments (contd)I A validation experiment should use any possible synergism
between experiment and computational approachesI Offset strength and weaknesses of computations and
experimentsI Use high confidence simulations for simple physics to
calibrate of improve the characterization of theexperimental facility
I Conduct experiments with a hierarchy of physics complexityto determine where the computational simulation breaks(remember, SimTool is the customer!)
I Maintain independence between computational andexperimental results
I Blind comparison, the computational simulations should bepredictions
I Neither side is allowed to use fudge factors, parameters
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Verification, Validation
Validation Experiments (contd)I Validate experiments on unit level problems, hierarchy of
experimental measurements should be made whichpresent an increasing range of computational difficulty
I Use of qualitative data (e.g. visualization) and quantitativedata
I Computational data should be processed to match theexperimental measurement techniques
I Experimental uncertainty analysis should be developedand employed
I Distinguish and quantify random and correlated bias errorsI Use symmetry arguments and statistical methods to identify
correlated bias errorsI Make uncertainty estimates on input quantities needed by
the SimTool
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Prediction
Outline
Before We Start
Why Do You Trust a Simulation?Verification, ValidationPrediction
Verification and ValidationThe T ExperimentsSelect V and V Examples
Summary
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Prediction
Prediction
I Prediction: use of computational model to foretell the stateof a physical system under consideration under conditionsfor which the computational model has not been validated
I Validation does not directly make a claim about theaccuracy of a prediction
I Computational models are easily misused (unintentionallyor intentionally)
I How closely related are the conditions of the prediction andspecific cases in validation database
I How well is physics of the problem understood
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Prediction
Relation Between Verification, Validation andPrediction
Quantification of confidence in a prediction:
I How do I quantify verification and its inference value in aprediction?
I How do I quantify validation and its inference value in apredictions?
I How far are individual validation experiments from physicalsystem of interest?
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Prediction
Application Domain Complete or Partial Overlap
System Parameter
Sys
tem
com
plex
ity
DomainValidation
DomainApplication Domain
Application
System Parameter
Sys
tem
com
plex
ity
DomainValidation
I Rarely applicable to infrastructure objectsI Environmental influences (generalized loads, conditions,
wear and tare) are hard to predictI Human factors (I-35 bridge failure, forgot to multiply loads
with safety factor...)
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Prediction
Application Domain – No Overlap
System ParameterS
yste
m c
ompl
exity
ApplicationDomain
DomainValidation
Inference
I Inference⇒ Based on physics or statistics (or both)I Validation domain is actually an aggregation of tests and
thus might not be convex (bifurcation of behavior)I Validation domain for infrastructure objects (bridges, dams,
buildings, ports...) is exclusively non–overlapping with theapplication domain.
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Prediction
Importance of Models and Numerical Simulations
I Verified and Validated models can be used for assessingbehavior of
I components orI complete systems,
I with the understanding that the environmental influencescannot all be taken into the account prior to operation
I but with a good model, their influence on system behaviorcan be assessed as need be (before or after the event)
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Prediction
Prediction under Uncertainty
I Ever present uncertainty needs to be estimated forpredictions
I Identify all relevant sources of uncertainty (see next lectureon uncertain soils)
I Create mathematical representation of individual sources
I Propagate representation of sources through modeling andsimulation process
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
Outline
Before We Start
Why Do You Trust a Simulation?Verification, ValidationPrediction
Verification and ValidationThe T ExperimentsSelect V and V Examples
Summary
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
Errors in Scientific SoftwareI Les Hatton, Oakwood Computing
I "Extensive tests showed that many software codes widelyused in science and engineering are not as accurate as wewould like to think."
I "Better software engineering practices wold help solve thisproblem,"
I "Realizing that the problem exists is an important firststep."
I Large experiment over 4 years measuring faults (T1) andfailures (T2) of scientific and engineering codes
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
The T1 ExperimentsI Measured defects without running the code, measuring
formal consistency of 3,305,628 lines f77 and 1,928,011lines of C
I 100 codes, 40 application areas: graphics, nuc. mech.chem. aero. civil engineering, comms, DBs, med. systems
I Safety–critical and non–safety–critical applicationsI Applications with and without internationally standardized
systems of quality controlI Mature codes (1 - 20 years old), in regular useI Errors (select), arguments sequence in function calls,
misunderstanding of finite precision arithmetic, codecomplexity
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
T1: C Sources
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
T1: f77 Sources
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
The T2 Experiments
I Specific application area: seismic data processing (inverseanalysis)
I Echo sounding of underground and reconstructing"images" of subsurface geological structure
I Nine mature packages, using same algorithms were usedon a same data set!
I 14 primary calibration points for results checkI Results "fascinating and disturbing"
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
T2: Disagreement at Calibration Points
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
The T Experiments
T2: Stage 14, Interpretation of Data
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Select V and V Examples
Outline
Before We Start
Why Do You Trust a Simulation?Verification, ValidationPrediction
Verification and ValidationThe T ExperimentsSelect V and V Examples
Summary
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Select V and V Examples
Constitutive Integration Error MapsNormalized error: δr =
√(σij − σ∗
ij )(σij − σ∗ij )/
√σ0
pqσ0pq
von Mises, linear isotropic hardening:
0 50 100 150 2000
20
40
60
80
100
p (kPa)
q (k
Pa)
5e−010
5e−0101e−009
1e−009
1.5e−009(a) At θ = 0
0 0.2 0.4 0.6 0.8 10
20
40
60
80
100
θ (rad)
q (k
Pa)
1e−010
2e−0103e−0104e−010
5e−0106e−010
(b) At p = 100 kPa
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Select V and V Examples
Constitutive Integration Error Mapsvon Mises, non–linear kinematic hardening(Armstrong-Frederick):
0 50 100 150 2000
20
40
60
80
100
p (kPa)
q (k
Pa)
0.002
0.0040.006
0.0080.01 0.010.0120.0140.016
(a) At θ = 0
0 0.2 0.4 0.6 0.8 10
20
40
60
80
100
θ (rad)
q (k
Pa)
0.002
0.0040.0060.008
0.01 0.010.0120.0140.016
(b) At p = 100 kPa
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Select V and V Examples
Constitutive Integration Error MapsDafalias-Manzari, rotational kinematic hardening, boundingsurface:
0 50 100 150 2000
20
40
60
80
100
p (kPa)
q (k
Pa)
0.05
0.05
0.1
0.1
0.1
0.1
0.15
0.15
0.15
0.15
0.15
0.2
0.2
0.2
0.2
0.2
0.2
0.25
0.25
0.25
0.3
0.3
0.35
0.4
(a) At θ = 0
0 0.2 0.4 0.6 0.8 10
20
40
60
80
100
θ (rad)
q (k
Pa)
0.05 0.05
0.10.1
0.150.15
0.20.2
0.25
(b) At p = 100 kPa
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Select V and V Examples
Coupled Formulation and Implementation
(after Gajo 1995)Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Select V and V Examples
Coupled Formulation and Implementation
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Select V and V Examples
Material Model Validation (Dafalias-Manzari)
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics
Before We Start Why Do You Trust a Simulation? Verification and Validation Summary
Summary
I Importance of proper verification and validation fornumerical predictions used in design of infrastructureobjects
I Public (yours and mine) safety issue!
I Would you trust (design using) numerical simulations ifSimTool did not follow (extensive) Verification andValidation procedures?
Jeremic Computational Geomechanics Group
Verification and Validation in Computational Geomechanics