a theoretical formalism for verification and validation ...oden/dr._oden... · verification and...

A THEORETICAL FORMALISM FORVERIFICATION AND VALIDATION

Iva Babuska and J.Tinsley Oden

Institute for Computational Engineering and SciencesThe University of Texas at Austin

VERIFICATION AND VALIDATION WORKSHOPAustin, TXApril 13, 2004

No theoqj call be e:l:JlOsedto all relevant tests' .. Il.sknot whdheT a the01:lJ has been verijir.rl, Imt mtlwT itsprobability, iu the light. of the evidence that actuallye:.rists. ",TIIOl\IAS S. KUIIN(lD22-1[)!!G)

OUTLINE

1. Some Fundamental Questions in Computer Simulation andPrediction

2. A View of Verification and Validation (V & V)

3. Definitions, Rules, and Processes in V & V

4. Theoretical Formalism

5. Sample Application in Mechanics

6. Summary and Conclusions

1

1. SOME FUNDAMENTAL QUESTIONS

• Why do we compute and what do we want to know?

• What is a model and how do we select a model?

• Is what we want to know related to what can we compute?

• What criteria can we use in a V & V process?

• To what degree do we believe our prediction?

2

2. A View of Verifi cation and Validation

1. While not absolutely validated, sustained success under severe testscan at least corroborate a theory and render it a legitimate basis fordecision-making (Popper).

2. So long as a mathematical model and a computational model of aphysical event and code implementing the computational modelwithstand detailed and severe tests, claims such that themathematical model is validated, the computational model is verified,and the code is verified, can be made, with respect to a specificseries of tests and tolerances.

3. Unlike the notion of falsifiability such as Popper's, we distinguishscientific theory and a mathematical model, the latter representingonly limited implications of a general theory. Thus, a mathematicalmodel can be validated in a relative sense with acceptability relativeto tolerances for a particular event of interest, although the model inother circumstances may be invalidated.

3

3.1. DEFINITIONS

VALIDATION:The process of determining if a mathematical model of a physicalevent represents the actual physical event with sufficient accuracy.

VERIFICATION:The process of determining if a computational model obtained bydiscretizing a mathematical model of a physical event and the codeimplementing the model represents the mathematical model withsufficient accuracy.

Other Defi nitions:

1. Physical event

3. Mathematical model

5. Discretization

7. Code

4

2. Simulation

4. Data

6. Computational model

8. Prediction

3.2. RULES AND PROCESSFOR MODEL SELECTION

Rule 1: Wellposed ness and Qualitative PropertiesRule 2: Quantities of Interest

Rule 3: Selection of Tolerances

Rule 4: Feedback Control

Rule 5: Verifi cation Independent of ValidationRule 6: Data DependenceRule 7: Convergence

Rule 8: Reproducibility of Experimental Results

5

4. Theoretical Formalism

A mathematical model consists of input data and a mathematicaltheory. The theory is an operator A that takes input data into output.To an extent, A is independent of the actual data: it will take differentdata into different outputs.

Prediction is the image of particular input data under the operator Acovering the events.

I A(D*) = Q*

where fDa} U {DII} = JI)ltotall 0' = 1,··· ,Tn and j = 1,··· ,n{Do} = admissible data {'DII} = inadmissible dataDtotat = total dataQ = Quantities of Interest

6


5.1. The Physical Problem Setting

A solid block is in equilibrium under the action of traction T, (t) applied over the surface II.-;;B2C2])2 andthe displacements are zero on the surface Al BICIDlo as shown. The region !1 occupied the material is!1 = {(Xl,X2,X3) Ell, S\II: S = closure{O < Xl < a,O < X2 < c,O < x" < b ,

II = (XI' X2, x,,) E cylinder of radius r() : r = ± (X2 - c/2)2 + (x" - b + d)2, 0 < " ~ r(J,tan 0 = (x" - (b - 11))/(xl - c/2), 0 < 0 :S 21l'}

where a, b, c, 11, "0 are the nominal dimensions indicated In the fi gure. The assumptions are:

• The nominal dimensions (geometric data) and the traction 1'1(t) are given and are exact.

• The body is in quasi-static equilibrium,

7


°1

5.2. The Physical Event and Quantities of Interest,We are obliged to identify a c, (Q )

list of quantities of interest(target outputs) and toleranceswe believe are acceptable forvalues of these quantities inorder that we can confi dentlymake decisions.Quantities of Interest:

Cauchy stress tensor at points A and C1E nQ2 = average stress tensor over l1e(A)

8


5.3. Tolerances

The analyst specifi es tolerances rQl' rQ2' rQ3' rQ4' etc. thatset limits to values in the quantities of interest suffi cientlysmall that he/she can use predicted values to makedecisions:

IllQi - reality of eventll < rQi I

9


5.4. The Mathematical Model = Input Data + Mathematical Theory

The analyst selects a mathematical theory to describe the physicalevents, linear elasticity, in this example.

I THEORYX)

div O'(x, t)O'(x,t)e(X, t)

(f(x, t))(E(x))e(X, t)Vu(x, t)lsy'/1l

X E (0)

O'(x, t)e(X, t)u(x, t)f (x, t)

( . )

Cauchy stress tens07' at point x

stmin tensor at point x

displacement vector at point x

body f01'ce at point x

place holders fOT data

The mathematical theory is thus:Hooke's Law + Equilibrium

10


5.4. The Mathematical Model (cont'd)

Based on the theoretical formalism in Section 4:

I A('D*) = Q* = {Qj} and 'D* E {'Do} I

In this example:

Input Data

V*={O*,E*,J*,(BC)*, ..· }

Mathematical::}- I Theory I::}-

A( . )

11

Output

Q*


5.4. The Mathematical Model: Remarks/Examples

• The mathematical theory is an integral part of the mathematicalmodel, but is separate (distinct) from the input data, which is theremaining part of the mathematical model.

• Validation involves the determination of the acceptability of thetheory operator and input set, not the acceptability of the map for aparticular set of input data.

• The admissibility of data (the sets 'Vi E 'Vo) depends on thequantities of interest: If E is not smooth (e.g., at micro-scale level),then Q1, Q4 are meaningless (do not necessarily exist), but Q2, Q3can be well-defined or exist.

12


5.4. The Mathematical Model: Admissible Data Set

The selection of admissible data thus depends upon thequantities of interest.

For B'"YQi C Q, if

Di E Do: : A(Di) 6; B'"YQi

then the model is invalid (for example Q = Q1 is invalid, if Eis not smooth).

13


5.5. Computational ModelThe Code. The computational code involves the construction of a familyof discretizations {Aft} ofthe mathematical theory operator {A}; itincludes a variety of features: solvers, pre- and post-processes, meshgenerators, algorithms for computing quantities of interest, etc.

Input Data

1)~={O~,E~,f~',(BC)~, ... }

1)i.={0i., Ei.,J,:,(BC)i.,·" }

Mathematical=> I Theory I =>

14

Output


5.6. The Verification Process

Code Verification. The performance, convergence, fidelity, accuracy of thecode is tested for particular suites of input data 1)11. (not the basic data 1)*

or 1)iJ

Solution Verification. The analyst (once again recognizing that the goal isthe prediction of specific quantities of interest), sets tolerances/~1' /82' /~3 on the numerical error that can be tolerated in the discreteapproximation of the quantities of interest:

15


5.6. The Verification Process (cont'd)

The choice of a quantity of interest Qi and a tolerance ,aidetermines a set of discrete models {Ah} C {All} that arecapable of achieving the tolerance using a posteriori errorestimates of quantities of interest and adaptive meshing.

0, O2 0,

16

5. Sample Application in Mechanics5.6. The Verification Process: Remarks/Examples

• The code is a realization of the computational model, which is morethan a simple implementation. It involves choices of computationalalgorithms (e.g. FEM), linear system solvers, code optimization andparallel or distributed or grid computing. It also involves issues oferror estimation, convergence as well as rate of convergence, etc.

• The computational model needs to be augmented by a posteriorierror estimators for quantities of interest, and must lead toapproximate values of quantities of interest in the range oftolerances given a priori. This is controlled by a posteriori errorestimation and is an essential part of the solution verification.

• The lower bound of an error estimate serves as a check foreffectiveness of the upper bound. A posteriori error estimates areassumed to be reasonably accurate with a certain level ofconfidence based on the corroboration principle, i.e. by testingaccuracy on a sufficiently large and properly selected set ofproblems.

17

• A posteriori error estimation characterizes the difference betweencomputed and mathematically exact quantities of interestpredictable by the theory. Obviously, these quantities must existand the computational models of them must converge. Theseestimates have nothing to do with the error between the computedquantities of interest and reality.

• Both the mathematical theory operator and input data can involveprobabilistic features which are measures of uncertainty. In thesecases, the output necessarily also involves uncertainty. Forexample, stochastic functions can be used in the analysis ofelasticity problems.

WARNING:Approximations of StochasticsCan Lead to Models with No Solutions!

18


5.7. The Validation Process

The Validation process consists of designing a set of validation problemsfor which experiments can be made (or observational data can beobtained) relevant to the prediction.

Validation problems involves applying the mathematical theory operatorA to restricted data set iJ E fDa} (= admissible data) to test variousassumptions used to develop the mathematical model.

A(iJi) = Qiwhere Qi = output for validation problems

The validation problems generally call for developing solution-orientedcomputational models of each problem by controlling approximation errorvia a posteriori error estimation and use of adaptive meshing.

I Ah(iJ:t) = Q~I~ Qi IThe Qi are compared with experimental values ei'

19


5.7. The Validation Process: Remarks/Examples

• To validate the mathematical model is to validate the theory andinput data, not predictions related to particular data sets.

• The selection of the measure and the acceptability thresholds arevery delicate problems. In fact, the proper selection of measuresand acceptability are tightly connected to the quantities of interest.

• For physical experiments, the V & V principles must also be applied:

- Analog of Validation - Do we measure what we wish to measure?- Analog of Verification -Is measurement accurate enough?

• One related yet important issue in the validation process isreproducibility, and another important issue is quantification ofuncertainty.

20


5.8. Reproducibility of Experiments

• Generally, N samples are fabricated to obtain information on theproperties of physical system (for example, E, and v).

• Verification of experiments involves characterizing the accuracy ofapparatus (experimental setting, sensors and probes, e.g straingauge) The important question is how many experiments need to beperformed in order to obtain a standard deviation of (say) 5% for themean values of quantities of interest.

• Dog bone validation problem:

P(I)

21


5.8. Reproducibility of Experiments (cont'd)

The strain gauge measures ell, e33. Two gauges are placed on both sidesof the specimen to compensate for bending,

_ 1 (1) (2) _ 1 (1) (2)en - 2"(eu + ell) E:33 - 2"(e33 + e33)

The load P(t) is applied cyclically, slowly, and periodically, with amplitudePo. By Hooke's law,

where IAI is the area of cross-section of the dog bone sample. Weassume (a) Material is isotropic, (b) Material is homogeneous, (c) Theform of the dog bone sample does not influence the results.

22


5.8. Reproducibility of Experiments (cont'd)

Given experimentally determined E and v, we compute quantities ofinterest ili, iii as a function of P(t), iii = KiP(t) (average over twogauges per specimen. Now, define reproducibility indices eRand ev, forN experiments

1l:t<;lX lIe(i)(t) - ii(t) IILOO(O,T)

e l:S',J:SN~v = II ii (t) IILOO(o,T)I

where e(k)(t) = ell(t) or e33(t) for sample k.

23


5.9. Rejection Criteria: An Example

lief{ - evl 2: 0.041

I levi 2: 0.061

The rejection criteria is not only applicable to the data, E and v, but alsoto P(t) (especially Po).

The argument for the introduction of particular criteria is that the solutionof the elasticity problem is continuously dependent on the perturbation ofthe coefficients measured in the LOO-norm. Note that the selection of themeasure is very essential, but not unique.

24

5. Sample Application in Mechanics5.10. Uncertainty and its Quantification

• Worst Case Scenario: A relatively simple analysis to treatuncertainty (can be incorporated easily into existing codes).

• Fuzzy Set: At the core of fuzzy set theory is the membershipfunction, which is an alternative view and description of probabilitydensity function.

• Stochastic Formulation: A widely-used approach in dealing withuncertainty.

• Sensitivity Analysis: Extend the confidence in the use of a validatedmodel.

• Safety Factor: a quantify to address features not accounted for aspart of the modeling process.

• Information needed in all approaches is seldom completelyavailable and some expert opinions are needed.

25

6. SUMMARY AND CONCLUSIONS

1. The goal of computer prediction is to obtain quantities ofinterest within specifi ed tolerances, which in turn serveas a basis for the decision-making.

2. The quantities of interest are part of the mathematicalproblem.

3. Validation is always related to tolerances measured in away related to the predicted event. This is a majordifference with the purely Popperian approach.

4. Selection of the set of validation problems and thespecifi c rejection criteria, which could be of a statisticalnature, as well as the parameters used in quantifyingreproducibility, are usually based on only heuristics andhave to be directly related to the goals of prediction.

26

5. The validation process provides information for feedbackcontrol.

6. The mathematical problem, which is necessarilywell-posed and well-formulated, involves both the theoryand data.

7. Any numerical treatment must be verifi ed (a posteriorierror estimates need to be obtained).

8. Physical experiments need to be submitted to an Analogof Validation and Verifi cation processes.

9. These are always uncertainties in available information,and these should be quantifi ed and taken into account inthe V & V processes.

27

a theoretical formalism for verification and validation ...oden/dr._oden... · verification and...

Documents