ansys quality assurance & asme v&v standards

© 2012 ANSYS, Inc. June 6, 2013 1 ANSYS Confidential

ANSYS Quality Assurance & ASME V&V Standards

Dimitri Tselepidakis

ASME 2013 Verification and Validation Symposium May 24th, 2013


Click to edit Master text styles

Introduction to ANSYS

ANSYS Products

ANSYS QA Program

Standards and Regulatory Compliance

Application of ASME V&V 20 Standard

Discussion and Summary

Agenda


Introduction to ANSYS

• ANSYS, Inc. develops and markets engineering simulation software and services used in the aerospace, automotive, manufacturing, electronics, biomedical, energy, defense, and many other industries.

• ANSYS is dedicated to engineering simulation and is the world's leading software provider.

• ANSYS, Inc. was founded in 1970 and is headquartered in Canonsburg, Pennsylvania.



Consolidated Company Geographic Revenue

Percent of Net Sales Revenue by Industry

Academic 8%

Aerospace & Defense 17%

Automotive 15%

BioMedical 2% Construction

2% Consumer Products

2%

Electronics 19%

Energy and CleanTech 10%

Industrial Equipment 13%

Materials & Chemical Processing

12%

Americas 34%

Asia 32%

Europe 34%

ANSYS Balance


Fluid Dynamics Structural Mechanics

ANSYS Simplorer

ANSYS Engineering Knowledge Manager

ANSYS HPC ANSYS Workbench

Electromagnetics

ANSYS DesignXplorer

Systems and Multiphysics

ANSYS FLUENT

ANSYS CFX

ANSYS POLYFLOW

ANSYS Icepak

ANSYS HFSS

ANSYS Maxwell

ANSYS Designer

Apache RedHawk

ANSYS Mechanical

ANSYS AUTODYN

ANSYS LS-DYNA

ANSYS nCode

Products Overview


Introduction to the ANSYS Quality System

ANSYS, Inc. signed first agreement with a nuclear power industry corporation in 1970

ANSYS, Inc.’s Quality System was developed and evolved to meet all requirements of the nuclear industry and later enhanced to meet ISO 9001 requirements

ANSYS Quality System is applicable to all ANSYS software products and services:

Development

Testing

Production

Delivery

Customer Training and Support

ANSYS end users have confidence in analysis results, meaning they can rely less on costly physical testing


ANSYS Quality Program History

1970s

ANSYS, Inc. formed in 1970

• Many of our first customers were in the nuclear industry

Development and testing processes established to meet 10CFR50 appendix B and later NQA-1

NQA developed by ASME, recognized as a consensus standard, is an interpretation of 10CFR50 Appendix B requirements plus good practice, first published in 1979

1980s

1983 - ANSYS established a quality department, Quality System, and published NQA compliant Quality Manual, Technical Procedures Manual, and policies

1990s

1994 – Expanded Quality System to address all elements of ISO 9001:1994

1995 - Achieved ISO 9001:1994 certification

2000s

ANSYS Quality System certified to each revision of ISO 9001; ISO 9001:2000 and ISO 9001:2008

Integrated single NQA-1 and ISO 9001 compliant Quality System



Standards Certification and Compliance

ISO 9001 : 2008

ASME NQA-1 2012 (subpart 2.7 is specific to software) is industry and NRC accepted approach (consensus standard) for meeting 10CFR50 Appendix B requirements

Regulatory Compliance

10CFR50 Appendix B – Title 10 (Energy) Code of Federal Regulations Part 50 Appendix B – Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants

10CFR21 – Title 10 (Energy) Code of Federal Regulations Part 21 – Reporting of Defects and Noncompliance



ANSYS Inc. has successfully passed over 100 customer quality system audits against NQA-1 and 10CFR50 Appendix B since company was founded and over 60 since 1997

ANSYS Inc. has successfully passed over 100 ISO 9001 assessments and has retained ISO Certification since initial ISO 9001 : 1994 certification in 1995


ANSYS QA Services

• ANSYS, Inc. runs tens of thousands of test cases to test the code at each release. We generate tests to verify product against high level product requirements and against lower level features and functionality.

• A subset of tests are included in the ANSYS QA Services Verification Testing Packages.

• If Verification Testing tests run on the customer’s installation the same as they did at ANSYS Inc.’s, it is reasoned that all tests would give results same as ANSYS, Inc.’s testing.


ANSYS Software Testing

• We use traceability analysis to ensure requirement and feature test coverage. We also make selective use of run time analysis tools to verify coverage at the code level.

• Many of our tests are automated to facilitate regression testing and cross platform testing. However we also make use of manual testing approaches including exploratory testing methods and end user testing on real world applications (in collaboration with our technical support staff and customers). There is also a preview program allowing evaluation by customers prior to release.


ANSYS QA Services

While ANSYS’ Quality System applies to all products, ANSYS QA Services are available for some selected products/licenses:

ANSYS Structural Mechanical Products (ANSYS Multiphysics, ANSYS Mechanical/Mechanical Emag/Mechanical CFX-Flo, ANSYS Structural, ANSYS Workbench Mechanical)

ANSYS Fluid Dynamics Products (ANSYS CFD, ANSYS CFX, and ANSYS Fluent)


Class 3 Error Reporting

ANSYS, Inc. was the first software company to classify and define errors in its products.

Our error classifications have become an industry standard.

A Class3 error is defined as an error that allows the program execution to completion and yield results that may be wrong, but are not easily identifiable as incorrect.

Class3 error handling is a key part of our defect handling process.

Nuclear Industry has particular reporting needs – expedited traceable Class3 Report delivery.

Class3 Errors per Thousand Lines of ANSYS Code

0

0.05

0.1

0.15

0.2

0.25

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

YTD

Year

Nu

mb

er

of

Cla

ss3 E

rro

rs p

er

/

KL

OC

Yearly GoalSpace Shuttle Software Errors per KLOC

Better


Class3 Error Reports and 10CFR21

ANSYS, Inc.’s Quality System meets 10CFR21:

10CFR21 “Reporting of Defects and Noncompliances” for commercial grade items. Relates to reporting of safety hazards to the USNRC.

Most important aspect – ANSYS, Inc. provides customers Class3 Error Reports as the means to meet THEIR 10CFR21 responsibilities.

• All Quality Assurance Service licensees who use ANSYS software products for applications addressed in 10CFR21 and who must themselves comply with 10CFR21 are informed of all Class3 errors.

• It is their responsibility to use information in Class3 Error Reports to determine if any bug impacted their work and if so, whether a reportable instance exists per 10CFR21.

• Class3 Error Reports also allow customers to avoid bugs in their future analyses.


CFD Quality Assurance Services

• CFD portion of ANSYS’ product suite are produced under our Quality System which meets the requirements of ISO-9001,10CFR50 Appendix B, 10CFR21 and NQA-1

• Testing is a significant and important part of our QA processes (ranging from basic to industrial strength & a mixture of public and proprietary cases) – user interface – numerics – physical models – computing platforms – parallel computing – post-processing

• Subsets of the verification tests are published or can be made available to customers

• Customer can suggest addition of specific tests to our test suite



ANSYS V&V Practices in CFD

• ANSYS uses ASME Standard and additional practices as appropriate to each situation

– Method of Manufactured Solutions

– monitoring of “conserved quantities”

– multi-level integration testing

– recourse to experimental results

– benchmark code-to-code comparisons

• Due to breadth of ANSYS CFD features and models, V&V is a continuous and joint endeavor between code developer and end-user

– ANSYS performs verification on the basic features of the code

– ANSYS performs validation on a more-limited set of carefully selected problems

– ANSYS customers extend this foundation as needed for application-specific V&V

– ANSYS offers an array of software features and services to assist



• Navier-Stokes Equations

– Laminar Couette flow

Applying ASME V&V 20 Code I. Code Verification

b

Input Parameter Value

Moving plate velocity 3 m/s

Distance between plates 1 m

Length of domain 1.5 m

Pressure drop -12 Pa/m

Fluid density 1 kg/m3

Fluid viscosity 1 kg/m-s




)

Mesh 4: 21 x 9 Mesh 3: 35 x 15

Mesh 2: 59 x 25 Mesh 1: 99 x 41

Loc-1

Loc-2




)

Loc-1

Loc-2




Solution Total

Number of

Cells

Cells

Along

x-direction

Cells

Along

y-direction

Velocity

at Loc-1

(m/s)

Velocity

at Loc-2

(m/s)

Mass Flow

Rate

(kg/s)

Mesh 4 189 21 9 3.0185 3.3889 2.5247

Mesh 3 525 35 15 3.0067 3.3800 2.5089

Mesh 2 1475 59 25 3.0024 3.3768 2.5032

Mesh 1 4059 99 41 3.0009 3.3757 2.5012

Analytical - - - 3.0000 3.3750 2.5000

Code Verification Results

Mesh Characteristic

Mesh Size, hi

(m)

Refinement

Factor,

hi+1/hi

Velocity

Error

at Loc-1

(m/s)

Velocity

Error

at Loc-2

(m/s)

Integrated

Mass Flow

Rate Error

(kg/s)

4 0.0891 - 0.0185 0.0139 0.0247

3 0.0535 1.67 0.0067 0.0050 0.0089

2 0.0319 1.68 0.0024 0.0018 0.0032

1 0.0192 1.66 0.0009 0.0007 0.0012

Error (Eh) in the Code Simulation during Mesh Refinement




Meshes Velocity at Loc-1 Velocity at Loc-2 Mass Flow Rate

3 and 4 1.99 2.00 2.00

2 and 3 1.99 1.98 1.98

1 and 2 1.94 1.87 1.94

All (1 to 4) 1.97 1.95 1.97

Observed Order of Convergence from Mesh Refinement

Error

Mesh size, h



Pipe Expansion with Heat Flux

• Double precision

• Second-order discretization

• SST k-ω turbulence model with low-Reynolds-number corrections

• 3 meshes

Applying ASME V&V 20 Code II. Solution Verification




)

Input Parameter Value

Pipe diameter before expansion 0.0381 m

Pipe diameter after expansion

Heat flux

Mass flow

0.09525 m

10.0 W/m2

0.0513 kg/s

Density 1.225 kg/m3

Viscosity 1.683e-5 kg/m-s

Specific heat 1006.43 J/kg-K

Thermal conductivity 0.0242 W/m-K

Variables in the evaluation of uncertainties

• Reattachment length, L/H

• Nusselt number at the heated wall

Mesh #3




)

L/H

N1 620,800 620,800

N2 155,200 155,200

N3 38,800 38,800

r21 2 2

r32 2 2

11.290 2.284

11.280 2.269

11.250 2.230

p 1.58 1.38

11.295 2.293

0.09% 0.7%

0.04% 0.4%

0.06% 0.6%

0.17% 1.3%



ASME V&V 20 Procedure

• Estimate the modeling error of a given mathematical model in relation to a given set of experimental data

• Validation uncertainty

• Validation comparison error

• If |E| >> uval the comparison error is probably dominated by the modeling error, which indicates that the model must be improved

• For |E| < uval , modeling error is likely within the “noise level” imposed by the numerical, experimental and input uncertainties

Applying ASME V&V 20 Code III. Validation




0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

0.00 10.00 20.00 30.00 40.00 50.00 60.00

Baughn et al exp. data

Simulation

x/H

x/H

Nu(x)/NuDB




0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.00 10.00 20.00 30.00 40.00 50.00 60.00

uval

|E|

x/H

Comparison of Validation Uncertainty with the Comparison Error



Discussion on ASME V&V 20

• ASME Standard involves a partitioning of non-numerical simulation error sources into “modeling assumptions” and “input parameters”

– input errors are combined with experimental and numerical error sources

• Alternative established definitions of verification and validation

– DoD: “the process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications.”

– IEEE: “providing objective evidence that the software and its associated products conform to requirements …”

• ANSYS typically adopts the “strong form” definition of the “model” so that, for example, BC uncertainties can be treated more explicitly.



Next Steps in CFD V&V

• A more general V&V approach is required, and further work is needed to establish robust factors of safety for the application of Richardson Extrapolation (RE) and Grid Convergence Method (GCI), or otherwise extend these methods, to more complex cases

– unstructured grids of widely varying sizes

– particle-based methods (typically associated with multiphase flows)

– moving-boundary problems

– transient fluid/thermal problems

– Large-Eddy Simulations (LES)


Summary

• ANSYS simulation tools are quality engineering simulation products produced under a quality system that meets the requirements of 10CFR50 Appendix B, 10CFR21 & NQA-1 as well as ISO-9001

• The ASME V&V procedures represent a significant step of progress and clarity compared to commonly-adopted methods of comparison of simulation predictions with experiments

• ANSYS V&V practices are generally compliant with ASME V&V procedures subject to some CFD issues (V&V 20), these CFD issues suggest promising topics for further research

• Note that while ANSYS’ Quality System applies to all products, QA Services are available only for selected products

ansys quality assurance & asme v&v standards

Documents