ansys quality assurance & asme v&v standards
Post on 05-Dec-2021
9 Views
Preview:
TRANSCRIPT
© 2012 ANSYS, Inc. June 6, 2013 1 ANSYS Confidential
ANSYS Quality Assurance & ASME V&V Standards
Dimitri Tselepidakis
ASME 2013 Verification and Validation Symposium May 24th, 2013
© 2012 ANSYS, Inc. June 6, 2013 2 ANSYS Confidential
Click to edit Master text styles
Introduction to ANSYS
ANSYS Products
ANSYS QA Program
Standards and Regulatory Compliance
Application of ASME V&V 20 Standard
Discussion and Summary
Agenda
© 2012 ANSYS, Inc. June 6, 2013 3 ANSYS Confidential
Introduction to ANSYS
• ANSYS, Inc. develops and markets engineering simulation software and services used in the aerospace, automotive, manufacturing, electronics, biomedical, energy, defense, and many other industries.
• ANSYS is dedicated to engineering simulation and is the world's leading software provider.
• ANSYS, Inc. was founded in 1970 and is headquartered in Canonsburg, Pennsylvania.
© 2012 ANSYS, Inc. June 6, 2013 4 ANSYS Confidential
Click to edit Master text styles
Consolidated Company Geographic Revenue
Percent of Net Sales Revenue by Industry
Academic 8%
Aerospace & Defense 17%
Automotive 15%
BioMedical 2% Construction
2% Consumer Products
2%
Electronics 19%
Energy and CleanTech 10%
Industrial Equipment 13%
Materials & Chemical Processing
12%
Americas 34%
Asia 32%
Europe 34%
ANSYS Balance
© 2012 ANSYS, Inc. June 6, 2013 5 ANSYS Confidential
Fluid Dynamics Structural Mechanics
ANSYS Simplorer
ANSYS Engineering Knowledge Manager
ANSYS HPC ANSYS Workbench
Electromagnetics
ANSYS DesignXplorer
Systems and Multiphysics
ANSYS FLUENT
ANSYS CFX
ANSYS POLYFLOW
ANSYS Icepak
ANSYS HFSS
ANSYS Maxwell
ANSYS Designer
Apache RedHawk
ANSYS Mechanical
ANSYS AUTODYN
ANSYS LS-DYNA
ANSYS nCode
Products Overview
© 2012 ANSYS, Inc. June 6, 2013 6 ANSYS Confidential
Introduction to the ANSYS Quality System
ANSYS, Inc. signed first agreement with a nuclear power industry corporation in 1970
ANSYS, Inc.’s Quality System was developed and evolved to meet all requirements of the nuclear industry and later enhanced to meet ISO 9001 requirements
ANSYS Quality System is applicable to all ANSYS software products and services:
Development
Testing
Production
Delivery
Customer Training and Support
ANSYS end users have confidence in analysis results, meaning they can rely less on costly physical testing
© 2012 ANSYS, Inc. June 6, 2013 7 ANSYS Confidential
ANSYS Quality Program History
1970s
ANSYS, Inc. formed in 1970
• Many of our first customers were in the nuclear industry
Development and testing processes established to meet 10CFR50 appendix B and later NQA-1
NQA developed by ASME, recognized as a consensus standard, is an interpretation of 10CFR50 Appendix B requirements plus good practice, first published in 1979
1980s
1983 - ANSYS established a quality department, Quality System, and published NQA compliant Quality Manual, Technical Procedures Manual, and policies
1990s
1994 – Expanded Quality System to address all elements of ISO 9001:1994
1995 - Achieved ISO 9001:1994 certification
2000s
ANSYS Quality System certified to each revision of ISO 9001; ISO 9001:2000 and ISO 9001:2008
Integrated single NQA-1 and ISO 9001 compliant Quality System
© 2012 ANSYS, Inc. June 6, 2013 8 ANSYS Confidential
Standards and Regulatory Compliance
Standards Certification and Compliance
ISO 9001 : 2008
ASME NQA-1 2012 (subpart 2.7 is specific to software) is industry and NRC accepted approach (consensus standard) for meeting 10CFR50 Appendix B requirements
Regulatory Compliance
10CFR50 Appendix B – Title 10 (Energy) Code of Federal Regulations Part 50 Appendix B – Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants
10CFR21 – Title 10 (Energy) Code of Federal Regulations Part 21 – Reporting of Defects and Noncompliance
© 2012 ANSYS, Inc. June 6, 2013 9 ANSYS Confidential
Standards and Regulatory Compliance
ANSYS Inc. has successfully passed over 100 customer quality system audits against NQA-1 and 10CFR50 Appendix B since company was founded and over 60 since 1997
ANSYS Inc. has successfully passed over 100 ISO 9001 assessments and has retained ISO Certification since initial ISO 9001 : 1994 certification in 1995
© 2012 ANSYS, Inc. June 6, 2013 10 ANSYS Confidential
ANSYS QA Services
• ANSYS, Inc. runs tens of thousands of test cases to test the code at each release. We generate tests to verify product against high level product requirements and against lower level features and functionality.
• A subset of tests are included in the ANSYS QA Services Verification Testing Packages.
• If Verification Testing tests run on the customer’s installation the same as they did at ANSYS Inc.’s, it is reasoned that all tests would give results same as ANSYS, Inc.’s testing.
© 2012 ANSYS, Inc. June 6, 2013 11 ANSYS Confidential
ANSYS Software Testing
• We use traceability analysis to ensure requirement and feature test coverage. We also make selective use of run time analysis tools to verify coverage at the code level.
• Many of our tests are automated to facilitate regression testing and cross platform testing. However we also make use of manual testing approaches including exploratory testing methods and end user testing on real world applications (in collaboration with our technical support staff and customers). There is also a preview program allowing evaluation by customers prior to release.
© 2012 ANSYS, Inc. June 6, 2013 12 ANSYS Confidential
ANSYS QA Services
While ANSYS’ Quality System applies to all products, ANSYS QA Services are available for some selected products/licenses:
ANSYS Structural Mechanical Products (ANSYS Multiphysics, ANSYS Mechanical/Mechanical Emag/Mechanical CFX-Flo, ANSYS Structural, ANSYS Workbench Mechanical)
ANSYS Fluid Dynamics Products (ANSYS CFD, ANSYS CFX, and ANSYS Fluent)
© 2012 ANSYS, Inc. June 6, 2013 13 ANSYS Confidential
Class 3 Error Reporting
ANSYS, Inc. was the first software company to classify and define errors in its products.
Our error classifications have become an industry standard.
A Class3 error is defined as an error that allows the program execution to completion and yield results that may be wrong, but are not easily identifiable as incorrect.
Class3 error handling is a key part of our defect handling process.
Nuclear Industry has particular reporting needs – expedited traceable Class3 Report delivery.
Class3 Errors per Thousand Lines of ANSYS Code
0
0.05
0.1
0.15
0.2
0.25
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
YTD
Year
Nu
mb
er
of
Cla
ss3 E
rro
rs p
er
/
KL
OC
Yearly GoalSpace Shuttle Software Errors per KLOC
Better
© 2012 ANSYS, Inc. June 6, 2013 14 ANSYS Confidential
Class3 Error Reports and 10CFR21
ANSYS, Inc.’s Quality System meets 10CFR21:
10CFR21 “Reporting of Defects and Noncompliances” for commercial grade items. Relates to reporting of safety hazards to the USNRC.
Most important aspect – ANSYS, Inc. provides customers Class3 Error Reports as the means to meet THEIR 10CFR21 responsibilities.
• All Quality Assurance Service licensees who use ANSYS software products for applications addressed in 10CFR21 and who must themselves comply with 10CFR21 are informed of all Class3 errors.
• It is their responsibility to use information in Class3 Error Reports to determine if any bug impacted their work and if so, whether a reportable instance exists per 10CFR21.
• Class3 Error Reports also allow customers to avoid bugs in their future analyses.
© 2012 ANSYS, Inc. June 6, 2013 15 ANSYS Confidential
CFD Quality Assurance Services
• CFD portion of ANSYS’ product suite are produced under our Quality System which meets the requirements of ISO-9001,10CFR50 Appendix B, 10CFR21 and NQA-1
• Testing is a significant and important part of our QA processes (ranging from basic to industrial strength & a mixture of public and proprietary cases) – user interface – numerics – physical models – computing platforms – parallel computing – post-processing
• Subsets of the verification tests are published or can be made available to customers
• Customer can suggest addition of specific tests to our test suite
© 2012 ANSYS, Inc. June 6, 2013 16 ANSYS Confidential
Click to edit Master text styles
ANSYS V&V Practices in CFD
• ANSYS uses ASME Standard and additional practices as appropriate to each situation
– Method of Manufactured Solutions
– monitoring of “conserved quantities”
– multi-level integration testing
– recourse to experimental results
– benchmark code-to-code comparisons
• Due to breadth of ANSYS CFD features and models, V&V is a continuous and joint endeavor between code developer and end-user
– ANSYS performs verification on the basic features of the code
– ANSYS performs validation on a more-limited set of carefully selected problems
– ANSYS customers extend this foundation as needed for application-specific V&V
– ANSYS offers an array of software features and services to assist
© 2012 ANSYS, Inc. June 6, 2013 17 ANSYS Confidential
Click to edit Master text styles
• Navier-Stokes Equations
– Laminar Couette flow
Applying ASME V&V 20 Code I. Code Verification
b
Input Parameter Value
Moving plate velocity 3 m/s
Distance between plates 1 m
Length of domain 1.5 m
Pressure drop -12 Pa/m
Fluid density 1 kg/m3
Fluid viscosity 1 kg/m-s
© 2012 ANSYS, Inc. June 6, 2013 18 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code I. Code Verification
)
Mesh 4: 21 x 9 Mesh 3: 35 x 15
Mesh 2: 59 x 25 Mesh 1: 99 x 41
Loc-1
Loc-2
© 2012 ANSYS, Inc. June 6, 2013 19 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code I. Code Verification
)
Loc-1
Loc-2
© 2012 ANSYS, Inc. June 6, 2013 20 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code I. Code Verification
Solution Total
Number of
Cells
Cells
Along
x-direction
Cells
Along
y-direction
Velocity
at Loc-1
(m/s)
Velocity
at Loc-2
(m/s)
Mass Flow
Rate
(kg/s)
Mesh 4 189 21 9 3.0185 3.3889 2.5247
Mesh 3 525 35 15 3.0067 3.3800 2.5089
Mesh 2 1475 59 25 3.0024 3.3768 2.5032
Mesh 1 4059 99 41 3.0009 3.3757 2.5012
Analytical - - - 3.0000 3.3750 2.5000
Code Verification Results
Mesh Characteristic
Mesh Size, hi
(m)
Refinement
Factor,
hi+1/hi
Velocity
Error
at Loc-1
(m/s)
Velocity
Error
at Loc-2
(m/s)
Integrated
Mass Flow
Rate Error
(kg/s)
4 0.0891 - 0.0185 0.0139 0.0247
3 0.0535 1.67 0.0067 0.0050 0.0089
2 0.0319 1.68 0.0024 0.0018 0.0032
1 0.0192 1.66 0.0009 0.0007 0.0012
Error (Eh) in the Code Simulation during Mesh Refinement
© 2012 ANSYS, Inc. June 6, 2013 21 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code I. Code Verification
Meshes Velocity at Loc-1 Velocity at Loc-2 Mass Flow Rate
3 and 4 1.99 2.00 2.00
2 and 3 1.99 1.98 1.98
1 and 2 1.94 1.87 1.94
All (1 to 4) 1.97 1.95 1.97
Observed Order of Convergence from Mesh Refinement
Error
Mesh size, h
© 2012 ANSYS, Inc. June 6, 2013 22 ANSYS Confidential
Click to edit Master text styles
Pipe Expansion with Heat Flux
• Double precision
• Second-order discretization
• SST k-ω turbulence model with low-Reynolds-number corrections
• 3 meshes
Applying ASME V&V 20 Code II. Solution Verification
© 2012 ANSYS, Inc. June 6, 2013 23 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code II. Solution Verification
)
Input Parameter Value
Pipe diameter before expansion 0.0381 m
Pipe diameter after expansion
Heat flux
Mass flow
0.09525 m
10.0 W/m2
0.0513 kg/s
Density 1.225 kg/m3
Viscosity 1.683e-5 kg/m-s
Specific heat 1006.43 J/kg-K
Thermal conductivity 0.0242 W/m-K
Variables in the evaluation of uncertainties
• Reattachment length, L/H
• Nusselt number at the heated wall
Mesh #3
© 2012 ANSYS, Inc. June 6, 2013 24 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code II. Solution Verification
)
L/H
N1 620,800 620,800
N2 155,200 155,200
N3 38,800 38,800
r21 2 2
r32 2 2
11.290 2.284
11.280 2.269
11.250 2.230
p 1.58 1.38
11.295 2.293
0.09% 0.7%
0.04% 0.4%
0.06% 0.6%
0.17% 1.3%
© 2012 ANSYS, Inc. June 6, 2013 25 ANSYS Confidential
Click to edit Master text styles
ASME V&V 20 Procedure
• Estimate the modeling error of a given mathematical model in relation to a given set of experimental data
• Validation uncertainty
• Validation comparison error
• If |E| >> uval the comparison error is probably dominated by the modeling error, which indicates that the model must be improved
• For |E| < uval , modeling error is likely within the “noise level” imposed by the numerical, experimental and input uncertainties
Applying ASME V&V 20 Code III. Validation
© 2012 ANSYS, Inc. June 6, 2013 26 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code III. Validation
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
0.00 10.00 20.00 30.00 40.00 50.00 60.00
Baughn et al exp. data
Simulation
x/H
x/H
Nu(x)/NuDB
© 2012 ANSYS, Inc. June 6, 2013 27 ANSYS Confidential
Click to edit Master text styles
Applying ASME V&V 20 Code III. Validation
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.00 10.00 20.00 30.00 40.00 50.00 60.00
uval
|E|
x/H
Comparison of Validation Uncertainty with the Comparison Error
© 2012 ANSYS, Inc. June 6, 2013 28 ANSYS Confidential
Click to edit Master text styles
Discussion on ASME V&V 20
• ASME Standard involves a partitioning of non-numerical simulation error sources into “modeling assumptions” and “input parameters”
– input errors are combined with experimental and numerical error sources
• Alternative established definitions of verification and validation
– DoD: “the process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications.”
– IEEE: “providing objective evidence that the software and its associated products conform to requirements …”
• ANSYS typically adopts the “strong form” definition of the “model” so that, for example, BC uncertainties can be treated more explicitly.
© 2012 ANSYS, Inc. June 6, 2013 29 ANSYS Confidential
Click to edit Master text styles
Next Steps in CFD V&V
• A more general V&V approach is required, and further work is needed to establish robust factors of safety for the application of Richardson Extrapolation (RE) and Grid Convergence Method (GCI), or otherwise extend these methods, to more complex cases
– unstructured grids of widely varying sizes
– particle-based methods (typically associated with multiphase flows)
– moving-boundary problems
– transient fluid/thermal problems
– Large-Eddy Simulations (LES)
© 2012 ANSYS, Inc. June 6, 2013 30 ANSYS Confidential
Summary
• ANSYS simulation tools are quality engineering simulation products produced under a quality system that meets the requirements of 10CFR50 Appendix B, 10CFR21 & NQA-1 as well as ISO-9001
• The ASME V&V procedures represent a significant step of progress and clarity compared to commonly-adopted methods of comparison of simulation predictions with experiments
• ANSYS V&V practices are generally compliant with ASME V&V procedures subject to some CFD issues (V&V 20), these CFD issues suggest promising topics for further research
• Note that while ANSYS’ Quality System applies to all products, QA Services are available only for selected products
top related