handbook for v&v of digital systems

EPRI • 3412 Hillview Avenue, Palo Alto, California 94304 • PO Box 10412, Palo Alto, California 94303 USA800.313.3774 • 650.855.2121 • [email protected] • www.epri.com

Handbook for Verification andValidation of Digital SystemsRevision 1

TR-103291-R1

Final Report, December 1998

CosponsorAutoridad Regulatoria NuclearAv. del Libertador 8250Buenos Aires, Argentina

Project ManagerC. Terrado

EPRI Project ManagerR. Torok

Effective December 6, 2006, this report has been made publicly available in accordance with Section 734.3(b)(3) and published in accordance with Section 734.7 of the U.S. Export Administration Regulations. As a result of this publication, this report is subject to only copyright protection and does not require any license agreement from EPRI. This notice supersedes the export control restrictions and any proprietary licensed material notices embedded in the document prior to publication.

pcdo001

Rectangle

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

THIS PACKAGE WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN ACCOUNT OFWORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC.(EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) NAMEDBELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM:

(A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITHRESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEMDISCLOSED IN THIS PACKAGE, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULARPURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNEDRIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (III) THAT THIS PACKAGE ISSUITABLE TO ANY PARTICULAR USER'S CIRCUMSTANCE; OR

(B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDINGANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISEDOF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THISPACKAGE OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED INTHIS PACKAGE.

ORGANIZATION(S) THAT PREPARED THIS PACKAGE

S. Levy Incorporated

ORDERING INFORMATION

Requests for copies of this package should be directed to the EPRI Distribution Center, 207 Coggins Drive, P.O.Box 23205, Pleasant Hill, CA 94523, (925) 934-4212.

Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc.EPRI. POWERING PROGRESS is a service mark of the Electric Power Research Institute, Inc.

Copyright © 1998 EPRI, Inc. All rights reserved.

iii

CITATIONS

This report was prepared by

S. Levy Incorporated3425 S. Bascom AvenueCampbell, California 95008-7006

AuthorsJ. WoolleyS. Levy Incorporated

C. TerradoNuclear Regulatory Authority of Argentina

This report describes research sponsored by EPRI and Autoridad Regulatoria Nuclear.

The report is a corporate document that should be cited in the literature in thefollowing manner:

Handbook for Verification and Validation of Digital Systems, EPRI, Palo Alto, CA, andAutoridad Regulatoria Nuclear, Buenos Aires, Argentina. 1998. TR-103291-R1.

v

REPORT SUMMARY

With the increasing use of digital instrumentation and control systems in power plants,utilities must determine the dependability and predictability of such systems and theirsoftware. This updated handbook provides a comprehensive guide to help utilitiesunderstand the verification and validation (V&V) process. The handbook presents agraded approach to select convenient V&V methods, develop a V&V plan, generatenecessary documentation, and conduct appropriate V&V activities.

BackgroundV&V is a systematic program of review and testing activities performed throughout thedevelopment life cycle for digital system hardware and software. Verification is theprocess of evaluating a system or component during development. Validation is theprocess of evaluating a system or component at the end of the development processunder conditions representative of its intended use. Although a formal V&V program isnot without costs, it can enhance the reliability and performance of a system, decreaselicensing risk, and reduce the often substantial costs of long-term softwaremaintenance.

Objectives• To explain how V&V requirements apply to the software life cycle for specific

projects.

• To help utilities develop and implement a V&V plan, with emphasis on appropriatetechniques for testing, review, inspection, and analysis.

ApproachThe project team consolidated information from national and international standards,EPRI research results and guidelines, open literature, and vendor reports. Using EPRI’soriginal Handbook for V&V of Digital Systems as their basis, they updated this revisionwith information from more recent standards and publications, reviewed it forconsistency with related EPRI V&V reports, and added new insights, as appropriate.

ResultsThis handbook describes the three categories of V&V techniques: 1) Reviews andinspections—involving independent scrutiny of system documentation, code, or

vi

output; 2) Analysis—involving mathematical or graphical modeling of systemrequirements, design, and/or implementation; and 3) Testing—involving actualexecution of all or part of the software, using input data representative of system usage.

As a comprehensive reference guide, the handbook presents a self-contained discussionof V&V principles, methods, issues, task selection, applicable standards, and regulatoryconsiderations. The handbook recommends a classification approach to determine howmuch V&V to apply and where to focus efforts within a project. It provides links torelated sections of other EPRI reports and includes four abstracts of utility case studies,corresponding to complete case histories available in Volume 2 of the original V&Vhandbook. EPRI intends for utilities to use this handbook as a starting point for V&Vactivities, with lead-ins to more detailed references as needed. The revised handbookwill be available in CD-ROM format, with hard copy as an option. The CD willaccompany Volume 2 of the original V&V handbook and replaces the original Volumes1 and 3, which will no longer be available.

EPRI PerspectiveV&V is effective only if utilities apply it systematically throughout the software lifecycle so that the concepts actively interface with the development effort. In complexprojects, following a formal V&V plan may result in earlier resolution of problems andavoidance of major modifications later in the project development cycle.

Related EPRI V&V work includes Guidelines for the V&V of Expert System Softwareand Conventional Software (TR-103331, Vols. 1-8), Software Fault Reduction UsingComputer-Aided Software Engineering (CASE) Tools (TR-105989, Vols. 1-2), andVerification and Validation Guidelines for High Integrity Systems (TR-103916, Vols. 1-2). EPRI developed this handbook in conjunction with the Nuclear RegulatoryAuthority of Argentina, which provided both cofunding and technical services insupport of its preparation.

TR-103291-R1

Interest CategoriesInstrumentation and controlPlant support engineering

KeywordsVerification/ValidationInstrumentation/Control systemsComputer programs/softwareDigital systemsReliabilityQuality assurance

EPRI Licensed Material

vii

ABSTRACT

Because of the discrete nature of software-based digital systems, it is generallyimpossible to thoroughly evaluate and demonstrate that such systems are completelyfree of all faults that might cause failure. Particularly insidious are faults that resultfrom inadequate specification, design, or implementation, as well as those that remainundiscovered because of incomplete testing. Useful digital applications have so manydiscrete states and conditions that it is not practical to exhaustively test allcombinations and prove there are no faults. Verification and validation (V&V) is aquality assurance discipline that considers the process of developing a software-baseddigital system, in addition to the final system product, so that the possibility ofintroducing faults is carefully managed.

EPRI published the original three volume V&V Handbook in 1994. This updated reportreplaces Volumes 1 and 3. It addresses all aspects of V&V with special attention todigital systems for instrumentation and control (I&C) in nuclear power plants. Itsobjective is to consolidate, interpret, and link established information to help electricutility engineers and managers understand and apply V&V for their digital systemprojects. Such information includes national and international standards, EPRI researchreports and guidelines, regulatory publications, technical books and periodicals,vendor information, and industry experience. A graded approach to V&V is developedaccording to the critical nature of the system under evaluation.

Section 1 introduces V&V by answering a series of questions that might be asked by atypical engineer. This section also provides an overview of the general principles andthe important standards that define V&V.

Section 2 explains that V&V is effective only if applied systematically throughout thedevelopment life cycle coincident with specification, design, implementation, andtesting activities. V&V planning is also discussed.

Sections 3, 4, and 5 present detailed explanations of a variety of methods, techniques,and tools that are available for performing V&V. These sections are divided accordingto the three major categories of V&V activities — reviews and inspections, analysis, andtesting.


viii

Section 6 describes a graded approach to V&V. It discusses classification of a systembased on the required level of integrity, then it recommends V&V activities to achievethe target integrity level. It also develops a practical approach for selecting V&Vmethods that are appropriate for each situation. Finally, it explains the relationshipbetween graded V&V and various technical and licensing issues, including comparisonwith the latest versions of national and international standards that support thismethodology.

Section 7 addresses other important issues that are related to V&V, including commonmode failure, defense-in-depth and diversity, software reliability, and metrics, amongothers.

Section 8 contains abstracts of four case studies covering the design and V&V processesfor digital upgrade of I&C systems at various nuclear power stations. These abstractsrelate to complete case histories that are available in Volume 2 of the original V&VHandbook.

Section 9 combines in alphabetical order the list of individual references that isincluded at the end of each previous section.

A detailed Table of Contents is included to help the reader locate material of interest.


ix

PREFACE

Objective

As illustrated below, this V&V Handbook draws on existing standards, EPRIguidelines, utility and regulatory experience, technical literature, and related sources.Its objective is to consolidate, interpret, and link established information to help electricutility engineers and managers

• Understand the important elements of V&V for a project.

• Prepare and execute a V&V Plan.

• Select the most appropriate V&V techniques.

EPRI V&VHANDBOOK

LINKS TO EPRI REPORTS

PRIMARY STANDARDS INDUSTRY EXPERIENCE

V&V HANDBOOK USERS

• Case studies included in Vol. 2 of the original V&V Handbook

OTHER EPRI ANDRELATED REPORTS

• EPRI TR-103331-1995• EPRI TR-103916-1995• EPRI TR-105989-1995• EPRI TR-106439-1996• EPRI TR-108831-1997

• Utility engineers and managers• Software engineers• System integrators• Vendors

• IEEE 7-4.3.2-1993• IEEE 1012-1998• IEC 61508-Draft 1998• IEC 880-1986• DO-178B/ED-12B-1992

• EPRI TR-102348-1993• EPRI TR-104595-1996• Digital Instrumentation and Control Systems in Nuclear Power Plants (National Research Council, 1997)

SELECTED BOOKS


x

Scope

The scope of this V&V Handbook can be broadly interpreted to include a wide range ofdigital applications ranging from real-time monitoring and control to engineeringanalysis to off-line diagnostic systems. The primary scope, however, is focused oninstrumentation and control (I&C) systems for nuclear power plants, with emphasis onhigh-integrity systems that can have a significant influence on the safety and/oroperation of the plant. This approach still encompasses a rather broad coverage of real-time monitoring, feedback and feed-forward control, protection, on-line diagnostic, andsimilar systems. For such applications, IEEE Std 7-4.3.2-1993 defines requirements fordesign and quality-related activities at the system level. This broad guidance issupplemented by more detailed information from other national and internationalstandards as well as EPRI reports and related technical literature.

Standards published by IEEE have been used for primary guidance in specific technicalareas. For example, IEEE Std 1012-1998 addresses V&V processes, while IEEE Std 830-1993 provides a starting point for specification of software requirements. This isconsistent with the approach taken by the US NRC in Regulatory Guides 1.168–1.173concerning software for nuclear plant safety systems. However, because of the broadrange of topics that are discussed here, as well as the dynamics of both technology andregulatory issues, several other domestic and international sources are also consulted.Some of these are indicated in the figure above.

It is anticipated that this V&V Handbook will be most useful to nuclear plant projectengineers who are responsible for planning and technical direction of digital upgradesto I&C systems. It should also provide valuable information for utility managers,software engineers, system integrators, and vendors who participate in such projects.This Handbook has been thoughtfully organized to facilitate locating the material ofinterest, and a detailed Table of Contents is included to support the reader.

Approach

The original Handbook for Verification and Validation of Digital Systems waspublished by EPRI in 1994. It included three volumes titled Summary, Case Histories,and Topical Reviews. Since 1994, EPRI and other organizations have sponsoredcontinuing efforts addressing V&V, several important standards were revised, andvarious regulatory issues have been resolved. Accordingly, EPRI determined that itwas desirable to update the V&V Handbook. As part of this update, it was recognizedthat the original structure of the Handbook was repetitious and difficult to follow, sothis was corrected by assembling a single volume arranged in a more logical way.Volume 2 of the original Handbook is retained to supply details of the Case Histories.

This V&V Handbook consolidates existing information from national and internationalstandards, EPRI research reports and guidelines, regulatory publications, technicalbooks and periodicals, vendor information, and industry experience to provide


xi

recommendations for V&V requirements, methods, and techniques. A graded approachto V&V is developed according to the critical nature of the system under evaluation.

Numerous EPRI reports related to digital I&C systems and published during recentyears were carefully considered to avoid any inconsistencies and to directly referencethem. Some of these reports are especially applicable to V&V and are given specialattention. These include

• EPRI TR-103331, Guidelines for the Verification and Validation of Expert System Softwareand Conventional Software, May 1995.

• EPRI TR-103916, Verification and Validation Guidelines for High Integrity Systems,December 1995.

• EPRI TR-105989, Software Fault Reduction Using Computer-Aided Software Engineering(CASE) Tools, June 1995.

• EPRI TR-106439, Guideline on Evaluation and Acceptance of Commercial Grade DigitalEquipment for Nuclear Safety Applications, October 1996.

• EPRI TR-108831, Requirements Engineering for Digital Upgrades: Specification, Analysis,Tracking, December 1997.

With respect to this V&V Handbook, each of the reports listed above has differentobjectives and characteristics; however, some themes overlap, and sometimes they aredeveloped in different ways. To facilitate comprehension of these reports and to guideuse of them in appropriate situations, graphical “i-buttons” are inserted throughout thisHandbook. These “i-buttons” point to specific sections in the other reports whererelated topics are also developed. Thus, this Handbook becomes a “home” document tolink with other V&V resources supplied by EPRI, allowing the reader to form a morecomplete understanding of the material that is presented. While these links lead toadditional discussion of a subject, the depth of that treatment may vary, sometimesgiving more or less detail or, perhaps, an alternative approach.

Where used, an “i-button” is included at the end of the section in this V&V Handbookthat discusses a related topic. Each one points to a specific part of the referenced EPRIreport where additional information concerning that particular subject may be found.An example “i-button” is presented below. The “i” displayed within the buttonsignifies additional information.

EPRI TR-103916 Vol. 1 Sec. 4.3


xii

Acknowledgments

The original V&V Handbook published in 1994 was prepared by S. Levy Incorporatedwith guidance from the EPRI/Utility Working Group on Verification and Validationand Related Issues, which participated in three meetings during 1992–1993. TheWorking Group included representatives from the following utilities, vendors,contractors, and other organizations:

Arizona Public ServiceBaltimore Gas & ElectricCarolina Power & LightCommonwealth EdisonDuke Power CompanyEntergyFlorida Power & LightGPU NuclearNortheast UtilitiesNorthern States PowerOntario HydroPacific Gas & ElectricSouthern Company ServicesTennessee Valley AuthorityToledo Edison

ABB/Combustion EngineeringAtomic Energy of Canada, Ltd.Electric Power Research InstituteFoxboroGeneral ElectricOak Ridge National LaboratoryS. Levy IncorporatedTriconexWestinghouse

In addition, R. S. May (SLI) and S. C. Bhatt (EPRI) were major contributors to theoriginal V&V Handbook.

Finally, Hugo Vallerga and Gustavo Caruso of the Nuclear Regulatory Authority ofArgentina deserve recognition for their generous support of the V&V Handbookupdate project.


xiii

CONTENTS

1 INTRODUCTION TO VERIFICATION AND VALIDATION .................................................. 1-1

1.1. What Is V&V? ............................................................................................................. 1-2

1.2. What Is the Relationship Between V&V and QA?....................................................... 1-4

1.3. What Is the Relationship Between V&V and CM? ...................................................... 1-6

1.4. Why Is V&V Important and What Are Its Costs and Benefits?.................................... 1-7

1.4.1. Regulator Perspective.......................................................................................... 1-7

1.4.2. Utility Perspective................................................................................................. 1-8

1.4.3. Vendor Perspective.............................................................................................. 1-9

1.5. How Much V&V Is Necessary? ................................................................................... 1-9

1.6. Who Does V&V?....................................................................................................... 1-10

1.7. What Is the Utility’s Role in V&V? ............................................................................. 1-11

1.8. General Principles..................................................................................................... 1-13

1.8.1. V&V Should Be Performed Throughout the Life Cycle....................................... 1-13

1.8.2. V&V Involves Advance Planning........................................................................ 1-13

1.8.3. V&V Should Address Both the Process and the Product ................................... 1-14

1.8.4. V&V Should Trace Results Relative to Requirements........................................ 1-15

1.8.5. V&V Effort Should Concentrate on High Value and High Risk ........................... 1-16

1.8.6. V&V Team Members Should Be Independent ................................................... 1-16

1.8.7. Development Products Should Be Scrutable ..................................................... 1-17

1.8.8. V&V Results Should Be Documented ................................................................ 1-17

1.9. Standards and Guides .............................................................................................. 1-17

1.10. References ............................................................................................................. 1-18

2 LIFE CYCLE VERIFICATION AND VALIDATION............................................................... 2-1

2.1. V&V for the Reference Life Cycle ............................................................................... 2-4

2.1.1. Concept and Planning.......................................................................................... 2-5

xiv

2.1.2. System Requirements.......................................................................................... 2-6

2.1.3. Software Prototyping............................................................................................ 2-6

2.1.4. Software Requirements........................................................................................ 2-8

2.1.5. Formal Requirements........................................................................................... 2-9

2.1.6. Software Design................................................................................................. 2-10

2.1.7. Software Implementation ................................................................................... 2-11

2.1.8. Software Integration........................................................................................... 2-11

2.1.9. System Integration ............................................................................................. 2-12

2.1.10. System Testing ................................................................................................ 2-12

2.1.11. Installation........................................................................................................ 2-13

2.1.12. Operation and Maintenance............................................................................. 2-14

2.1.13. Summary.......................................................................................................... 2-14

2.2. V&V Plan .................................................................................................................. 2-17

2.3. References ............................................................................................................... 2-19

3 V&V TECHNIQUES — REVIEWS AND INSPECTIONS ..................................................... 3-1

3.1. Elements of Reviews and Inspections ........................................................................ 3-2

3.2. Independent Reviews ................................................................................................. 3-3

3.3. Inspections.................................................................................................................. 3-5

3.4. Independent Witnessing ............................................................................................. 3-7

3.5. Regulatory Audits ....................................................................................................... 3-7

3.5.1. Standard Review Plan.......................................................................................... 3-8

3.5.2. Thread Path Audit ................................................................................................ 3-9

3.6. Summary .................................................................................................................. 3-11

3.7. References ............................................................................................................... 3-12

4 V&V TECHNIQUES — ANALYSIS...................................................................................... 4-1

4.1. Requirements Specification ........................................................................................ 4-2

4.1.1. Software Requirements Specification .................................................................. 4-2

4.1.2. Relation of SRS to V&V ....................................................................................... 4-4

4.2. Methods for Describing Requirements and Design..................................................... 4-4

4.2.1. Natural Language Text......................................................................................... 4-6

4.2.2. Structured Methods.............................................................................................. 4-7

4.2.2.1. Extensions for Real-Time Systems — Ward-Mellor and Hatley-Pirbhai ...... 4-13


xv

4.2.2.2. Extensions for Object-Oriented Design — Coad & Yourdan andSommerville............................................................................................................... 4-14

4.2.2.3. Impact of Structured Methods on V&V......................................................... 4-15

4.2.3. Formal Methods ................................................................................................. 4-16

4.2.3.1. Characteristics of Formal Methods.............................................................. 4-16

4.2.3.2. Formal Languages ...................................................................................... 4-18

4.2.3.3. Finite State Machines Using Graphical or Tabular Languages ................... 4-19

4.2.3.4. Impact of Formal Methods on V&V ............................................................. 4-23

4.2.4. Semi-Formal Methods ........................................................................................ 4-25

4.2.4.1. Statecharts .................................................................................................. 4-26

4.2.4.2. Impact of Semi-Formal Methods on V&V .................................................... 4-27

Simulation.............................................................................................................. 4-27

4.2.5. Function Block Languages................................................................................. 4-28

4.3. Other Analysis Techniques ....................................................................................... 4-32

4.3.1. Traceability Analysis........................................................................................... 4-33

4.3.2. Animation and Prototyping ................................................................................. 4-33

4.3.2.1. Requirements Animation ............................................................................. 4-33

4.3.2.2. Design Animation ........................................................................................ 4-34

4.3.2.3. Prototype Evaluation ................................................................................... 4-34

4.3.3. Static Analysis.................................................................................................... 4-34

4.3.4. Formal Proofs..................................................................................................... 4-36

4.4. Summary .................................................................................................................. 4-36

4.4.1 Formal (Mathematical) Languages...................................................................... 4-37

4.4.2 Tabular/Graphical Languages............................................................................. 4-37

4.4.3 Semi-Formal Methods and Tools ........................................................................ 4-38

4.4.4 Function Block Languages.................................................................................. 4-38

4.4.5 Final Considerations ........................................................................................... 4-38

4.5. References ............................................................................................................... 4-39

Appendix 4.A. Suggested Contents of SRS and SDD ..................................................... 4-42

4.A.1. Software Requirements Specification ................................................................ 4-42

4.A.2. Software Design Description.............................................................................. 4-43

Appendix 4.B. Application of Z to a System Performance Requirement .......................... 4-45

4.B.1. Performance Requirement................................................................................. 4-45

4.B.2. Z Specification of the Performance Requirement .............................................. 4-47


xvi

4.B.3. Comments on the Z Language .......................................................................... 4-56

4.B.4. Conclusions and Recommendations ................................................................. 4-57

Appendix 4.C. Statecharts Method and STATEMATE Modeling ..................................... 4-58

4.C.1. Activity-Charts — The Functional View.............................................................. 4-58

4.C.2. Statecharts — The Behavioral View .................................................................. 4-60

4.C.3. When Are Statecharts Appropriate? .................................................................. 4-64

5 V&V TECHNIQUES — TESTING ........................................................................................ 5-1

5.1. Terminology ................................................................................................................ 5-2

5.1.1. What Is Testing? .................................................................................................. 5-2

5.1.2. What Is Review? .................................................................................................. 5-3

5.1.3. What Is a Fault? ................................................................................................... 5-4

5.1.4. What Is Debugging?.............................................................................................. 5-5

5.1.5. What Are Component, Module, and Unit?............................................................ 5-6

5.1.6. What Is a Traceability Matrix? ............................................................................... 5-7

5.1.7. What Is the Relationship between Design and Testing?...................................... 5-9

5.1.8. What Are Types of Testing?............................................................................... 5-10

5.2. Testing Techniques .................................................................................................. 5-11

5.2.1. Functional Testing.............................................................................................. 5-13

5.2.1.1. Requirements Based Testing ...................................................................... 5-14

5.2.1.2. Equivalence Class Testing .......................................................................... 5-19

5.2.1.3. System Testing ........................................................................................... 5-20

5.2.1.4. Comparison Testing .................................................................................... 5-22

5.2.2. Structural Testing ............................................................................................... 5-22

5.2.2.1. Path Testing................................................................................................ 5-24

5.2.2.2. Branch Testing ............................................................................................ 5-27

5.2.2.3. Condition Testing ........................................................................................ 5-30

5.2.2.4. Data Flow Testing ....................................................................................... 5-31

5.2.2.5. Loop Testing ............................................................................................... 5-31

5.2.3. Other Testing Techniques.................................................................................. 5-33

5.2.3.1. Regression testing ...................................................................................... 5-34

5.2.3.2. Random Testing.......................................................................................... 5-34

5.2.3.3. Abnormal Conditions and Events ................................................................ 5-34

5.3. Testing Activities ....................................................................................................... 5-35


xvii

5.3.1. Test Design........................................................................................................ 5-35

5.3.2. Use of Tools ....................................................................................................... 5-38

5.3.3. When to Stop Testing......................................................................................... 5-39

5.3.4. Test Documentation ........................................................................................... 5-40

5.5. Standards for Nuclear Power Stations...................................................................... 5-43

5.5.1. Independence .................................................................................................... 5-44

5.5.2. Test Techniques................................................................................................. 5-45

5.6. Summary .................................................................................................................. 5-47

5.7. Selected Bibliography ............................................................................................... 5-48

5.8. References ............................................................................................................... 5-49

Appendix 5.A. Classification of Faults............................................................................... 5-51

Appendix 5.B. Types of Testing....................................................................................... 5-53

6 GRADED V&V FOR I&C SYSTEMS ................................................................................... 6-1

6.1. Technical Approach .................................................................................................... 6-1

6.1.1. System Classification ........................................................................................... 6-3

6.1.2. Software Classification......................................................................................... 6-4

6.1.2.1. Adjusting Factors .......................................................................................... 6-7

6.1.2.2. Restrictions on Application of Adjusting Factors ........................................... 6-9

6.1.3. Grading Process .................................................................................................. 6-9

6.1.3.1. Identification of Activities and Attributes...................................................... 6-10

6.1.3.2. Graded Recommendations for Activities ..................................................... 6-13

6.2. Practical Approach.................................................................................................... 6-20

6.2.1. What to Do: Determining the V&V Class............................................................ 6-21

6.2.2. How to Do It: Selecting Applicable V&V Methods .............................................. 6-23

6.2.3. Summary of the Practical Approach................................................................... 6-33

6.3. Relationship to Technical and Licensing Issues ....................................................... 6-35

6.3.1. Software Reliability, Common Mode Failure, and Diversity ................................ 6-35

6.3.2. Graded Software Development and V&V Practices ........................................... 6-36

6.3.3. Acceptance of Commercial Off-the-Shelf Products ............................................ 6-39

6.3.4. Relation to 10 CFR 50.59 Screening Criteria ..................................................... 6-41

6.4. Summary .................................................................................................................. 6-41

6.5. References ............................................................................................................... 6-42


xviii

7 ISSUES RELATED TO V&V................................................................................................ 7-1

7.1. System and Hardware vs. Software............................................................................ 7-1

7.2. System Engineering.................................................................................................... 7-1

7.3. Failure Analysis........................................................................................................... 7-3

7.4. Common Mode Failure ............................................................................................... 7-4

7.5. Redundancy, Diversity, and Defense-in-Depth ........................................................... 7-5

7.6. Probabilistic-Based Approach for Software Reliability Assessment ............................ 7-6

7.6.1. What Is Software Reliability?................................................................................ 7-6

7.6.2. Software Reliability Models .................................................................................. 7-7

7.6.3. Selection of a Software Reliability Model ........................................................... 7-10

7.6.4. Limitations for Application of Software Reliability Theory ................................... 7-10

7.6.5. Proposed Use of Software Reliability Theory in Nuclear Applications................ 7-11

7.7. Metrics ...................................................................................................................... 7-12

7.8. Tools ......................................................................................................................... 7-15

7.9. Licensing Digital I&C Upgrades ................................................................................ 7-15

7.10. Comparison of EPRI Reports Referring to V&V...................................................... 7-16

7.11. References ............................................................................................................. 7-19

8 ABSTRACTS OF DIGITAL I&C CASE STUDIES ............................................................... 8-1

8.1. Case #1 Westinghouse Eagle 21 Replacement Hardware for ReactorProtection Systems............................................................................................................. 8-2

8.1.1. Description ........................................................................................................... 8-2

8.1.2. Safety Significance............................................................................................... 8-2

8.1.3. Key Technical Issues ........................................................................................... 8-2

8.1.4. Classification ........................................................................................................ 8-3

8.1.5. V&V Organization................................................................................................. 8-3

8.1.6. V&V Plan.............................................................................................................. 8-4

8.2. Case #2 Arizona Public Service Palo Verde Diverse Auxiliary FeedwaterActuation System................................................................................................................ 8-4

8.2.1. Description ........................................................................................................... 8-4



8.2.4. Classification ........................................................................................................ 8-7

8.2.5. V&V Organization................................................................................................. 8-7

8.2.6. V&V Plan.............................................................................................................. 8-7


xix

8.3. Case #3 FPL Emergency Bus Load Sequencer ........................................................ 8-8

8.3.1. Description ........................................................................................................... 8-8



8.3.4. Classification ...................................................................................................... 8-10

8.3.5. V&V Organization............................................................................................... 8-10

8.3.6. V&V Plan............................................................................................................ 8-10

8.4. Case #4 B&W Owners Group Advanced Control System........................................ 8-11

8.4.1. Background........................................................................................................ 8-11

8.4.2. Description ......................................................................................................... 8-12

8.4.3. Safety Significance............................................................................................. 8-12

8.4.4. Key Technical Issues ......................................................................................... 8-12

8.4.4. Classification ...................................................................................................... 8-13

8.4.5. V&V Organization............................................................................................... 8-13

8.4.6. V&V Plan............................................................................................................ 8-14

8.5. References ............................................................................................................... 8-14

9 COMBINED LIST OF REFERENCES ................................................................................. 9-1


xxi

LIST OF FIGURES

Figure 1-1 Relationship of V&V, Quality Assurance, and System Significance...................... 1-6

Figure 1-2 Simple Example of Requirements Traceability Matrix......................................... 1-15

Figure 2-1 Reference Life Cycle for Digital System (Adapted from IEEE 7-4.3.2) ................. 2-2

Figure 2-2 Reference Life Cycle for Software Related V&V Activities................................... 2-3

Figure 3-1 Common-Sense Categories of V&V Techniques.................................................. 3-1

Figure 3-2 Reviews and Inspections in IEEE 7-4.3.2............................................................. 3-3

Figure 3-3 Concept of Thread Audit: Broad View ................................................................ 3-10

Figure 3-4 Concept of Thread Audit: Deep View ................................................................. 3-11

Figure 4-1 Analysis Techniques............................................................................................ 4-1

Figure 4-2 Example Data Flow Diagram ................................................................................ 4-8

Figure 4-3 Example Structure Chart ...................................................................................... 4-9

Figure 4-4 Example of High-Level Data Flow Diagram ....................................................... 4-10

Figure 4-5 Example of Lower Level (More Detailed) Data Flow Diagram ............................ 4-10

Figure 4-6 Example of Structure Chart ................................................................................ 4-11

Figure 4-7 Example of Process Specification (P-Spec) ....................................................... 4-13

Figure 4-8 Requirements and Design Specification Methods .............................................. 4-17

Figure 4-9 State Transition Diagram for an Electric Garage Door Opener........................... 4-20

Figure 4-10 Ontario Hydro Requirements Format — Function Overview............................. 4-22

Figure 4-11 Process of Refinement and Formal Verification................................................ 4-24

Figure 4-12 Function Block Definition for a Comparator ...................................................... 4-29

Figure 4-13 Example of a Trip Function Composed from Basic Function Blocks................. 4-30

Figure 4-14 Part of a Control System Specification using SAMA Diagram Language ........ 4-31

Figure 4-15 Meaning of Acceptable Band for Evaluating Transient Response.................... 4-46

Figure 4-16 Top-Level Activity-Chart for ACS...................................................................... 4-59

Figure 4-17 Lower Level Activity-Chart for Control Algorithm .............................................. 4-60

Figure 4-18 Top-Level Statechart for ACS........................................................................... 4-61

Figure 4-19 Generic Statechart for Operator Interface ........................................................ 4-63

Figure 4-20 Statechart that Invokes Interface Logic for Three Controllers........................... 4-64

Figure 5-1 Testing Techniques ............................................................................................ 5-12

Figure 5-2 Cause-Effect Graph Symbols ............................................................................. 5-16


xxii

Figure 5-3 Cause-Effect Graph for Example........................................................................ 5-17

Figure 5-4 Equivalence Class Test Data.............................................................................. 5-19

Figure 5-5 Simple Decision Paths........................................................................................ 5-25

Figure 5-6 Complex Decision Paths..................................................................................... 5-26

Figure 5-7 Flow Graph Notation........................................................................................... 5-27

Figure 5-8 Flow Graph for Example Module ........................................................................ 5-28

Figure 5-9 Graph Matrix for Example in Figure 5-8.............................................................. 5-30

Figure 5-10 Three Kinds of Loops ....................................................................................... 5-32

Figure 5-11 Data Flow Diagram........................................................................................... 5-37

Figure 5-12 Overlap During Testing Phase.......................................................................... 5-38

Figure 5-13 Illustration of Document Hierarchy.................................................................... 5-41

Figure 6-1 Overview of Technical Approach .......................................................................... 6-2

Figure 6-2 Selection of the V&V Class................................................................................. 6-21

Figure 6-3 Distribution of V&V Methods During Development Life Cycle for Each V&VClass ............................................................................................................................. 6-32

Figure 6-4 Factors Involved in Selection of V&V Class........................................................ 6-34

Figure 6-5 Factors Involved in System Acceptance............................................................. 6-35

Figure 6-6 Comparison of Graded Approach to Software Integrity ...................................... 6-39


xxiii

LIST OF TABLES

Table 1-1 Common-Sense Categories of V&V ...................................................................... 1-3

Table 1-2 Division of Responsibilities for Several Example Digital I&C UpgradeProjects ......................................................................................................................... 1-12

Table 1-3 Division of V&V Responsibilities for I&C Upgrade Projects.................................. 1-12

Table 2-1 Summary of Recommended Products for the Reference Life Cycle.................... 2-16

Table 2-2 V&V Plan Outline Recommended in IEEE 1012.................................................. 2-18

Table 3-1 Excerpts from Westinghouse Eagle 21 Review Checklists .................................... 3-5

Table 3-2 Comparison of Reviews and Inspections............................................................... 3-7

Table 4-1 Relative Cost for Repair of Software...................................................................... 4-2

Table 4-2 Desirable Attributes of an SRS (from IEEE 830).................................................... 4-3

Table 4-3 Desirable Attributes of an SRS (from Other Sources)............................................ 4-3

Table 4-4 Real-Time Extensions to Structured Methods...................................................... 4-14

Table 4-5 Ontario Hydro Requirements Format — State Transition Table .......................... 4-21

Table 4-6 Ontario Hydro Requirements Format — Structured Decision Table .................... 4-22

Table 4-7 Benefits of Employing Formal Methods ............................................................... 4-23

Table 4-8 Realization of the Three Specification Views for Several Methods...................... 4-26

Table 4-9 Typical Outline for Software Requirements Specification .................................... 4-42

Table 4-10 Typical Outline for Software Design Description................................................ 4-44

Table 4-11 Terminology and Notation Appearing in the Z Specification .............................. 4-48

Table 4-12 Abstract Specification of Typical Performance Requirements in the ZLanguage...................................................................................................................... 4-49

Table 5-1 Sample Traceability Matrix..................................................................................... 5-8

Table 5-2 Decision Table for Example................................................................................. 5-18

Table 5-3 Recommended Document Contents.................................................................... 5-42

Table 5-4 Characteristics of Functional and Structural Testing............................................ 5-48

Table 5-5 Classification of Faults......................................................................................... 5-52

Table 6-1 System Classification for Digital I&C Upgrades .................................................... 6-6

Table 6-2 Adjusting Factors for Software Integrity Level ...................................................... 6-8

Table 6-3 Common (Ungraded) Activities Applicable to All Software Integrity Levels.......... 6-10


xxiv

Table 6-4 Activities and Attributes That Can Be Adjusted to Achieve the Target Levelof Software Integrity ...................................................................................................... 6-12

Table 6-5 Development Activity Recommendations to Achieve the Target SoftwareIntegrity Level................................................................................................................ 6-16

Table 6-6 Explanation of Terms Used in Table 6-5.............................................................. 6-17

Table 6-7 Concepts of V&V Classification ........................................................................... 6-20

Table 6-8 Attributes for Selecting Applicable V&V Methods ................................................ 6-25

Table 6-9 Selection of Some Specific V&V Methods ........................................................... 6-26

Table 6-10 Description of Some Specific V&V Methods ...................................................... 6-28

Table 6-11 Relation to 10 CFR 50.59 (NSAC-125) Screening Criteria ................................ 6-41

Table 7-1 Software Reliability Models.................................................................................... 7-8

Table 7-2 Some Software Metrics........................................................................................ 7-14

Table 7-3 EPRI Documents Concerning V&V of Digital Systems ........................................ 7-17


1-1

1 INTRODUCTION TO VERIFICATION AND VALIDATION

During the 1960s and 1970s, digital technology began to assume an increasinglyimportant role in large systems for aerospace, national security, telecommunications,and energy applications. As digital systems were applied to increasingly important andcomplex tasks formerly handled manually or by analog systems, developers and usersof digital systems had to come to grips with issues of dependability and predictability.Such issues ranged from clearly expressing what a new system must do, to confirmingthat the delivered system actually performed as intended. This period witnessedemergence of the new discipline of software engineering that attempted to emulatesystematic approaches used in more traditional engineering specialties. The softwareengineering concept of Verification and Validation (V&V) was originally devised toimprove the quality of military software that was critical to success of a mission or tosafety of the public.

For the nuclear power industry, conversion to digital instrumentation and control (I&C)systems started to gain momentum during the late 1980s and early 1990s. The nuclearpower industry and its regulatory authorities have considered digital conversioncautiously, recognizing the high visibility of nuclear power plants and their potentialimpact on public safety. Today, however, many techniques and tools for V&V havematured, and industry standards governing the quality of critical digital systems arerapidly becoming established by consensus.

This V&V Handbook supports a broad methodology to achieve the greatest possibleconfidence that a digital system will not produce some extremely undesirablecondition. This methodology includes analysis of the system and its environment todetermine its target safety level, with the application of graded methods and processesto insure its quality. These methods and processes include:

• Quality assurance

• Configuration management

• Verification and validation

• Careful specification of requirements


Introduction to Verification and Validation

1-2

• Design for defense-in-depth

• Assessment of reliability

• Evaluation of the development environment

Finally, it is important to agree that when a software-based product is related tosignificant consequences, we should accept nothing less than a complete and pervasiveintolerance to any compromise of safety and security.

1.1. What Is V&V?

V&V is a systematic program of review and testing activities performed throughout thesystem development life cycle. IEEE 610.12 [1] and IEEE 7-4.3.2 [2] both define

verification and validation (V&V). The process of determining whether therequirements for a system or component are complete and correct, the productsof each development phase fulfill the requirements or conditions imposed by theprevious phase, and the final system or component complies with specifiedrequirements.

The first two parts of this definition are verification; the third part is validation.Verification includes detailed review and testing at each life cycle phase anddetermines whether the project is ready to move to the next phase. Validation, on theother hand, involves evaluating the overall behavior of the system under conditionsrepresentative of its intended use. Verification answers “Did we build the productcorrectly?” Validation answers “Did we build the correct product?”

The latest version of IEEE 1012 [3] simplifies the definitions of verification andvalidation somewhat, while also demanding evidence of the process

verification. Confirmation by examination and provisions of objective evidencethat specified requirements have been fulfilled.

validation. Confirmation by examination and provisions of objective evidencethat the particular requirements for a specific intended use are fulfilled.

IEEE 610.12 also defines

system. A collection of components organized to accomplish a specific functionor set of functions.

software. Computer programs, procedures, and possibly associateddocumentation and data pertaining to the operation of a computer system.



1-3

Although the latter definition seems to make it optional, we definitely should includedocumentation and data when considering software V&V. In particular, softwareincludes firmware, microcode, documentation, configuration data, executionparameters, and any other products of the software development process. We willfocus on software V&V because it has several unique aspects, but it should beremembered that software V&V is really a part of system V&V; i.e., the processes ofsoftware V&V are intended to assess the software in the context of the system.

Despite the large number of available techniques for performing V&V (cf. [3,4]), thesetechniques can be grouped into the three common-sense categories identified inTable 1-1 — Reviews and Inspections, Analysis, and Testing. For simplicity, this V&VHandbook will use Reviews and Inspections to categorize the three independent V&Vactivities that are listed in Annex E of IEEE 7-4.3.2 — Independent Reviews,Independent Witnessing, and Inspection. For detailed information concerning reviewsand inspections, see IEEE 1028 [5].

Table 1-1Common-Sense Categories of V&V

Category Description Examples

Reviews andInspections

Independent scrutiny of systemdocumentation, implementation (e.g., code),and/or results.

Design reviewCode inspectionTest witnessing

Analysis A mathematical or graphical model of systemrequirements, design, and/or implementationis constructed, then properties of the modelare analyzed. Often called static testing.

Data flow diagramsGraphical analysisFormal specification languageFormal design languageFormal proofs

Testing Execution of all or part of the software usingrepresentative input data. Often calleddynamic testing.

Structural (white-box) testingUnit testingAcceptance (black-box) testing

EPRI TR-103916 Vol. 1 Sec. 4.3 1

1 This is the first appearance of an “i-button,” which is used to link with related discussion in another EPRI report.See the Preface for a detailed explanation of the “i-button” concept.



1-4

1.2. What Is the Relationship Between V&V and QA?

V&V is a part of Quality Assurance (QA). It is concerned with the process as much asthe product. The verification part of V&V (Did we build the product correctly?) focuseson the process.

The relationship between V&V and QA can be a source of confusion when discussingsoftware-based digital systems for the nuclear power industry. Nuclear QA usuallyrefers to the documentation and review processes defined in 10 CFR 50 Appendix B(Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants).Appendix B of 10 CFR 50 strictly applies only to QA of “safety-related functions of …structures, systems, and components that prevent or mitigate the consequences ofpostulated accidents that could cause undue risk to the health and safety of the public.”However, it is prudent to apply a QA program that is appropriately derived from 10CFR 50 Appendix B for important non-safety nuclear systems, also.

Software QA is addressed by IEEE 730 [6]. In the software engineering community aswell as the IEEE standards, software QA usually refers to all of the activities that areperformed to build-in and insure software quality, whether the software is consideredcritical or non-critical. Software QA includes elements such as the softwaredevelopment life cycle, coding standards, management control, and V&V.

To avoid confusion, this V&V Handbook uses the following terms:

Nuclear QA. Quality assurance activities required for safety-related nuclearsystems as dictated by 10 CFR 50 Appendix B.

Software QA. Quality assurance activities for critical and non-critical softwareas described in IEEE 730.

V&V can be considered a subset of QA. So if a digital I&C project is performed under aNuclear QA program, then careful V&V is implied. But the reverse is not true.Performing V&V does not imply application of Nuclear QA. V&V is appropriate forinsuring quality in a software-based digital system even when Nuclear QA is notrequired.

Software QA includes a V&V plan and some level of V&V activities. Certainrequirements for nuclear safety systems may be regarded as an extra layer of activitiesover and above software QA and V&V. Figure 1-1 illustrates the relationships amongthese activities and between different classes of systems. It shows layers of effort thatbuild from the foundation of software QA (layer 4) to the highest integrity required forsafety-critical systems (layer 1). Figure 1-1 also suggests that deciding how much V&Veffort to apply depends on the intended function of the software.



1-5

IEEE 1012 [3] prescribes a minimum set of V&V activities for critical software (i.e.,safety or high-integrity applications). Four software integrity levels are defined toquantify software criticality. In addition, IEEE 1012 suggests optional V&V activitiesthat depend on the nature and risk level of the software and upon available resourcesand skills. For critical software, V&V should be formal and explicitly documented.

V&V of safety software should be performed by a team that is independent of thesoftware development team. The degree of independent V&V needed depends on thedegree of risk associated with the application. For critical software, this combination ofintensified V&V activities together with use of an independent V&V team implies ahigh level of design discipline and documentation. Some of these activities involvebroader management questions of overall direction and adequacy of resources.

Section 6 of this V&V Handbook presents a classification approach that clarifies termssuch as critical and recommends specific activities for different classes of software.


EPRI TR-103916 Vol. 1 Sec. 6.2.4



1-6

Critical Software Minimum V&V Tasks

Independent Reviews, Audits, and Testing

Optional / Advanced V&V Tasks

GeneralPurpose

Safety-Critical(e.g., RPS)

1 State-of-the-Art Design and V&V Activities

Safety

Non-Safety

3 Critical Software Minimum V&V Tasks (IEEE 1012)

Good Software Engineering Practices

(Life Cycle, Documentation, Coding Standards, etc.)

2Additional Requirements for Safety Systems (10CFR50 App. B, IEEE 7-4.3.2, ASME NQA-2a Part 2.7)

Legend:

Layers of Effort to Support System and Software Quality

4 Software Quality Assurance (IEEE 730)

System Significance

Important toOperations

Defense-in-Depth Analysis

Hazards AnalysisSafety

1

2

3

4

Figure 1-1Relationship of V&V, Quality Assurance, and System Significance

1.3. What Is the Relationship Between V&V and CM?

Configuration Management (CM) must be addressed by the QA plan. In its simplestterms, CM insures that the correct version of each document is used duringdevelopment and that the correct versions of hardware, software, and systemparameters are used during testing. CM and V&V are parallel activities. V&V is notefficient or effective unless CM is carefully performed. CM is particularly importantwhen using the spiral life cycle model (iterative development).

Recall that our broad definition of software (Section 1.1) includes programs,procedures, documentation, and data pertaining to the operation of a computer system.Within each of these elements, there may be many modules. For example, the finalsoftware may depend on many separately maintained files containing source code andparameters. Source files are frequently merged with globally included files, compiledinto object files, and linked with system library files to build an executable program



1-7

that subsequently references parameter data files. All of these elements must becarefully controlled to maintain consistency and insure that V&V results support thecurrent configuration. Knowing that a system was successfully developed withappropriate V&V is of little value if we cannot determine whether the installedconfiguration matches the one that existed during development. In the same way, thereshould be a complete trail of documentation that supports the installed configuration.And to support the maintenance phase of the life cycle, the development environmentmust be retained, including tools such as compiler, linker, software library, computeroperating system, and test rig.

Although configuration management is essentially a routine bookkeeping task, it is animportant one that presents many opportunities for error. IEEE 828 [7] and IEEE 1042[8] address software CM. Some projects that have a system level CM plan can stillbenefit from a separate software CM plan because there are many unique issues relatedto software.


EPRI TR-103916 Vol. 1 Sec. 6.2.4

1.4. Why Is V&V Important and What Are Its Costs and Benefits?

In this section, we look at the importance of V&V from three perspectives — the nuclearplant regulator, the electric utility company, and the vendor of digital systems. Forutilities and vendors in particular, we consider the costs and benefits that can beexpected with a good V&V program.

1.4.1. Regulator Perspective

V&V is clearly important for nuclear plant safety systems. This is evident by thefollowing US Nuclear Regulatory Commission (NRC) Regulatory Guides:

• Criteria for Digital Computers in Safety Systems of Nuclear Power Plants(Regulatory Guide 1.152, Rev. 1, January 1996) [9]

This Regulatory Guide endorses IEEE Standard Criteria for Digital Computers inSafety Systems of Nuclear Power Generating Stations (IEEE 7-4.3.2-1993) [2], whichaddresses V&V in Annex E (pp. 17–25).



1-8

• Verification, Validation, Reviews, and Audits for Digital Computer Software Usedin Safety Systems of Nuclear Power Plants (Regulatory Guide 1.168, September1997) [10]

This Regulatory Guide endorses IEEE Standard for Software Verification andValidation Plans (IEEE 1012-1986, Reaffirmed 1992, Updated 1998) [3] and IEEEStandard for Software Reviews and Audits (IEEE 1028-1988, Reaffirmed 1993,Updated 1997) [5]. This is the first in a series of Regulatory Guides (1.168–1.173)[10–15] issued in September 1997 that are related to digital computer software usedin safety systems of nuclear power plants.

V&V records, including clear and complete documentation, provide transparencynecessary for regulatory review and support the Instrumentation and Controls sectionof the NRC Standard Review Plan (NUREG-0800, Section 7) [16].

1.4.2. Utility Perspective

As described above, V&V is a mandate for nuclear plant safety systems. A wellexecuted and documented V&V program reduces the possibility that operation of anew digital safety system might be delayed by regulatory concerns.

Adequate V&V is prudent even if the nuclear plant I&C system is not considered to besafety-related. (It is always possible that the system will become the subject of alicensing evaluation.) Note that V&V is only practical if it is performedcontemporaneous with the development life cycle; it is very difficult to add after thefact.

V&V records, including clear and complete documentation, support long-termoperation and maintenance of the system. So the V&V activities performed during thedevelopment phases of the life cycle have continued value to the electric utilitycompany during the operation and maintenance phases.

For a system or software that is purchased from a vendor, V&V records help to ensurethe quality of the product. Therefore, V&V should be addressed in the purchasespecification, along with other QA issues. Utility representatives should review vendorQA, CM, and V&V records related to specification, design, test, operation, andmaintenance of the purchased product.

(When a utility directly performs its own system development, there is no vendor. Inthat case, the next section should be considered an extension of this section.)



1-9

1.4.3. Vendor Perspective

V&V practices (for example, a requirements traceability matrix) are useful beyond theirrole with respect to QA. Successful software development organizations usuallyperform such activities even if they are described by different terms, but it isparticularly desirable to adopt the standard jargon established by IEEE softwareengineering standards (e.g., IEEE 610.12).

Front-end V&V investment will usually reduce back-end maintenance cost. This hasbeen confirmed by many studies. V&V helps to ensure that a quality product isproduced. But formal V&V can be very expensive. Developers of high-integrity digitalsystems typically allocate 25–50% of the overall budget to V&V related tasks such astesting.

V&V frequently improves efficiency during development by forcing earlier resolutionof latent problems and avoiding the need for major corrections later in the project.Fixing an error in system requirements can cost one or two orders of magnitude more ifthe error remains undiscovered until acceptance testing (see Section 4.1). So V&V mightbe expensive but is probably cost effective, even though it is difficult to quantify costavoidance.

V&V should be addressed by any vendor targeting the market for nuclear plantsystems because a utility company’s purchase specification will probably identify V&Vas a requirement. As we have noted, V&V is very difficult to add after the product hasalready been developed.

1.5. How Much V&V Is Necessary?

Throughout this V&V Handbook, a classification approach is recommended todetermine how much V&V effort should be applied and where within a project to focusthe effort. Each software function has a “degree of criticality” that is related to itscontribution toward risk — either the increase in risk that can result from softwarefailure or the reduction of risk that can be achieved when the software functionsproperly.

Risk for an electric utility company is not restricted to the issue of public safety. Acompany may consider other risks, such as its financial results or its public credibility,to be worth the extra costs associated with a rigorous V&V effort. For example, a plantcontrol system may not be required for safety, but it may contribute significantly toplant availability. Furthermore, the failure of an important non-safety control system(e.g., feedwater control) may also challenge safety systems.

The risk of an event is the combination of its probability and its consequences, whichinclude lost opportunities for improvement. If the performance of a software-based



1-10

system is associated with significant consequences (either unfavorable or favorable),then the likelihood that it will perform successfully must be raised to a level such thatthe overall risk is acceptable. A high level of quality should be demonstrated for acritical software system. Because the probability of an event may be difficult todetermine quantitatively, and because a deterministic analysis may be prescribed byregulation, this V&V Handbook will provide the opportunity to make judgments basedupon consequences alone.

EPRI TR-103916 Vol. 2 App. A

1.6. Who Does V&V?

Generally, individuals who perform V&V activities should be independent of thosewho perform development activities. The development team consists of the projectmanager, system engineers, software engineers, and programmers who are actuallyinvolved in developing the software-based system. Independence from thedevelopment team can be achieved by assembling a V&V team composed of employeesor contractors who are not directly involved in the development effort that is beingexamined. However, there are certain analysis and testing activities requiring so muchspecialized knowledge or skill that it is only practical for them to be done by membersof the development team itself. In those cases, independence can be achieved by havingthe V&V team independently review or witness such activities.

The V&V team may include a variety of managers, engineers, and programmers. Itusually is not necessary for everyone on the V&V team to become familiar with all ofthe details of the system. The V&V team should provide the breadth to cover all aspectsof the system, as well as the depth to probe for the details needed to supportconclusions. In any case, the background and expertise of the V&V team should becomparable to that of the development team. For small projects or for certain elementsof a project, the V&V team may consist of a single reviewer. The V&V team should begiven the time, resources, and authority needed to adequately evaluate the software-based system, convey results to the development team, and influence changes toimprove quality.

Members of the V&V team will normally perform verification reviews and inspectionsdirectly, including review of analysis results provided by the development team. Withregard to testing, the extent of independent effort by the V&V team depends on thedegree of risk associated with the application as well as the level of technical skillneeded for effective testing. Usually V&V team members do not directly perform thetesting. It is more common for the V&V team to plan the test program, specify what



1-11

tests need to be executed by the development team, possibly witness the tests, andfinally review the test results.

1.7. What Is the Utility’s Role in V&V?

In the United States, most power plant systems and software are not developed directlyby the electric utility company. The typical digital I&C system is either procured as acustomized system, or it is configured using commercial off-the-shelf (COTS)equipment such as a programmable logic controller (PLC). What, then, is the utility’sinterest in and responsibility for development and V&V? This question can beanswered in terms of the three basic roles that must be filled for every digital I&Csystem:

• The customer is always the utility company, which is ultimately responsible for safeand reliable operation of the plant and all of its systems.

• The vendor supplies hardware and software components that become part of thesystem.

• The system integrator is responsible for designing, assembling, and delivering thesystem components to satisfy plant specific requirements.

It is rare, but not impossible, for a utility company to assume all three of these roles. Asone example, the Commonwealth Edison process computer staff developed thesoftware for its safety monitoring systems, such as the safety parameter display andpost accident monitoring systems. However, it is more common for the utility companyto rely on a system integrator and/or an equipment vendor. The division ofresponsibilities for several example digital I&C upgrade projects is shown in Table 1-2.



1-12

Table 1-2Division of Responsibilities for Several Example Digital I&C Upgrade Projects

Project Customer Vendor SystemIntegrator

Eagle 21 at Diablo Canyon PG&E Westinghouse Westinghouse

Diverse Auxiliary FeedwaterSystem at Palo Verde

APS Modicon ABB/CE

Emergency Bus Load Sequencerat Turkey Point

FP&L Allen-Bradley United ControlsBarker Engineering

B&W Owners’ Group AdvancedControl System

Arkansas P&LDuke PowerFlorida PowerGPU NuclearToledo Edison

Foxboro Control SystemTriconex PLC

B&WOGB&W Nuclear

Each of the three basic roles (customer, vendor, and system integrator) should assumeV&V responsibilities according to Table 1-3.

Table 1-3Division of V&V Responsibilities for I&C Upgrade Projects

V&V Responsibility Customer Vendor SystemIntegrator

Component Verification Audit Perform Audit

System Verification Review, Audit None Perform

System Validation Plan, Witness, Review None Perform

The sole responsibility of the equipment vendor is to assure the quality of the COTSproduct. Of course, the vendor must accurately communicate product specifications tothe other participants, also. For a system that is important to plant safety or operations,both the system integrator and the customer should audit the vendor’s design andquality assurance processes.



1-13

Any verification activities during system integration, including integration testing,should be performed by the system integrator. However, the customer should performboth technical reviews and process audits of the integrator’s verification activities.

Finally, the customer should take an active role in system validation testing byindependently planning, witnessing, and reviewing test results. However, the systemintegrator usually performs the actual validation testing. Recall that system validationis usually accomplished with functional tests that should not require detailedknowledge of the system internals.

For the customer to take an active role in testing, they must have access to testing toolsthat can be used independently of the development team. They should also be aware ofthe performance history of the application and/or its underlying equipment, so that thetests can explore areas of possible weakness.

This V&V Handbook will describe important methods and techniques used forverification and validation. Depending on its role in a particular project, many of thesetechniques may never be used directly by the electric utility company. However, theutility should be sufficiently familiar with these recommended V&V practices todescribe them in a purchase specification and to audit the performance of contractedvendors and system integrators.

1.8. General Principles

This section summarizes various characteristics and practices that are representative ofa good V&V program.

1.8.1. V&V Should Be Performed Throughout the Life Cycle

A discussion of V&V in relation to development processes requires definition of anorderly development life cycle. A reference life cycle is discussed in Section 2.

V&V is effective only if it is applied systematically throughout the development lifecycle so there can be an active and constructive interaction between the developmentstaff and the V&V team. Any development problems can be addressed much moreefficiently if they are discovered early in the life cycle rather than later.

1.8.2. V&V Involves Advance Planning

A V&V Plan document should be prepared at the beginning of the developmentproject. It should be coordinated with other project planning documents such as the QAPlan.



1-14

The first step in planning V&V is to assess the importance of the system with respect toplant operation and safety. This assessment will identify what level of integrity isdesired for the system and its software. The V&V Plan should describe what activitiesare necessary to demonstrate that the desired integrity level is achieved during thedevelopment process. Planning will help to anticipate any special considerations (suchas software tools or plant simulation facilities) that are needed to support V&V testingactivities. It may even be desirable to modify the design approach to accommodatesuch testing. The V&V Plan should also define what personnel and resources areneeded for a successful V&V effort. Many projects have stumbled because V&V andother important life cycle activities (such as the definition of system and softwarerequirements) have been budgeted inadequately.

Further discussion of V&V planning will be presented in Section 2.

1.8.3. V&V Should Address Both the Process and the Product

Everybody will agree that the correctness of the completed system should bechallenged and demonstrated by extensive validation testing. However, focusing onlyon the final product is insufficient because no organization has either the insight or theresources to exhaustively test for all possible conditions. Even an extensive final reviewof the product may miss subtle flaws in design or implementation. Besides, waitinguntil late in the life cycle to uncover latent errors makes them much more expensive tocorrect. Furthermore, there presently is no accepted objective method for measuring thereliability of a software-based product to the level of confidence that is required for asafety-critical application.

To add confidence, V&V should promote a thorough and disciplined developmentprocess and thereby encourage thoughtful design and frequent independent reviewthroughout the life cycle. The V&V process should not be simply an administrativeformality. Rather, it should include adequate resources and management attention toencourage the development team to embrace practices that will lead to a high qualityproduct.

It is important to distinguish between reviews and audits. Both are required for criticalsoftware-based systems. Reviews focus on the actual technical work that is producedby a process. On the other hand, audits focus on the process to make sure that thedevelopment team is thinking through and documenting all of the steps that arenecessary to build a sound and reliable product.2

2 Different terminology may be used in some utility organizations. For example, an audit may be called acompliance audit, and a review may be called a performance audit.



1-15

1.8.4. V&V Should Trace Results Relative to Requirements

The definition of V&V in Section 1.1 stresses comparison with requirements. Aneffective technique to ensure thorough evaluation is to prepare a requirementstraceability matrix that links each specified requirement to corresponding results ofdesign, implementation, and testing. Using this method to clearly trace requirementsthrough all phases of the life cycle, the V&V team can determine if all of the system andsoftware requirements are satisfied.

Figure 1-2 shows a simple example of a requirements traceability matrix.3 Columns areadded and entries are completed during each life cycle phase, so at the end of theproject there is a clear relationship between every requirement, design element, and testcase. Each requirement may spawn more than one design or test item.

RequiredCapability

RequirementsReference

DesignReference

DesignVerificationReference

SystemTest PlanReference

ValidationTest PlanReference

ValidationResultsReference

1. Availability of 0.99 p. 15 p. 121–130 p. 14 NT NT NT

2. Update every 5 sec. p. 18 p. 140–146 p. 18 p. 18 p. 6 p. 9

3. Human factorsengrg.

p. 19 p. 202–260 p. 7 NT NT NT

•

•

•

•

From NSAC-39, Verification and Validation for Safety Parameter Display Systems, December 1981 NT = not to be tested

Figure 1-2Simple Example of Requirements Traceability Matrix

Validation tests should be directly associated with requirements; verification tests maybe associated with intermediate results such as a design description. As illustrated inFigure 1-2, it may not be practical to test every requirement. A requirement that is notcrisply specified (e.g., good human factors engineering) may have to be judgedsubjectively by design review or operator evaluation. Such non-testable requirementsare problematic because it is difficult to define objective acceptance criteria. It is betterto invest more effort early in the project to improve the specification of requirements, sothat V&V will be more efficient and effective.

3 More detailed requirements traceability matrix examples are included with the case studies presented in Volume2 of the original EPRI V&V Handbook [17]. Abstracts of those case studies are included in Section 8 herein. Also,see Section 5.1.6 for further discussion of the requirements traceability matrix.



1-16

EPRI TR-108831 Sec. 4

1.8.5. V&V Effort Should Concentrate on High Value and High Risk

A typical system may be reasonably straightforward but still have a few specificfunctions with high complexity, significant failure consequences, or uncertainperformance. V&V attention to these critical areas should be more intense, so that V&Vresources are used most effectively. This extra attention may include special designreviews or augmented testing to reduce uncertainty and risk in such critical areas. Notethat such concentration is warranted only if the critical areas can be accuratelyidentified.

The risk identified above can be characterized as functional risk, or the risk that thecompleted system may not satisfy all of its functional requirements. Also to beconsidered, however, is the project risk associated with uncertainties in schedule,budget, or technical feasibility that may compromise delivery of the system as planned.It is also important for project risks to be carefully managed.

1.8.6. V&V Team Members Should Be Independent

The individuals that perform V&V should be independent of the development team, sothat they are intellectually free to challenge the developers’ assumptions andconclusions, and so that they are politically free to identify problems without undueconcern for the developers’ plans, budgets, and schedules. It is useful to define twoconcepts of independence. The more strict concept of organizational independencerequires that developers and verifiers belong to different organizations and that theydo not share the same manager (except perhaps at the highest levels of the enterprise).The basic concept of individual independence requires that the verifier should not havedirect participation in the development product that is under consideration; however,for this concept the verifier may be part of the same organization as the developer.

Standards applicable to the US nuclear power industry only require individualindependence. However, international standards and Defense Departmentorganizations have required organizational independence for critical software systems,and the US NRC has insisted on organizational independence in certain cases —notably application of the Westinghouse Eagle 21 Process Protection System to a reactorprotection system. As a general rule, the degree of independence should be based onthe degree of risk associated with the application, with consideration given to specialfamiliarity or experience that may be needed to perform thorough and informed V&V.In some cases, activities such as analysis or testing may require knowledge or skill that



1-17

makes it impractical to strictly enforce V&V independence. When necessary, such V&Vactivities can be reviewed or witnessed by an independent person in order to obtain theindependence that is desired.

1.8.7. Development Products Should Be Scrutable

The need for independent assessment implies that the V&V team should be able tounderstand and critique the results of specification, design, implementation, andtesting without assistance by the development team. Depending on the degree ofindependence that is required, V&V team members may need to scrutinize softwaresource code and detailed test results, or they may even have to design and execute theirown test cases. Such needs dictate a very high level of clarity and completeness indocumentation, including the self-documenting comments that are placed withinsource code. A logical and modular design also contributes to clarity.

1.8.8. V&V Results Should Be Documented

Throughout the formal V&V effort, every discrepancy that is noted by the V&V teamshould be documented along with its eventual resolution by the development team.The obvious reason for maintaining such documentation is to ensure that all issues areproperly closed. However, this documentation is also important for trending thenumber of discrepancies identified during the life cycle and (hopefully) confirming asteady improvement in quality. V&V records are also useful during regulatory review.Finally, such data can be used to discover areas of difficulty and to recommendimproved processes for future projects.

1.9. Standards and Guides

In 1992, the US NRC published NUREG/CR-5930 [18] containing a review of ten USand international standards relating to the quality of high-integrity software. Thisstudy concluded that no single standard existed with the breadth, depth, and objectivecriteria for compliance that is necessary to guide development of software for high-integrity applications. Refinement of consensus standards at the national andinternational levels has continued (and is expected to continue in the future), but theconclusion is still valid.

In January 1996, the US NRC published Regulatory Guide 1.152 [9]. This RegulatoryGuide endorses IEEE 7-4.3.2-1993 [2], which addresses V&V in Annex E (pp. 17–25).

In September 1997, the US NRC published Regulatory Guide 1.168 [10], which endorsesIEEE 1012-1986 (reaffirmed in 1992 and updated in 1998 [3]) and IEEE 1028-1988(reaffirmed in 1993 and updated in 1997 [5]). This is the first in a series of Regulatory



1-18

Guides issued in September 1997 [10–15] that are related to digital computer softwareused in safety systems of nuclear power plants and endorse applicable IEEE standards.

This V&V Handbook uses IEEE 7-4.3.2 as a base standard. It relies on related IEEEstandards, especially IEEE 1012, as principal sources of additional detail. Internationalstandards such as IEC 880 [19], IEC 987 [20], IEC 1226 [21], and IEC 61508 [22] may beconsidered in areas where IEEE standards lack specifics or timely information. EPRIguidelines for V&V of software for high-integrity systems, expert systems, andconventional systems are integrated and referenced in this V&V Handbook.

Within its scope of digital computers in safety systems, IEEE 7-4.3.2 focuses on thesystem level to cover a broad range of issues peculiar to digital technology; it defersmore detailed discussions of software issues to other referenced standards. In particular,IEEE 7-4.3.2 defers to IEEE 1012 and to IEC 880 for guidance in V&V planning. ASMENQA-2a, Part 2.7 [23], is also referenced for important topics such as quality assurancerequirements, configuration management, and independence of review. Because theemphasis of this V&V Handbook is on software, frequent references will be made tosections of these software related standards. Also, since IEEE 7-4.3.2 is directedspecifically toward safety applications, recommendations will be included in this V&VHandbook to identify activities that can be curtailed or eliminated for less critical non-safety applications.

The particular techniques and tools that are available to support activities associatedwith digital system requirements, design, implementation, and V&V evolve too rapidlyfor inclusion in even the most recent standards. Therefore, this V&V Handbook refersdirectly to open literature, research reports, and vendor information for descriptions ofsuch techniques and how they can be used to improve the quality of digital systems.

Finally, NUREG/CR-4640 [24] is a document that has been used by the NuclearRegulatory Commission as a basis for software quality assurance audits. WhereasNUREG/CR-4640 covers the full range of nuclear industry applications, this V&VHandbook emphasizes real-time digital systems used for instrumentation and control.As such, it is consistent with Section 7 of the US NRC Standard Review Plan NUREG-0800 [16].

1.10. References

1. IEEE Std 610.12-1990, Standard Glossary of Software Engineering Terminology.

2. IEEE Std 7-4.3.2-1993, Standard Criteria for Digital Computers in Safety Systems ofNuclear Power Generating Stations.

3. IEEE Std 1012-1998, Standard for Software Verification and Validation.



1-19

4. EPRI TR-103331, Guidelines for the Verification and Validation of Expert System Softwareand Conventional Software, Vol. 2: Survey and Assessment of Conventional SoftwareVerification and Validation Methods (Revision 1), May 1995.

5. IEEE Std 1028-1997, Standard for Software Reviews.

6. IEEE Std 730-1989, Standard for Software Quality Assurance Plans.

7. IEEE Std 828-1990, Standard for Software Configuration Management Plans.

8. IEEE Std 1042-1987, Guide to Software Configuration Management.

9. US NRC Regulatory Guide 1.152, Criteria for Digital Computers in Safety Systems ofNuclear Power Plants, Rev. 1, January 1996.

10. US NRC Regulatory Guide 1.168, Verification, Validation, Reviews, and Audits forDigital Computer Software Used in Safety Systems of Nuclear Power Plants, September1997.

11. US NRC Regulatory Guide 1.169, Configuration Management Plans for DigitalComputer Software Used in Safety Systems of Nuclear Power Plants, September 1997.

12. US NRC Regulatory Guide 1.170, Software Test Documentation for Digital ComputerSoftware Used in Safety Systems of Nuclear Power Plants, September 1997.

13. US NRC Regulatory Guide 1.171, Software Unit Testing for Digital Computer SoftwareUsed in Safety Systems of Nuclear Power Plants, September 1997.

14. US NRC Regulatory Guide 1.172, Software Requirements Specifications for DigitalComputer Software Used in Safety Systems of Nuclear Power Plants, September 1997.

15. US NRC Regulatory Guide 1.173, Developing Software Life Cycle Processes for DigitalComputer Software Used in Safety Systems of Nuclear Power Plants, September 1997.

16. NUREG-0800, US NRC Standard Review Plan, Section 7: Instrumentation and Controls,Rev. 4, 1997.

17. EPRI TR-103291, Handbook for Verification and Validation of Digital Systems, Vol. 2:Case Histories, December 1994.

18. NUREG/CR-5930, High Integrity Software Standards and Guidelines, September 1992.

19. IEC 880, Software for Computers in the Safety Systems of Nuclear Power Stations, 1986.



1-20

20. IEC 987, Programmed Digital Computers Important to Safety for Nuclear Power Stations,1989.

21. IEC 1226, Nuclear Power Plants — Instrumentation and Control Systems Important forSafety — Classification, 1993.

22. IEC 61508, Functional Safety of Electrical/Electronic/Programmable Electronic Safety-Related Systems, Draft, 1998.

23. ASME NQA-2a-1990, Part 2.7, Quality Assurance Requirements of Computer Software forNuclear Facility Applications.

24. NUREG/CR-4640, Handbook of Software Quality Assurance Techniques Applicable to theNuclear Industry, August 1987.


2-1

2 LIFE CYCLE VERIFICATION AND VALIDATION

Numerous authors (cf. [1,2]) have offered alternative models of the life cycle fordevelopment of digital systems and software. Each model assumes that the developerfollows a sequence of steps leading from concept definition and requirementsspecification to operation and maintenance of a delivered system. The main differencesbetween such models are in the number of distinct steps that are involved and thedegree to which those steps must be revisited as uncertainties in system behavior areresolved.

There are two definitions describing particular development life cycle models:

waterfall model. Development phases are performed in a sequential mannerfrom system requirements to acceptance testing. Hardware and software phasesmay be performed in parallel (simultaneously).

spiral model. Development phases are performed in an iterative manner.Requirements, design, implementation, and testing may be repeated severaltimes to refine the system.

In reality, most projects have some phases performed sequentially, then there isiteration among other phases. For example, there might be iteration among softwaredesign, implementation, and testing phases while the requirements remain inviolate.

Still other life cycle models recognize that much contemporary software is built usingsophisticated software tools such as ladder logic diagrams, spreadsheets, databaseprograms, or expert system shells. Since these tools, in a sense, generate the applicationcode, the implementation step may be reduced in importance or even eliminated. Moreemphasis then shifts to correct specification of the problem and design of the solution.

The principles and techniques of V&V do not depend on a particular life cycle model,but there does need to be some reasonable, step-by-step approach to development, sothat V&V can be applied systematically throughout. Figure 2-1 shows a simplifiedversion of the reference life cycle suggested in Annex E of IEEE 7-4.3.2 [3].


Life Cycle Verification and Validation

2-2

Hardware Design and Implementation

Digital System Requirements

IntegrationRequirements

System Integration

System Testing(Factory Acceptance Test)

Site Acceptance Test

HardwareRequirements

Software Requirements

SystemValidation

System Accepted for Use

SoftwareRequirementsVerification

Development Path

Selected V&V Activities(for areas of focus only)

Shaded boxes are thefocus of this Handbook

Shadowed boxes areexpanded in Fig. 2-2

Software Design and Implementation

SoftwareV&V

Figure 2-1Reference Life Cycle for Digital System (Adapted from IEEE 7-4.3.2)

To avoid cluttering the illustration, Figure 2-1 shows only a few of the V&V activitiesand related feedback paths. Although, for perspective, we begin with the system lifecycle shown in Figure 2-1, this V&V Handbook focuses on the software path depictedby the shaded boxes. In particular, the shadowed boxes will be expanded to illustratemore details of the software development life cycle. This expanded software path isshown in Figure 2-2, which identifies the major software-related V&V activities andfeedback paths.



2-3

Expansion ofSoftwareRequirementsPhase

Software Implementation

System Requirements

Software Prototyping (optional)

Software Requirements

Formal Requirements (optional)

Software Design

Software Integration

System Integration

System Testing

Installation

Operation and Maintenance

DevelopmentPath

System Requirements Review

SoftwareRequirements Review

Software Validation Testing

Software Design Review

Software Unit Testing

Software Integration Testing

System Integration Testing

System ValidationFactory Acceptance Test

Site Acceptance TestSystem Accepted for Use

Revisionsand Changes

Change Assessment,Repeat V&V Activities,Regression Testing

Repeat Life CyclePhases as Necessary

Expansion ofSoftwareDesign andImplementationPhase

V&V Activityand Feedback

System

Software

Figure 2-2Reference Life Cycle for Software Related V&V Activities



2-4

The following observations are apparent from Figures 2-1 and 2-2:

• V&V activities consider both the system and its software. In particular, it isdesirable to perform validation testing on the integrated system in a testenvironment that is representative of its final installation.

• Verification activities normally have short-range effects. That is, an error foundduring verification can usually be efficiently corrected by going back just one step inthe life-cycle.

• Validation activities may have a long-range effect because they compare resultsrelative to requirements. Errors discovered during validation may necessitatecorrections to any step in the life-cycle all the way back to the specification ofrequirements. Such errors can be very expensive, particularly if they are notdiscovered until after the factory acceptance test.

Although Figures 2-1 and 2-2 generally depict a waterfall model for the reference lifecycle, V&V activities can easily be incorporated into a spiral model or any otherreasonably structured life cycle model.

EPRI TR-103331 Vol. 2 Sec. 4

EPRI TR-105989 Vol. 1 Sec. 3.2 and Vol. 2 App. B

2.1. V&V for the Reference Life Cycle

This section follows the guidance of IEEE 1012 [4] to expand on Figure 2-2. BecauseV&V processes should be considered in relation to development processes, much of thediscussion concerns development activities as well as V&V activities. Various V&Vtechniques will be detailed in Sections 3, 4, and 5, which are organized according to thethree common-sense categories identified in Table 1-1 — Reviews and Inspections,Analysis, and Testing. IEEE 1012 also specifies a graded approach to V&V, which isaddressed in Section 6.

Following the organization of IEEE 1012, both development and V&V products areidentified for each phase. Note that the V&V products in some situations are actuallyproduced by the V&V team, while in other situations (e.g., unit test results) they maybe produced by the development team and reviewed or witnessed by the V&V team.



2-5

IEEE 1012 prescribes that anomaly reports must be produced during each phase. Theseanomaly reports are not shown explicitly in the discussion below. Tracking ofanomalies and errors is an important mechanism for trending the quality through aproject, assessing areas where more resources are needed, and providing data forplanning future projects.

IEEE 1012 emphasizes advance planning; each activity is assigned to the earliest phaseduring which it can be performed. For example, the validation test plan can beprepared during the system requirements phase because validation testing considersthe functional behavior of the system with respect to requirements. However, there isflexibility to defer such planning to later phases as long as the plans are completedbefore dependent activities. For example, the test plan should be completed before testprocedures are prepared.

A separate document does not have to be generated for each activity mentioned; onlysufficient documentation is required to ensure that each activity is performedadequately. The traceability of the documentation from one step to the next is crucial sothat its logical flow is obvious to a reviewer.

2.1.1. Concept and Planning

Although not shown explicitly in Figure 2-2, an initial concept phase of the life cycle isrecommended in IEEE 1012. During this phase, the goals and success criteria for thesystem are established, and the selection of technologies is narrowed. At this point,both financial and personnel resources should be allocated. IEEE 1012 considers boththe acquirer (requester) and the supplier (developer), which may be the same entity.

It is recommended that a V&V Plan should be completed during this phase, so thatV&V requirements may be defined and resources for V&V may be allocated. Also, theV&V approach should be established early, so that testability and documentation needscan be factored into the system requirements specification. However, it may be morerealistic to draft a skeleton of the V&V Plan at this point and complete it later duringthe system requirements phase when the general system characteristics and reliabilityrequirements are better understood. More detailed information regarding the V&VPlan is presented in Section 2.2.

Development Products V&V Products

• System Concept

• Project Plan, QA Plan

• Concept Review

• Draft V&V Plan



2-6

2.1.2. System Requirements

During this phase, the functional and performance requirements for the system shouldbe established from the point of view of the plant engineering, licensing, operations,and maintenance staffs. This activity is the foundation for all subsequent work, in thatit establishes what the system should do. It is also the last opportunity for utilitymanagement to review the direction of the project at a high level before detailsrequiring more specialized knowledge of digital technology begin to dominate theagenda. In addition, the system requirements document may establish an overalltechnical approach (e.g., custom designed versus off-the-shelf hardware), a generalarchitecture, and allocation of functions to hardware and software. However, it shouldnot introduce any unnecessary restrictions on the software or hardware designs.

The primary V&V product is the Validation Test Plan, which identifies the kinds oftests and the equipment and personnel that are needed to confirm the systemrequirements. This plan should be completed as early as possible, so that time isavailable to reserve or acquire any special resources (e.g., training simulator or testfacilities). The V&V Plan should be finalized, also.

As the requirements are specified, a Requirements Traceability Matrix (RTM) should beinitiated to link each requirement to corresponding results of design, implementation,and testing. (See Section 1.8.4.) The RTM must be updated and maintained throughoutthe development life cycle.


• System Requirements Specification • Requirements Review

• Validation Test Plan

• V&V Plan

• Requirements Traceability Matrix


2.1.3. Software Prototyping

This is an optional phase of the life cycle depending on the selection of developmenttools. Modern software development tools such as simulation languages, graphicalinterface builders, and expert system shells make possible rapid and cheap



2-7

development of prototypes. Such rapidly developed prototypes are not subjected torigorous design practices; therefore, they should usually be discarded after servingtheir purpose rather than being refined and accepted as the final product. Prototypeevaluation leads to an increased understanding both of the requirements and thedetailed design approaches that can be followed to implement the requirements. Thereare two principal types of prototypes — demonstration and functional.

A demonstration prototype emphasizes the user interface and has little underlyingfunctionality. It provides a vehicle for making the ideas concrete to potential end usersand for soliciting their feedback, without incurring much development cost. Evaluationof a demonstration prototype usually involves exposing it to end users to helpdetermine the following:

• Whether the proposed software fulfills useful functions

• Whether the proposed interface is appropriate for end users

• What level of training will be necessary for effective use

In contrast, a functional prototype is intended to determine the feasibility of a proposedapproach or to compare alternative approaches for one or more functions. Tostreamline the prototyping process, the functions may be prototyped in a simpler ormore convenient software environment. For example, consider a plant control systemwith a requirement to raise the reactor power at a specified rate without tripping theplant. A simulation language could be used to model the plant and a control algorithm.Based on simulation runs, the algorithm could be refined and better specified for actualimplementation. Evaluation of a functional prototype usually involves scrutiny of theprototyped method to evaluate its accuracy, efficiency, and difficulty ofimplementation before deciding to apply that approach to the full-scale softwareproduct.

Prototyping is unnecessary for systems that have fairly simple functional requirementsand use established technology. For example, it is quite straightforward to calculate atrip state from the instantaneous values of several inputs, so that software requirementscan be generated directly from the system requirements. However, it is more difficultto establish detailed software requirements for systems that have the followingcharacteristics:

• Utilize complex mathematical methods (e.g., iterative algorithms or simulation)

• Affected by feedback from the plant (e.g., feedback control systems)

• Interface with human operators



2-8

• Logic that may be interrupted by or is dependent on events

Under any of these conditions, it is generally useful to perform prototyping, simulationor mathematical analysis informally before establishing software requirements.

Development Products (optional) V&V Products (optional)

• Prototype of System, Algorithm,etc.

• Prototype Evaluation

2.1.4. Software Requirements

During this phase, the development team documents those functions that have beenassigned to software and adds additional requirements that will help guide thesoftware design. To the extent possible, the Software Requirements Specification (SRS)should not assume specific hardware selection or prescribe details of the softwaredesign. The link from system requirements to software requirements should bedocumented by the requirements traceability matrix.

IEEE 830 [5] provides guidance on the contents and organization of the SRS, whichshould include the following:

• Functional requirements (e.g., input, output, and processing for each function)

• External interfaces (human, hardware, and software)

• Performance requirements

• Design constraints and attributes (e.g., security and maintainability)

If prototyping has been performed, the prototype itself (e.g., interface screens) andresults of the prototype evaluation should be factored into the SRS. The V&V teamreviewing the SRS should have software engineering experience and should be familiarwith the system requirements.


• Software RequirementsSpecification

• Software Requirements Review




2-9

2.1.5. Formal Requirements

This is an optional phase of the life cycle depending on the selection of developmenttools. It is recognized that the most costly and pervasive errors in digital systems occurat the requirements stage. Many of these errors occur because a natural language suchas English allows much room for interpretation and ambiguity. Others occur becausenarrative requirements may not cover all possible conditions exhaustively andconsistently.

Among developers and regulators of high-reliability software, there is a growing trendto express requirements in a more precise and more easily analyzed format based onmathematical or graphical rules. The use of such formal specification methods forsafety-critical applications is a requirement within the nuclear power industry inCanada and the defense establishment of the United Kingdom.

The development of formal requirements is an optional intermediate step during whichthe narrative requirements are translated into a precise mathematical or graphical form.The subsequent steps to translate the formal requirements into a software design andimplementation can then be accomplished with less interpretation and with a higherlevel of confidence.

The term formal methods will be used to include mathematical, tabular, and graphicalapproaches to characterize requirements and designs. Formal specification languagesare still a subject of research and have, to date, been successfully applied only torelatively simple software systems. Other methods based on tabular and graphicalrepresentations are more easily understood by practitioners and have been shown bythe aerospace industry and by non-US nuclear industries to be practical even forcomplex systems. Furthermore, a selection of software tools that help developers applysuch tabular and graphical methods is available (e.g., CASE).

Formal requirements can be regarded as clarification of narrative requirements, so bothshould be reviewed together.

Development Products (optional) V&V Products (optional)

• Formal Expression ofRequirements

• Software Requirements Review(combined with review of narrativerequirements specification)



2-10

2.1.6. Software Design

During this phase, the development team prepares a detailed description of thesoftware design. IEEE 1016 [6] and IEEE 1016.1 [7] suggest the contents andorganization of the Software Design Description (SDD), which should include thefollowing:

• Functional decomposition of software into module, process, and data entities

• Description of the interdependence of module, process, and data entities

• Description of the interfaces between module, process, and data entities

• Detailed design for each module, process, and data entity

Beginning with this phase, the documentation becomes more detailed and orientedtowards software design techniques. The V&V team that reviews the SDD shouldinclude technical personnel familiar with software design, programming, and testing.After the design has been reviewed, preparation of the Verification Test Plan forindividual software units, such as module and data entities, is possible.

The functional decomposition is a crucial step in arriving at a maintainable, modularstructure. This is essential for making the design reviewable and for facilitating unittests, so it is equally important to the V&V effort.

If formal methods are used to specify requirements, it may also be possible to preparethe software design in a similarly formal manner. Therefore, a formal design activitycould optionally be added after, or even in place of, the software design activity.However, for simplicity we have chosen not to explicitly include a formal design phase.

This V&V Handbook makes no distinction between a Software Design Specificationdocument and a Software Design Description document. However, Software DesignDescription is the title used in software engineering standards, so it is preferable to thealternative and is recommended.


• Software Design Description • Software Design Review

• Verification Test Plan



2-11

2.1.7. Software Implementation

During this phase, the development team turns the software design into source codeand data. For systems that use off-the-shelf software elements (e.g., a library of controlblocks), this step includes development of the configuration information that is neededto make use of and combine such elements.

The implementation review includes not only a comparison of code and data againstdesign documents but also an assessment of conformity to any programming standardsthat are applicable to the project. For example, programming standards may dictate theliberal use of descriptive comments and may forbid constructs such as GO TOstatements.

This phase includes software unit testing. The V&V team’s implementation reviewshould consider the test case definitions and test results, as well as the code and dataitself.

During this phase, it is also desirable to prepare and review the Operation Manual.Considering only the software, it should be possible to derive the Operation Manualdirectly from the SRS and the SDD. The Operation Manual is especially useful whenpreparing test procedures.


• Source Code

• Configuration Information

• Operation Manual

• Software and System TestProcedures

• Code Inspection, Walkthrough

• Software Unit Tests

• Operation Manual Review

• Review Test Procedures

2.1.8. Software Integration

This phase is discussed separately from system integration because extensive softwaretests are more easily performed within a development environment and prior tointegration with the final system hardware. For example, embedded software may haveto be delivered as firmware on chips that have limited software libraries; this mayseverely limit the possibilities for testing. Such software can be integrated and tested onthe development computer that was used during implementation; this developmentcomputer may include an emulator that simulates the delivery environment.



2-12

Furthermore, system integration and testing is more easily performed if most of thesoftware bugs have been eliminated during software integration testing.

Software integration tests are verification tests to confirm that the software interfaces(integration between module, process, and data entities) have been correctlyimplemented. In addition, software validation tests should be performed to assureconformance with the original software requirements. Frequently, simulation methodsmay be used to supply input data for validation tests in lieu of actual plant inputs.


• Integrated Software

• Installation Manual

• Software Integration Tests

• Software Validation Tests

2.1.9. System Integration

During this phase, the hardware and software are integrated. System integration testsconfirm proper implementation of hardware and software interfaces. They frequentlyrepeat a subset of the software integration tests to confirm that the software has beenproperly installed.


• Integrated System • System Integration Tests

2.1.10. System Testing

This phase is the culmination of the V&V program for a typical development project.The system validation test, often referred to as the Factory Acceptance Test (FAT), isperformed to confirm that the final system conforms to the original systemrequirements. System validation testing should be performed in an environment that isrepresentative of the delivery environment and should include as much of thedelivered system as possible. In order to cover the range of conditions dictated by thesystem requirements, it is usually necessary to drive the system using simulated inputsderived from a signal generator or test computer.

The V&V team may participate in the FAT either by reviewing test results, bywitnessing tests performed by the development team, or by directly performing the



2-13

tests independently of the development team. In any case, the V&V team that isinvolved in validation testing should include individuals that are familiar with theplant application, the system requirements including user needs, and the generalapproach of the system design.

If all verification steps have been performed adequately, then the validation effortshould not reveal too many new problems and should not involve any code debugging.However, validation tests may identify problems with the original requirements, suchas feasibility or clarity; for resolution, such discrepancies may require additionalreview at the management level.

When the FAT is complete, a V&V Report should be prepared to summarize all resultsof the V&V effort.


• Maintenance Manual • Factory Acceptance Test

• V&V Report

2.1.11. Installation

An installation test, which is called the Site Acceptance Test (SAT) in IEEE 7-4.3.2,should be performed after the completed system is installed in its plant environment.The SAT is usually confined to confirmation that the hardware and software arefunctioning, that each input channel is correctly accepting data from the appropriatesensor, and that each output channel is correctly driving its actuator or display device.

Any reasonable combination of FAT and SAT activities may be used to satisfyvalidation testing. However, because the plant environment is generally less convenientfor extensive tests, the FAT is normally used for most of the validation testing. It is notnecessary for the SAT to repeat all of the validation tests performed during the FAT,but the SAT is frequently a subset of the FAT.


• Completed System • Site Acceptance Test



2-14

2.1.12. Operation and Maintenance

During the life of a software-based system, operation and maintenance can account formore than the original development cost. Maintenance activities are usually initiated byrequests from users who discover errors or find ways of operating the system that werenot anticipated. Usage and performance should be monitored and analyzed by thesystem maintenance staff to determine whether improvements are necessary.Alternatively, the users may identify needs for extended capabilities that dictatechanges to the original requirements.

As far as V&V is concerned, a maintenance change is simply another iteration in the lifecycle. The software maintenance team should review anomaly reports and requests forextended capabilities, decide on priorities, and support modifications to requirements,design, and implementation with appropriate attention to V&V. Only the softwarerevisions need to be subjected to additional V&V, but it is also good practice to performa standard set of regression tests, which are simply intended to confirm that changes donot introduce any unexpected side effects.


• Changes to Requirements, Design,Implementation, Integration, etc.

• Anomaly Evaluation

• Change Assessment

• Revised V&V Plan

• Repeated V&V Activities

2.1.13. Summary

A reference life cycle (Figure 2-2) has been presented in this section. It follows theguidance of IEEE 1012 and suggests two useful optional phases that have recentlybecome more feasible with modern software development tools. Table 2-1 is asummary of the major deliverable products described for each phase. Each V&Vproduct includes documentation; it is desirable, although it is not necessary, to producea separate document for each V&V product. The development products includedocumentation, code, data, and, of course, the completed software and system.

For simple projects that are not critical to safety, documents from closely related phasesof the life cycle may be combined if all of the prescribed activities have been performedand clearly documented in some reasonable manner. For example, it may make sense tocombine the Software Requirements Specification (SRS) and the Software Design



2-15

Description (SDD) for a simple control system, provided that the combined documentdistinguishes between the true requirements and the design choices. Since controlsystem function block diagrams lie in a gray area between the SRS and the SDD, it mayactually clarify the presentation to consolidate them within a single document.

The purpose for such a detailed breakdown of development and V&V products inTable 2-1 is to clarify all of the activities and issues that should be addressed by V&V,but it is not to dictate a specific set of documents or sequence of actions. In fact, thegoverning standard IEEE 7-4.3.2 does not prescribe any such level of detail; instead, itallows considerable latitude to accommodate the circumstances of each project. Thedevelopment team and the V&V team for a particular project may define different stepsor even a different life cycle model, as long as the V&V Plan addresses the issues thatare relevant to that project.

On the other hand, IEEE 1012, which is referenced in IEEE 7-4.3.2 and has subsequentlybeen updated, now prescribes a minimum set of V&V activities depending on theintended software integrity level. IEEE 1012-1986 (reaffirmed in 1992, one year beforethe current issue of IEEE 7-4.3.2) was a product standard defining the contents of aSoftware V&V Plan. IEEE 1012-1998 is a process standard defining software V&Vprocesses in addition to the contents of a Software V&V Plan. It should be noted thatIEEE 7-4.3.2 only refers to IEEE 1012 for guidance in developing the V&V Plan. Foradditional discussion of the V&V Plan, see Section 2.2 below. See Section 6 forinformation on V&V activities that depend on software integrity level.



2-16

Table 2-1Summary of Recommended Products for the Reference Life Cycle

Life Cycle Phase Development Products V&V Products

Concept and Planning System ConceptProject Plan, QA Plan

Concept ReviewDraft V&V Plan

System Requirements System Requirements Specification Requirements ReviewValidation Test PlanV&V PlanRequirements Traceability Matrix

Software Prototyping (optional) Prototype of System, Algorithm, etc. Prototype EvaluationSoftware Requirements Software Requirements Specification Software Requirements ReviewFormal Requirements (optional) Formal Expression of Requirements Software Requirements Review (combined with

review of narrative requirements specification)Software Design Software Design Description Software Design Review

Verification Test PlanSoftware Implementation Source Code

Configuration InformationOperation ManualSoftware and System Test Procedures

Code Inspection, WalkthroughSoftware Unit TestsOperation Manual ReviewReview Test Procedures

Software Integration Integrated SoftwareInstallation Manual

Software Integration TestsSoftware Validation Tests

System Integration Integrated System System Integration TestsSystem Testing Maintenance Manual Factory Acceptance Test

V&V ReportInstallation Completed System Site Acceptance TestOperation and Maintenance Changes to Requirements, Design,

Implementation, Integration, etc.Anomaly EvaluationChange AssessmentRevised V&V PlanRepeated V&V Activities



2-17

2.2. V&V Plan

IEEE 1012 [4] recommends that a V&V Plan be completed at the start of a project. ThisV&V Handbook (Section 2.1.1) makes the less restrictive recommendation that while apreliminary V&V Plan may be drafted during the concept phase, a final V&V Plan neednot be completed until the system requirements are well understood, so that thenecessary emphasis on review, testing, and analysis can be better determined. The V&VPlan can and should be updated throughout the life cycle as new information becomesapparent.

There is sometimes a tendency to make the V&V Plan extremely detailed and even toinclude within it some test plans and procedures. This is entirely unnecessary, as thepurpose of the V&V Plan is to provide an overall approach to V&V and to anticipatethe necessary resources and schedule constraints. Detailed definitions of test cases, aswell as selection of specific testing tools and methods, can come later and should beincluded in test plan and test procedure documents.

Table 2-2 lists verbatim the eight-point outline recommended by IEEE 1012 for a V&VPlan; each of the eight items is explained in detail in IEEE 1012. The first four sectionsprovide a high-level background and overview of the V&V effort. Section 4 describesthe organization, schedule, resources, responsibilities, and techniques planned forV&V, plus determination of the target integrity level. Most of the technical substanceshould be in Section 5, including detailed plans for V&V activities throughout the lifecycle. Section 6 covers the required documentation for reviews and inspections,analysis, and testing, including task reports, anomaly reports, and the final V&V Reportthat is prepared at the conclusion of the project. Section 7 addresses administrativedetails such as methods for recording and resolving anomalies, determining when tasksshould be repeated, criteria for deviation from the plan, configuration managementprocedures, and governing standards. Section 8 addresses test documentation,including test plans, test designs, test cases, test procedures, and test results. For mostprojects, many of the sections may simply refer to or echo corporate or departmentprocedures concerning software quality assurance. With appropriate justification, it isacceptable to indicate that any section is not applicable for the specific project. Forfurther guidance on preparing a V&V Plan, see IEEE 1059 [8].

Volume 2 of the original EPRI V&V Handbook [9] contains case histories of V&Vactivities for several digital I&C projects at nuclear power plants. Each case studyincluded either an outline or discussion of the V&V Plan created for that project. Inparticular, the complete V&V Plan for the Westinghouse Eagle 21 case study isprovided as a concrete example from a successful safety system development project.These representative V&V Plans can provide a useful starting point when preparing asimilar document either for a custom software-based digital system or for a system that



2-18

utilizes commercial off-the-shelf (COTS) equipment. See Section 8 of this V&VHandbook for abstracts of the case studies.

Table 2-2V&V Plan Outline Recommended in IEEE 1012

1. Purpose2. Referenced Documents3. Definitions4. V&V Overview

4.1 Organization4.2 Master Schedule4.3 Software Integrity Level Scheme4.4 Resources Summary4.5 Responsibilities4.6 Tools, Techniques, and Methodologies

5. V&V Processes5.1 Process: Management

5.1.1 Activity: Management of V&V5.2 Process: Acquisition

5.2.1 Activity: Acquisition Support V&V5.3 Process: Supply

5.3.1 Activity: Planning V&V5.4 Process: Development

5.4.1 Activity: Concept V&V5.4.2 Activity: Requirements V&V5.4.3 Activity: Design V&V5.4.4 Activity: Implementation V&V5.4.5 Activity: Test V&V5.4.6 Activity: Installation and Checkout V&V

5.5 Process: Operation5.5.1 Activity: Operation V&V

5.6 Process: Maintenance5.6.1 Activity: Maintenance V&V

6. V&V Reporting Requirements7. V&V Administrative Requirements

7.1 Anomaly Resolution and Reporting7.2 Task Iteration Policy7.3 Deviation Policy7.4 Control Procedures7.5 Standards, Practices, and Conventions

8. V&V Documentation Requirements



2-19

2.3. References

1. Pressman, R. S., Software Engineering: A Practitioner’s Approach, McGraw-Hill, 1992.

2. Boehm, B. W., A Spiral Model of Software Development and Enhancement, IEEEComputer, Vol. 21, No. 5, pp. 61–72, May 1988.



5. IEEE Std 830-1993, Recommended Practice for Software Requirements Specifications.

6. IEEE Std 1016-1987, Recommended Practice for Software Design Descriptions.

7. IEEE Std 1016.1-1993, Guide to Software Design Descriptions.

8. IEEE Std 1059-1993, Guide for Software Verification and Validation Plans.



3-1

3 V&V TECHNIQUES — REVIEWS AND INSPECTIONS

A variety of methods, techniques, and software tools are available for performing V&Vfor nuclear power plant I&C systems. Most of them fall into one of the three common-sense categories described in Table 1-1 and illustrated in Figure 3-1 — Reviews andInspections, Analysis, and Testing.

VERIFICATION & VALIDATION TECHNIQUES

REVIEWS &INSPECTIONS

ANALYSIS TESTING

Figure 3-1Common-Sense Categories of V&V Techniques

Recall that our broad definition of software includes not only the code itself, but alsothe supporting design, test, and user documentation as well as configuration data,execution parameters, and any other products of the software development process.Reviews and inspections emphasize scrutiny of the software products themselves or ofthe processes used to generate the software products. Techniques for reviews andinspections do not include execution of the code. In contrast, testing techniques involveexecution of the code either in its development environment or on its delivery platform.

The remaining category of analysis emphasizes ways for modeling the softwarerequirements and design so that important properties can be checked before the systemis completed. Because analysis implies study by a human and, in some cases, executionof a discrete or continuous simulation, it may be considered to occupy a middle groundbetween the other two categories.


V&V Techniques — Reviews and Inspections

3-2

A few techniques, such as prototyping (see Section 2.1.3) and beta testing (unstructureduser testing with a pre-delivery version), lie in gray areas between these categories.Also, there are some oversight activities, such as V&V planning and requirementstraceability, that cut across these categories and tie the specific techniques together.

This section discusses the category of Reviews and Inspections. Sections 4 and 5 willelaborate on the other two V&V categories — Analysis and Testing.


3.1. Elements of Reviews and Inspections

By lumping reviews and inspections into a single category, we are aggregating threeseparate categories that are described in IEEE 7-4.3.2 [1] — Independent Reviews,Inspections, and Independent Witnessing. These are illustrated in Figure 3-2. Thebreakdown into separate categories is intended to recognize that certain detailedinspection processes may best be carried out by highly specialized members of thedevelopment team that may be criticized for lacking independence. In this situation,IEEE 7-4.3.2 suggests a combination of independent reviews of documents and/orindependent witnessing of activities (e.g., testing activities) to satisfy the ASME NQA-1[2] design verification requirement.



3-3

REVIEWS &INSPECTIONS

INDEPENDENTREVIEWS

INSPECTIONSINDEPENDENTWITNESSING

PROCESSAUDITS

REGULATORYAUDITS

TECHNICALREVIEWS

WALKTHROUGHS SOFTWAREINSPECTION

Figure 3-2Reviews and Inspections in IEEE 7-4.3.2

The scope of this V&V Handbook includes non-safety systems that are not subject toNQA-1 requirements. The recommended level of independence (if any) for theactivities listed in Figure 3-2 depends on the integrity level of the software-basedsystem as determined using the classification procedure described in Section 6.

It should be noted that IEEE 1028 [3] provides specific guidance for performing reviewsand inspections. IEEE 1028 also describes detailed procedures for management reviews.However, management reviews typically focus on project schedules and allocation ofresources rather than technical content, so they are not considered further in this V&VHandbook. Likewise, this Handbook concentrates on V&V techniques for assessing thetechnical content of the software-based system rather than on particular formats forpresenting those assessments; the latter may be better determined by a qualityassurance specialist.

3.2. Independent Reviews

IEEE 7-4.3.2 says, “Independent reviews of documented work, such as testing, meet theverification requirements of ASME NQA-1-1989. Reviews should provide adequateconfidence that the safety system design basis requirements are traceable to theinstalled safety system.”



3-4

Reviews come in two flavors, depending on whether they emphasize the softwareproduct or the process of developing that product. Everyone agrees that the quality ofthe final product must be challenged through extensive review and testing. However,to focus only on performance of the product is not enough because even an extensivetechnical review may miss subtle flaws in design or implementation. In order to addfurther confidence, V&V promotes a thorough and disciplined development processand thereby encourages thoughtful design and frequent checking throughout thedevelopment cycle. Herein lies the distinction between independent technical reviewsand independent audits of critical software.

Technical reviews focus on the substance of the actual technical product. A technicalreview must include a sufficient cross-section of specialists to judge whether aproposed technical solution actually meets the requirements and specifications.

Audits, on the other hand, focus on the form of the process to develop the product. Anaudit tries to confirm that knowledgeable staff are performing the correct activitiesnecessary to build a sound and reliable software product by assessing compliance withstandards, procedures, contractual agreements, etc. The special subject of regulatoryaudits will be considered separately in Section 3.5. For additional discussion of audits,see IEEE 1028.

IEEE 7-4.3.2 does not prescribe that reviews must necessarily be conducted at formalmeetings. Therefore, they may be conducted informally simply by scrutiny ofdocumentation, so long as the appropriate level of independence is provided.However, it is advisable to have formal technical review meetings at major milestonesof a project, such as the points when requirements, design, implementation, and testdocuments are issued. The content of such meetings should follow a detailed outline orchecklist to ensure that all issues are considered.

Table 3-1 shows excerpts from two review checklists used during development of theWestinghouse Eagle 21 Reactor Protection System described in Section 8. Note thatTable 3-1 provides only very general lists of issues, for consideration by highlyexperienced reviewers. It is often desirable to define more focused and detailedchecklists in order to more effectively guide the reviewer and produce responses thatare less open-ended.



3-5

Table 3-1Excerpts from Westinghouse Eagle 21 Review Checklists

Software Design Requirements (SDR) Review Checklist

…

2. Does the SDR reflect an understanding of the problem to be solved?

a. Are the software design requirements consistent with the system design requirements?

b. Are the models that are specified appropriate for the problem being solved?

c. Are the numerical techniques that are specified appropriate for the problem being solved?

…

3. Are the requirements complete?

…

e. INTERFACES:

i. Are the man-machine interfaces and operational procedures clearly defined?

…

iv. Are external system interface definitions accurate and complete?

…

Source Code Review Checklist

…

IV. COMPARISON DEFECTS

A. Are there any comparisons between variables having inconsistent data types?

…G. For expressions containing more than one Boolean operator, are the assumptions about the

order of evaluation and the precedence of operators correct?

…

VI. INTERFACE DEFECTS

A. Does the number of parameters received by a module equal the number of arguments sent byeach of the calling modules? Is the order correct?

B. Do the attributes of each parameter in the calling routine match the attributes of eachcorresponding argument in the subroutine?

…

3.3. Inspections

An inspection implies a greater depth of technical scrutiny than a review. Therefore, itis an activity that may require the specialized skills and project familiarity that are onlyavailable among members of the development team. In contrast to broad technicalreviews, which cover a large amount of material at major milestones, inspections arerestricted to a narrow focus on a small amount of material. This is intended to increasethe likelihood of finding defects.



3-6

An inspection relies on visual examination of a software product to detect errors,violations of standards, and other problems. Types include code inspection and designinspection. A walk-through is a special type of inspection in which a developer leadsinterested parties through a segment of the code or documentation, and the participantsask questions or comment about possible problems.

IEEE 1028 prescribes detailed procedures for performing inspections and walk-throughs. The important common elements of these procedures are:

• The activity should focus on only a small element (or portion) of the softwareproduct. The time allotment should be limited to encourage efficiency.

• A specific individual should be designated as leader to administer the activity andanother as recorder to document anomalies and conclusions. Other participants,including the author or technical contributor, should be present to representappropriate technical specialties as well as varied points of view.

• The activity is most effective if every member prepares in advance and if the leaderfollows a predetermined checklist.

• The emphasis of the activity should be anomaly detection, not resolution.

• The activity must be performed constructively in an effort to improve the product,not to criticize its author.

Note that walk-throughs may be performed to assess a variety of software productsincluding requirements, specifications, code, test plans, and test results. Obviously eachof these products has its own peculiarities, so a different checklist or procedure may beappropriate for each. Inspections are primarily for source code. For most practicalpurposes, the terms code inspection and code walk-through may be consideredsynonymous in this V&V Handbook.

IEEE 1028 implies the comparison shown in Table 3-2 between the main types ofreviews and inspections considered so far.



3-7

Table 3-2Comparison of Reviews and Inspections

Characteristic Technical Review Inspection Walk-through

Objective Conformance tospecifications andplans

Find anomalies andverify resolutions

Find anomalies andexamine alternatives

Scope Broad Narrow Narrow

Recommended group size 3+ 3–6 2–7

Can the author be theleader?

Yes No Yes

Alternatives considered? No No Yes

Applicable to code? No Yes Yes

Applicable todocumentation?

Yes Yes, but primarily forcode

Yes

3.4. Independent Witnessing

Some activities, such as verification testing or requirements analysis, may requirespecial skills or familiarity with details of the software product. Therefore, it may bevery difficult to find independent personnel that can perform such activitiesmeaningfully, or it may be very inefficient to train them to do so. In such cases, it issufficient to assign independent witnesses to passively observe the activity and confirmthat correct procedures were applied and that results were properly documented.


3.5. Regulatory Audits

Under conditions where an unreviewed safety question may be involved, the digitalupgrade of a safety system in a US nuclear facility may be subject to reviews and auditsby the Nuclear Regulatory Commission. A knowledge of NRC audit practices is helpfulnot only to anticipate the requirements for passing the NRC audit, but also as guidancefor identifying issues and selecting techniques appropriate for a thorough systemdesign and V&V effort. This section considers regulatory audit practices with particularattention to the thread path audit.



3-8

3.5.1. Standard Review Plan

In Section 7 of the Standard Review Plan [4], the NRC has defined guidelines forreviews and audits of digital I&C safety systems used in nuclear facilities. In suchreviews, some of the topics that are pertinent to V&V include the following:

• Life cycle process planning — Do plans have commitments to coordinatedevelopment activities, with checkpoints where product and process characteristicsare verified?

• Life cycle process implementation — Do samples of V&V results for various phasesof the life cycle confirm that activities were implemented as planned?

• Life cycle process design outputs — Do samples of the software products addresstheir requirements with the expected implementation characteristics? Do validationactivities confirm the adequacy of test procedures and results to provide assurancethat the system satisfies its functional requirements?

The Standard Review Plan notes that functional requirements describe what the systemis to perform, while process requirements describe how the system is to be developed.These functional and process requirements come together as a result of thedevelopment activities; therefore, the design outputs, which we have called softwareproducts in this V&V Handbook, exhibit both functional and process characteristics. Inparticular, according to the Standard Review Plan, “Process characteristics end up inthe design outputs as an artifact of the development process. Their presence is evidencethat a disciplined development process was employed, and the goal of high-qualitysoftware has been achieved.” It goes on to describe audits to confirm that “functionalrequirements are traceable through all intermediate design products to the finalproduct” and that “development process characteristics… are present.”

Examples of questions that might be asked during regulatory review include:

• Requirements — Have the requirements for interfaces with operators andmaintenance personnel been adequately defined along with appropriate trainingneeds?

• Design — Does the timing analysis used in the design allow for adequate margins?

• Implementation — Has a requirements traceability matrix been prepared andmaintained throughout the development life cycle to indicate which design outputsaddress each requirement? (This is particularly useful to support a regulatoryaudit.)



3-9

• V&V — Does the V&V team have organizational independence from thoseresponsible for system design? Does the V&V team have technical qualificationscomparable to those of the design team? Are technical reviews, inspections, andaudits well documented? Are the acceptance test criteria clear, adequate, andquantitative?

The manager who is responsible for development of a safety-related digital I&C systemto be used in a US nuclear facility is strongly urged to become intimately familiar withSection 7 of the NRC Standard Review Plan.

3.5.2. Thread Path Audit

One technique for technical review is of particular note because it is frequently used bythe NRC when auditing software-based safety systems. Recall that an audit emphasizesthe adequacy of the life cycle processes used for designing, implementing, verifying,and validating a system. An audit focuses on the software products, such asdocumentation, that are developed during these life cycle processes. Typically, the timeand resources that would be necessary to check every line of documentation are notavailable. Instead, a representative sample of the system and its documentation isselected and systematically examined.

Figure 3-3 illustrates the concept of a representative thread that traces a single inputsignal through analog-to-digital conversion, calculation functions, trip logic, andactivation of an output channel. The documentation and design processes for each ofthe major components associated with this thread can be examined in detail.

In a digital system, the conversion and calculation functions will probably beperformed by software. Although the software may perform the same basictransformation between input and output as an existing analog system that is beingreplaced, it is frequently more complex because of added functionality, moresophisticated calculations, and/or enhanced system capabilities. A software threadpath audit starts with a representative system function, which is then traced throughthe software development life cycle. The purpose of this is to confirm that the selectedfunction has been adequately addressed by each of the major development and V&Vproducts that are described in Table 2-1.

Figure 3-4 illustrates the software thread path audit concept in terms of several of themajor software products that are developed at different phases of the life cycle.Demonstrating traceability of a functional requirement through design, analysis,review, and testing is an important aspect of V&V that is supported by the thread pathaudit.



3-10

The type of documentation that might be reviewed during a thread path audit ispresented in some detail as part of the Westinghouse Eagle 21 Reactor ProtectionSystem case study described in Section 8.

•••

•••

Conversionand

Calculation

Trip Logic

A

B

C

D

Thread Path Through aProtection System:Broad ViewSafety

SystemSensors

(Adapted from presentation to Advisory Committee on Reactor Safeguards by R. Ets of Smartware Associates)

Figure 3-3Concept of Thread Audit: Broad View



3-11

•••

•••

Trip Logic

A

B

C

D

Documentation Thread PathAudit of Software V&V:Deep View

System Function

SystemRequirements

SoftwareRequirements

Coding and/orConfiguration

V&V ThreadValidationTests

VerificationTests

SafetySystemSensors

SoftwareRequirements

(Adapted from presentation to Advisory Committee on Reactor Safeguards by R. Ets of Smartware Associates)

Figure 3-4Concept of Thread Audit: Deep View

3.6. Summary

The main topics of this section are summarized in the following list.

• A scheme for classifying the broad range of V&V techniques is presented. Threecategories are defined using a common sense approach. These categories range fromtechniques that do not require execution of software code (reviews and inspections)to those that do (testing). The middle category includes analysis techniques.

• Concepts and definitions related to reviews and inspections (the first category),including different methods for performing them, are discussed.

• Special concern with regulatory audits is reviewed to anticipate NRC requirementsand to enhance the software V&V process.



3-12

3.7. References


2. ASME NQA-1-1989, Quality Assurance Program Requirements for Nuclear Facilities.

3. IEEE Std 1028-1997, Standard for Software Reviews.

4. NUREG-0800, US NRC Standard Review Plan, Section 7: Instrumentation andControls, Rev. 4, 1997.


4-1

4 V&V TECHNIQUES — ANALYSIS

The degree to which analytical methods can be used in V&V depends almost entirelyon the form in which the software elements are expressed. For example, there ispresently very little that can be done to analyze design documents in the form of flattext (i.e., natural language), but there is a wide range of analyses that can be performedon specifications expressed in more systematic forms such as tables, databases, orgraphs. Of particular importance is the form of expression used for softwarerequirements because most of the methods and tools available for later stages in the lifecycle can only be applied if the appropriate requirements specification tool is used first.Furthermore, the very act of expressing requirements in a rigorous format is itself animportant part of requirements analysis. Therefore, much of the material in this sectionis intended to summarize the options for representing and analyzing requirements.Figure 4-1 illustrates the various techniques that will be discussed. The degree offormality of these techniques generally increases going from left to right in the figure.

REQUIREMENTS AND DESIGNSPECIFICATION METHODS

OTHER ANALYSISMETHODS

FORMALMETHODS

SEMI-FORMAL

METHODS

STRUCTUREDMETHODS

NATURAL LANGUAGE

TEXT

TRACEABILITYANALYSIS

ANALYSIS

ANIMATION ANDPROTOTYPING

STATIC ANALYSIS

OF PROPERTIES

PROOFSFUNCTION BLOCK

LANGUAGES

REQUIREMENTSANIMATION

DESIGNANIMATION

PROTOTYPEEVALUATION

Figure 4-1Analysis Techniques


V&V Techniques — Analysis

4-2

4.1. Requirements Specification

The system and software requirements provide the foundation for all successive stepsin the development life cycle, including design, implementation, and testing.Specifying a complete set of requirements is necessary to answer the question “Did webuild the correct product?” The requirements for systems and software provide thebases for both verification and validation, in that they define the standards againstwhich V&V techniques judge the acceptability of the evolving system and software.

Historical studies of software errors (cf. [1]) show that a substantial fraction areassociated with the requirements themselves but are not detected until the coding ortesting phases. Moreover, many authors have shown that the cost of finding and fixingerrors later in the life cycle rises dramatically with each stage. Table 4-1 (adapted from[2]) shows the relative cost (effort) for repairing software at each stage in the life cycle.

Table 4-1Relative Cost for Repair of Software

Life Cycle Stage Relative Cost for Repair

Requirements 1–2

Design 5

Coding 10

Unit Test 20

Acceptance Test 50

Operation and Maintenance 200

Therefore, the motivations to develop high quality requirements are extremely strong.This section provides general guidance about what must be included in therequirements (with emphasis on software) together with an introduction to methodsuseful for improving the precision and completeness of requirements.

4.1.1. Software Requirements Specification

IEEE 830 [3] provides guidance on what should (and should not) be included in aSoftware Requirements Specification (SRS). Most importantly, the SRS should describewhat the software must do and no more; it should not prescribe how the functions willbe performed or describe any details of the actual design solution, except for requiredconstraints on the design. Questions of how should be reserved for the Software DesignDescription (SDD), which is described in IEEE 1016 [4] and in IEEE 1016.1 [5].



4-3

Table 4-2, adapted from IEEE 830, lists desirable attributes of an SRS. Other sourcessuggest additional desirable attributes as listed in Table 4-3.

Table 4-2Desirable Attributes of an SRS (from IEEE 830)

Attribute Elaboration

Correct The software shall meet every requirement that is included in the SRS.

Unambiguous Because natural language is subject to interpretation, it tends to introduceambiguities. The desire to eliminate ambiguity motivates the use of graphicaland mathematical techniques for expressing requirements.

Complete The response to any possible class of input data should be defined as arequirement.

Consistent The SRS should agree with higher level specifications and have no internalconflicts.

Ranked for Importanceand/or Stability

Some requirements may be essential, while others may be desirable oroptional. Some may be likely to change during operation and maintenance,while others may be considered more permanent.

Verifiable There must be an objective and cost-effective method for confirming thateach requirement is met.

Modifiable It should be possible to change the requirements document easily, withoutintroducing inconsistencies. Therefore, redundancy within the specificationshould be avoided, as it compromises modifiability.

Traceable It must be possible to trace each requirement forward through the life cycleto confirm it has been covered by design, implementation, and testing.

Table 4-3Desirable Attributes of an SRS (from Other Sources)

Attribute Elaboration

Understandable byCustomers

Specification techniques, however desirable from a technical point of view,must not inhibit communication between customers, specifiers, anddesigners.

Design Independent The requirements document should not presume any particular design; itshould allow the designer to pick the best approach.

Focused on SoftwareProduct, not Project

Project requirements (e.g., cost, schedule, organization, developmentenvironment, QA) should be covered in other documents.

Concise No document should be longer or more complicated than necessary.



4-4

Several attributes are crucial to support V&V. A requirement is verifiable if there is somecost-effective, objective way that the software design and end product can be confirmedto meet the requirement. The traceability of requirements through each downstreamphase of design, implementation, and testing is equally important. The requirementsshould be decomposed into sufficient detail, so that each requirement can be associatedwith a particular software unit (e.g., procedure or subsystem). Traceability is importantfor confirming the completeness of design, implementation, and test programs.

Appendix 4.A describes suggested contents of a Software Requirements Specification(SRS) as well as a Software Design Description (SDD).


4.1.2. Relation of SRS to V&V

The person that specifies requirements should be very familiar with the end user needsand with the plant environment with which the application must interact. The users ofthe requirements include the designers, who must transform the requirements into aspecific design, and the V&V team, who will use the requirements as the basis for allfurther downstream V&V activities.

Techniques for expressing and analyzing requirements specifications are describedbelow. The available menu of V&V techniques and their effectiveness depends heavilyon the choice of requirements method. For example, formalized reviews such asstructured walkthroughs are more effective if the requirements are expressed in anunambiguous tabular or graphical form rather than simply as traditional text that issubject to interpretation and semantic debate. Similarly, most systematic methods fordesigning test cases depend on the requirements being expressed in an easilyunderstood (and preferably machine readable) form; this issue will be discussed inSection 5.

4.2. Methods for Describing Requirements and Design

IEEE 830, 1016, and 1016.1 only discuss what kinds of information should go into theSRS and SDD, with little guidance on what form the information should take. The leftside of Figure 4-1 describes a range of graphical, tabular, and mathematical methodsfor preparing a requirements specification that help to improve its comprehension andprecision. These methods generally become more formal going from left to right in thefigure.



4-5

The use of natural language text is obviously the oldest and most commonly usedapproach. Pseudo code is also discussed as a combination of computer programminglanguage and natural language.

By the 1970s, software systems had become large and complex, and the problems ofambiguity and comprehension with natural language had been recognized. In supportof more complex requirements and designs, developers began to adapt structuredmethods that emphasize:

• Top-down approach allows detail to be added gradually

• Easily understood graphical presentation

• Clear delineation of functions, data flows, data stores, and modules

During the last 9–11 years, a variety of Computer Aided Software Engineering (CASE)tools have emerged that support the application of structured methods.

At the top of the formality scale lie formal methods, which began appearing in practicalapplications within the last 9–14 years. Their reliance entirely on abstract mathematicsallows for mathematically precise definition of functions and opens the possibility forproving conclusively that a detailed design meets its underlying requirements.

Figure 4-1 introduces another category of semi-formal (or pseudo-formal) methods, whichlie in the gray area between structured and formal methods. The semi-formal methodsuse formal techniques such as finite state machine models to precisely represent asystem’s required structure, but they may be somewhat informal in their definitions ofthe functions performed while within an individual state.

Figure 4-1 also identifies function block languages, which synthesize softwarespecifications in terms of basic building blocks (e.g., ON, OFF, AND, OR, etc.). Manycommercial controllers, for example, maintain libraries of such building blocks, so thatspecifications written in this way can be directly implemented by invoking blocks fromthe library.

All of these methods for describing requirements and designs will be discussed in thefollowing subsections.4

4 Most of the material in this section was taken from the original EPRI V&V Handbook TR-103291 Vol. 1 Sec. 3and Vol. 3 Sec. 2 (December 1994). Structured and formal methods are dynamic subjects and may not be fullycovered by this material after four years of continued development, but it is believed that this discussion adequatelyrepresents the sense of analytical techniques that are currently available. For a more complete exposition of moderntechniques, the reader should consult the latest technical literature.



4-6

EPRI TR-108831 Sec. 3 and App. A

4.2.1. Natural Language Text

Of course, natural language (e.g., English text) is still the most commonly usedapproach for describing requirements. To date, the US nuclear power industry hascontinued to depend mostly on natural language specifications.

For very simple systems with few requirements and design entities, a simple naturallanguage description may be adequate. However, as systems become more complex, itbecomes more difficult for a reviewer (or a specifier, designer, or programmer, for thatmatter) to comprehend all the pieces and how they fit together. Furthermore, for criticalsystems it is difficult to develop a natural language text that conveys exactly the samemeaning unambiguously to all the document’s users. The literature is full of exampleswhere subtle shadings in meaning have caused requirements to be interpreted andimplemented in significantly different (and incorrect) ways.

Pseudo code is a combination of programming language constructs together withnatural language text that provides more structure than simple narrative text and isespecially useful for expressing sequences of logic. For example, the following is a top-level description of the logic used to determine monthly kW-demand from a set of kWhmeters by communicating with pulse counters, where each pulse output by the meterrepresents a quantity of kWh energy:



4-7

Perform initializationAssign communication portCreate database fileFOREVER

FOR each kWh counterIF new day AND designated monthly reset day

Set need-to-reset-this-counterCollect data from kWh counterIF the kWh counter did not respond

Set data to represent failed kWh counterELSEIF need-to-reset-this-counter

Reset kWh counterReset need-to-reset-this-counterSet data to represent reset kWh counter

Calculate kWh, kW, kW-demand, peak kW-demandIF 2 minutes since last update

Update local memory backup fileNEXT kWh counterUpdate database file

CONTINUE

Pseudo code can be a useful way to reduce the ambiguity of natural language text.Further discussion of pseudo code is continued in the next section.

4.2.2. Structured Methods

During the 1960s and earlier, software development was largely the domain of theindividual programmer who would take vague statements of goals and simply beginprogramming from the lowest level toward progressively higher levels (bottom-upapproach). The resulting code may or may not have met the loosely statedrequirements, let alone any unstated expectations of the customers. Documentation wasskimpy and, if available at all, usually took the form of detailed flow charts (also,bottom-up in character) that were often incomprehensible to all but the programmer.

By the 1970s, software systems had become large and complex. The problems ofcomprehension and ambiguity associated with natural language were recognized.There arose glaring needs to improve communication between customers, specifiers,engineers, and programmers. In support of more complex requirements and designs,developers began to adapt structured methods that emphasize:

• Top-down approach that first introduces the broad requirements of end users thengradually adds layers of detail

• Easily understood graphical presentation



4-8

• Clear delineation of functions, data flows, data stores, and modules

Also, such methods often support a central data dictionary for storing a summary ofrequirements and/or design information.

Tools for structured analysis and design emerged to help define and design systems.Several authors developed structured design methodologies, all of which rely heavilyon pictorial devices for communication.

Figure 4-2 (adapted from [6]) is an example of a data flow diagram (DFD) for a rentreceipt process. The bubbles identify functions, the lines and their labels data flows,and the open boxes data stores. All of these elements are described in more detail in adata dictionary (not shown here). The data flow diagram describes only the functionsthemselves without any mention of how or by what components they will beimplemented.

CreditInvoice

DepositFunds

RecordPayment

Check

Credited-InvoiceCopy

RentReceivables File

Archive File

Bank-Deposit

Figure 4-2Example Data Flow Diagram



4-9

Issue Pay Checks

for Employees

Get Employee

Pay Record

Employee

Pay Record

Calculate

Net PayIssue Check

Employee

Pay Record Net Pay

Employee Name

Employee Pay

Figure 4-3Example Structure Chart

Structured methods also provide techniques and guidelines to aid in system design.Figure 4-3 (adapted from [6]) is an example of a structure chart used for top-downdesign decomposition. Here the functions are broken down into a sequence of tasks(corresponding to procedures or subroutines), with the input and output parametersshown. While the decomposition is left to the designer, the design methods recommendhigh cohesion within each module (i.e., each module should perform only a singlefunction) and simple coupling among the modules (i.e., minimal moduleinterdependence).

A more meaningful example of a data flow diagram comes from the WestinghouseEagle 21 Process Protection System described in Section 8. Figure 4-4 shows the highestlevel DFD for a principal subsystem — the Loop Calculation Processor. This DFD isalso referred to as a context diagram because it only shows data flows for thesubsystem to and from its environment. Figure 4-5 shows a more detailed breakdowninto more basic functions. The DFDs are accompanied by a data dictionary (not shown)that specifies all of the data items and describes the transformations implied by thefunctions. Figure 4-6 is a structure chart used for top-down design decomposition of theEagle 21 software. The structure chart captures the functional decomposition neededfor arriving at software that is reviewable, testable, and maintainable.



4-10

Figure 4-4Example of High-Level Data Flow Diagram

Figure 4-5Example of Lower Level (More Detailed) Data Flow Diagram



4-11

Figure 4-6Example of Structure Chart



4-12

At the basic level of an individual procedure, corresponding to a single function orsoftware routine, a process specification (p-spec) is defined. A p-spec includes aninput/output list as well as a pseudo code description of the procedure’s function.(Pseudo code was introduced in Section 4.2.1.) As an intermediate step, the pseudocode can be easily understood by reviewers assessing the p-spec against its requiredfunction. Pseudo code is also sufficiently unambiguous that a programmer can use itdirectly as a basis for generating source code.

An example of the p-spec for a simple function is shown in Figure 4-7. This examplewas taken directly from the Westinghouse Eagle 21 Loop Calculation Processor p-specfor Perform Comparator Checks. Note that the pseudo code in this example is quite easilyread by a non-programmer. However, it does tend to prejudge how the functions will beimplemented in code and reduce the number of options available to the designer.



4-13

NAME:5.1.5.3;5

TITLE:Perform Comparator Checks

INPUT/OUTPUT:Standard_AT_Point_Value : data_inStandard_Quality: data_inStandard_Comparator_Data: data_inStandard_Partial_Trips: data_out

BODY:BeginFor each comparator

BeginSet Standard_Deadband_EU = ( Standard_Deadband / 100% ) *

Standard_Input_RangeIf comparator_type = Trip_On_Low

then if Prev_Comp_Output =Setthen if ( Standard_AT_Point_Value – Standard_Deadband_EU ) >

Comparator_Setpointthen Comparator_Qutput = Resetotherwise Comparator_Output = Set

otherwise if Standard_AT_Point_Value < Comparator_Setpointthen Comparator_Output = Setotherwise Comparator_Output = Reset

Figure 4-7Example of Process Specification (P-Spec)

4.2.2.1. Extensions for Real-Time Systems — Ward-Mellor and Hatley-Pirbhai

Structured methods were originally designed for business-oriented applicationssystems that sequentially process multitudes of routine, essentially identicaltransactions. Power plant instrumentation and control systems, in contrast, are real-time, “reactive” systems that must respond appropriately to changes in monitored dataor operator commands.

Structured methods have been extended to accommodate the needs of real-timemonitoring and control systems. The most influential extensions are Ward-Mellor [7]



4-14

and Hatley-Pirbhai [8], which use different notation but are conceptually nearlyidentical. These extensions are summarized in Table 4-4.

Table 4-4Real-Time Extensions to Structured Methods

Extension Hatley-Pirbhai Ward-Mellor

Introduction of control flowdistinct from data flows.

Introduces a separate controlflow diagram used in parallelwith the data flow diagram(DFD).

Uses a distinct notation forcontrol flow (dashed line), buton the DFD itself.

Introduction of a controlspecification consisting of a finitestate machine (FSM).

FSM describes behavior ofprocesses in the DFD. Allowseither state transition diagramsor tables to represent them.

FSM describes primitivetransforms.

Extension of data dictionary tomore data types (e.g., events).

Self-explanatory. Self-explanatory.

Each approach has three distinct views of the system specification:

• The Functional View defines only what functions are to be done, not where, how, orin what sequence. It is typically embodied in data flow diagrams.

• The Architectural View specifies the structure of the subsystems, as well as theirhardware and software components. It usually resembles an engineering drawing,the primary tool of hardware engineers.

• The Behavioral View defines when functions will be started and stopped and how thesequence of computations will progress, depending on measured conditions anduser input. It is based on a form of a finite state machine model.

4.2.2.2. Extensions for Object-Oriented Design — Coad & Yourdan and Sommerville

The most influential current trend in software development is the shift toward object-oriented design and programming. Both Coad & Yourdan [9] and Sommerville [10]provide good introductions to object-oriented design.

Some elements of the object-oriented approach are:

• Object — Objects are separate individual entities that typically model the physicalobjects of the problem. Each object may have one or more attributes. An object mayalso contain executable code in the form of a special attribute called a “method”. Anobject may be a member of one or more classes.



4-15

• Class — Classes are templates for objects. A class may group together objects orsubclasses with the same general structure; for example, objects within a class mayhave the same attributes but different values for those attributes.

• Inheritance — Objects and classes may inherit their general structure (e.g., attributes)from higher level classes. Structure need only be defined once at the highest level.

• Message Passing — Objects may communicate by passing messages; for example, amethod in one object may be executed in response to a message sent by a secondobject.

As a result of these features, object-oriented designs have some desirable properties:

• Modularity — Because the data structures and executable code are packagedtogether, they can be easily changed without affecting the behavior of other objects.

• Encapsulation (Information Hiding) — Only the interface information is available toother objects, so there can be no accidental use of internal information by otherobjects. Therefore, the internals of any object can be changed with no effect on thebehavior of other objects, as long as the interface is unchanged.

• Potential for Reuse — This is a consequence of modularity and encapsulation, as wellas inheritance.

The emphasis on independent objects, with combined data structures and code, is quitedifferent from the traditional separation of data and functions in a data flow diagramand the decomposition of functions through a structure chart. Therefore, extensions tostructured design methods have been introduced to accommodate object-orientedtechnology [9].

4.2.2.3. Impact of Structured Methods on V&V

The use of structured methods, especially graphical representations such as data flowdiagrams, greatly increases the ability of a reviewer to comprehend the requirementsand design details of a complex system. Reviews can be organized beginning with thebroad, top-level requirements and gradually proceeding down the hierarchy toconsider increasing detail until the pseudo code description of low-level functions isaddressed.

The structured methods provide much of the information needed to design functionaltest cases. For any particular function:

• The data flow diagram defines all of the input and output data flows that must betested.



4-16

• The data dictionary defines all possible values of these input flows.

• The p-spec defines the low-level functions for the data transformations.

This information is sufficient to determine a set of black-box validation tests confirmingthat the final implementation meets the requirements. Section 5 includes morediscussion on methods and tools for test planning based on structured methods forspecification of requirements and design.

4.2.3. Formal Methods

Wing [11] provides the following down-to-earth definition (emphasis added):

“Formal methods used in developing computer systems are mathematically basedtechniques for describing system properties. Such formal methods provideframeworks within which people can specify, develop, and verify systems in asystematic, rather than ad hoc, manner.”

The most important point is that formal methods are based on mathematics (of oneform or another) which is by nature unambiguous.

4.2.3.1. Characteristics of Formal Methods

The term “formal methods” is sometimes used loosely to include all mathematical,tabular, and graphical approaches for specifying requirements and designs. Inparticular, software engineers often confuse formal methods with structured methods.However, in Figure 4-8 and the following discussion we make some clearerdistinctions. As shown in Figure 4-8, the strictly formal methods can be characterized asproperty-oriented or model-oriented.

The model-oriented group takes the direct approach of building a model of systemfunction that can be refined for more detail and checked for its implications. Thedominant model form is the finite state machine (FSM), which presumes that, at anypoint in time, the system is in a particular state or mode. As time progresses, the systemmust pass from one of these states to another, depending both on its internal controlbehavior and on its interactions with the environment. The FSM model thusemphasizes identification of all required states as well as the conditions and events thatshould cause transitions between the states. Examples of model-oriented formalmethods include the VDM and Z languages, Petri Nets based on graph theory, andParnas’ approach, which emphasizes tabular representations of state transitions.

In contrast, the property-oriented formal methods describe the system only indirectly bymaking statements about properties that the finished system must have. The mostcommon of these are axiomatic, in that they rely on predicate logic to make assertions.



4-17

Not only specification languages such as OBJ and Larch are useful, but the resultingformal specification can be prototyped in more general programming languages (e.g.,Prolog) and executed to evaluate their behavior. Alternatively, algebraic methods maybe used to reduce the properties to a series of algebraic equations.

Requirements and Design Specification Methods

Formal Methods

Property-Oriented Model-Oriented

Semi-Formal Structured Methods

Prolog

Algebraic Axiomatic

VDM,Z

CSP PetriNets

Parnas Hatley-Pirbhai,Ward-Mellor

Yourdon, et al.

State-Machine

Statecharts

Formal (Mathematical) Languages Graphical/Tabular Languages

= Primary relationship

= Secondary or weak relationship

= Decreasing degree of formality

Jackson... LOTOS OBJ Larch

Figure 4-8Requirements and Design Specification Methods

Model-oriented formal methods such as Z and VDM have received significant attentionamong users of formal methods. However, these methods are best established forsequential processes and protocols, and they exhibit weakness for time-dependent,real-time systems. Therefore, they must be supplemented with graphics-oriented, tool-based approaches that deal with these issues and handle more complexity. The semi-formal methods described in Section 4.2.4 are beginning to fill this need, and theircapabilities are rising to approach the level of formal methods.



4-18

4.2.3.2. Formal Languages

As shown in Figure 4-8, most of the methods that we have just discussed rely onabstract mathematical language based on set theory, algebra, and/or predicate calculus.These mathematical languages have the advantage of being quite general andpowerful, but the disadvantage of requiring a high level of mathematical skill to beused effectively.

Formal languages have been especially embraced by defense [12] and nuclear powerindustries within the United Kingdom, with a high level of interest in the Europeannuclear community as well. Especially outside of academia, model-oriented formallanguages such as Z [13] and VDM [14] have received the most attention. However,these languages are best established for sequential processes and protocols and are lessuseful for time-dependent, real-time systems. Although the LOTOS and CSP languagesaddress concurrency, they are not yet mature enough to have accumulated muchexperience. An excellent introduction to formal languages, with some good annotatedexamples, was assembled in the September 1990 issue of IEEE Software (cf. [15]).

Of the strictly formal languages described above, the Z (pronounced “Zed”) languageis relatively new, but it has rapidly accumulated case studies and skilled users at afaster rate than other such languages. The September 1990 issue of IEEE Softwareincluded two case studies that provide a very clear introduction to Z.

EPRI sponsored a small exploratory research study to evaluate formal and semi-formalmethods such as Z for use in specifying control system requirements [16]. Appendix 4.Bpresents a Z specification of a performance requirement for the B&W AdvancedControl System. Several conclusions can be drawn from the literature and from thisfirsthand experience with Z:

• It is indeed feasible to represent a performance requirement in a formal specificationlanguage and in a form that can define the basis for automatic evaluation ofvalidation test cases. However, the Z language software tools available during thestudy were not mature enough to make the actual automation a straightforwardtask.

• A surprising amount of formal infrastructure, through definition of data structuresand utility functions, must be built to support the formal definition of even a simplerequirement.

• Practicing utility or vendor engineers should expect to invest considerable effort tomaster a language such as Z. Proponents of Z and VDM have suggested that 1–2weeks of formal training should be sufficient. While this may be adequate for areading knowledge to enable review by a software engineer, the amount of trainingand experience needed for a good writing knowledge to produce high quality



4-19

specifications efficiently will no doubt be much greater. Other authors havesuggested that becoming proficient in a specification language takes the sameamount of time and effort as developing skills in a new programming language —certainly several months even for an experienced programmer.

4.2.3.3. Finite State Machines Using Graphical or Tabular Languages

An alternative approach to working with formal languages is to accept the limitationsand assumptions of a particular model, which can then be represented in a less generalbut more easily understood notation. The finite state machine (FSM) is a well-established model for describing and analyzing the behavior of digital systems. Whileit was originally limited to simple sequential systems, methods and tools have emergedthat permit it to be used for modeling more complex systems as well as concurrentprocesses. The basic premise of the model is that at any time, the machine can be inonly one of a finite number of possible states. In response to an input, which may be aninstantaneous event (e.g., pushing a button) or a Boolean condition (e.g., a measurablevalue greater than a setpoint), the machine produces an output and moves to a differentstate.

There are several ways to represent these concepts. Figure 4-9 (adapted from Figure 6.4of [8]) shows a simple state transition diagram for an electric garage door opener. Thisdiagram does not explicitly show the outputs (e.g., electric signal) associated with eachstate. The main idea is that the next state depends not only on the current inputs, but onthe current state as well. Note that certain transitions are forbidden in this particularspecification. For example, while the door is opening, a press_button event has noeffect.



4-20

STATE Denotes a state

event Denotes an instantaneous event

[condition] Denotes the value of a Booleanvariable or expression

CLOSED

OPENED

OPENINGCLOSING

press_bu tton

press_bu tton

bottom_sensor

[ time > ( time_at_opened +10 minutes ) ] top_sensor

Figure 4-9State Transition Diagram for an Electric Garage Door Opener

In the 1970s, Parnas [17] developed an easily understood tabular method forrepresenting FSM behavior, and he and others at the Naval Research Laboratoryapplied it to embedded systems for the A7-E aircraft. While this method is somewhatrestricted in scope compared to a general mathematical language, it can be learned veryeasily, and its tabular format is amenable to developing supporting tools forautomating some of the verification tasks.

Ontario Hydro [18] has adopted a form of Parnas’ method for specifying their safety-critical digital systems. The Ontario Hydro method triggers state transitions only bychanges in conditions (i.e., instantaneous events are not explicitly allowed), and thenotation for defining actions is somewhat simplified. Several tabular formats areavailable in the Ontario Hydro method. Table 4-5 shows a state transition table fordefining a reactor trip meter. In the table, ai_m and sp_k represent an analog input andits setpoint, respectively, and it is required to inhibit state transitions within adeadband db_k below the setpoint. This table defines the final state trip_state_f.



4-21

Table 4-5Ontario Hydro Requirements Format — State Transition Table

untrip trip trip_state_s := (Initial State)

ai_m >= sp_k trip trip

[ai_m < sp_k]and

[ai_m >= (sp_k – db_k)]— —

trip_state_f := (Final State)

ai_m < (sp_k – db_k) untrip untrip

Conditions

The leftmost column of Table 4-5 lists all possible conditions based on the current valueof the input ai_m. The second and third columns show transitions to a new statetrip_state_f given the previous state trip_state_s. No transition is specified wheneverthe input lies within the deadband; therefore, the final state remains the same as theinitial state. Note that Table 4-5 represents exactly the same type of question as Figure4-9: Given the external inputs and the current state, what is the next state?

Table 4-6 illustrates a structured decision table providing an alternative format thatlends itself more readily to review and verification. Within the action statements, eachcolumn must have one and only one entry to ensure that the specification defines theoutputs in a complete and unambiguous manner. Furthermore, each row must have atleast one entry to ensure that every defined state is indeed reachable. Table 4-6 specifiesthe response of the pb_f pushbutton function. The prescribed action depends on thecombination of the three input conditions. A blank in an input box indicates that thecondition is not important to that particular state transition.

FSM tables such as Tables 4-5 and 4-6 are developed by Ontario Hydro to define everyrequired state and state transition. The tables are accompanied by a function overviewdiagram (e.g., Figure 4-10) that ties together these details of required behavior.



4-22

Table 4-6Ontario Hydro Requirements Format — Structured Decision Table

Condition Statements

select_pb_m = pressed_k T F F F F T T T — — —

inc_pb_m = pressed_k F T F F F T — — T T —

dec_pb_m = pressed_k F F T F F — T — T — T

enter_pb_m = pressed_k F F F T F — — T — T T

Action Statements

pb_f := select X

pb_f := inc X

pb_f := dec X

pb_f := enter X

pb_f := none X

pb_f := invalid X X X X X X

select_pb_m

inc_pb_m

dec_pb_m

enter_pb_m

actual_setpt_f

pb_state_s

ai_m

1

pb_f

5

6

eu_f

7

trip_state_ftrip_state_s

trip_do_c

actual_setpt_s

setpt_disp_s

4setpt_disp_f

2pb_state_f

3setpt_change__led_f

setpt_change_led_c

setpt_disp_c

eu_disp_c

Figure 4-10Ontario Hydro Requirements Format — Function Overview



4-23

4.2.3.4. Impact of Formal Methods on V&V

Employing formal methods entails a non-trivial commitment of resources to preparethe formal specifications of requirements and design, plus initial training in the use offormal methods. The potential benefits to offset these costs are summarized in Table4-7.

Table 4-7Benefits of Employing Formal Methods

Activity Benefit to V&V Feasibility

Precision and unambiguity ofrequirements specification.

Low/Medium Feasible now.

Refinement of formal specifications toformal design and implementation.

Medium Feasible only for small, specializedproblems; otherwise, in R&D stage forseveral years.

Formal proof that design andimplementation satisfy requirements.

High Feasible only for small, specializedproblems; otherwise, in R&D stage forseveral years.

The ultimate long-term benefits of employing formal methods are:

• To have a computer-comprehensible specification that can be refined into a detaileddesign and synthesized into an executable implementation. The process ofrefinement is indicated conceptually in Figure 4-11.

• To be able to formally prove that the design and implementation meet therequirements.

• To be able to animate (or execute) the formal specification to test it prior to anydesign or implementation work.



4-24

FormalRequirements

FormalRequirements

Formal Design(High Level)

Formal Design(High Level)

Formal Design(Detailed)

Formal Design(Detailed)

Implementation(Coding)

Implementation(Coding)

Formal Verification Formal Verification Walkthrough and Unit Testing

Validation Testing

Figure 4-11Process of Refinement and Formal Verification

At this time, however, these benefits are still largely in the R&D stage. In particular, thesupport tools for formal languages are weak and generally limited to syntax checking.Most authors agree that for the near term, refinement and formal proof will be possibleonly for small, specialized problems. Nevertheless, Table 4-7 indicates that even nowthere is considerable motivation for employing formal methods in preparation ofrequirements specifications, including:

• The formal specification provides a more precise and unambiguous basis for design,implementation, testing, and V&V. It can be used in conjunction with an informal,natural language specification (serving as explanatory information) for review ofthe requirements. In addition, the formal specification can be subjected to a “semi-formal proof,” essentially walkthroughs of the logic, as part of the verificationprocess.5

• The process of translating from informal to formal specifications forces carefulthought about the completeness and consistency of the specification.

• Depending on the format (e.g., tabular vs. formal language), some automatedchecking tools can be used for evaluating completeness, test coverage, and otherproperties.

As an example of automated requirements checking, one reference [19] describes how aZ specification can be used to find unreachable states, which are symptoms that eitherthe specification has unnecessary aspects or that one or more necessary transitions ismissing.

5 This process is called a “rigorous proof” and is prescribed by some European standards such as [12].



4-25

Another reference [20] describes a utility for translating back and forth between dataflow diagrams (the primary tool of structured methods) and the Z language. Suchcapability is of considerable interest because ideas can first be developed in the familiarand intuitive DFDs, then translated to a formal language where they can besupplemented, refined, and subjected to checks for important properties such ascompleteness and reachability.

4.2.4. Semi-Formal Methods

As illustrated in Figure 4-8, there is considerable overlap between model-orientedformal methods and real-time structured methods. For example, applications of Parnas’method (see Section 4.2.3.3) combine a functional representation with a graphicaloverview to show how the functions are tied together. Similarly, the Hatley-Pirbhaimethod (see Section 4.2.2.1) and the method of Statecharts [21] combine easily graspedgraphical representations of state transition diagrams with data flow diagrams for ahybrid description of functional requirements.

The Statecharts method (described below in Section 4.2.4.1) is closely tied to thecommercial tool STATEMATE [22]. Another approach is Requirements DrivenDevelopment, which is supported by the RDD-100 tool [23]. RDD-100 places itsemphasis on documentation and traceability of requirements and automaticallygenerates documentation that satisfies DOD-STD-2167A [24]. RDD-100 also providescapabilities for discrete simulation of a specification to test its completeness andfeasibility for different scenarios.

These semi-formal (or pseudo-formal) methods are of particular interest because they:

• Retain some of the precision of formal methods

• Require less mathematical background to apply

• Are well supported by several commercial software tools

It should be noted that the term “semi-” has to do with the precision (vs. informality) towhich individual operations are defined. The structure of operations and theirinteractions may be very precisely described by a semi-formal method.

Table 4-8 indicates how the Hatley-Pirbhai, RDD-100, and STATEMATE methods (ortools) implement the three distinct views of a specification previously introduced inSection 4.2.2.1.



4-26

Table 4-8Realization of the Three Specification Views for Several Methods

Method or Tool

Functional ViewWhat functions are to bedone? How does thedata flow betweenfunctions?

Architectural ViewWhat is the structure ofcomponents (hardwareand software) that areused to implement thefunctions?

Behavioral ViewWhen and how are thefunctions sequencedand managed?

Hatley-PirbhaiTool-IndependentMethod for Real-TimeSystems

Requirements Model(data flow diagrams)

Architecture FlowDiagram, ArchitectureInterconnect Diagram

Control Flow Diagram,Control Specification(finite state machine)

RDD-100Tool for RequirementsSpecification, Analysis,and Documentation

Behavior Graph(time functions anddiscrete functions)

Component Model Behavior and StimulusResponse Graphs(processing sequence)

STATEMATETool for Analysis ofReactive Systems

Activity-Charts(data flow diagrams)

Module-Charts Statecharts

4.2.4.1. Statecharts

Several semi-formal methodologies, such as Ward-Mellor [7], Hatley-Pirbhai [8], andRequirements Driven Development [23], exist with support from commercial tools. Thissection focuses on Statecharts [21] as an example of one of the more prominent semi-formal methods.

Statecharts take the basic state transition diagram (e.g., Figure 4-9) as a starting point,then add some powerful extensions:

• Statecharts can be nested as a hierarchy, so that broad behavior can first be definedand then expanded into more detailed underlying behavior.

• Concurrent processes can be modeled explicitly, with communication betweenprocesses.

• Statecharts are not simply diagrams; they are expressions of a rich and syntacticallyprecise graphical language that can be checked for grammatical properties or evenexecuted to test the specification.

Statechart technology is embodied in the commercial tool STATEMATE [22], whichconsists of an underlying database that can be accessed through graphical interfaces tothe three views listed in Table 4-8. Its functional view and data dictionary are nearly



4-27

classical implementations of data flow diagrams. In addition, an architectural view isprovided, so that functions can be assigned to specific hardware and softwarecomponents.

Functions can be described with text such as pseudo code. Alternatively, functions canbe described using a C-like language that can be executed to provide simulation tests ofthe specification for representative input scenarios. These capabilities have not yet beenfully exploited to produce truly formal specifications, but the potential is there. (Ofcourse, such a specification would tend to be quite prescriptive about the algorithmimplementation.)

Appendix 4.C expands the Statecharts approach using STATEMATE modeling andcompares it with the Parnas model.

4.2.4.2. Impact of Semi-Formal Methods on V&V

The attractions of semi-formal methods to improve V&V arise from the capabilities ofthe software tools used to implement them. We will discuss these capabilities forsimulation and static checks in terms of the STATEMATE tool, simply because it wasused for the illustrative examples presented in Appendix 4.C. However, similarcapabilities are present in other CASE tools available in the commercial marketplace.

Simulation

For many years, system and software designers have used graphical methods such asstate transition diagrams to help in defining system and software specifications. Thedesigners would use these graphical representations to aid in “thought experiments” orwalkthroughs of system operation. Such walkthroughs help designers find flaws in thespecification and improve their understanding of how the actual system will behave.STATEMATE provides discrete simulation6 capabilities to automate this walkthroughprocess. The statechart of the behavioral model, together with the connected activity-charts, can be executed to visualize the state transitions of a complex system under avariety of operational scenarios. Particularly powerful is the ability to visualize theinteractions among concurrent processes. These capabilities also provide a powerfultool for verification of the requirements and design, not only as an aid for the reviewerbut also to assist with developing test cases.

6 The reader should be careful not to confuse the discrete simulation at the specification level performed bySTATEMATE’s analyzer with continuous numerical simulation of the control algorithm and plant response. It ispossible, however, that the software tools will be extended so that these two simulations can be combined.



4-28

Static Checks and Tests

Because the semi-formal specification model is built as a structured database, ratherthan simply as text, important properties such as consistency and completeness can betested directly. In addition to the simulation capabilities provided for testing aspecification, STATEMATE provides several features for evaluating the specificationmodel.

• The Check Model feature examines the database for syntactic correctness, consistencyamong related charts, and completeness of representation.

• Reachability of Conditions — A search technique determines whether it is feasible toreach some condition or state. Failure of such a test indicates problems with thecorrectness or completeness of the state transition logic.

• Deadlock — This test searches for states, or groups of states, for which there is noexit. In essence, it is looking for traps or infinite loops.

• Non-Determinism — Ambiguous behavior must be detected and eliminated from thespecification.

• Unused Elements — This test looks for transitions that can never be encountered orconditions that are not needed. Such unused elements are at best unnecessary and atworst symptoms of errors.

• Problems with Concurrent Processes — This test searches for troublesome “race”conditions that occur when interacting concurrent states are not properlysynchronized or require information from each other that may not be available atthe appropriate point in the execution cycle.

Building the specification in the form of a software model (as with STATEMATE)opens the possibility of checking for such defects automatically and enhances the levelof verification that can be done to confirm the validity of the specification.

4.2.5. Function Block Languages

The software requirements methods discussed so far have been purely descriptive,with the emphasis on unambiguously defining functional behavior without prejudgingthe way in which the requirements will be implemented in software. A complementaryapproach is to make use of a function block language, which synthesizes softwarespecifications in terms of basic building blocks. Each building block describes acommon operation, such as logical AND, logical OR, differentiation, or proportional-integral-differential (PID) control. The attraction of this approach is that some commercial



4-29

products maintain libraries of such building blocks, so that implementation of thespecifications can be done directly by invoking blocks from the library. Such productsinclude programmable logic controllers (PLCs) and modular control systems. Twoexamples of function block languages follow:

The first example (from [25]) illustrates the use of a function block language forsynthesizing a trip function. Figure 4-12 is the specification of a basic block forcomparison (with hysteresis) to a high limit. This specification for the COMPH blockunambiguously defines output Out1 given the current value of input In1. Note that ‘Out1

is the previously computed value of the output.

Figure 4-12Function Block Definition for a Comparator

Such blocks can then be combined to synthesize the more complex functions requiredof the application itself. As shown in Figure 4-13, the value of the trip outputC_NOP_PTRIP can be constructed in terms of COMPH and other function blocks.Because the block definitions, as well as utilities for combining them, are embodied in acommercial PLC product line, the need for low-level coding of these functions iseliminated. Instead, the block diagram of Figure 4-13 may be represented by aconfiguration file that invokes and connects the blocks as shown and assigns each



4-30

variable or parameter to an input of the block. In this example, AECL used this formatto provide a formal design description. AECL transformed this formal description fromformal requirements supplied in a tabular/graphical format.

Figure 4-13Example of a Trip Function Composed from Basic Function Blocks

A second example (from [26]) is part of the specification for a complex control systemfor a class of nuclear power plants. The specification in Figure 4-14 is constructed fromSAMA Diagrams [27], which have been used for many years to provide functionaldescriptions for both analog and digital control systems. In this example, the SAMAspecification was constructed directly from text-based system requirements, with theaid of insights provided by extensive software prototyping and simulation testing.Therefore, it was used in a combined role of formal requirements and formal designspecification.

Figure 4-14 utilizes a variety of blocks important to continuous control, includingdifferentiation (d/dt), integration (I), and deadband (DEAD). Because many of theseblocks are more complex than those of the previous example, the parameters andsetpoints are not explicitly included as part of the diagram. Several vendors of modularcontrol system products allow control logic to be synthesized directly from suchfunctional diagrams, again eliminating the need for low-level coding of these basicfunctions.



4-31

Figure 4-14Part of a Control System Specification using SAMA Diagram Language



4-32

Function block languages can provide an important, practical bridge between systemrequirements and software design. Block diagrams are conceptually close to otherengineering diagrams, so they produce software specifications that are quite reviewableby a variety of engineers, not just by highly specialized computer scientists. Someverification steps associated with design and implementation may be greatly simplifiedbecause much of the application-specific coding is eliminated. On the other hand, thequality of the generic commercial product (including the function block library)assumes greater importance, and the application loses some of its transparency toindependent scrutiny.

Because the language syntax presumes a class of products (e.g., PLCs), block diagramspecifications are often somewhat biased toward a particular implementation approach;i.e., they usually contain design decisions as well as pure requirements. Becausefunction block diagrams lie in a gray area between requirements and design, they canbe useful contributions to either the Software Requirements Specification (SRS) or theSoftware Design Description (SDD).

The vocabularies of existing block languages may not be sufficiently expressive todefine all possible operations. For example, SAMA diagrams do not have any blocksthat explicitly specify iterative operations or that recognize the sequence of operationsimportant to certain numerical approximations. Consequently, there may be a tendencyto force fit new functions into an existing block language, with the penalty ofinefficiency or unnecessary complexity. Of course, new blocks may be defined anddocumented as in Figure 4-12 and, in some cases, may be added to the commerciallibraries (e.g., as macros).

In summary, block diagram languages provide a well-established means fordocumentation and specification of a large class of functions, and they can be closelytied to implementation through standard commercial products. Whether they shouldbe used in combination with or in place of the requirements methods described earlierdepends on the relative importance of the issues discussed in this section.

4.3. Other Analysis Techniques

The analysis techniques presented in this section are applicable primarily to front-endlife cycle activities — i.e., the system requirements, software requirements, andsoftware design phases shown in Figure 2-2. Although activities such as code analysisand database analysis do apply to the software implementation phase, by that point inthe life cycle, testing techniques involving execution of code begin to assume moreimportance.

Because these analysis techniques are highly dependent on the method used forrepresenting requirements as well as on the capabilities of specific commercial tools,



4-33

general methods will be emphasized over detailed techniques. To help make thediscussion specific, we will consider the capabilities of STATEMATE as an example,although other commercial tools provide similar capabilities.

4.3.1. Traceability Analysis

Regardless of the methods and tools used, a traceability matrix (cf. Figure 1-2 and Table5-1) should always be maintained and traceability analysis should be performed. Thetraceability analysis proceeds both in forward and backward directions.

Forward traceability analysis starts with each system requirement and follows itthrough every phase of the life cycle (see Figures 2-1 and 2-2). The purpose is todemonstrate that each requirement is covered by specific design elements and byspecific test cases. (The thread path audit described in Section 3.5.2 is related to forwardtraceability analysis.)

Backward traceability analysis starts with each design element to confirm that it ismotivated by a specific software and system requirement. This helps to reduce thepossibility of introducing unintended functions into the design.

4.3.2. Animation and Prototyping

Although animation and simulation are nearly synonymous terms, simulation iscommonly associated with either prototyping or testing, where both involve executionof completed code. It may avoid confusion to use the term animation when referring toanalysis of front-end life cycle activities such as requirements and design, but it shouldbe recognized that IEEE 1012 [28] refers to this task as a simulation analysis.

4.3.2.1. Requirements Animation

When requirements are represented in an unambiguous language, whether a semi-formal graphical language or a formal mathematical language, there exists the potentialfor executing the specification itself, before any source code exists, and even before adetailed design is completed. Animation of requirements can be used by developersand reviewers to challenge the ability of the requirements specification to deal withvarious combinations of input data and external commands. It is also useful inproviding the developers with additional insights regarding how a complex systemwill behave.

Animation of requirements presumes the existence of software tools for compiling andexecuting the specification language. For formal languages, such tools may not yet beavailable, but they are surely an active subject of research and development. Severalcommercial tools supporting semi-formal methods already provide facilities for



4-34

requirements animation. For example, as discussed in Section 4.2.4.2, STATEMATEprovides discrete simulation capabilities to automate this walkthrough for checking thebehavior of the finite state machine (FSM) model.

4.3.2.2. Design Animation

During the design phase, functions are allocated to hardware and software modules.For instrumentation and control systems, it is common to distribute some functionsacross parallel computers while other functions may be performed serially on a singlecomputer. Such complex architectures presume that interacting tasks are completed in atimely fashion. For example, the B&W Advanced Control System (described in Section8) split its integrated control algorithm over three parallel computers. To maintainstable response, outputs from each segment must be completed before they are neededas inputs by other segments.

Design animation (also known as performance simulation) models the capacity andtime response of components deterministically. The model may use either assumed orrandomly varying system loads to evaluate the adequacy of time dependentperformance. Such models can be developed with the aid of a commercial tool such asSES/Workbench [29].

4.3.2.3. Prototype Evaluation

Development and evaluation of prototypes can be extremely valuable in refining userrequirements and in determining their feasibility. The software life cycle shown inFigure 2-2 identifies prototyping as a distinct (but optional) phase. Section 2.1.3discusses both the demonstration prototype and the functional prototype. A functionalprototype is evaluated to determine its accuracy, efficiency, and implementationdifficulty before committing to a specific approach for the final design. Evaluation of ademonstration prototype usually involves exposure to end users for help indetermining whether the proposed product performs useful functions, whether its userinterface is appropriate, and how much user training will be necessary.

4.3.3. Static Analysis

Techniques for static analysis attempt to check important properties of a design withoutactually executing the system. Therefore, static analysis can evaluate such propertiesearly in the life cycle before any code is available for testing. Static analysis may beespecially useful during the requirements phase, depending on what method or tool isused to express the requirements.

Because a semi-formal specification model is built as a structured database, rather thansimply as text, important properties such as consistency and completeness can be tested



4-35

directly. For example, as discussed in Section 4.2.4.2, STATEMATE provides severalfeatures for static analysis of the specification model, including:

• Syntactic correctness, consistency, and completeness of representation

• Reachability of conditions

• Deadlock

• Non-determinism

• Unused elements

• Problems with concurrent processes

Other properties, such as low coupling and high cohesion, help when evaluating the waythe system is partitioned and designed into modules (cf. [30]). These properties maynot be specifically identified in the functional requirements specification, but they areindications of a design that will be easy to implement, test, and maintain. Coupling andcohesion are described in Section 4.2.2. Low coupling helps to ensure that no moduledepends on the internals of another module. This quality is essential for maintainabilityand, parenthetically, is one of the underlying principles of object-oriented designmethods discussed in Section 4.2.2.2. High cohesion within a module means that all theelements of that module are closely associated with a single function. The data flowdiagrams and structure charts used in structured methods provide a convenientmechanism for evaluating cohesion and coupling.

IEEE 1012 suggests several analysis activities as optional V&V tasks, which are selecteddepending on the individual characteristics of the software system, available tools, andpersonnel skills. Control flow analysis ensures that the proposed control flow is free ofproblems, such as design or code elements that are unreachable or incorrect. Data flowanalysis ensures that the input and output data and their formats are properly defined,and that the data flows are correct. Each of these analyses assumes that requirementsand design have been expressed through a structured or semi-formal method.

With particular software development methods, special analyses may be possible. Forexample, rule-based (expert) systems do not really express logic as code. The logic isincluded in a rule base that is treated as an extended form of data. This rule base can beanalyzed externally to look for anomalies, such as redundant or incomplete sets ofrules, and for inconsistent rules, circular logic, or other defects. As another example, itis possible to perform extensive static analyses using tools designed for databasesystem development.



4-36

4.3.4. Formal Proofs

One of the strongest motivations for using formal methods is the potential for provingproperties of the specifications or, more precisely, proving that a detailed designsatisfies a high-level requirements specification. Because it is very difficult to provehigh reliability by testing alone, the ability to prove correctness of a specification orimplementation would be extremely useful for certain critical applications having verystringent requirements with respect to failure rate. An example, which has been tackledby researchers sponsored by both EPRI and the Department of Defense, would be toprove that the design for a fault-tolerant processor satisfies the property of insensitivityto a single failure.

The development of methods and automated tools for formal proofs (also called formalverification) is an active area of research. The most mature “automated proving” tools donot actually discover the proofs; they merely check and document the steps. Therefore,trained mathematicians would be needed to perform formal proofs. Formal proofs tosupport digital I&C systems are unlikely at the present time.

Less demanding than a formal proof is a rigorous proof prescribed for critical softwareby the UK defense establishment [12]. In contrast to a formal proof, which wouldresemble an unimpeachable set of steps that might be found in a mathematical journal,a rigorous proof could be an outline of the steps or, essentially, a critical walkthroughof the logic.

4.4. Summary

This section provides an introduction to the wide range of methods for softwareanalysis, and particularly for requirements representation. In addition to traditionalapproaches such as natural language text and structured analysis, they include:

• Formal (Mathematical) Languages

• Tabular/Graphical Languages

• Semi-Formal Methods and Tools

• Block Diagrams

In contrast to the neutral technical descriptions presented in this section, the summarybelow states some normative conclusions about the practical roles of these newermethods.



4-37

4.4.1 Formal (Mathematical) Languages

It is crucial that system and software requirements be understandable by experiencedengineers, not just by mathematicians or computer scientists. Appendix 4.B is includedto illustrate how a formal language may be applied to a typical performancerequirement for a control system. Despite the extensive comments provided, the readermay question whether the formal specification actually clarifies the meaning of thisparticular requirement.

Despite the enthusiasm for formal languages in the academic community, mostpublished successes with such languages have been limited to small, specializedproblems. To be practical for industrial efforts by practicing engineers, commercialquality software tools must be available to help construct, diagnose, and animate thespecification. Existing tools are quite limited in their functionality to simple tasks suchas syntax checking. They typically do not deal with more subtle attributes such ascompleteness or consistency. Furthermore, existing tools are of research quality, androbust commercial tools are not widely available.

One of the prime motivations for pursuing formal languages is the possibility forirrefutable formal proofs. Candid authors agree that formal proofs and the tools tosupport them are still in the research stage. The current state of the practical art stops at“rigorous” proofs, which are essentially sound arguments that require much training,engineering labor, and mathematical judgment to outline, perform, and review eachproof.

The conclusion is that formal languages and supporting tools must be given more timeto mature before they can be used cost-effectively and with confidence for significantnuclear power digital I&C applications. The project engineer wishing to applyanalytical techniques for requirements and design need not be too concerned about thisunsettled situation with respect to formal languages because several other usefulapproaches are available. While these more mature methods are not as powerful or all-encompassing as formal languages, they can help address the most prominent singleconcern in design of high integrity systems, which is the need to specify more completeand usable functional requirements with a minimum of ambiguity. Some conclusionsregarding such methods are given below.

4.4.2 Tabular/Graphical Languages

Parnas’ method, adapted by Ontario Hydro and discussed in Section 4.2.3.3, issomewhat restricted in scope compared to a more general formal mathematicallanguage. In addition, it has limited tool support. However, utility company engineershave successfully applied this method to safety-critical systems in nuclear powerplants. Within its scope of coverage, its descriptions can be made unambiguous. Most



4-38

important, the tabular descriptions of simple functions can be understood by someonehaving minimal training and no special knowledge or tools. For example, the meaningof Table 4-6 can be easily grasped without extensive explanation. For simple safety-critical systems, this is a significant statement. Descriptions based on this method canbe independently reviewed by both verifiers and regulators.

4.4.3 Semi-Formal Methods and Tools

For major systems that include many states, complex interactions with the plant or withoperators, or hard-to-analyze features such as event driven logic, powerful methodssupported by automated tools should be considered. Such methods may providehierarchical representations and semantics that are more expressive, allowing therequirements of more complex systems to be modeled and comprehended. Section 4.2.4described several semi-formal methods, such as Statecharts, that combine finite statemachine (FSM) models with more traditional structured analysis methods. Some toolsprovide the capability to animate the specification before any code is generated,introducing analysis and testing very early in the life cycle. Other capabilities includechecking for completeness and consistency of the underlying database.

4.4.4 Function Block Languages

Function block languages provide another approach for achieving more complete andunambiguous software specifications. If an early project decision is made to restrictattention to an appropriate class of commercial products (e.g., PLCs), then availablefunction block languages are especially useful. When requirements and design arerepresented using such methods, both implementation and verification are simplifiedas well. If a function block language is used as a tool for specification of requirements,it should be understood that the results tend to integrate certain design decisionsinstead of focusing only on requirements. In general, it is desirable to keep therequirements specification independent of the design.

4.4.5 Final Considerations

With the dynamic evolution of methods and tools, it appears that all of these diverseapproaches are beginning to converge. Some research papers frankly acknowledge thatformal languages are hard to use and, to be practical, must be combined with graphicaltools borrowed from structured analysis. Likewise, semi-formal tools are beginning tointroduce simple languages for defining low-level functions in detail. Consequently,semi-formal methods may approach the rigor and clarity of tabular/graphical methodsor the rigor and flexibility of formal methods.



4-39

Finally, the choice of approach should not be based on technical capability alone, but itshould also consider:

• Level of rigor demanded by the application

• Level of rigor provided by the method or tool

• Costs of acquiring, using, and training for the method or tool

• Capabilities of the staff to use the method and to review its results

• Proven experiences within the nuclear power industry

At present, no “optimization algorithm” exists for making this choice. It can only bedone by informed judgment of the project development staff and the V&V team.

4.5. References

1. Boehm, B. W., et al., Some Experience with Automated Aids to the Design of Large-ScaleReliable Software, IEEE Transactions on Software Engineering, Vol. 1, pp. 125–133,March 1975.

2. Davis, A. M., Software Requirements, Prentice Hall, 1993.

3. IEEE Std 830-1993, Recommended Practice for Software Requirements Specifications.

4. IEEE Std 1016-1987, Recommended Practice for Software Design Descriptions.

5. IEEE Std 1016.1-1993, Guide to Software Design Descriptions.

6. DeMarco, T., Structured Analysis and System Specification, Prentice Hall, 1978.

7. Ward, P., and S. Mellor, Structured Development for Real-Time Systems, Prentice Hall,1985.

8. Hatley, D., and I. Pirbhai, Strategies for Real-Time System Specification, Dorset House,1987.

9. Coad, P., and E. Yourdon, Object-Oriented Analysis, Yourdon Press, 1991.

10. Sommerville, I., Software Design for Reliability, Software Reliability Handbook, P.Rook, Ed., pp. 21–50, Elsevier, 1990.



4-40

11. Wing, J. M., A Specifier’s Introduction to Formal Methods, IEEE Computer, pp. 8–22,September 1990.

12. MOD 00-55, Requirements for Safety Related Software in Defence Equipment, Part 1:Requirements, Part 2: Guidance, UK Defence Standard 00-55/Issue 2, 1 August 1997.

13. Potter, B., et al., Introduction to Formal Specification and Z, Prentice Hall, 1991.

14. Jones, C. B., Systematic Software Development using VDM, Prentice Hall,1990.

15. Delisle, N., and D. Garfan, A Formal Specification of an Oscilloscope, IEEE SoftwareSpecial Issue on Formal Methods, pp. 29–36, September 1990.

16. May, R. S., Formal Methods and Tools for Nuclear Power Plant Control Systems: AnExploratory Study, Report to EPRI Nuclear Power Division Exploratory ResearchCommittee, August 1993.

17. Parnas, D. L., et al., Assessment of Safety-Critical Software in Nuclear Power Plants,Nuclear Safety, Vol. 32, No. 2, pp. 189–198, April–June 1991.

18. Joannou, P. K., et al., Software Requirements Specification for Critical Applications,Proceedings: Methodologies, Tools, and Standards for Cost-Effective, ReliableSoftware Verification and Validation, EPRI TR-100294, January 1992.

19. Nicholl, R. A., Unreachable States in Model-Oriented Specifications, IEEE Transactionson Software Engineering, Vol. 16, No. 4, April 1990.

20. DRIC-BR-115966, Translating Data Flow Diagrams into Z (and Vice Versa), Royal RadarEstablishment, United Kingdom, October 1990.

21. Harel, D., Statecharts: A Visual Formalism for Complex Systems, Scientific ComputerProgramming, Vol. 8, pp. 231–274, 1987.

22. Marketing Literature, The STATEMATE Approach to Complex Systems, i-LogixCorporation.

23. Marketing Literature, Requirements Driven Developer, Ascent Logic Corporation,December 1989.

24. DOD-STD-2167A (superseded by MIL-STD-498), Defense System SoftwareDevelopment.



4-41

25. Ichiyen, N. M., V&V Experience — AECL/Ontario Hydro, Presentation to EPRI/UtilityWorking Group Meeting on V&V and Related Issues, Chattanooga, TN, May 4–5,1993.

26. Wilson, T. L., PCS Design Description Report, EPRI and B&W Owners’ GroupAdvanced Control System Task Force, February 1992.

27. SAMA Std PMC 22.1-1981, Functional Diagramming of Instrument and Control Systems.


29. Forte, G., Tools Fair: Out of the Lab, Onto the Shelf, IEEE Software, May 1992.

30. Page-Jones, M., Practical Guide to Structured Systems Design, Yourdon Press, 1988.

31. Mahony, B. P., and I. J. Hayes, A Case-Study in Timed Refinement: A Mine Pump, IEEETransactions on Software Engineering, Vol. 18, No. 9, pp. 817–825, September 1992.



4-42

Appendix 4.A. Suggested Contents of SRS and SDD

4.A.1. Software Requirements Specification

IEEE 830 [3] presents the following outline of a Software Requirements Specification(SRS). The standard includes eight alternative “templates” for Section 3 of the SRSdepending on whether the specific requirements should be organized by mode (withtwo versions), user class, object, feature, stimulus, functional hierarchy, or with multipleorganizations. (In the outline illustrated below, Section 3 is organized by feature.) Suchwide latitude in the standard reinforces the notion that the developer must useknowledge of the system requirements to select an appropriate format for the SRS aswell as provide its content. IEEE 830 elaborates on each item suggested by the outline.

Table 4-9Typical Outline for Software Requirements Specification

1. Introduction1.1 Purpose1.2 Scope1.3 Definitions, acronyms, and abbreviations1.4 References1.5 Overview

2. Overall Description2.1 Product perspective2.2 Product functions2.3 User characteristics2.4 Constraints2.5 Assumptions and dependencies

3. Specific requirements (organized by feature)3.1 External interface requirements

3.1.1 User interfaces3.1.2 Hardware interfaces3.1.3 Software interfaces3.1.4 Communications interfaces

3.2 System features3.2.1 System feature 1

3.2.1.1 Introduction/purpose of feature3.2.1.2 Stimulus/response sequence3.2.1.3 Associated functional requirements

3.2.1.3.1 Functional requirement 1…3.2.1.3.n Functional requirement n

3.2.2 System feature 2…3.2.m System feature m…

3.3 Performance requirements3.4 Design constraints3.5 Software system attributes3.6 Other requirements



4-43

4.A.2. Software Design Description

IEEE 1016 [4] describes recommended practice for the Software Design Description(SDD). According to the standard, “The SDD shows how the software system will bestructured to satisfy the requirements identified in the software requirementsspecification….” It is not necessary, or even desirable, for the SDD to spell out everyminute detail of how the design is to be implemented. Its greatest value comes from:

• Partitioning the system into manageable subsystems, components, processes, orother design entities

• Defining the attributes of and relationships among these entities

• Identifying a home for each requirement in one or more of these entities

It is particularly important to define the interfaces among the design entities.

Ideally, the SDD should be completed and verified before actual coding is started.However, for many demanding applications, small-scale software prototyping may beneeded to prove the feasibility of the design before a full commitment is made to thedesign approach.

IEEE 1016 presents the following outline as “… only one of many possible ways toorganize and format…” an SDD. The standard identifies modules, concurrentprocesses, and data as classes of design entities within a software system. Section 4 ofthe outline addresses dependencies among these entities. Design complicationsfrequently arise when considering subtle side effects on other entities and unstatedassumptions about the effects of other entities. IEEE 1016.1 [5] provides additionalguidance regarding preparation of an SDD.



4-44

Table 4-10Typical Outline for Software Design Description

1. Introduction1.1 Purpose1.2 Scope1.3 Definitions and Acronyms

2. References3. Decomposition Description.

3.1 Module Decomposition3.1.1 Module 1 Description3.1.2 Module 2 Description…

3.2 Concurrent Process Decomposition3.2.1 Process 1 Description3.2.2 Process 2 Description…

3.3 Data Decomposition3.3.1 Data Entity 1 Description3.3.2 Data Entity 2 Description…

4. Dependency Description4.1 Intermodule Dependencies4.2 Interprocess Dependencies4.3 Data Dependencies,

5. Interface Description5.1 Module Interface

5.1.1 Module 1 Description5.1.2 Module 2 Description…

5.2 Process Interface5.2.1 Process 1 Description5.2.2 Process 2 Description…

6. Detailed Design6.1 Module Detailed Design

6.1.1 Module 1 Detail6.1.2 Module 2 Detail…

6.2 Data Detailed Design6.2.1 Data Entity 1 Detail6.2.2 Data Entity 2 Detail…



4-45

Appendix 4.B. Application of Z to a System Performance Requirement

EPRI sponsored [16] a small exploratory research study to evaluate formal and semi-formal methods such as Z for use in specifying control system requirements. In thisappendix, we take an actual performance requirement from a nuclear plant I&C systemand rewrite it formally in the Z (pronounced “Zed”) language [13]. This exerciseillustrates in concrete terms how the Z language is applied. It enables evaluationwhether such a specification can be understood. By obtaining firsthand experience withthis Z specification, we can draw independent conclusions regarding its use in nuclearplant I&C systems.

4.B.1. Performance Requirement

The B&W Owners’ Group Advanced Control System (ACS) is described in Section 8.Consider the following performance requirement #4.3.3 from the ACS RequirementsDocument.

“4.3.3 The ACS shall provide a fast power reduction feature by which the operator mayinitiate a power reduction from the initial operating power level to lower end of theautomatic control range (1%) at rates up to the capacity of the reactor’s response (about20%/minute) and within ability of the ACS to maintain control of the key variables.”

Before expressing this requirement formally, let us break it up into some key chunks fordiscussion.

“4.3.3 The ACS shall provide a [fast power reduction feature] by which the [operator mayinitiate a power reduction] [from the initial operating power level to lower end of the automaticcontrol range (1%)] [at rates up to the capacity of the reactor’s response (about 20%/minute)]and [within ability of the ACS to maintain control of the key variables.]”

At the risk of belaboring the obvious, consider how each of these must be evaluated inthe final system.

[fast power reduction feature]This is really the subject of the requirement.

[operator may initiate a power reduction]The feature is initiated by operator action, not automatically.

[from the initial operating power level to lower end of the automatic control range (1%)]This defines the set of initial conditions, as well as the final target power level, underwhich the feature must work.



4-46

[at rates up to the capacity of the reactor’s response (about 20%/minute)]This bounds the rate of change of demanded power; i.e., –20% < d/dt(DemandPower) <0 for this feature.

[within ability of the ACS to maintain control of the key variables.]It is clear that this acceptance criterion must be defined more precisely to allow for anobjective evaluation and, especially, to allow any automatic screening of validation testresults. We interpret this acceptance criterion to mean that both of the following criteriashall apply, together:

1. For all operational and upset transients, there shall be no reactor trip.

2. For all operational transients, key process variables must stay within an acceptableband of a target trajectory that is defined in advance of the event.

The meaning of an acceptable band is show in Figure 4-15. The exercise of describingsuch a band (in abstract terms) is included in the Z specification that follows.

Simulation Time

Pro

cess

Var

iabl

e

Desired End State

Acceptable Band

Transient

Note: The Target Trajectory runs through the middle of the band.

Figure 4-15Meaning of Acceptable Band for Evaluating Transient Response



4-47

4.B.2. Z Specification of the Performance Requirement

Table 4-11 contains an explanation of the terminology and notation used in thisexample of a Z language specification of performance requirement #4.3.3. The Zspecification is given on the left side of Table 4-12. Some explanatory comments areincluded next to each item of the specification; nevertheless, a few general commentsare in order.

The actual definition of the FastPowerReduction feature (requirement #4.3.3) liescompletely on one page of Table 4-12 directly under the heading PerformanceRequirements. However, it takes six pages to write the complete specification (plus onepage for requirement #4.3.5 that was added as a second example). There are two goodreasons for this apparent imbalance. First, we must define the data structures, which inessence define the grammar for the system specification, before any single part of thespecification makes sense. Declarations of data and general function types consume twopages. Second, the definition of requirement #4.3.3 refers to operations and functionsthat must be further defined. For example, “ran case ⊆ AcceptableBand” means that therange (or outcome) for each key variable in the simulation test case lies within anacceptable band of results, but then we must define what we mean by an acceptableband. Defining such supporting functions consumes three pages.

Referring to the Performance Requirements section of Table 4-12, notice the standardform of a Z schema or mini-specification of a function or operation. The intention is foreach such definition to be as self-contained as possible. The first block consists purelyof declarations of data and functions that are specific to that operation, namelyFastPowerReduction. For example, it indicates that FinalPower is an input variable, andFastestRampRate is a parameter that must be specified before the operation can beevaluated. The actual operation or function is defined below the line in the schema interms of predicates, which are simply statements that can be true or false. For evaluatingthe FastPowerReduction function, the predicate states that for all power reductiontransients within the desired ramp rate and initial and final power levels, there can beno trip condition, and the key variables must lie within the acceptable band.

Note that the specification is quite simplified and has much room for improvement. Forexample, tolerances on the key variables are constant with time. For another, theacceptance criterion includes nothing to ensure that changes of discrete modes occur atappropriate times. However, this Z language specification is a simple start thatillustrates the principles for a formal specification of requirements.



4-48

Table 4-11Terminology and Notation Appearing in the Z Specification

Terminology/Notation Comments

N-Tuples (Sequences) <x1,..,xN> denotes the “n-tuple” or sequence of N ordered items. N=2 yieldsan ordered pair, etc.

Concatenation If A is an n-tuple and B is an m-tuple, then A^B is an (n+m)-tuple.

Function I/O

x? Declares a variable x that is the input to a function.

y! Declares a variable y that is the output of a function.

y,y’ The values of y before and after the operation, respectively.

Variable Typing

[A] Denotes that A is a basic variable type.

== Used to denote a new type definition (based on one or more basic types).

Æ Declares a functional relationship (i.e., a mapping) from one set to another. We willgloss over subtle distinctions of types of functions (e.g., one-to-one mapping, partialfunction, etc.). Note that in Z, a function is just another type of variable (namely, arelation between sets).

Additional Definitions

Ø Denotes an empty set (i.e., a set with no elements in it).

∀ “for all”

∃ “there exists”

# Operator that determines the number of elements in a set.

∧,∨ AND, OR

A⊂B A is a proper subset of B.

A⊆B A is a subset of B.

A⇒B A implies B (i.e., if A is true, then B is true).

“Predicate” An assertion that may have a value of TRUE or FALSE depending on thevalue of the variables within it.

• A placeholder to separate conditions from a predicate.

A∪B Denotes the union of two sets A and B.

A∩B Denotes the intersection of two sets A and B.

dom R The domain of the relation R.

ran R The range of the relation R.

F S The set of subsets of the set S.



4-49

Table 4-12Abstract Specification of Typical Performance Requirements in the Z Language

Z LANGUAGE ATTRIBUTE DESCRIPTION

GLOBAL DECLARATIONS Global declarations define sets and data types that apply throughout thespecification. Note that they are not restricted to the example performancerequirements. They constitute the basic vocabulary of the specification.

BASIC VARIABLE TYPES

[Action] All operator-initiated actions not modeled as part of the normal, automaticresponse of the plant.

[Variable] Any plant process variable (e.g., reactor pressure). Dependent variables intransient test case.

[TestCase] Abstract definition of a scenario (e.g., reactor trip, power increase from onepower level to another, etc.)

[DesignParameter] Includes constants or setpoints established during design.

SimulationTime == Z Simulation Time, dependent variable of type integer. Note that floatingpoint numbers are not represented explicitly.

[Message] Text message (e.g., describing results of an operation).

[List] List (i.e., sequence) of text information.



4-50

CONSTRUCTED TYPES, SUBTYPES, AND GLOBAL DECLARATIONS

KeyVariable : F Variable The key variables compose the subset on which the success or failure of thetest is evaluated. They may be redefined for each transient as a subset of thepossible process variables.

Power: KeyVariable The thermal power is always a key variable.NV = #KeyVariable This defines the size of the “n-tuple” that characterizes the transient

response at each time.SimulationOutput == ⟨v1,..,vNV ⟩

where ∀i, i=1,..,NV, vi ⊆ KeyVariableand v1 ≡ Power

This is the NV-dimensional simulation SimulationOutput at each time. Forexample, if Power, ReactorWaterLevel, and Drywell Pressure were the keyvariables, then NV = 3 and SimulationOutput = <Power, ReactorWaterLevel,Drywell Pressure>Note: SimulationOutput is only a set of possible outcomes, or, if you like, atemplate. For each simulation time during a given transient, there will be anew set of specific values.

DemandPower: Variable DemandPower may be regarded sometimes a driving input and sometimesa result of the transient (as in the ACS).

InitialCondition: SimulationOutput Certain key variables (e.g., power) may be defined at initial points.TestCase = NormalOperation ∪ Upset ∪ Accident Test cases are categorized according to severity, and different acceptance

criteria may apply to each level of severity.Upset = SteamPressureEvent ∪ FeedwaterFlowEvent ∪ …

∪ ReactivityEventUpset transients can be further broken down into subclasses for definingtransient test cases.

Tolerancei: DesignParameter, ∀i = 1,..,NV This will be the acceptable tolerance used to judge whether the computedprocess variables lie within an acceptable distance of the target trajectory.Because these tolerances are constant, the criterion is very simple, but it canbe easily generalized.

LowerTripLimiti, UpperTripLimiti: DesignParameter, ∀i =1,..,NV

These are trip boundaries for the key variables. Note that the trip criterion isonly static and does not allow for time smoothing, etc.; however, byredefining another state (i.e., KeyVariable) in the simulation, suchrefinements are easily done.

Transient == TestCase X InitialCondition → SimulationTime→ SimulationOutput

This declaration abstracts a “transient” as a set of mappings from theSimulationTime to process variables. The actual mapping, of course, is doneby a simulation code or training simulator, so this relation is a black-boxabstraction of the simulator. Therefore, this should be regarded as aspecification for an external function.



4-51

PERFORMANCE REQUIREMENTS

4.3.3 FastPowerReduction

PowerChangeTransient ==InitialCondition? X FinalPower? X DemandPowerRateofChange→ SimulationTime → SimulationOutput

FastestRampRate, MinimumAutoPower, FullPower: DesignParameterFinalPower?: Variable

PowerRateofChange = Derivative (Power)

∀ case: PowerChangeTransient |FastestRampRate < DemandPowerRateofChange < 0∧ MinimumAutoPower < FinalPower < InitialPower < FullPower;•ran case ⊆ AcceptableBandran case ⊆ NoTripCondition

This is the mini-specification or “schema” for an operation. In this case, the operationis a constraint on acceptable SimulationOutputs of test cases, as implied by ACSFunctional Requirement 4.3.3.

This defines the type of test cases defining the conditions for the requirement to beevaluated. Note that this criterion is very specific, so it can be pinned down even atthe abstract level.

The “inputs” to the operation are denoted by “?”, and they are simply the quantitiesthat define the case: InitialCondition? and FinalPower?. It is assumed thatDemandPowerRateofChange may be varied within the case.

The power derivative variable is derived from the power variable.

The meat of the performance requirement comes here. Consider all fast reduction powertransients between operating power level InitialPower and FinalPower, which areconstrained as shown. For every case considered for fast power reduction transients,the predicate states that no trip condition should be generated and that key variablesmust remain within an acceptable band around a target response.

The notation “ran case” is simply the range of that transient case or predicate (i.e., theoutcome of the simulation test case).

The details about acceptance criteria are defined under Supporting Functions belowin the definitions of AcceptableBand and NoTripCondition and in later refinementswhere specific numerical values may be specified.



4-52

4.3.5 UpsetControlability

Upset ⊆ Transient

∀ case : Upset • ran case ⊆ NoTripCondition

As a second example, this specification is similar to 4.3.3, except that it applies to alltransients in the “Upset” category rather than applying to power reduction transients.Also, the acceptance criteria may be less stringent and, as shown here, only involvetrip avoidance.

Note that this test class is much more general, so the specific definition of each testcase must wait until later and may involve specific assumptions of equipment failure,operator actions, and other details.

The text description of 4.3.5 follows:

“The ACS shall be capable of controlling the plant under the following transients:Steam Pressure transients such as

Turbine trip,Load rejection,TB valve open,MSIV closure.

Feedwater flow transients such as….Reactor coolant flow upset transients such as….Reactivity transients such as

One dropped rod.”



4-53

SUPPORTING FUNCTIONS

QuasiStaticBalance

d?: DemandPowerp!: power∀i, i=1,..,NV, vi!: KeyVariable

Static == DemandPower → SimulationOutput

d' = dp' = d

⟨v1',..,vNV'⟩ = Static (d)

This function computes a quasistatic energy balance based on the demand power.

It takes current DemandPower as input (denoted by “?”).It produces remaining “demand” values for other key variables on output (denotedby “!”).

DemandPower is not modified by the function.Set the quasistatic power equal to the demand power.

Static is an external function that would be realized as a steady state computer codeor by running the simulator to achieve a converged steady state.

Note that there are several ways to compute such a balance depending onassumptions of boundary conditions. Also, the balance may be generalized toaccommodate time lags.

TargetTrajectory

DemandPowerTrajectory == SimulationTime → DemandPowerdp?: DemandPowerTrajectory

TargetTrajectory == SimulationTime → SimulationOutput

∀t: SimulationTime | t ∈ dom DemandPowerTrajectoryTargetTrajectory(t) = Static (dp(t))

As input, this function needs a function setting the DemandPower for each timeduring the transient.

Note that this is one approach to computing the trajectory: From the demand thermalpower and a static energy balance, one can derive the feed forward terms that definetargets for each variable throughout the transient. Therefore, TargetTrajectory is acomposite of two other functions — DemandPowerTrajectory and Static.



4-54

AcceptableBand

case?: Transientrep!: message

∀i, i=1,..,NV, ∀t: SimulationTime | t ∈ dom case|casei(t) – TargetTrajectoryi(t)| ≤ Tolerancei

rep! = “All Transient Variables within Acceptable Bounds”

This function determines whether the simulation result is within acceptabletolerance of the target.

This straightforward predicate simply says that the transient must remain within atolerance of the target trajectory for all times during the transient and for all of theKeyVariables. If so, a message is generated indicating that the transient is withinacceptable bounds.

Note: casei(t) may deviate from standard Z-notation, but it simply denotes the ith

SimulationOutput evaluated at time t.

UnAcceptableList


UnacceptableList!: List

∃i, i=1,..,NV, ∃t: SimulationTime | t ∈ dom case|casei(t) – TargetTrajectoryi(t)| > Tolerancei

rep! = “At least one KeyVariable is outside bounds”

UnacceptableList = ∅

∀i, i=1,..,NV, ∀t: SimulationTime | t ∈ dom case|casei(t) – TargetTrajectoryi(t)| > Tolerancei ⇒UnacceptableList' = UnacceptableList ∪ ⟨t,i,casei(t)⟩

This is, of course, just the complement of AcceptableBand. The reasons for having aseparate function are to generate a message indicating that the bounds are violatedand identify the variable that has strayed too far.

This predicate is true if there exists at least one time during the transient when atleast one key variable lies outside the bounds.

Initialize the Unacceptable list to an empty set.

The next operation selects all errant points (time variable pairs) and adds them to thebad list. The new list item will include (a) the time, (b) the index indicating whichvariable, and (c) its value.



4-55

NoTripCondition


∀i, i=1,..,NV, ∀t: SimulationTime | t ∈ dom caseLowerTripLimiti < casei(t) < UpperTripLimiti

rep! = “Transient Proceeds with no Trip”

This function defines the conditions under which no trip signal is generated. It issimilar to AcceptableBand operation, so see further comments there.

TripList


∃i, i=1,..,NV, ∃t: SimulationTime | t ∈ dom casecasei(t) < LowerTripLimiti ∨ casei(t) > UpperTripLimiti

rep! = “At least one KeyVariable is outside bounds”

TripList = ∅

∀i, i=1,..,NV, ∀t: SimulationTime | t ∈ dom casecasei(t) < LowerTripLimiti ∨ casei(t) > UpperTripLimiti ⇒TripList' = TripList ∪ ⟨t,i,casei(t)⟩rep! = “Transient response would trip on variable #”

This function generates a list of variables and times when trip signals would begenerated. It is similar to UnacceptableList operation, so see further comments there.



4-56

4.B.3. Comments on the Z Language

This section contains some observations about the use of Z and, in particular, aboutawkward aspects of the Z grammar. One reason Z may be hard to get used to is that itis a purely declarative language that describes what is to be done, not how to do it.Most engineers are accustomed to procedural programming languages, such asFORTRAN or C, where the emphasis is on how an algorithm should be implemented.In contrast, Z specifications do not state how, so that the specification can beimplemented optimally in a variety of programming languages. Consequently,effective use of a formal language such as Z may require a change of mindset.7

The abstract nature of the Z language is partially necessary in order to restrain thespecifier from defining how a function is to be implemented. The language is intendedto be purely descriptive, not prescriptive, so it should be possible to implement a Zspecification in several different languages.

One of the more appealing features of Z is the schema format, for which the datadeclarations and operations are defined in a concise black-box format.

The standard form of Z used here does not explicitly allow real (floating point)variables; instead, it only allows positive and negative integers. One rationalization ofthis shortcoming [15] notes that real numbers must be sampled and converted to binaryintegers, but this argument is not satisfying. However, at least one application paper[31] recognizes the need for real variables in continuous processes and introduces anew data type.

The treatment of vector information (i.e., n-tuples) and multi-variable operations isextremely awkward. Such information is pervasive in control systems; for example, asimulation result is best considered as a time-dependent vector. Consequently,variable-by-variable definitions of acceptable band and transient response seemverbose and obscure.

Readers may make their own judgment regarding readability of the examplespecification. The purpose of a formal specification is to provide a precision-forcingbridge between natural language text requirements, which may be written by I&Cengineers, and software design and code, which may be written and used by softwareengineers and programmers. For a formal specification to promote communicationbetween these two groups, it must certainly be understandable by both groups. It maybe that Z is not appropriate for direct use by most utility company engineers. However,

7 A similar change of mindset is necessary to effectively use expert system development tools, which are alsodeclarative in nature.



4-57

such engineers may still appreciate the capabilities of formal specifications and mayrequire their use by more specialized staff from vendor organizations.

4.B.4. Conclusions and Recommendations

Several conclusions can be drawn from the results and experience of this exercise:

• It is indeed feasible to represent a performance requirement in a formal specificationlanguage and in a form that can define the basis for automatic evaluation ofvalidation test cases. However, the Z language software tools available during thestudy were not mature enough to make the actual automation a straightforwardtask.

• A surprising amount of formal infrastructure, through definition of data structuresand utility functions, must be built to support the formal definition of even a simplerequirement. The supporting structure seems out of proportion for this exampleprimarily because we are concentrating only on the single requirement #4.3.3.However, this structure is also applicable to additional requirements that may beadded.

• Practicing utility or vendor engineers that are not “formally” educated in formalmethods should expect to invest considerable effort to master a language such as Z.Proponents of Z and VDM have suggested that 1–2 weeks of formal training shouldbe sufficient. While this may be adequate for a reading knowledge to enable reviewby a software engineer, the amount of training and experience needed for a goodwriting knowledge to produce high quality specifications efficiently will no doubtbe much greater. Other authors have suggested that becoming proficient in aspecification language takes the same amount of time and effort as developing skillsin a new programming language — certainly several months even for anexperienced programmer.

We have not explored an important concept of the formal methods approach —refinement. The abstract formal specification can be used to generate a sequence ofspecifications that are increasingly more detailed and closer to an implementation form.When appropriate, the specification language (e.g., Z) can be converted to aprogramming language (e.g., C or Pascal) that can actually be compiled on a computer.Potter [13] defines rules for refinement that guarantee that each step is consistent withthe one before. Such rules form the basis for formal verification proofs.



4-58

Appendix 4.C. Statecharts Method and STATEMATE Modeling

EPRI sponsored [16] some limited STATEMATE modeling of the B&W Owners’ GroupAdvanced Control System (ACS) described in Section 8, which has some characteristicsthat complicate its specification and design:

• A complex, integrated control algorithm

• Triplicated hardware, with voting

• A user interface

• Performance requirements that depend on closed loop response

Some charts from this effort are included as examples of the Functional and BehavioralViews represented by STATEMATE. For brevity, the Architectural View is not shownhere.

4.C.1. Activity-Charts — The Functional View

Figure 4-16 show the top-level activity-chart for the ACS. Notice that the chart definesboundaries of the ACS and describes its interactions with the outside world of sensors,actuators, and human operators. This top-level chart is often called the context diagramof the system. In addition, basic functions (activities) such as USER_INTERFACE,CONTROL_ALGORITHM, and VOTING_LOGIC are defined in terms of theirexchange of data with each other.

Within the data dictionary, facilities are provided for defining mini-specs. These mini-specs, which can be used to precisely describe a function, can be executed during thetest of a specification model.

The @ symbol in a box indicates that the box is actually represented by a more complexchart located “off the page”. This is a mechanism for defining a hierarchy of functionsfrom very general to more detailed. Figure 4-17 expands the@CONTROL_ALGORITHM activity into more detail. Notice that the data flows in andout of @CONTROL_ALGORITHM are consistent between the two charts.



4-59

USER_INTERFACE

@CONTROL_ALGORITHM

OPERATOR OUTPUT

HEARTBEAT_LOGIC

@ACS_SUPERVISOR

@VOTING_LOGIC

@PERFORMANCE_VALIDATOR

PERFORMANCE

INPUT

ALGORITHM_INPUTS

ALGORITHM_OUTPUTS

OPERATOR_CTP

COMMAND

HEARTBEATS

ACTUATORS

SENSORS

MEASUREMENTS

ACTUATOR_SIGNALS

TRANSIENT_RESPONSE

FINAL_OUTPUTS

EVALUATION

DEFICIENCIES

OPERATOR_CMDS

DISPLAYS

Figure 4-16Top-Level Activity-Chart for ACS



4-60

FW_MASTER

FEEDWATER_VALVE

FEEDWATER_PUMP

FW_MASTER_DEMANDS

ALGORITHM_OUTPUT_DATA

FW_PUMP_SPEED_DMDS

FEEDWATER_CONTROL

TOTAL_FEEDWATER_DEMAND

@INTEGRATED_MASTER

CTP_DEMAND

@CTP_DEMAND_CONTROL

CTP_DEMAND

REACTOR_CONTROL

ROD_CONTROL_SIGNALS

FLUX_DEMAND

FW_VALVE_POSN_DMDS

STEAM_CONTROL

STEAM_ERROR

STEAM_CTL_OUTPUTS

INPUT

ALGORITHM_INPUTSVOTING_LOGIC

ALGORITHM_OUTPUTS

USER_INTERFACE OPERATOR_CTP

COMMANDS

CONTROL_ALGORITHM

Figure 4-17Lower Level Activity-Chart for Control Algorithm

4.C.2. Statecharts — The Behavioral View

The real power of STATEMATE (and similar tools) lies in its ability to model andsimulate system behavior. Figure 4-18 shows the top-level statechart for the ACSmodel. Each rounded box represents a state of the system, some states may exist inparallel, while others cannot be active at the same time and can only be activated insequence.

The arrows between states in Figure 4-18 represent state transitions, which inSTATEMATE may be triggered by either events or conditions. When theINIT_COMPLETE event allows a transition to the ACS_RUNNING state, threeconcurrent states are entered — MAIN_LOOP, @INTERFACE_CONTROL, andINPUT_POLLING. Concurrent states are separated by a dotted line. MAIN_LOOPcontains three substates that are always visited in sequence — first executing thecontrol algorithm, then performing the voting logic, and finally outputting signals to



4-61

the field actuators. Concurrently, INPUT_POLLING is taking place, and the userinterface (@INTERFACE_CONTROL) is either awaiting or processing user commands.

T

TERMINATING_EVENTINITIALIZING

ACS_RUNNING

EXECUTING_ALGORITHM

@INTERFACE_CONTROL>INPUT_POLLING>

EXECUTING_VOTING> OUTPUTTING_TO_FIELD>VOTING_COMPLETE

ALG_CYCLE_COMPLETE

MAIN_LOOP

ACS_SUPERVISOR

INIT_COMPLETE

CYCLE_COMPLETE

Figure 4-18Top-Level Statechart for ACS

Within the data dictionary, states and activities may be connected, so that one activatesor deactivates the other. In this way, the connections between function and behaviorcan be investigated and simulated. More detailed statecharts may either be referencedin the statechart hierarchy or invoked from activity-charts.

Figure 4-19 provides a further example of more detailed statecharts, as applied tospecification of a control interface, with functions similar to hardware hand/autostations. The ACS has many variables that can be controlled by an operator fromhand/auto control stations or from graphic displays that mimic such a control station.In each case, the operator may use push buttons to manually input information aboutthe controller’s input,8 output, or setpoint. The operator first selects one of these three

8 In ACS terminology, the controller input is called meas for measurement.



4-62

variables, then each push of the up or down button changes the appropriate variable by1%. It should be obvious that the logic behind such changes is essentially identical ineach case and need only be specified once. The GENERIC_INTERFACE chart shown inFigure 4-19 is used as an example of a generic chart, the logic of which applies to eachcontrolled variable and for each of the three hardware implementations of thecontroller logic.

When the OPERATOR_CTL_CMD in Figure 4-19 is received, the generic interface goesinto a state awaiting the operator’s selection of one of the three variables. However, ifthe operator delays more than five seconds before deciding, the interface times out andreturns to the AWAITING_OPERATOR_CMDS state. If the operator pushes a button toselect the output; then the CHOOSE_OUTPUT event is generated, causing a transitionto the ADJUST_OUTPUT state, and in particular to its default sub-state indicated bythe arrow symbol •→.

When an event of UP or DOWN is received, a transition is made temporarily to a statethat increments the variable of interest. For example, the RAISE_O state increments theoutput by 1% as follows:

RAISE_O

Defined in Chart:GENERIC_INTERFACE

Type: BASIC

Reactions:entering/OUTPUT:=OUTPUT+0.0l

Following this increment, a transition is made to the idle state, where another pushbutton is awaited. If no additional push is sensed within one second, that state timesout, returning back to the AWAITING_OPERATOR_CMDS state.

The generic chart is characterized by formal parameters that have to be associated withspecific parameters when the chart is used in a specific instance. The statechart inFigure 4-20 shows how the generic chart can be invoked concurrently by three of thevariables of interest. When the generic chart is used in a specific instance for the reactordemand controller, the formal parameters must be associated with (or bound to) actualparameters.



4-63

SELECTING

AWAITING_OPERATOR_CMDS

RAISE_S>

IDLE_S

LOWER_S>

ADJUST_SETPOINT

on(RAISE_S)UP

DOWN on (LOWER_S)

CHOOSE_SETPOINT

RAISE_M>

IDLE_M

LOWER_M>

ADJUST_MEASUP

on(RAISE_M)

on(LOWER_M)DOWN

RAISE_O>

IDLE_O

LOWER_O>

ADJUST_OUTPUT

UP

on(RAISE_O)

on(LOWER_O)

DOWN

CLOSING_MANUAL_CONTROL

tm(on(IDLE_S),1)tm(on(IDLE_O),1)

tm(on(IDLE_M),1)

CHOOSE_OUTPUT

tm(on (SELECTING), 5)

MANUALLY_CONTROLLING

GENERIC_INTERFACE

OPERATOR_CTL_CMD

CHOOSE_MEAS

Figure 4-19Generic Statechart for Operator Interface



4-64

USE_GENERIC_INTERFACE

IDLE

IDLE

IDLE

RX_DMD<GENERIC_INTERFACE

FW_LOOP_1_DMD<GENERIC_INTERFACE

OTHER_PARAMETERS_GENERIC_INTER

(not LOCAL_CONTROL(1)) /(s) (RX_IN_MANUAL) , FMC

not LOCAL_CONTROL(2) /(s) (FW_A_UNAV) , FMC

(LOCAL_CONTROL (1)) / (r) (RX_IN_MANUA L)

(LOCAL_CONTROL (2)) / (r) (FW_A_UNAV)

(LOCAL_CONTROL(3))

/(s) (LOCAL_CONTROL (3)), FMC

Figure 4-20Statechart that Invokes Interface Logic for Three Controllers

4.C.3. When Are Statecharts Appropriate?

Comparing the above discussions of statecharts with the preceding discussion oftabular finite state machines, it should be apparent that:

• Statecharts are somewhat more complex and, therefore, more difficult tocomprehend compared to the simpler tabular formats introduced by Parnas.

• Effective use of the Statechart method requires purchase or lease of an expensivesoftware tool such as STATEMATE.

• Statecharts subsume the principles of finite state machines and add a variety ofexpressive capabilities for representing more general and complex requirements.For example, time dependence and concurrency can be represented explicitly.



4-65

Consequently, it is recommended that a tabular approach (e.g., Parnas) should be usedfor smaller, simpler problems where the static tabular description of transformations isadequate. This would include most safety-critical applications, which tend to involvevery simple static transformations, minimal (if any) user interface, and once-throughcomputations without concurrency. For more complex applications involving manystates, concurrency, feedback from the environment (or links to simulation), continuoustransformations, and other complications, investment in the Statechart approach forspecifying requirements is justified. This would include, for example, availability-critical control systems regulating the behavior of the plant and its systems duringnormal operation and transients. Note that the hierarchical structure of statechartsnaturally allows for successive refinement of specifications (i.e, a high-levelrequirement can be introduced in a statechart, then subsequently resolved into moredetailed underlying statecharts).


5-1

5 V&V TECHNIQUES — TESTING

The early part of the software development life cycle includes definition of the concept,specification of requirements, description of the design, and implementation of thedesign in terms of program code. These are mostly human activities, so problems maybe expected. There are many opportunities for misunderstanding, misinterpretation,miscommunication, imperfect translation of the design into code, or other errors tooccur. In recognition of these possibilities, software development must be accompaniedby verification and validation (V&V) activities.

V&V includes review of the products of each phase of the development process, but asignificant element of V&V involves testing the software. That is, the softwarerequirements specification (SRS), the software design description (SDD), andindividual components of code must be reviewed to identify faults as early as possible,but ultimately the software must be tested under controlled conditions until there is anappropriate level of confidence that it performs as required.

It is not unusual for testing to demand 40 or 50 percent of the total effort during atypical software development project. Safety-related software might require even moreresources devoted to testing. Therefore, testing must be carefully planned andscheduled. Fortunately, it is possible to begin defining these activities very early in theproject. The SRS is a primary source of information for definition of test cases; that is,the software must be tested until it can be demonstrated to satisfy all of itsrequirements. The SDD identifies how the code must be structured, so it tells the testerwhat must be done to check that the code faithfully reflects its intended design. Testcases should be prepared well in advance, so that testing may be performed as codingis completed. It is desirable to identify faults as early as possible, so that correctionsmay be made most efficiently.

Software development is a constructive process where ideas are transformed into codedinstructions that make a machine perform some task. Testing, on the other hand,involves a destructive attempt to make the software fail. Software developers must betrained to make a “Jekyll and Hyde” personality transformation for testingeffectiveness. They must overcome their desire to demonstrate how well they havedesigned or coded the software, and plunge into a ruthless pursuit of errors. For someprojects, this may be considered too much to ask, so testing should generally beperformed by individuals who are not involved in the development process.


V&V Techniques — Testing

5-2

It is important to remember that the only objective of testing is to identify the presence offaults. If testing fails to locate any faults, it is not possible to conclude that the softwareis free of faults; instead, it is only possible to say that the testing was incapable ofidentifying any faults that may have been present. (It is possible to make a predictionregarding the number of potential faults remaining, however, with sufficient test data.)This is why testing must be combined with reviews, inspections, and analysis toachieve the high level of confidence needed for digital I&C systems.

5.1. Terminology

IEEE 610.12 [1] provides definitions for about 1300 terms applicable to softwareengineering. This is particularly useful because many of the terms are adapted fromcommonly used English words that are likely to have a different meaning outside thefield of software engineering. Other engineering disciplines frequently haveterminology based on unusual words that are specific to the profession. For example,words like adiabatic, enthalpy, and entropy are not often used in conversations with thefamily during dinner. On the other hand, everyday words like branch, bug, code, fault,and even coupling might have a generally understood meaning to your children but amore particular definition to the software engineer. In addition, common engineeringterms like static and dynamic take on special meaning to the software engineer involvedin testing.

In this section, we will list some of the terminology of software engineering applicableto testing. It is desirable for each of us to be familiar with these terms to facilitateprecise communication.

5.1.1. What Is Testing?

Testing has been described as “the fiendish and relentless process of exercising asystem with the intent of locating faults” [2]. From IEEE 610.12:

testing. (1) The process of operating a system or component under specifiedconditions, observing and recording the results, and making an evaluation ofsome aspect of the system or component. (2) The process of analyzing a softwareitem to detect the differences between existing and required conditions (that is,bugs) and to evaluate the features of the software items.

The objective of testing is to find faults. Therefore, testing that finds faults is successful.If testing fails to find faults, then the testing may be at fault. A good tester is “creativelydestructive” [2] and dedicated to making the software fail.

We will focus on software testing. In particular, we will not consider hardwarequalification testing, such as environmental, seismic, electromagnetic interference



5-3

(EMI), or radio-frequency interference (RFI) testing. However, to properly test thesoftware it usually must be integrated with hardware to form a system.

5.1.2. What Is Review?

A review is a static analysis that can be performed whether or not any program code hasbeen prepared. Testing is usually accomplished by dynamic analysis of code. From IEEE610.12:

static. Pertaining to an event or process that occurs without computer execution.

dynamic. Pertaining to an event or process that occurs during computerexecution.

code. In software engineering, computer instructions and data definitionsexpressed in a program language or in a form output by an assembler, compiler,or other translator.

Note that dynamic refers to active execution of code, not to passage of time. As usual,static is the opposite of dynamic.

Two common review techniques are walk-through and inspection (see Section 3). Eachmay be applied to evaluate either documentation or code; however, an inspectionusually refers to code review. From IEEE 610.12:

walk-through. A static analysis technique in which a designer or programmerleads members of the development team and other interested parties through asegment of documentation or code, and the participants ask questions and makecomments about possible errors, violation of development standards, and otherproblems.

inspection. A static analysis technique that relies on visual examination ofdevelopment products to detect errors, violations of development standards, andother problems. Types include code inspection and design inspection.

It is desirable to perform walk-through reviews to confirm the software requirementsspecification (SRS) and the software design description (SDD) before coding is initiated.Walk-through and inspection reviews should also be performed to confirm interimcode before testing begins. A review normally demands fewer resources than testingdoes. It is important to recognize again that faults found early in the life cycle by staticreview are much easier to correct than those discovered during dynamic testing.



5-4

5.1.3. What Is a Fault?

In the definition of error, IEEE 610.12 includes these useful interpretations:

mistake. A human action that produces an incorrect result.

fault. An incorrect step, process, or data definition; the manifestation of amistake.

failure. An incorrect result; the manifestation of a fault.

error. An amount by which the manifestation of a fault (the failure) is incorrect.

Therefore, an error is the measurement of a failure that results from a fault caused by amistake. In common usage, the terms error and bug are often used interchangeably withfault. Nevertheless, we will use the more precise word fault to describe the item that weintended to identify and correct by means of testing. The word bug may be consideredslang for fault.

IEEE 610.12 uses the term error (incorrectly, according to the previous discussion) todefine two particular faults:

semantic error. An error resulting from a misunderstanding of the relationshipof symbols or groups of symbols to their meanings in a given language.Improper application of language rules. Usually a runtime error; for example,array size exceeded.

syntactic error. A violation of the structural or grammatical rules defined for alanguage. Usually a translation or compilation error; for example, missingparenthesis.

We would prefer to use the more precise word fault instead of error in these definitions.To locate a semantic fault usually demands testing. A syntactic fault is commonlydetected by review or, perhaps, by use of code development tools (e.g., a languagecompiler).

Faults congregate at decision points, at boundary values, and within complex sections ofcode. Decision points relate to path and branch. From IEEE 610.12:

path. A sequence of instructions that may be performed in the execution of acomputer program.

branch. (1) A computer program construct in which one of two or morealternative sets of program statements is selected for execution. (2) A point in a



5-5

computer program at which one of two or more alternative sets of programstatements is selected for execution. (3) Any of the alternative sets of programstatements in (1). (4) To perform the selection in (1).

Perhaps the third definition of a branch is most applicable to testing. It refers toseparate paths (branch paths) resulting from a decision (branch statement, the firstdefinition) at a certain location in the program (branch point, the second definition) asimplemented by the programmer (execute a branch, the fourth definition).

Boundaries are characterized by data domain (e.g., alphabetic or integer) and range(e.g., all upper case letters or positive numbers from 0 to 100 inclusive). From IEEE610.12:

boundary value. A data value that corresponds to a minimum or maximuminput, internal, or output value specified for a system or component.

Complex code can be characterized by certain metrics (e.g., McCabe’s cyclomaticcomplexity metric). From IEEE 610.12:

complexity. (1) The degree to which a system or component has a design orimplementation that is difficult to understand and verify. (2) Pertaining to any ofa set of structure-based metrics that measure the attribute in (1).

Cyclomatic complexity is a measure of the number of independent paths in a programor module. It may be determined by adding one to the number of binary (two-way)decisions in a program. An N-way decision (e.g., a “case” statement) is equivalent toN – 1 binary decisions. It is generally agreed that a module with cyclomatic complexitygreater than 10 is too complex and should be partitioned if possible. (See Section 5.2.2below for further discussion of cyclomatic complexity.)

During testing, it is useful to classify faults as they are located. This permits collectionof certain records. Beizer [3] has prepared a Taxonomy of Bugs, which is summarized inAppendix 5.A. Another classification scheme may be more applicable for any givenproject. The point is that some scheme should be devised for classifying faults so that itis possible to accumulate and maintain appropriate records. For additional informationon classification, see IEEE 1044 [4] and 1044.1 [5].

5.1.4. What Is Debugging?

It has been noted that bug is commonly used slang for fault. Therefore, debuggingmeans correcting faults. From IEEE 610.12:

debug. To detect, locate, and correct faults in a computer program.



5-6

Debugging is more art than science. Certain steps are typical:

1. Stabilize (make repeatable).

2. Isolate (identify).

3. Correct.

4. Test.

These steps usually require iteration because (1) some faults may hide other faults, and(2) correcting one fault may introduce a new fault. The software developmentenvironment usually includes tools to aid debugging.

Faults must be corrected as they are identified during testing. Debugging is the leastpredictable part of the testing process. This uncertainty makes it difficult to reliablyschedule testing.

Some people incorrectly think of the testing phase of the software development lifecycle as the debugging phase. Testing is not debugging. Debugging is subservient totesting. The purpose of testing is to locate faults. The purpose of debugging is toeliminate the faults. Together, these efforts improve the quality of the software.

Obviously, it is best to eliminate faults as early as possible (i.e., by careful specification,design, and implementation) so that the need for debugging is minimized. Asdescribed by Dyer [6], the cleanroom approach is a software development methodologyintended to virtually eliminate debugging. However, we all may have witnessed theother approach, which is to throw together software with inadequate review and hopethat sufficient testing and debugging will eventually make it work. We also may haveobserved the situation where frantic debugging and excessive patching has resulted incomplex code with new faults that are harder to locate.

5.1.5. What Are Component, Module, and Unit?

As noted in IEEE 610.12:

The terms module, component, and unit are often used interchangeably or definedto be sub-elements of one another in different ways depending on the context.The relationship of these terms is not yet standardized.

The referenced standard glossary also provides this useful definition:

test unit. A set of one or more computer program modules together withassociated control data (for example, tables), usage procedures, and operating



5-7

procedures that satisfy the following conditions: (a) All modules are from asingle computer program; (b) At least one of the new or changed modules in theset has not completed the unit test; (c) The set of modules together with itsassociated data and procedures are the sole object of a testing process.

For our purposes, the term unit will relate to early phases of testing program elements,while component and module will refer to the fundamental elements of a program. Wewill generally prefer the term module when discussing software and component whendiscussing hardware.

5.1.6. What Is a Traceability Matrix?

A traceability matrix enables one to tie modules to requirements and tests to modulesin order to identify the relationship of each to the other. From IEEE 610.12:

traceability matrix. A matrix that records the relationship between two or moreproducts of the development process, for example, a matrix that records therelationship between the requirements and the design of a given softwarecomponent.

A traceability matrix facilitates confirmation of complete coverage. That is, eachrequirement should be addressed by at least one module and one test. Similarly, thereshould be at least one test to verify each module and each requirement. Finally, if amodule or a test is not related to a requirement, perhaps it should be eliminated.

The following requirements should be considered:

Functional Performance TimingData Validation External Interface SecurityAvailability Reliability MaintainabilityDiagnostic Failure Handling Failure RecoveryDesign Constraints Standards Compliance

Requirements should be presented in such a way that they are testable.

The traceability matrix may be presented in a table as illustrated in Table 5-1, whereSRS and SDD refer to the software requirements specification and the software designdescription, respectively.



5-8

Table 5-1Sample Traceability Matrix

Requirements SRS Reference SDD Reference Modules Test Cases

Requirement 1 Section A Paragraph X

Paragraph Y

Module X1

Module Y1Module Y2

Tests X1a-c

Tests Y1a-dTests Y2a-b

Requirement 2 Section B Paragraph Z Module Y1

Module Z1Module Z2

Tests Y1c-g

Tests Z1a-bTests Z2a-d

etc. … … … …

The traceability matrix also makes it possible to visualize how well the design has beenpartitioned. Ideally, there should be a one-to-one relationship between requirementsand modules. This ideal is frequently impractical, as illustrated in the example above.

A more specific example is included in the Westinghouse Eagle 21 case study describedin Section 8. Westinghouse called their traceability matrix the System Design Matrix. Itcontained the following columns:

1. Functional Requirement (reference by ID number)

2. Subject (brief description of key variables or items)

3. Requirement For (brief description of requirement)

4. System Design Requirement (reference to requirement specification document)

5. System Design Specification (reference to design description document)

6. Functional Decomposition (reference to functional decomposition document)

7. FAT/Validate Test (reference to test design description document)

8. Dominant Software Units (reference to software configuration/test units)

9. Procedure Name (reference to software modules)

See Figure 1-2 for another example.

For applications having both safety-related and non-safety-related requirements, it isuseful to distinguish between these in the traceability matrix. This makes it easier to



5-9

identify which modules and tests address safety issues as well as the relationshipbetween safety and non-safety elements.

It is convenient to maintain the traceability matrix in a computer database orspreadsheet, permitting various sorts and cross-correlation. Additional columns androws should be added to the matrix as necessary to more clearly identify theconnections between various parts of the application. As simple as it may appear, thetraceability matrix can be an extremely useful tool.

5.1.7. What Is the Relationship between Design and Testing?

Software design is frequently performed by partitioning the problem into modules, ormodular decomposition. Good modular decomposition (hence, good design) may becharacterized by maximum cohesion and minimum coupling. Good design promotestestability:

Maximum Cohesion + Minimum Coupling = Maximum Testability

From IEEE 610.12:

modular decomposition. The process of breaking a system into components tofacilitate design and development.

cohesion. The manner and degree to which the tasks performed by a singlesoftware module are related to one another.

coupling. The manner and degree of interdependence between softwaremodules.

testability. (1) The degree to which a system or component facilitates theestablishment of test criteria and the performance of tests to determine whetherthose criteria have been met. (2) The degree to which a requirement is stated interms that permit establishment of test criteria and performance of tests todetermine whether those criteria have been met.

Notice that independent modules mean low coupling, while interdependence amongmodules means high coupling.

From the requirement-centric view, each module should focus on a limited set ofrequirements (maximum cohesion) and each requirement should be satisfied by alimited set of modules (minimum coupling). The traceability matrix can facilitateidentification of how well the design satisfies this tenet.



5-10

There are only three basic programming constructs: sequence, selection, and iteration.The arrangement of these basic constructs is determined by design of each module.Various techniques, such as structure chart, data flow diagram, entity/relationshipdiagram, decision table, state diagram, etc., are available to support software design.These design techniques also lead directly to early definition of test cases. Forexamples, see Section 5.3.1 below.

5.1.8. What Are Types of Testing?

For completeness, various types of testing are listed in Appendix 5.B. Here we areprimarily concerned with unit, interface, integration, and system testing. From IEEE610.12:

unit testing. Testing of individual hardware or software units or groups ofrelated units. [See test unit defined previously. Frequently refers to a subset ofall software modules.]

interface testing. Testing conducted to evaluate whether systems or componentspass data and control correctly to one another. [Frequently refers to theinteraction between software modules.]

integration testing. Testing in which software components, hardwarecomponents, or both are combined and tested to evaluate the interactionbetween them. [Frequently refers to a combination of all parts of the system.]

system testing. Testing conducted on a complete, integrated system to evaluatethe system’s compliance with its specified requirements. [Frequently refers tofinal proof and acceptance.]

Structural testing is a technique used primarily for unit and interface testing, whilefunctional testing is applicable to integration and system testing. These topics arediscussed in Section 5.2.

Thorough testing may be performed by incremental unit testing. This is done by addingone untested module to a set of previously tested modules, testing the new set ofmodules, and repeating until all modules have been tested. Each incremental set ofmodules is a test unit (defined previously). As modules are added, subsequent unittesting includes interface testing; i.e., the interface between each new module andpreviously tested modules is scrutinized during this process.

Incremental unit testing may be performed top-down or bottom-up. In top-down testing,stubs are created to represent lower level modules. Bottom-up testing requiressubstituting drivers for higher level modules. Sandwich testing refers to a combination of



5-11

both top-down and bottom-up approaches. Critical-first is a method where the mostcritical elements of a system are implemented and tested first, using stubs and driversas necessary. All of these incremental testing techniques are facilitated by modulardesign with loose coupling. From IEEE 610.12:

stub. A skeletal or special-purpose implementation of a software module, usedto develop or test a module that calls or is otherwise dependent on it [on thestub].

driver. A software module that invokes and, perhaps, controls and monitorsexecution of one or more other software modules.

In this context, a driver is different from a device driver, which is a program that controlsa peripheral device.

Exhaustive testing means testing with every possible condition; except for simplediscrete systems, this is not feasible. A more manageable type is equivalence class testing,which means testing with a representative sample of all possible conditions. Therefore,testing is frequently performed with a representative sample of inputs (e.g., boundarytesting). The strategy to select this sample will be mentioned below in functionaltesting, where it is mainly used; however, it may also be applied during structuraltesting.

It is convenient to package each test into a test case. From IEEE 610.12:

test case. (1) A set of test inputs, execution conditions, and expected resultsdeveloped for a particular objective, such as to exercise a particular programpath or to verify compliance with a specific requirement. (2) Documentationspecifying inputs, predicted results, and a set of execution conditions for a testitem.

In particular, packaged test cases facilitate documentation, tracking, and re-testing.

5.2. Testing Techniques

The major techniques of testing are shown in Figure 5-1.



5-12

FUNCTIONALTESTING

STRUCTURALTESTING

OTHER TESTINGTECHNIQUES

TESTING

REQUIREMENTSBASED TESTING

EQUIVALENCECLASS TESTING

SYSTEMTESTING

COMPARISONTESTING

PATHTESTING

BRANCHTESTING

CONDITIONTESTING

DATAFLOW

LOOPTESTING

REGRESSIONTESTING

RANDOMTESTING

ABNORMALCONDITIONSAND EVENTS

OPEN-LOOPTESTING

CLOSED-LOOPTESTING

Figure 5-1Testing Techniques

Functional testing is often called black-box testing because it is concerned only whether aset of inputs produces the correct outputs, as prescribed by the requirements, and notwith how the software operates internally. Structural testing takes a complementaryapproach by focusing on the correctness of implementation details; therefore, it is oftencalled white-box or glass-box testing. Finally, there are other testing techniques that canbe used along with functional and structural testing.

Validation testing is always functional in nature because it relates directly torequirements. The step-by-step emphasis of verification, in contrast, may involve acombination of structural testing and functional testing at the module and subsystemlevels.

Because functional testing is based on requirements, it can (and should) be plannedearly in the development life cycle. Structural testing cannot be planned until design ofthe code is evident. On the other hand, structural testing usually begins early in theimplementation phase of the life cycle before functional testing. Final validation testingshould not be completed until all structural tests have been performed satisfactorily.

Software engineering terminology is sometimes confusing. The term static testing issynonymous with review activities that do not involve operation of the software. In



5-13

contrast, dynamic testing refers to actually executing the code in order to identify faults.(This terminology is unfortunate because is not consistent with the usual interpretationof these terms as time-independent and time-dependent, respectively.) When wediscuss testing here, we usually mean dynamic testing. The remainder of this sectiondescribes the techniques listed in Figure 5-1.


5.2.1. Functional Testing

Functional testing is usually a form of integration testing performed by non-programmers after successful structural testing in order to demonstrate compliancewith specified requirements. From IEEE 610.12:

functional testing. Testing that ignores the internal mechanism of a system orcomponent and focuses solely on the outputs generated in response to selectedinputs and execution conditions.

If a fault is found during functional testing and subsequently corrected, then it isusually prudent to repeat structural testing before resuming functional testing.

Functional test cases attempt to define input data representative of all computationalcombinations. They can be designed very early because they do not depend on thedetails of implementation. For tests at the system level, validation test cases can bedeveloped directly from the software requirements specification (SRS). For functionaltests of lower level components or subsystems, verification test cases can be defined oncompletion of the system decomposition and the software design description (SDD).

Functional testing identifies the following:

• Incorrect or missing units

• Problems in the interface between units

• Problems in the data used by units

• Performance problems

• Initialization problems

• Termination problems



5-14

Functional testing may be performed at the unit level, but it usually requires anintegrated system. Small units can be tested by supplying stubs and drivers thatsimulate the balance of the system.

Ultimately, the objective of functional testing is to evaluate whether the system satisfiesits specified requirements. This final system functional test, which may be called anacceptance test, represents the validation part of V&V. That is, while all of structuraltesting and much of functional testing is concerned with verification that the system hasbeen implemented correctly, the acceptance test validates that the correct system hasbeen implemented.

Techniques to facilitate functional testing are described below.

5.2.1.1. Requirements Based Testing

The fundamental purpose of functional testing is to verify that the system satisfies itsrequirements. Sometimes it is difficult to interpret the requirements, depending on howwell they were specified. Complex requirements may have to be decomposed intosmaller elements. Once elementary requirements are identified, test cases must becarefully developed to properly test variations of each element without influence fromother elements that may invalidate the test.

For example, the functional requirement IF X AND Y AND Z, THEN A might havebeen implemented incorrectly as if Z were always TRUE, resulting in the functionalequivalent of IF X AND Y, THEN A. It takes four separate tests to confirm that all threeconditions X, Y, and Z have been implemented correctly. This point is illustrated belowwith a cause-effect graph (Figure 5-3) and a decision table (Table 5-2).

A more pertinent example is included in the Westinghouse Eagle 21 case studydescribed in Section 8. Decomposition of functional requirements led to the followingelementary requirement, which was presented along with a recommendation fortesting.

Elementary Functional Requirement Recommendation for Testing

The T hot Redundant Sensor Algorithm logicshall examine the quality codes of the incomingT hot estimate signals, reject those withqualities of BAD or DISABLED, and generategroup qualities and group values according tothe logic provided in....

This series of tests [cases 1–14] will verify that forboth normal and abnormal input conditions theT hot RSA assigns the proper quality codes to theindividual T hot estimate inputs, and assigns theproper quality codes and values to the T hot groupvalues. When performing this series of tests, theindividual T hot estimate values and quality codesand the T hot group values and quality codesshould be observed and recorded to verify properoperation.



5-15

The test design description goes on to specify fourteen test cases that represent allpossible combinations of input signal quality and of signals that are in mutualagreement or disagreement. For illustration, the first few of those are included below.

Case Required LogicInput

QualityGroupQuality Group Value Comments

1 |Thj – Thavg| ≤ DELTAHj = 1,2,3

3G GOOD (Th1 + Th2 + Th3)/3

2 |Thg1 – Thg2| ≤ DELTAH 2G,1D GOOD Avg of 2 GOOD

3 |Thp – Thavg| > DELTAH|Thg1 – Thg2| ≤ DELTAH

2G,1P POOR Avg of 2 GOOD Verify that Thp isfurthest from Thavg,entered as GOOD,reassigned to POOR

Such a table provides a straightforward guide for identifying the test cases in terms oftheir inputs, as well as a basis for independent determination of the expected outputvalue. Other approaches for requirements-based testing can use various representationsadapted from structured methods (e.g., data flow diagrams) or formal methods (e.g.,decision tables). (See Section 4 for further discussion of these points.)

To better understand the requirements specification in order to define test cases, it maybe useful to prepare a cause-effect graph and/or a decision table. This effort can beperformed before implementation of the code has even begun. In fact, illustrating therequirements in this manner facilitates verification of the requirements specificationand contributes to presentation of the design description. Note that preparation fortesting may proceed independent of implementation. This is discussed in Section 5.3.1with respect to test design.

A subset of symbols used in a cause-effect graph is illustrated in Figure 5-2. For eachsymbol, causes are presented on the left with effects on the right. The cause-effect graphmakes it easier to visualize logic and identify dependencies.



5-16

C1

C1

E1

C2

E1v Or

Not

C1

C2

E1^ And

A

B

E

Exclusive

A

B

R

Requi res

Figure 5-2Cause-Effect Graph Symbols

As an example, consider the following requirements for a system:

1. If the pump is running or flow is greater than 10 or pressure is greater than 20, thenopen the valve.

2. If the fan is stopped and temperature is less than 100, then close the damper.

3. If the valve is open or the damper is open, then start the process.

4. If the damper is closed, then turn the light on.

A cause-effect graph for this system is illustrated in Figure 5-3. It is instructive to readfrom right-to-left to determine what cause(s) are necessary to realize a specific effect.



5-17

P u m pRun

Flow> 10

Pres> 20

OpenVa lve

v

FanStop

T e m p< 100

CloseDamper^ Light

O n

StartProcessv

Figure 5-3Cause-Effect Graph for Example

To properly test each set of N causes (predicates) combined by OR, the following N+1test cases are necessary, where Ti or Fi represent TRUE or FALSE for cause i:

1. IF T1 OR F2 ... OR FN, THEN EFFECT

2. IF F1 OR T2 ... OR FN, THEN EFFECT

…

N. IF F1 OR F2 ... OR TN, THEN EFFECT

N+l. IF F1 OR F2 ... OR FN, THEN NOT EFFECT



5-18

The following N+1 test cases are necessary to properly test each set of N causescombined by AND:

1. IF F1 AND T2 ... AND TN, THEN NOT EFFECT

2. IF T1 AND F2 ... AND TN, THEN NOT EFFECT

…

N. IF T1 AND T2 ... AND FN, THEN NOT EFFECT

N+l. IF T1 AND T2 ... AND TN, THEN EFFECT

This principle leads to a decision table listing all of the variations that must be tested.Each column represents a test case. A blank in the table means “don’t care”. Table 5-2represents the decision table for the example in Figure 5-3.

The required number of tests for each effect would usually be equal to the number ofcauses plus one; e.g., 2 for Light On, 3 for Start Process, 3 for Close Damper, and 4 forOpen Valve. However, cascading frequently permits one test case for more than oneeffect. This must be done carefully to insure that one cause does not mask another andresult in an unexpected effect. In other words, the number of independent test cases foreach effect must be equal to at least one plus the number of causes for that effect.

Table 5-2Decision Table for Example

Cause-Effect Test Cases

Pump Run T F F F F

Flow > 10 F T F F F

Pres > 20 F F T F F

Fan Stop T T T F

Temp < 100 T T F T

Open Valve T T T F F

Close Damper T T F F

Start Process T T T F T T

Light On T T F F

Tools are available that automatically determine the minimum set of requirementsbased test cases, as well as non-testable and infeasible requirements (cf. SoftTest fromBender & Associates and T from Interactive Development Environments).



5-19

It should be noted that a given set of requirements based test cases will be applicableregardless of the implementation. This might be useful if redundant software isnecessary to insure high reliability, for example, or if the system is replaced by afunctional equivalent sometime in the future.

5.2.1.2. Equivalence Class Testing

Since exhaustive testing of all possible conditions from the input domain is usuallyimpossible (except in the case of simple discrete systems), a manageable set of testconditions may be derived by equivalence partitioning coupled with boundary valueanalysis. Therefore, testing is frequently performed with a representative sample ofinput (an equivalence class) as follows:

• In-bound — a positive test

• On-bound — a positive test

• On-bound ± ε — one positive and one negative test

• Out-of-bound — a negative test

The term positive refers to test data in the normal range, while negative refers to test datathat is not in the normal range. The symbol ±ε indicates slightly more and slightly less.This concept is represented in Figure 5-4.

Condit ions

BoundaryBoundaryIn-BoundOut-of-Bound

Test Data

Figure 5-4Equivalence Class Test Data

This approach accounts for the observation that more faults seem to occur near the“edges” of the input domain than in the “center”.

Besides equivalence classes for the input domain, test cases should be derived thatcreate equivalence classes for the output domain, also. For example, if the output valueis supposed to be an integer in the range from 1 to 100 inclusive, then testing shouldattempt to force the output values of 0, 1, 2, 50, 99, 100, 101, and perhaps 150.



5-20

Internal program data structures should also be tested at their boundaries. Forexample, if an array variable is dimensioned with 100 entries, then test cases shouldattempt to access the array at its extremities as well as its middle. It is instructive toinvestigate what happens if an attempt is made to access (read or write) the internaldata structure outside of its normal address limit.

Equivalence class testing is particularly suitable for functional testing, but it may alsobe applied during structural testing.

5.2.1.3. System Testing

The time-dependent and asynchronous nature of real-time systems adds significantly tothe testing challenge. In some situations, a given test case will yield correct resultswhen the system is in one time-dependent state but will fail when the system is inanother state. For example, software designed to close a valve could work fine if testedwhen a pump is stopped, but an implementation fault might be observed if the sametest is executed while the pump is running. This might occur, for example, if the datastructure representing the valve’s state was incorrectly shared with the pump’s state.

Also, there is frequently an intimate relationship between hardware and software inreal-time systems. The software must accommodate hardware faults or degradedperformance. These effects may be very difficult to simulate in the test environment.

Real-time systems frequently involve multiple parallel tasks. Comprehensive testdesign methods for such systems have not yet been devised, so such complexarchitectures should be avoided for the most critical safety applications; however, thefollowing recommendations by Pressman [7] should be considered:

Task Testing — Thoroughly test each task independently to uncover logic andfunctional faults. This will not identify timing or behavioral faults, however.

Behavioral Testing — Using models created with certain CASE (computer aidedsoftware engineering) tools, it is possible to simulate the behavior of real-time systemsand analyze their response to external events (e.g., interrupts, control signals, orchanges in data). Such analysis can be used to identify test cases. Events may becategorized in a manner similar to equivalence partitioning. Each event categoryshould be tested individually. The behavior of the actual system and that of the modelcan then be compared for each category. Once testing has been completed for eachevent category, additional testing should be performed with events presented inrandom order and with random frequency.

Intertask Testing — After correcting faults discovered during task and behavioraltesting, time related faults should be identified. Tasks that communicate with each



5-21

other asynchronously should be tested with varying data rates and processing load touncover intertask synchronization faults. Other intertask communication errors mayresult from invalid data structures such as message queues or data stores used totransfer information.

System Testing — Finally, hardware and software must be integrated for a full rangeof system tests to uncover any faults associated with the hardware/software interface.System testing might be either live or based upon simulation.

Live testing refers to tests of the fully integrated system installed in its actual workingenvironment. While such tests are the most complete and realistic, there are often costor safety constraints that prevent all the system’s functions from being tested in thisway. For example, it would not be practical to live test a feedwater control system forits ability to recover from the full range of operational transients. In addition, someexperts believe that live testing reduces the probability of finding faults because criticaland/or complex functions are less likely to be exercised.

Using simulation to provide test data enables more complete system functional testingthan would be feasible with live tests. Numerical simulation of physical processes maybe used to supply time dependent input to control and monitoring systems, such as

• Monitoring or diagnostic systems designed to display or interpret trends

• Actuation systems, including safety or protection systems

• Control systems accounting for feedback effects

• Operator interface systems

For monitoring, diagnostic, or actuation systems, it is generally appropriate to performthe simulations in advance, record the simulated results, and subsequently use thoserecordings to drive the system through a series of tests. This open-loop testing method isadequate whenever feedback is not a significant effect. It is relatively easy to perform,assuming the simulation environment is available. Simulated test cases should berepresentative of the real-time performance requirements of the system being tested.

Normally control systems are closely coupled by feedback terms to the physicalresponse of the equipment being controlled. Also, the acceptance criteria defined forthat equipment often require either a rapid response or a stable response, or both,demanding detailed analysis. Therefore, meaningful validation of control systemperformance must include closed-loop tests, where the control system interacts directlywith the simulation environment.



5-22

Closed-loop simulation testing is also important for operator interface evaluationbecause the test must provide information regarding the clarity and timeliness ofoperator displays as well as the system’s support for real-time human interaction.

5.2.1.4. Comparison Testing

In situations where reliability is critical, it may be decided to develop redundantsoftware. In this case, separate development teams would independently design andimplement software using the same requirements specification. It is also possible tohave two independent implementations when an obsolete system is being replaced by anew system with identical functionality; i.e., comparisons could be made between theold system and the new system. This can be a benefit during testing because bothversions can be executed with the same test cases and the results compared forconsistency. The actual comparison of results can be automated using software tools sothat determining the results of each test is simplified. If independent software yieldsthe same results, then either both were implemented correctly or the same mistake wasmade in each implementation. The latter situation will be identified only by detailedanalysis of test results, as would be required without comparison testing. (If functionaldiversity is required, see Section 7 for further discussion.)

Comparison testing or back-to-back testing can be particularly useful for testing real-timesystems that present a difficult challenge for ordinary test methods (see Section 5.2.1.3).Because of this and other considerations, it has been suggested that independentversions of software should be developed for critical requirements testing even if onlyone version will be used in the completed system. The benefits associated with testingmust be weighed against the costs associated with redundant implementation.

5.2.2. Structural Testing

Structural testing is usually an approach to unit and/or interface testing performedduring implementation by programmers (or by others with equivalent skill) todemonstrate that the structure of the code is sound. From IEEE 610.12:

structural testing. Testing that takes into account the internal mechanism of asystem or component. Types include branch testing, path testing, and statementtesting.

As discussed in Section 6, the safety category of the system, together withconsiderations such as its complexity, determines the importance of structural testingwith respect to V&V. Structural testing is complementary to functional testing byfocusing on implementation details rather than the broader issue of requirementsvalidation. Thoroughness demands that each unit be tested to demonstrate that allstatements, all branches, and selected paths (e.g., using equivalence class test data) have



5-23

been executed at least once without a fault. (Branch and path are defined in Section5.1.3.) From IEEE 610.12:

statement testing. Testing designed to execute each statement of a computerprogram.

branch testing. Testing designed to execute each outcome of each decision pointin a computer program.

path testing. Testing designed to execute all or selected paths through acomputer program.

In general, full path testing can be very difficult and is seldom accomplished, whilestatement testing requires the least effort.

A loop construct has a branch either at the beginning or at the end of the loop. Eachrepetition of the loop must be counted as a new path when considering all possiblepaths. Therefore, loop testing is a special form of branch and path testing.

At the unit level, structural testing should be performed until at least

• All instructions have been executed once (all statements)

• All decisions have been taken in each direction (all branches)

Most experts agree that this represents the minimum amount of structural testingnecessary. On the other hand, IEEE 1008-1987 [8] only requires full statement coverage.Statement and branch testing will insure that normal, but not all, paths are tested.Additional paths should be tested as deviations from the normal, starting with thesimplest and adding more paths as practical.

Path testing, however, can involve a large number of test cases (except for the simplestsoftware units). Section 5.2.2.1 contains examples of decision constructs illustrating howthe total number of paths can grow geometrically. A control flow graph can help toidentify the basis set, which is the set of independent paths in a module. This representsthe set of test cases that must be designed to insure full branch testing. That is, the basisset requires taking all decisions in each direction (all branches). The basis set of testcases will also insure that all statements have been executed. McCabe’s cyclomaticcomplexity metric (defined in Section 5.1.3) identifies the number of independent pathsin the basis set; therefore, it is fairly easy to determine the minimum number of testcases required for structural testing. This is discussed further in Section 5.2.2.2.

As noted in Section 5.1.3, faults congregate at decision points, within complex sectionsof code, and at boundary values. Structural testing should focus on all of these areas.



5-24

Branch and path testing consider decision points and complexity. Boundaries arecharacterized by data domain (e.g., alphabetic or integer) and range (e.g., all upper caseletters or positive numbers from 0 to 100 inclusive). Test cases that probe theboundaries should be included during structural testing. For example, what happens ifupper case letters are supplied when positive numbers from 0 to 100 were expected?Equivalence class testing is a useful technique for examining the boundaries (seeSection 5.2.1.2).

Section 5.2.2.5 discusses loop testing. Convenient methods are described for limitingthe number of test cases necessary for adequate testing of loop constructs. In general,loops should be tested at the boundaries. If the loop permits a variable number ofpasses and the maximum number of passes is 10, for example, then test cases shouldinclude 1, 2, 9, and 10 passes. A typical number of passes should also be tested. Finally,an attempt should be made to skip the loop entirely and to try 11 passes, if possible.

5.2.2.1. Path Testing

Often it is not feasible to test all paths. Notice that the simplest logic segment with 3binary decision points has 6 branches and 4 paths, as illustrated in Figure 5-5.



5-25

Decis ion 1

Decis ion 2

Branch 2

Branch 1Path 1

Branch 3Path 2

Branch 4

Decis ion 3

Branch 5Path 3

Paths 1,2,3,4

Branch 6Path 4

Figure 5-5Simple Decision Paths

Figure 5-5 has N binary decisions, 2N branches, and N+l paths. For this example, thereis a linear relationship between the number of decisions and the number of paths.Normal branch testing also tests all paths.

(If there were no process statements along branches 2 and 4, then Figure 5-5 wouldcorrespond to a 4-way “case” statement, including the “default” case along branch 6. Inthis instance, there would be the equivalent of 4 branches and 4 paths.)

However, a more complex logic segment with the same number of binary decisions Nand branches 2N may have 2N paths, as illustrated in Figure 5-6. In this case, as thenumber of decisions increases, the number of paths grows geometrically.Independently testing each branch does not also test each path. Suppose that thenumber of decisions was N = 20, so that the number of possible paths would be 220 =1,048,576? It is obvious that complete path testing may be impossible for general software.



5-26

Decision 1

Path 1Path 2

Decision 2

Paths 1,2

Paths 1,2Paths 3,4

Decision 3

Paths 1,2,3,4

Paths 1,2,3,4Paths 5,6,7,8

Paths 1,2,3,4,5,6,7,8

Figure 5-6Complex Decision Paths

McCabe’s cyclomatic complexity metric was described earlier. It is defined as thenumber of independent paths in a program or module and may be determined byadding one to the number of binary (two-way) decisions. Contrary to the concept of allpossible paths associated with Figure 5-6, an independent path from any given point isone that ignores all previous paths taken prior to that point. Therefore, the number ofindependent paths is 4 in both Figures 5-5 and 5-6.



5-27

5.2.2.2. Branch Testing

Pressman [7] uses the term basis path testing to describe testing that is designed toexercise all of the independent paths (the basis set). This represents the set of test casesthat must be designed to insure full branch testing. That is, the basis set requires takingall decisions in each direction (all branches). Basis path testing and branch testing (aswe have used the term) are synonymous. Only independent paths are considered,ignoring previous paths, so full branch testing does not necessarily correspond to fullpath testing. However, the basis set of test cases will insure that all statements havebeen executed at least once, so the minimum objectives of structural testing will besatisfied.

It is convenient to construct a control flow graph (or simply flow graph) to illustrate thebasis set. A flow graph is like a flow chart that includes only control structures. Flowgraph notation for the usual structured programming constructs is illustrated in Figure5-7.

Sequence If. . .Then...Else Whi le Unti l

Figure 5-7Flow Graph Notation

Circles are called nodes and arrows are called links or edges. A decision is a node withmore than one arrow leaving; this is also called a predicate node. A junction is a nodewith more than one arrow entering. Entry and exit are also denoted by nodes.Processing occurs along links. Therefore, the flow graph for a module with no decisionsor loops (no program control statements) would simply consist of the sequence notationillustrated in Figure 5-7. A region is an enclosed area of the flow graph bounded byedges and nodes. Figure 5-8 (adapted from Pressman) illustrates the flow graph for anexample program module with 10 nodes, 12 edges, and 3 regions



5-28

1: Begin2: Do While Not end-of-input read record3: If field 1 is blank4: Then process record store in buffer increment counter5: ElseIf field 2 is blank6: Then skip record7: Else process record store in buffer write buffer to output file clear buffer reset counter8: EndIf9: EndDo10: End

1

2

5

467

8

3

9

10

Figure 5-8Flow Graph for Example Module

There are several ways to calculate a module’s cyclomatic complexity, C, from its flowgraph. These are described using the example in Figure 5-8:

1. C = R + 1, where R is the number of regions in the flow graph; therefore, C = 3 + 1 = 4.

2. C = E – N + 2, where E is the number of edges and N is the number of nodes in theflow graph; therefore, C = 12 – 10 + 2 = 4.

3. C = P + 1, where P is the number of predicate nodes in the flow graph; therefore,C = 3 + 1 = 4.

As stated previously while discussing complexity, studies have indicated that C shouldnot be allowed to go above 10; if that happens, then an effort should be made topartition the module into two or more sub-modules.

Some authors (cf. IEEE 982.1 [9]) describe cyclomatic complexity in terms of a stronglyconnected flow graph, which has an additional link from the exit node to the entry node.Therefore, a strongly connected flow graph will have one more region and one more



5-29

edge than we have considered, and the above expressions becomeC = R = E – N + 1 = P + 1.

It is important to note that predicate nodes in the flow graph must represent simplepredicates (binary decisions). If the code has a compound predicate, such as two relationalexpressions joined by a boolean operator as in A < B AND B < C, then it must beseparated into its parts and represented in the flow graph as individual simplepredicate nodes. For example, the compound predicate

If A < B AND B < CThen X = B/2Else X = 0EndIf

when separated into simple predicates becomes

If A < BThen If B < C Then X = B/2 Else X = 0 EndIfElse X = 0EndIf

The corresponding flow graph has two predicate nodes and a cyclomatic complexity of3. It is sufficient to recognize that a compound predicate consisting of P simplepredicates contributes P+1 to cyclomatic complexity.

The cyclomatic complexity metric C defines the number of independent paths thatcomprise the basis set. This is the minimum number of test cases that must be provided toinsure coverage of all branches and all statements. The test cases must be designed to forceexecution of each path in the basis set. It is possible that an independent path cannot betested in stand-alone fashion because the necessary combination of logic cannot beachieved during normal program flow; such a path must be identified and tested aspart of another test case in the basis set.

It is possible to represent the flow graph as a graph matrix. A graph matrix for theexample given in Figure 5-8 is illustrated in Figure 5-9. Note that the graph matrix maybe derived directly from the program code (without first preparing a flow graph) bysimply numbering all control flow nodes (as we have done in Figure 5-8), thenrecognizing how the nodes are connected.



5-30

In the graph matrix, a connection from a node defined by its row to a node defined byits column is illustrated by a link weight at the intersection of the row and column. (Thisform of graph matrix is also called a connection matrix.) In Figure 5-9, links have beenassigned a uniform weight of 1; however, other values may be assigned to the linkweight to represent probability of execution, processing time, memory required, orwhatever. Predicate nodes are identified by rows with more than one connection link.As before, cyclomatic complexity C may be determined by adding one to the number ofpredicate nodes. An exit node is a row with no connection link.

Connected to Node

Node1 2 3 4 5 6 7 8 9 10

Predicates

1 1 1 − 1 = 0

2 1 1 2 − 1 = 1

3 1 1 2 − 1 = 1

4 1 1 − 1 = 0

5 1 1 2 − 1 = 1

6 1 1 − 1 = 0

7 1 1 − 1 = 0

8 1 1 − 1 = 0

9 1 1 − 1 = 0

10 C = 3 + 1 = 4

Figure 5-9Graph Matrix for Example in Figure 5-8

5.2.2.3. Condition Testing

As we have discussed, branch testing involves executing each outcome of each decisionpoint. Pressman calls branch testing the simplest form of condition testing. (Herecondition, decision, and predicate may be considered synonymous terms.) Other formsof condition testing described by Pressman are domain testing and branch and relationaloperator (BRO) testing. In domain testing, the relational expression E1 Ω E2, Where E1and E2 are arithmetic expressions and Ω is a relational operator (<, ≤, =, ≠, ≥, >), mustbe tested with the three values E1 = E2 and El = E2 ± ε, where ε represents a smallvalue. This will insure detection of an error in Ω when El and E2 are correct, as well as



5-31

errors in El and E2 when Ω is correct. See Pressman for further information on domainand BRO testing.

5.2.2.4. Data Flow Testing

Pressman also discusses data flow testing, where test paths are selected according to thelocations of program variable definition and use. A definition-use (DU) chain describes thepath from definition of a variable X to use of X, with no intervening redefinition of X.The DU testing strategy requires that every DU chain be covered at least once. QuotingPressman [7], “Since the statements in a program are related to each other according todefinitions and uses of variables, the data flow testing approach is effective for errorprotection. However, the problems of measuring test coverage and selecting test pathsfor data flow testing are more difficult than the corresponding problems for conditiontesting.”

5.2.2.5. Loop Testing

A loop construct has a decision and branch either at the beginning or at the end of theloop. Each repetition of the loop must be counted as another path when considering allpossible paths (as opposed to independent paths).

Four kinds of loops may be defined: simple loops, nested loops, concatenated loops, andloops-from-hell. Examples of the first three are illustrated in Figure 5-10. Loops-from-hellresult from failure to use structured programming constructs and should be redesignedwhenever possible.



5-32

Nested Loops Conca tena ted LoopsS imp le Loops

Figure 5-10Three Kinds of Loops

Simple Loops — Assuming that the loop permits a variable number of passes and thatthe maximum number of allowable passes is M, all of the following tests should beapplied to each simple loop:

1. Attempt to skip the loop entirely (make zero passes through the loop).

2. Make a single pass through the loop.

3. Make two passes through the loop.

4. Make a typical number of passes N through the loop, where N < M.

5. Make M – 1 passes through the loop.

6. Make M passes through the loop.

7. Attempt to make M + 1 passes through the loop.



5-33

Naturally, it is not necessary to perform a given test more than once, so if M is small,some tests may be skipped.

If the simple loop has an excluded range, then it can be tested in two parts. Forexample, suppose that the loop control variable ranges from 1 to 20 inclusive, exceptthat it is not allowed to take the values 7, 8, 9, and 10. Then test cases should attempt toset the loop control variable to the following values: 0, 1, 2, 4, 6, 7 for the first part and10, 11, 15, 19, 20, 21 for the second part. The attempt to set the values that areunderlined should fail.

Nested Loops — Extending the simple loop test strategy to nested loops would causethe number of test cases to grow geometrically as the level of nesting increased.Assuming that zero and M+1 passes are not possible, 25 tests would be required fortwo nested loops and 125 for three nested loops. The following alternative keeps thenumber of tests manageable:

1. Begin with the innermost loop. Set all outer loops to their minimum number ofpasses. Apply the simple loop test strategy to the innermost loop.

2. Work outward, testing the next loop with all outer loops set to their minimumnumber of passes and all inner loops set to their typical number of passes (as in step4 of the simple loop test strategy).

3. Continue until all nested loops have been tested, without repeating any previoustests.

4. Do each of the seven simple loop tests for all loops in the nest simultaneously,without repeating any previous tests.

This strategy works out to 12 tests for two nested loops, 16 tests for three nested loops,and 19 tests for four nested loops, assuming as before that zero and M+1 passes are notpossible.

Concatenated Loops — If each of the loops is independent of the others, then thesimple loop test strategy can be applied to concatenated loops. However, when the loopcounter variable for one loop depends on the other loop(s) in some way, then thenested loop test strategy is recommended.

5.2.3. Other Testing Techniques

In this section, we discuss regression testing, random testing, and testing for AbnormalConditions and Events (ACEs), which are described in Section 6.



5-34

5.2.3.1. Regression testing

From IEEE 610.12:

regression testing. Selective retesting of a system or component to verify thatmodifications have not caused unintended effects and that the system orcomponent still complies with its specified requirements.

It is good practice to design a set of regression tests that challenge already establishedfunctions and to run them periodically to prevent any errors from inadvertentlycreeping in. Regression testing should be carefully considered after each change orcorrection to verify that new faults were not created.

5.2.3.2. Random Testing

With random testing (also called statistical testing), the domain of possible inputs ischaracterized in terms of a probability distribution, then random samples are selectedfrom that distribution for testing. This is very useful in situations where it is impossibleto exhaustively test a module or system for every possible combination of inputs. Forexample:

• Any floating-point input can assume a continuous set of possible values within adiscrete range.

• Except for the simplest logic, exhaustive path testing is prohibitively expensive.

• A control or diagnostic algorithm may depend on the time-history of inputs, whichcan have many possibilities.

In the first example, a floating point input variable may be assumed uniformlydistributed within some fixed range, then a random number generator can be used toselect one or more input values for testing. Likewise, input time-histories can beselected by knowing the mean transient response and its standard deviation (i.e., ameasure of its uncertainty) at each point in time.

By selecting a statistically significant sample and carefully monitoring for any faults, itis possible to use random testing to estimate the reliability of software. See Section 5.5.2for further discussion of random testing.

5.2.3.3. Abnormal Conditions and Events

For high-integrity digital systems, it is good practice to perform tests that probe beyondthe boundaries specified by the requirements. A variety of terms (e.g., fault testing,



5-35

robustness testing, stress testing, and prudence testing) have been applied to thispractice, which is an important part of Abnormal Conditions and Events (ACEs)analysis. Consideration of ACEs is essential for the most critical systems to provide anadditional level of confidence that the requirements are well defined and that thedesign can accommodate all possible modes of operation, including the possibility ofinternal or external mishaps. See Section 6 for further discussion of ACEs.

5.3. Testing Activities

This section explores the issues of test design, using tools for testing, when to stoptesting, and test documentation.

5.3.1. Test Design

Beizer [3] claims, “The test design process, at all levels, is at least as effective at catching[faults] as is running the test designed by that process.” Naturally, finding andcorrecting faults during test design make subsequent testing activities proceed moreefficiently.

Beizer also makes a good analogy between testing and playing pool. A child playing“kiddie pool” claims success whenever any ball drops into a pocket. Adults playing thegame first decide which balls must be pocketed in order to win. Similarly, “kiddietesting” is characterized by claims that whatever test result is observed, that was theexpected one. For proper testing, the correct result is defined and documented beforerunning the test, and success is determined by comparison with that prediction.

There are two major parts to designing tests:

• Identification of test cases

• Preparation of test procedures

Each test case is identified by the input necessary to test some particular aspect of thesoftware and the output expected from that test, with consideration of the conditionsrequired for a proper test. Generally, the difficulty lies in determining what aspectsmust be examined for thorough testing of the software. Test procedures are usuallyspecific to the system under development and are only discussed here in terms ofrecommendations for documentation.

For functional testing, it is often possible to derive test cases directly from the softwarerequirements specification (SRS) and/or the software design description (SDD). Inparticular, if the requirements have been decomposed into elementary form and if thedesign has been expressed using data flow diagrams, decision tables, or similar



5-36

techniques, then it is easier to identify suitable test cases. Table 5-2 is an example of adecision table used for determining test cases.

Adequate structural testing normally requires many test cases. Software modulesfrequently have a large number of potential paths. Such paths may be difficult toidentify, and it may be even more difficult to devise input that will force traversal ofthe desired paths. Finally, the task of demonstrating suitable statement, branch, andpath coverage has to be considered. Therefore, structural testing is most efficient whenthe units are planned as relatively small modules with a small number of branches. Themodules should be designed and implemented with consideration of testability. Aspreviously discussed, maximum cohesion and minimum coupling are desirablecharacteristics. Certain design techniques, such as control flow graphs, help tominimize complexity as well as define what must be tested.

Figure 5-11 is an example data flow diagram. To derive test cases for a particularprocess unit (circle), define combinations of possible data input and expected dataoutput (arrows). Use equivalence classes when the data is continuous (as opposed todiscrete). More detailed test cases may be successively derived from lower level dataflow diagrams. Such diagrams also facilitate the design of interface testing (definedpreviously).



5-37

Data Acquis i t ion

ProcessData

CompareSet Point

SignalA la rm

Display

History Fi le

Parameter F i le

R a wData

RequestScan

DataError

A la rm

AlarmVal idData

A la rm

SetPoints

Cal ibrat ionFactors

Figure 5-11Data Flow Diagram

In summary, test design should begin early in the life cycle. All of the design forfunctional testing can be completed before the implementation phase is even started.With proper consideration during the design phase, most of the test cases necessary forstructural testing can be prepared then so that structural testing may proceed in parallelwith implementation. As structural testing is completed for certain units, functionaltesting may proceed. In other words, the testing phase of the life cycle does not have tobegin after the implementation phase has ended. This is illustrated in Figure 5-12.



5-38

RequirementsPhase

DesignPhase

ImplementationPhase

Functional Test Design

Structural Test Design

Structural Testing

Functional Testing

Figure 5-12Overlap During Testing Phase

As noted by many, it is much less costly to detect and correct faults early in the lifecycle than later (cf. Table 4-1). Therefore, carefully planning and designing tests shouldhave a favorable impact on the budget.

5.3.2. Use of Tools

Testing demands tools. For a typical project, testing may be impractical without toolssupporting

• Instrumentation of code

• Coverage analysis

• Scripted input

• Captured output

• Comparison to expected output

• Regression testing



5-39

A database should be created to keep track of the following elements for each test case:

1. Identifier

2. Input

3. Expected output

4. Actual output

5. Fault classification

6. Reference to requirement/module/unit

7. Type of test (e.g., structural or functional)

EPRI TR-103916 Vol. 2 App. C and D

EPRI TR-105989 Vol. 2 App. D

5.3.3. When to Stop Testing

Testing never ends, in the sense that we cannot plan to continue testing until all thefaults are removed. The reasons for this are both practical limitations and theoreticalimpossibility. Deciding how much testing is enough is subjective and is based on theconsequences of residual faults. Systems with high failure consequences require moretest cases and greater emphasis on testing. Systems that have a limited impact of failuredo not warrant the same concern.

For structural testing, a good test completion criterion may be to require a particular setof test case design methodologies. For system functional testing, the completion rulemight be to stop when a predefined number of faults have been detected or when thescheduled time has elapsed, whichever comes later (see Myers [10]). The main problemwith this rule is specifying the number of faults that must be detected. Methods fordetermining this number include:

• Use experience with similar systems.

• Use a predictive model. Some such models require recording test results during aperiod of time, then applying that data with certain equations and tools to predict



5-40

failure rate, number of remaining faults, etc. These predictions can be used for atradeoff of reliability vs. total test time. See Schneidewind [11]. Also, see theprobabilistic based approach discussed in Section 7.

• Use an industry average (e.g., multiply the number of code statements by thenumber of faults per code statement that are usually discovered in typical programsduring exhaustive functional testing).

Sometimes, the best approach is a combination of these methods.

Functional and structural testing are related. If a fault is found and subsequentlycorrected during functional testing, it is good practice to suspend that test phase, repeatstructural testing, and then resume functional testing. The level of faults requiringrepetition of structural testing is called the suspension criterion. The resumption criterionrefers to the reduction of faults necessary before functional testing is resumed.


5.3.4. Test Documentation

Information for this section has been adapted from IEEE 829 [12]. That standardrecommends preparation of the following documents:

• Test Plan (each software system)

• Test Design Description (each group of tests)

• Test Case Description (each test case)

• Test Procedure (each test case or set of test cases)

• Test Item Transmittal (each item to be tested)

• Test Log (each test)

• Test Incident Report (each incident)

• Test Summary Report (each group of tests)

A software system may have a different group of tests for a set of requirements or for atest unit. Each group of tests may consist of many test cases. A given test procedure maybe applicable to several test cases, where the primary differences are specific input and



5-41

expected output. Reference to a test item is necessary for configuration control. Theremay be many tests (and test logs) for each test case, since a given test case may requirerepeating several times. This document hierarchy is illustrated in Figure 5-13, wheregroup B test case B3 has been repeated for test items X and Y.

Plan

Design A

Design B

Case B1

Case B2

Case B3Procedure N

Procedure M

Log B3NX

Inc ident B3NX1

Inc ident B3NX2

Log B3NY Item Y

I tem X

Summary B

Figure 5-13Illustration of Document Hierarchy

The recommended contents of each document are listed in Table 5-3. Additionaldiscussion may be found in IEEE 829. Note that we have interpreted the IEEE standard;although the words may not be identical, the intent is the same.

Although every enterprise is free to structure its test documentation differently, it isdesirable to include all of the information indicated. The recommended documenthierarchy provides a useful way to organize the information with assurance ofcompleteness. It is also convenient in terms of workflow, from revising test cases andprocedures to reporting results.

A specific example of test documentation is included in the Westinghouse Eagle 21 casestudy described in Section 8. A V&V Software Test Specification illustrating bothstructural and functional unit testing is included. This is described as verification,compared to validation, testing. There is also a Validation Test Procedure related torequirements for the redundant sensor algorithm. Each Westinghouse document is, ineffect, a combination of several of the recommended test documents presented above.Each Westinghouse document contains most of the material recommended for the TestDesign Description, Test Case Description, Test Procedure, and Test Log.



5-42

Table 5-3 Recommended Document Contents

Test Plan

IntroductionItems To Be TestedRequirements To Be TestedRequirements Not To Be TestedApproachPass/Fail CriteriaSuspension/Resumption CriteriaIdentifier ConventionsDeliverables

Test Design Description(s)Test Case DescriptionsTest ProceduresTest Item TransmittalsTest LogsTest Incident ReportsTest Summary Report(s)

EnvironmentToolsTasksResponsibilitiesStaffing and TrainingScheduleRisks and ContingenciesApproval

Test Design Description

IdentifierItems To Be TestedRequirements To Be TestedApproachPass/Fail CriteriaToolsReference to Test Cases and Procedures

Test Case Description

IdentifierItemsEnvironmentSpecial ConsiderationsInputExpected OutputReference to Test Procedure

Test Procedure

IdentifierPurposeReference to Test Case Description(s)Special ConsiderationsProcedures

LogSet UpBeginPerformSuspendResumeCompleteClean Up

Contingencies

Test Item Transmittal

IdentifierItemConfiguration Control

StatusLocation

Approval

Test Log

IdentifierReference to Test Case and ProcedureReference to Test Item TransmittalActivities

Time/DateEnvironmentEventsResultsAnomalies

Reference to Test Incident ReportInput/Output

Test Incident Report

IdentifierReference to Test LogDescriptionImpact

Test Summary Report

IdentifierResultsVariancesEvaluationAssessmentApproval



5-43

5.5. Standards for Nuclear Power Stations

It is instructive to consider requirements for verification and validation (V&V) ofsafety-related applications at nuclear power stations, with the understanding thattesting is a subset of V&V. These requirements may be adapted as appropriate forsystems that are not safety-related.

The following standards and guides include material that is relevant to the subject oftesting:

• ASME NQA-1-1989, Quality Assurance Program Requirements for Nuclear Facilities.

• ASME NQA-2a-1990, Part 2.7, Quality Assurance Requirements of Computer Software forNuclear Facility Applications.

• IEC 880, Software for Computers in the Safety Systems of Nuclear Power Stations, 1986.

• IEC 61508, Functional Safety of Electrical/Electronic/Programmable Electronic Safety-Related Systems, Draft, 1998.

• IEEE Std 7-4.3.2-1993, Standard Criteria for Digital Computers in Safety Systems ofNuclear Power Generating Stations.

• IEEE Std 603-1991, Standard Criteria for Safety Systems for Nuclear Power GeneratingStations.

• IEEE Std 610.12-1990, Standard Glossary of Software Engineering Terminology.

• IEEE Std 829-1983, Standard for Software Test Documentation.

• IEEE Std 982.1-1988, Standard Dictionary of Measures to Produce Reliable Software.

• IEEE Std 982.2-1988, Guide for the Use of IEEE Standard Dictionary of Measures toProduce Reliable Software.

• IEEE Std 1008-1987, Standard for Software Unit Testing.

• IEEE Std 1012-1998, Standard for Software Verification and Validation.

• IEEE Std 1028-1997, Standard for Software Reviews.

• IEEE Std 1044-1993, Standard Classification for Software Anomalies.

• IEEE Std 1044.1-1995, Guide to Classification for Software Anomalies.



5-44

• IEEE Std 1061-1992, Standard for a Software Quality Metrics Methodology.

• US NRC Regulatory Guide 1.168, Verification, Validation, Reviews, and Audits forDigital Computer Software Used in Safety Systems of Nuclear Power Plants, September1997.

• US NRC Regulatory Guide 1.170, Software Test Documentation for Digital ComputerSoftware Used in Safety Systems of Nuclear Power Plants, September 1997.

• US NRC Regulatory Guide 1.171, Software Unit Testing for Digital Computer SoftwareUsed in Safety Systems of Nuclear Power Plants, September 1997.

Some material from these standards and guides is discussed in the remainder of thissection.

5.5.1. Independence

Independence of the V&V team is defined by reference to ASME NQA-1 [13], whichspecifies the following:

Design verification shall be performed by any competent individual(s) orgroup(s) other than those who performed the original design but who may befrom the same organization. This verification may be performed by theoriginator’s supervisor, provided the supervisor did not specify a singulardesign approach or rule out certain design considerations and did not establishthe design inputs used in the design or, provided the supervisor is the onlyindividual in the organization competent to perform the verification.

Section 5.3.4 of IEEE 7-4.3.2 [14] specifies, with reference to NQA-1:

A V&V plan shall be prepared to confirm the correctness and completeness ofthe design. The V&V plan shall specify activities and tests that shall beinspected, witnessed, performed, or reviewed by competent individual(s) orgroup(s) other than those who developed the original design. The V&V planshall be reviewed by competent individual(s) or group(s) other than those whodeveloped the plan.

This seems to say that testing should be either inspected, witnessed, performed, orreviewed by an independent group, which allows some flexibility in planning. On theother hand, the V&V plan must be independently reviewed. With respect toindependent V&V activities, Appendix E of IEEE 7-4.3.2 describes review, witnessing,inspection, analysis, and testing, then it says, “A combination of activities should bechosen, based upon the system development process.” It goes on to say:



5-45

Testing may be performed by the designer or by person(s) other than theoriginal designer. If testing is performed by the designer, the results must bedocumented and independent review subsequently performed.

5.5.2. Test Techniques

Although not an official part of the standard, Appendix E of IEEE 7-4.3.2 indicates,“Test techniques can be divided into two categories: functional, and structural.” Itdescribes these techniques using the software engineering terminology discussedpreviously. It states that functional testing “cannot be used to conclusively determinethat there are no internal characteristics of the code that would cause unintendedbehavior unless all combinations of inputs are exhaustively tested.” Exhaustivefunctional testing is generally impossible. Therefore, structural testing is usuallynecessary and always desirable to supplement functional testing. The referencedAppendix affirms that structural branch and path testing is most useful for individualunits that have been designed with a small number of conditional statements (decisionpoints). It states:

All branches or paths that affect the safety function should be tested. Theremaining branches and paths should be evaluated and tested as appropriate.[emphasis added]

Notice the implicit recognition that complete path testing may not be feasible. Areasonable interpretation of this recommendation is that structural testing shouldinclude at least all branches that affect the safety function.

IEC 880 [15] discusses software testing in terms of systematic and statistical approaches.A table is used to briefly describe each approach. Although expressed with differentterminology, the systematic approach includes:

• Equivalence class testing (in-bound, on-bound, on-bound ± ε, out-of-bound)

• Requirements testing (all)

• Hardware/software interface testing (all)

• Exhaustive testing (if feasible)

• Statement testing (all)

• Branch testing (all)

— Every predicate (condition) to each branch



5-46

— Each loop with min, max, and (at least) one intermediate number of repetitions

• Path testing (if feasible)

— Each loop repetition is a new path

• Assignment and reference to each memory location used (at least once)

• All input-to-output mappings (if feasible)

• Time constraints (all)

• Significant combinations of interrupt sequences (all)

• Maximum combinations of interrupt sequences (if feasible)

• Interface testing (all)

— Every module invoked (at least once)

— Every invocation of a module (at least once)

At the system level, the referenced Appendix suggests systematic testing “with arealistic simulator, or with a realistic version of the system to be monitored orcontrolled by the safety system.” It also recommends performance testing.

The statistical approach (also called random testing) should only be used tocomplement, not replace, systematic testing. The following assumptions apply:

• The number of test cases is large (e.g., more than 1,000).

• The result of each test case is independent of the sequence and number of test cases.

• Few of the test cases result in failures.

• Each failure resulting from a test case is detected.

Statistical testing is particularly useful for:

• Loop structures

• Standard functions (supplied by others)

• Operating systems (supplied by others)

• Code interface to standard functions and operating systems



5-47

The following statistical testing is described:

• Each input data combination with realistic probability

• Each input data combination with equal probability

• Each program property with equal probability

• Any potential fault detected with equal probability for each test case

• Random selection of multiple program properties

• Random selection of multiple input data

Statistical testing can lead to determination of a failure probability.

5.6. Summary

The main topics of this section are summarized in the following list.

• Concepts and definitions related to testing are presented.

• Different testing methods, their effectiveness, and their appropriate use arediscussed, with special emphasis on functional and structural testing.

• Testing activities are considered, including the design and performance of tests.

• Topics of special interest to nuclear electric utilities are addressed, includingstandards that are applicable to testing.

As indicated above, functional and structural testing are the two major categories oftesting techniques that are developed in this section. Table 5-4 provides a summary ofthe main characteristics of these important subjects.



5-48

Table 5-4Characteristics of Functional and Structural Testing

Functional Testing Structural Testing

Requirements Design

Acceptance Implementation

User Developer

Engineer Programmer

Domain, Range, Result Statement, Branch, Path

Evaluate Analyze

What How

Validation or Verification Verification

Right Work Work Right

For electric utility instrumentation and control projects, it is likely that most of thestructural testing will be performed by equipment vendors or system integrators undercontract to the utility company. Utility engineers are more likely to be involved infunctional testing.

5.7. Selected Bibliography

The following limited bibliography has been selected for those wishing to furtherpursue the subject of testing.

• Baber, R., Error-Free Software, 1991.

• Beizer, B., Software Testing Techniques, 1990.

• Beizer, B., Software System Testing & Quality Assurance, 1984.

• Boehm, B., Software Engineering Economics, 1981.

• Brooks, F., Mythical Man-Month, 1982.

• Dunn, R., Software Defect Removal, 1984.

• Dyer, M., The Cleanroom Approach to Quality Software Development, 1992.

• Evans, M., Productive Software Test Management, 1984.



5-49

• Hetzel, W., The Complete Guide to Software Testing, 1988.

• Hollocker, C., Software Reviews and Audits Handbook, 1990.

• Marks, D. M., Testing Very Big Systems, 1992.

• Muss, Iannino, and Okumoto, Software Reliability, 1989.

• Myers, G., The Art of Software Testing, 1979.

• Ould, M., and C. Unwin, Testing in Software Development, 1986.

• Parrington, N., and M. Roper, Understanding Software Testing, 1989.

• Perry, W., How to Test Software Packages, 1986.

• Perry, W., A Structured Approach to Systems Testing, 1988.

• Pressman, R., Software Engineering: A Practitioner’s Approach, 1992.

• Romberg, F., Can Software Quality be Quantified?, 1989.

• Royer, T., Software Testing Management: Life on the Critical Path, 1992.

• Spencer, R., Computer Usability, 1985.

• Stitt, M., Debugging: Creative Techniques & Tools for Software Repair, 1992.

• Single Source (1992 catalog of best books on testing and QA).

• Software Management News (special testing issue quarterly).

• Software Quality World (newsletter).

5.8. References


2. Harrison, R., Effective Techniques for Software Testing Seminar Workbook, Data-TechInstitute, 1992.

3. Beizer, B., Software Testing Techniques, Van Nostrand Reinhold, 1990.



5-50

4. IEEE Std 1044-1993, Standard Classification for Software Anomalies.

5. IEEE Std 1044.1-1995, Guide to Classification for Software Anomalies.

6. Dyer, M., The Cleanroom Approach to Quality Software Development, Wiley, 1992.

7. Pressman, R. S., Software Engineering: A Practitioner’s Approach, McGraw-Hill, 1992.

8. IEEE Std 1008-1987, Standard for Software Unit Testing.

9. IEEE Std 982.1-1988, Standard Dictionary of Measures to Produce Reliable Software.

10. Myers, G. J., The Art of Software Testing, Wiley, 1979.

11. Schneidewind, N., Reliability Modeling for Safety-Critical Software, IEEE Transactionson Reliability, Vol. 46, No.1, March 1997.

12. IEEE Std 829-1983, Standard for Software Test Documentation.

13. ASME NQA-1-1989, Quality Assurance Program Requirements for Nuclear Facilities,Supplement 3S-1, Section 4.


15. IEC 880, Software for Computers in the Safety Systems of Nuclear Power Stations,Appendix E, Software Testing, 1986.



5-51

Appendix 5.A. Classification of Faults

During testing, it is useful to classify faults as they are located. This permits collectionof certain records. For example, Beizer reported on 6,877,000 program statements(comments included) from various sources, with 16,209 discovered faults (2.36 faultsper 1000 statements) [3]. He refers to this as a Taxonomy of bugs. These data describefaults discovered during independent testing, integration testing, and system testing.They do not include faults discovered and corrected by the programmer during unittesting. These faults were classified by Beizer as indicated in Table 5-5 on the next page.

Another classification scheme may be more applicable for any given project. The pointis that some scheme should be devised for classifying faults so that it is possible toaccumulate and maintain appropriate records. Such records may identify whereadditional work should be focused to reduce the number of faults and improve thequality of software development. In particular, it is worth noting that structural anddata problems account for the largest number of faults according to Beizer’s data.

EPRI TR-103331 Vol. 2 Sec. 6.1.1


EPRI TR-105989 Vol. 2 Sec. E.3



5-52

Table 5-5 Classification of Faults

Fault Classification Number of Faults Percent of Total

Requirements 1317 8.1%Requirements Incorrect 649 4.0%Requirements Logic 153 0.9%Requirements, Completeness 224 1.4%Presentation, Documentation 13 0.1%Requirements Changes 278 1.7%

Features and Functionality 2624 16.2%Feature/Function Correctness 456 2.8%Feature Completeness 231 1.4%Functional Case Completeness 193 1.2%Domain Bugs 778 4.8%User Messages and Diagnostics 857 5.3%Exception Condition Mishandled 79 0.5%Other Functional Bugs 30 0.2%

Structural 4082 25.2%Control Flow and Sequencing 2078 12.8%Processing 2004 12.4%

Data 3638 22.4%Data Definition and Structure 1805 11.1%Data Access and Handling 1831 11.3%Other Data Problems 2 0.0%

Implementation and Coding 1601 9.9%Coding and Typographical 322 2.0%Style and Standards Violation 318 2.0%Documentation 960 5.9%Other Implementation 1 0.0%

Integration 1455 9.0%Internal Interfaces 859 5.3%External Interfaces, Timing, etc. 518 3.2%Other Integration 78 0.5%

System, Software Architecture 282 1.7%O/S Call and Use 47 0.3%Software Architecture 139 0.9%Recovery and Accountability 4 0.0%Performance 64 0.4%Incorrect Diagnostics, Exceptions 16 0.1%Partitions, Overlays 3 0.0%Sysgen, Environment 9 0.1%

Test Definition and Execution 447 2.8%Test Design Bugs 11 0.1%Test Execution Bugs 355 2.2%Test Documentation 11 0.1%Test Case Completeness 64 0.4%Other Testing Bugs 6 0.0%

Other, Unspecified 763 4.7%



5-53

Appendix 5.B. Types of Testing

For completeness, various types of testing are listed here as defined in IEEE 610.12.These are presented approximately in the order that such activities might beaccomplished during the testing phase of the development life cycle.

development testing. Formal or informal testing conducted during thedevelopment of a system or component, usually in the development environmentby the developer.

unit testing. Testing of individual hardware or software units or groups of relatedunits.

component testing. Testing of individual hardware or software components orgroups of related components.

interface testing. Testing conducted to evaluate whether systems or componentspass data and control correctly to one another.

mutation testing. A testing methodology in which two or more program mutationsare executed using the same test cases to evaluate the ability of the test cases todetect differences in the mutations.

program mutation. (1) A computer program that has been purposely altered fromthe intended version to evaluate the ability of test cases to detect the alteration. (2)The process of creating an altered program as in (1).

integration testing. Testing in which software components, hardware components,or both are combined and tested to evaluate the interaction between them.

system testing. Testing conducted on a complete, integrated system to evaluate thesystem’s compliance with its specified requirements.

functional testing. Testing conducted to evaluate the compliance of a system orcomponent with specified functional requirements.

performance testing. Testing conducted to evaluate the compliance of a system orcomponent with specified performance requirements.

qualification testing. Testing conducted to determine whether a system orcomponent is suitable for operational use.

operational testing. Testing conducted to evaluate a system or component in itsoperational environment.



5-54

acceptance testing. (1) Formal testing conducted to determine whether or not asystem satisfies its acceptance criteria and to enable the customer to determinewhether or not to accept the system. (2). Formal testing conducted to enable a user,customer, or other authorized entity to determine whether to accept a system orcomponent.

stress testing. Testing conducted to evaluate a system or component at or beyondthe limits of its specified requirements.


6-1

6 GRADED V&V FOR I&C SYSTEMS

Because of the wide range of digital instrumentation and control (I&C) systems thatmight be implemented in nuclear power plants, it is difficult to make specificrecommendations for design or verification and validation (V&V) activities that mustbe applied in all situations. Therefore, it is appropriate to restrict attention to certainsystem classes. By classifying each system according to a few important attributes, thegeneral requirements defined in standards can be translated into gradedrecommendations. This assures adequate confidence that the rigorous process used todevelop a system is commensurate with the system’s importance. It is helpful toprescribe costly design or V&V activities only when actually necessary.

For each classification of system attributes, this section identifies gradedrecommendations for design and V&V activities that are needed to achieve anappropriate level of confidence in the system’s software.

6.1. Technical Approach

The technical approach to classification considers plant instrumentation, monitoring,and control systems, including safety systems, safety-related systems, and some non-safety systems.9 Figure 6-1 illustrates the overall approach. There are three principalsteps:

1. System Classification

2. Software Classification

3. Grading Process

The last step leads to recommendations for software project activities.

9 The original technical approach was developed during 1993 by the Classification Subcommittee of theEPRI/Utility V&V Working Group. The Classification Subcommittee included Ron Reeves of Tennessee ValleyAuthority, Richard Blauw of Commonwealth Edison Company, Ray Rettberg of GPU Nuclear Corporation, PaulJoannou of Ontario Hydro, and Randall May of S. Levy Incorporated, with assistance by Siddharth Bhatt of EPRI.


Graded V&V for I&C Systems

6-2

IEEE 603 [1] and IEEE 7-4.3.2 [2] are used as the basis for requirements governingsystem design and V&V for safety systems in US nuclear facilities. IEEE 7-4.3.2provides considerable latitude in its implementation. The classification approach allowsattention to be focused on certain classes of software, so that specific recommendationscan be made for each class. These recommendations address the appropriate degree ofrigor in software engineering, design considerations, and V&V activities. For the safetyclasses, such recommendations constitute an interpretation of IEEE 7-4.3.2.

System Safety Role (Failure Consequences)

SoftwareClassification

System Category

(A) Safety/RPS(B) Safety(C) Important Non-Safety(D) Other

Software Integrity Level

Required Integrity

Defense-in-Depth

Probabilistic Data* Reliability Data Reliability Target

* Used only for probabilistic approach (PRA)

SystemClassificationEvent Frequency*

Recommended activities for each Integrity Level (1, 2, 3)

Adj

ustin

g F

acto

rs

STEPS

(1) Most Rigorous(2) Intermediate(3) Less Rigorous

GradingProcess

Flexibility for acceptingexistin g com ponents

FunctionalComplexity

DevelopmentEnvironment

Figure 6-1Overview of Technical Approach

The approach is equally applicable whether probabilistic or deterministic methods areused to establish design requirements. With deterministic design bases, theconsequences of initiating events and system failures determine the systemclassification, without explicit consideration of the probability that they may actuallyoccur. Similarly, the consequences of a software failure are factored into the softwareclassification. However, if a probabilistic approach is adopted, then the frequency of



6-3

initiating events as well as the required reliability10 of the system, its components, andits software are also considered.

The term software is used in its broad sense to include not only program source code,but also other results of the software development life cycle, such as specifications,algorithms, design, configuration, and data values. For example, an applicationinvolving a programmable logic controller (PLC) has configuration representations(e.g., ladder logic diagrams) and parameter values that are considered applicationsoftware even though no program source code may be involved.

The scope of this classification approach is focused on digital I&C systems that havewell-defined functional requirements at the plant system level. This approach does notaddress overall design and analysis of the nuclear plant architecture to establish plantsystem requirements. The end product of this approach is a set of recommendations fordesign and V&V activities not only for the software, but also for the digital system tothe degree that it influences or is influenced by the software.

Early in the system design, functionality of the digital system must be allocated amongthe hardware, the software, and the operator. Although this document focuses onactivities needed to ensure software quality, the hardware must also be carefullydesigned, implemented, and tested to provide the necessary level of confidencecommensurate with its role in the system. In addition, operation and maintenanceprocedures and training are necessary to ensure that the operational approachconceived by the designers is implemented in practice.

6.1.1. System Classification

The system classification depends on the role that the system plays in plant safety.Various standards [3,7,8,10,12,34] and guidelines [4,9] used in the US andinternationally classify systems in this way, using either risk or safety consequences asa classifying attribute.

The approach is to tie the system classification as closely as possible to domestic designstandards that are familiar to utility engineering staff. IEEE 603 provides a basis fordetermining what supporting features must be designed to the same quality level as theunderlying safety feature. However, IEEE 603 is narrowly focused on reactor protectionsystems and engineered safety features, so it is necessary to reference another standardto broaden the coverage.

10 For a reactor protection system, reliability corresponds to availability per demand.



6-4

ANS-58.14 [3] defines “Supplemental Grade” items that, although not Class 1E, havebeen assigned varying safety significance through regulatory guides or licensingcommitments. ANS-58.14 also provides guidance and many examples for determiningto which category a given plant system or component belongs. In addition, certain non-safety systems that are important to operations assume a comparable significancebecause of their importance to plant availability as well as their potential forchallenging safety systems.

Based upon these two standards, the classification approach recommends the followingsystem categories. Note that categories A and B correspond to Class 1E systems.

System Category Description

(A) Safety/RPS Reactor Protection System

(B) Safety All other safety systems

(C) Important Non-Safety

ANS-58.14 Supplemental, and Important to Operations

(D) Other All other systems

Table 6-1 elaborates on these categories, lists examples, and relates them to thecategories of IEEE 603 and ANS-58.14. Note that category B is the nominal category forsafety systems. The reactor protection system (RPS) has been elevated to its owncategory because its special importance warrants additional activities over and abovethose necessary for the other categories.

Performing system classification is an intermediate step toward establishing theintegrity level and appropriate degree of rigor for the digital system software.


6.1.2. Software Classification

The principal input to software classification is the system category described in Figure6-1. The goal of software classification is to determine the necessary software integritylevel. The higher the integrity level, the more rigorous will be the requirements for



6-5

engineering, verifying, and validating the software, and the more restrictive will bedesign constraints and practices.11

As a first approximation, we directly map each system category to a software integritylevel. That is, software that is part of a digital RPS in category A is assigned softwareintegrity level 1, safety system software in category B is assigned integrity level 2, andimportant non-safety software in category C is assigned integrity level 3. All othersoftware (i.e., category D) is assigned integrity level 4, the lowest level. 12

System Category Software Classification (preliminary)

(A) Safety/RPS Integrity Level 1

(B) Safety Integrity Level 2

(C) Important Non-Safety

Integrity Level 3

(D) Other Integrity Level 4

11 This terminology does not imply that the quality of software at lower integrity levels should be compromised,only that the documentation and life cycle activities to demonstrate quality need not be as rigorous.12 Notice that the highest integrity level is called 1, and the lowest level is 4. This backward numbering scheme isunfortunate, but it is historically consistent with other references (cf. [31,32]). See Section 6.3.2 for furtherdiscussion on this point.



6-6

Table 6-1System Classification for Digital I&C Upgrades

System Category Example Explanation IEEE 603 ANS 58.14

(A) Safety/RPS Reactor Protection System(RPS)

Because of the crucial role of the RPS and the short timeresponse required of reactor trips, certain additional stepswill be recommended for the RPS that are not necessaryfor other Class 1E safety systems. This category alsoincludes other auxiliary features (as defined by IEEE 603)that cannot be isolated from RPS critical functions andwhose failure may lead to RPS failure.

Safety Features

Certain OtherAux. Features

Safety

(B) Safety

Engineered Safety Features

Aux. Supporting Features

Other Auxiliary Features

Core Spray Initiation

Diesel Generator LoadSequencer

Reg. Guide 1.97 Type APost Accident Variables

Maintenance Interface forRPS

With the exception of RPS, this category includes all Class1E safety systems plus post-accident monitoring functionsthat must be designed to Class 1E standards. Otherauxiliary features that are not required for operation ofsafety systems but whose failure could interfere withcritical functions are also included. Auxiliary features ofthe RPS that cannot cause failure of RPS critical functionsalso belong in this category.

Safety Features

Aux. SupportingFeatures

Remaining OtherAux. Features

Safety

(C) Important Non-Safety ANS-58.14 Supplemental13

SPDS Diverse ATWS Systems Security System

Important to Operations Alarm System Plant Control Systems

ANS-58.14 Supplemental is “an item that does not performa safety-related function, but to which a significantlicensing requirement or commitment applies.” Importantto Operations includes non-safety systems that areimportant to plant availability, help operators respond tooff-normal events, or whose failure may challenge safetysystems.

N/A Supplementalor Non-Safety

(D) Other Plant Chemistry System All other systems are included in this category. No specificV&V recommendations are made for this category.

N/A Non-Safety

13 Based on the Supplemental category of ANS-58.14, except that Reg. Guide 1.97 Type A variables are elevated to Category B (Safety).



6-7

6.1.2.1. Adjusting Factors

This first approximation for grading the required level of software integrity can beimproved by considering the adjusting factors listed in Figure 6-1. That is, thepreliminary software integrity level based on system category should be adjusted byconsidering certain factors. These factors, which are primarily system characteristics,are explained below.

• The required integrity of the system is determined by answering the question: “Howimportant is it that the system not fail?” This is closely related to the consequencesof software failure. But the consequences of software failure are not always the sameas the consequences of system failure. That is why it is necessary to classify thesoftware and the system separately when defining software integrity level. If thesoftware introduces a system failure mode that is not already accounted for in thesystem classification, then the software integrity level should be raised. On the otherhand, if the role of the software is such that its failure cannot impact the systemsafety function, then the software integrity level may be lowered. Adequateisolation must exist between the software and the system components thatimplement its safety function in order to claim that the software cannot affect thesystem safety function. For example, the role of the software is reduced if the criticalfunctionality of the system has been allocated to its hardware or to manual action.The consequences of software error can also be reduced by use of fault tolerance inthe design. For example, mitigating features can return the system to a safe state iferroneous software behavior is detected.

• The functional complexity of the system should be considered when determiningsoftware integrity level. This may be characterized by how difficult or intricate arethe functions that the software is required to perform. For example, it may bepossible to exhaustively test a simple set of Boolean functions, permitting areduction in the software integrity level. But a more complicated control algorithmmay require extensive random testing in a plant simulator environment, so a higherlevel of software integrity is indicated. Several authors (cf. EPRI TR-103916 [4]) haveargued that complexity can be directly related to the probability of software failure.Only functional complexity is considered here as an adjusting factor for softwareintegrity level because only functional requirements are known at this stage of thelife cycle. The complexity of a system’s design and implementation will beaddressed separately as a graded design attribute, such that high software integritylevels demand simple design and implementation.

• The development environment involves all aspects related to the softwaredevelopment team and its organization. Some of the primary topics to consider arethe characteristics of the technical staff, their knowledge and experience, theirmanagement, their use of tools, established practices and procedures, etc. Better



6-8

development environment characteristics may justify a reduction in the necessarysoftware integrity level.

• The last adjusting factor, defense-in-depth, considers diversity, redundancy, andbackup systems. The presence of functional diversity may permit a lower softwareintegrity level. For example, if there are two diverse systems performing the samefunction, then the software in each of the two systems might be assigned a lowerintegrity level. More discussion of software diversity and common mode softwarefailures is presented in Section 6.3.1.

These adjusting factors, summarized in Table 6-2, define considerations for translatingthe system category into an appropriate software integrity level.14

Table 6-2Adjusting Factors for Software Integrity Level

Required integrityMitigating systems and featuresAllocation of functionality to softwarePossible introduction of new failure modes

Functional complexityQuantity of external interfacesReal-time process monitoringReal-time process controlLevel of operator interaction

Development environmentExpertise of the development teamOrganization and managementToolsQuality assuranceConfiguration managementPractices and procedures

Defense-in-depthComponent diversity (within a safety system)Functional diversity (among safety systems)Backup systems (including non-safety systems)

14 Application of these adjusting factors will differ when using either a deterministic or a probabilistic approach toestablish design requirements. For the deterministic approach prevalent in the U.S., each adjusting factor willmodify the software integrity level that prescribes recommended design and V&V activities. For the probabilisticapproach, each adjusting factor will modify the software reliability requirements.



6-9

6.1.2.2. Restrictions on Application of Adjusting Factors

Comparison to minimum requirements — This section endorses the use of adjustingfactors to modify the software integrity level only if there is a characteristic that exceedsthe nominal design and licensing bases for the corresponding system category. Inparticular, if the design or licensing bases for a system category assume a minimumamount of functional diversity, then the software integrity level may be lowered only ifthere is additional diversity beyond that minimum.

Reduction by at most one level — The software integrity level may not be lowered bymore than one level below the nominal level of the corresponding system category,regardless of how many adjusting factors apply favorably.

Exclusion of specific activities prescribed by regulation — For a system in a certaincategory, there may be activities associated with that category that are required byregulation and cannot be reduced even if a lower software integrity level is technicallyjustified. For example, consider a category B safety system that requires a 10 CFR 50Appendix B quality assurance program. Even if extra diversity and extreme functionalsimplicity justified its demotion to software integrity level 3, it is still necessary toapply 10 CFR 50 Appendix B.

6.1.3. Grading Process

The system and software classification technical approach specifies the softwareintegrity level. This is useful only if it leads to practical recommendations concerningwhat activities should be (or need not be) performed to achieve the desired softwareintegrity. A grading process is used to develop such recommendations.

The first step in the grading process is to identify activities and attributes that can bevaried to yield an appropriate software integrity level. In essence, these are the “knobs”that can be adjusted to produce the required software quality. These activities andattributes are grouped into three major categories:

1. Technical (analysis and design)

2. V&V (qualification)

3. Administrative (management)

Because these aspects of the software development process are really inseparable, thegraded recommendations must cut across all three.



6-10

6.1.3.1. Identification of Activities and Attributes

It is not possible or advisable to grade all of the activities associated with software andsystem quality. Some activities are applicable to all projects regardless of the targetsoftware integrity level, with the understanding that responsible engineers andmanagers must use judgment as to how they are applied in a particular project.Examples of such common activities are given in Table 6-3. It is very difficult torecommend discrete levels for such activities without overly restricting a projectmanager’s ability to accommodate the specific needs of the project.

Table 6-3Common (Ungraded) Activities Applicable to All Software Integrity Levels

Codingstandards

Coding standards should encourage accepted practices (e.g., extensivecomment statements, restricted use of GOTO statements) to keep thesource code simple and reviewable.

Functionaltesting

For all systems, functional (black-box) testing should be performed toconfirm that each requirement is correctly implemented.

Planning Planning ensures an adequate schedule and commitment of resources.It is appropriate to prepare project, QA, V&V, and configurationmanagement (CM) plans, as well as plans and procedures for eachstage of software development and testing. The degree of formaldocumentation and prior approval can be graded as described below.

Configurationmanagement

Whether a system is in the development or the maintenance stage, it isimportant to have a consistent package of code, documentation, andsupport tools. Identification and control of revisions and changes isgood practice for all software projects. This extends to firmware onPROM (or similar device).

Personneltraining andevaluation

Personnel must be qualified to perform their responsibilities. Thisapplies to both the development team and the V&V team, as well asother tasks such as configuration management. Training of operatorsand maintenance personnel may also be necessary to ensure properuse of the system.



6-11

For certain other activities, it is easier to identify appropriate tasks or degrees of rigornecessary to achieve the desired software integrity level. Table 6-4 presents a list ofnine activities that can be adjusted depending on the target level. These fall into threemajor categories:

1. Technical (analysis and design)

2. V&V (qualification)

3. Administrative (management)



6-12

Table 6-4Activities and Attributes That Can Be Adjusted to Achieve the Target Level of Software Integrity

Category Activity or Attribute Description

Technical Design process Rigorous analysis and design techniques can improve confidence in high integrity software. Forexample, specifications can be made clear and unambiguous by using graphical, tabular, ormathematical methods instead of depending on narrative descriptions.

Abnormal conditionsand events (ACEs)

Analysis of ACEs helps define software requirements, reveals hidden faults in the design, anddemonstrates whether the design is robust.

Design attributes The extent to which the system design is robust, as measured by its tolerance of abnormalconditions or faults, is an important characteristic of highly reliable software.

V&V Verification reviews Verification review effectiveness can be improved by use of structured walk-through, codeinspection, requirements traceability matrix, thread path audit, and other techniques.

Verification testing Structural (white-box) testing is concerned with tests of the internal structure of code. Itcomplements functional (black-box) testing, providing additional confidence that theimplementation is correct and free of unintended features, especially under abnormal conditions.

Validation testing Validation testing may range from simple open-loop tests to closed-loop dynamic tests with theplant response modeled by high-fidelity simulation.

V&V independence The extent to which reviewers of plans, designs, documents, coding, and testing are independent ofthe primary developers is an issue for assuring systems with high integrity.

Administrative Software QA The extent of software quality assurance activities, with documented results that are easily audited,can influence confidence in a digital system.

Documentation Certain standards prescribe many specific separate documents, including plans, procedures,personnel qualification records, specification, design, implementation, testing, resolution of V&Vissues, configuration management records, etc. The flexibility to combine, condense, or reorganizedocuments depends on interpretation of form vs. function with respect to such standards.



6-13

As an example from Table 6-4, consider verification testing, which involves choicesregarding a V&V technique. For systems that do not require a high level of softwareintegrity, functional (black-box) testing may provide sufficient confidence, so structural(white-box) testing may not be necessary. However, for a high integrity system such asthe RPS, extensive structural testing should be performed to cover all possible branchesof program execution.15

6.1.3.2. Graded Recommendations for Activities

Table 6-5 presents a set of recommendations that is based upon the engineeringjudgment of the Classification Subcommittee with review and approval by theEPRI/Utility V&V Working Group.16 Criteria for these recommendations includerecommendations of other standards, precedents from nuclear industry projects, andtechnical feasibility. These recommendations describe development activities that areappropriate to achieve different software integrity levels.

Many of the terms used in Table 6-5 may be somewhat terse because of the desire tocompress several complex ideas and recommendations into a single table that is moreeasily visualized. Therefore, Table 6-6 supplies a more detailed explanation of eachsuch term.

Some discussion is useful to add perspective to the recommendations in Table 6-5. Firstconsider integrity level 2, which may usually be associated with a safety system(category B in Table 6-1) such as core cooling. Recommendations include:

• Structured analysis and design methods, such as those supported by commercialCASE17 tools, should be used to enhance the clarity and precision of specifications.

• Software abnormal conditions and events (ACEs) should be analyzed as describedin IEEE 7-4.3.2 Annex F [2] and in EPRI TR-104595 [14]. Design of the system shouldaddress and resolve all issues raised by the ACEs analysis.

• The complexity of software design and coding should be restricted to encouragesimple, easily reviewed source code and documentation.

15 Structural (white-box) testing is concerned with checking the internals of software source code. Structural testingcomplements functional (black-box) testing, which may be considered more traditional.16 See the footnote at the beginning of Section 6.1 for more information on the Classification Subcommittee.17 CASE is an acronym for Computer Aided Software Engineering.



6-14

• A thorough traceability matrix should be prepared and maintained to support anefficient thread path audit. The traceability matrix should readily verify that allrequirements have been adequately addressed during each part of the life cycle.

• Software source code should be reviewed using structured walk-through and/orindependent code inspection techniques performed by personnel with appropriateskills using defined procedures.

• Structural (white-box) software testing should verify that each statement in theprogram has been correctly executed at least once. Correct implementation of allrequirements must be confirmed by functional (black-box) testing.

• Validation testing should include dynamic tests using inputs that are representativeof relevant plant transients.

• Individual members of the V&V team should not participate in the design workthey are verifying.

• 10 CFR 50 Appendix B applies to QA of safety-related functions.

• Suggested documents include QA plan, V&V plan, CM plan, requirementsspecification, design description, implementation record(s), test procedure(s), testreport(s), V&V report, installation manual, operation manual, and maintenancemanual.

Some of these recommendations are relaxed or left to the discretion of the individualproject manager for integrity level 3, which will usually apply to an important non-safety system (category C in Table 6-1) such as a feedwater controller. Flexibility isallowed with respect to the rigor of design, coding, and documentation practices.Structural testing is considered optional, and it may be possible to perform adequatevalidation testing with static inputs or open-loop tests. 10 CFR 50 Appendix B strictlyapplies only to QA of safety-related functions; however, a software QA program that isconsistent with 10 CFR 50 Appendix B should be applied when developing a non-safetysystem. The project manager should consider whether the non-safety system isimportant to operations or associated with a licensing commitment.

For integrity level 1 (e.g., reactor protection system), some additional recommendations(relative to level 2) add confidence for both the utility and the regulator. These include:

• Unambiguous formats, using tabular or mathematical representations, should beused in the requirements document. Reliance on natural language should beminimized.



6-15

• More extensive structural testing, with complete coverage of branches, should beperformed.

• Validation testing should include randomly generated test cases, to increasecoverage and avoid any bias in test case definition. Tests that cover significantabnormal and faulted conditions should be included, also.

• It might be appropriate for the V&V team to be organizationally independent of thedesign team.



6-16

Table 6-5Development Activity Recommendations to Achieve the Target Software Integrity Level

Activity Level 1 (Highest Integrity) Level 2 (Medium Integrity) Level 3 (Lowest Integrity)

Design process Unambiguous mathematical or tabularspecifications

Structured analysis methods Text specifications18

Abnormalconditions andevents (ACEs)

In-depth and formal analysis ofsoftware ACEs

Formal analysis of software ACEs Informal consideration of softwareACEs

Design attributes Design resolves ACEs analysis issuesDesign emphasizes simplicity withlimited functionality

Design resolves ACEs analysis issuesDesign emphasizes simplicity withlimited functionality

Error handling capabilitiesDesign and implementation inaccordance with functionality

Verification reviews Traceability matrixPreparation for thread path auditStructured walk-through and/orinspection

Traceability matrixPreparation for thread path auditStructured walk-through and/orinspection

Informal reviews

Verification testing Branch coverage (structural testing)Requirements (functional) testing

Line coverage (structural testing)Requirements (functional) testing

Structural testing optionalRequirements (functional) testing

Validation testing Dynamic testing and random testingEnhanced validation testing

Dynamic testing and random testing Depends upon application

V&V independence Individual independence(organizational independenceoptional)

Individual independence Peer review (independence optional)

Software QA Strict adherence to 10 CFR 50Appendix B

Strict adherence to 10 CFR 50Appendix B

Appropriate software QA (SQA)19

Documentation Rigorous documentation, withapproval required before next lifecycle step

Rigorous documentation, withapproval required before next lifecycle step

Flexibility to combine and reorderdocuments

18 Control systems can be quite complex, so special design methods (e.g., function block languages) should be considered.19 An augmented quality program may be required by regulatory guides applicable to specific non-safety systems.



6-17

Table 6-6Explanation of Terms Used in Table 6-5

Term Used in Table 6-5 Explanation

Unambiguousmathematical or tabularspecifications

Functional requirements should be expressed in an unambiguous format instead of depending on narrative textdescriptions. Such formats may employ graphical, tabular, or mathematical methods.

Structured analysismethods

Structured analysis methods use data flow diagrams, structure charts, and similar methods to define thefunctional structure of complex software. Yourdon methodology is one prominent example. Unified ModelingLanguage (UML) is another example.

Text specifications Natural language narrative text specifications should be concise, complete, consistent, verifiable, traceable,understandable, and modifiable.

Formal analysis of softwareabnormal conditions andevents (ACEs)

See IEEE 7-4.3.2 Annex F. For safety systems, analysis of ACEs should be formally conducted using specialtechniques such as fault tree analysis. Each identified ACE and its resolution should be documented. The analysisshould be performed by individuals who have appropriate training and experience. Also, see EPRI TR-104595.

In-depth and formalanalysis of software ACEs

In addition to formal treatment, a deeper analysis of ACEs is necessary for Safety/RPS systems. Besidesconsidering ACEs caused by external events or hardware faults, critical modules of the software should beindividually analyzed for ACEs that are internal to the system.

Informal consideration ofsoftware ACEs

Regardless of the software integrity level, good design practice includes consideration of possible failure modesand an attempt to identify abnormalities (e.g., bad input data) that might interfere with system operation.Informal consideration does not require any special ACEs analysis techniques or documentation.

Design resolves ACEsanalysis issues

For high integrity systems, consideration of ACEs should be integrated into specification and design. When newexternal hazards or internal error modes are identified, they may be resolved by mitigating features orsupplementary requirements.

Error handling capabilities Good software engineering practices include built-in error handling features. Robust error handling reduces thesensitivity of system behavior in the presence of abnormal conditions such as bad data, failed sensors, or highprocessing demand.

Design emphasizessimplicity with limitedfunctionality

Other things being equal, a simple system is more scrutable and more demonstrably reliable than a complexsystem. This concept pervades international standards as well as NRC correspondence. For example, IEC 1226states that for safety-critical systems, “The Design shall aim to ease the verification … by maintaining simplicity….The functionality of category A [systems] shall be restricted….”



6-18


Design andimplementation inaccordance withfunctionality

Restrictions on functionality can be relaxed for low-integrity systems. It is good practice for the design to be assimple as practicable, recognizing that additional functionality may increase complexity.

Traceability matrix A traceability matrix is essential for integrity levels 1 and 2, so that it is easy for an independent reviewer to tracespecification, design, implementation, and testing of each requirement. The traceability matrix identifies whereeach requirement is addressed in documents, source code, and test reports.

Preparation for thread pathaudit

A thread path audit starts with a specific requirement, function, or feature, which is then traced through thesystem and software development life cycle. The purpose is to confirm that the requirement, function, or featurehas been adequately addressed by each of the major life cycle activities, including V&V. The documentation andtraceability matrix should be organized so that a thread path audit can be performed efficiently.

Structured walk-throughand/or inspection

Structured walk-through and code inspection are established techniques for rigorous review of software. Theyinclude formal procedures and carefully prescribed activities.

Informal reviews Informal reviews do not require a formal procedure or any special techniques. They may be conducted by simplyreviewing the documentation, as long as an appropriate level of independence is observed.

Line coverage Complete line coverage testing confirms that each statement in the program has been correctly executed at leastonce. Line coverage is synonymous with statement coverage.

Branch coverage Complete branch coverage testing confirms that each statement in the program and, in particular, every possibleresult (branch) of each decision statement (e.g., IF statement) has been correctly executed at least once.

[Extent of validationtesting] depends uponapplication

Validation (functional) testing is an important activity that should be applied at all integrity levels. Systems withintegrity level 3 may be more complex than systems with higher integrity levels (1 or 2); therefore, validationtesting for non-safety systems may be more demanding than for safety systems. For example, control systemfunctions often depend on the details of both plant response and operator commands, so it may be essential to testcontrol systems using a high-fidelity plant simulator.

Dynamic testing In the context of real-time systems, dynamic testing refers to tests with time dependent inputs that arerepresentative of the transient response of the plant. Such inputs may be supplied by a plant simulator, recordedplant transient data, engineering transient modeling programs, or related sources.

Random testing For random test cases, input values are randomly selected from the set of all possible values. With a reasonablenumber of random test cases, it may be possible to provide a high level of test coverage even though exhaustivetesting is not feasible.



6-19


Enhanced validationtesting

For Safety/RPS systems, the highest possible attention should be given to validation testing. In addition todynamic testing, random testing should be performed for increased coverage. The results of ACEs analysis shouldbe factored into the test plan to ensure that abnormal behavior is tested. It may be appropriate to haveindividually independent verifiers perform, witness, and/or review the tests. (This may be called a prudence testor review.)

Individual independence Individual independence requires that the reviewing individual should not have direct involvement with thedesign or implementation under review; however, the V&V reviewer may belong to the same organization as thedeveloper.

Organizationalindependence

Strict organizational independence requires that developers and verifiers report to separate organizations that donot share the same manager. Organizational independence is not required in this recommendation, nor is itrequired in domestic standards such as IEEE 7-4.3.2.

Peer review (independenceoptional)

This phrase describes a situation where a development work product may be reviewed by any qualifiedindividual who did not prepare that work product. However, the reviewer may belong to the same developmentteam or may have prepared other work products on the same project.

Strict … 10 CFR 50 App. B Self-explanatory.

Appropriate software QA(SQA)

Appendix B of 10 CFR 50 strictly applies only to QA of “safety-related functions of … structures, systems, andcomponents that prevent or mitigate the consequences of postulated accidents that could cause undue risk to thehealth and safety of the public.” However, it is prudent to apply a QA program that is appropriately derived from10 CFR 50 Appendix B for important non-safety systems. Refer to industry standards such as IEEE 730, 1012, 1028,and 1298 for guidance with respect to good software QA practices.

Rigorous documentation This implies documentation with the degree of rigor needed to satisfy external reviewers (e.g., NRC). The projectmanager should consider preparing a separate document for each major life cycle phase. Each document shouldbe reviewed and approved before proceeding to the next life cycle phase. Plans should be prepared and reviewedin advance of each major activity. Note that the life cycle may be defined differently for each project.

Flexibility to combine andreorder documents

For integrity level 3, it is not necessary to generate a separate document for each life cycle activity, but it isimportant to have each activity described in documentation that can be easily traced. Traceability of the life cyclefrom one activity to the next is crucial, so that its logical flow is apparent to a reviewer. Note that the life cyclemay be defined differently for each project.



6-20

Although these recommended practices might not have been used for every level 1system development to date, each of them has been used practically and successfully inat least one application.

Notice that no specific recommendations are presented for integrity level 4, which willusually apply to “other” systems (category D in Table 6-1). The project manager shouldchoose an appropriate subset of the activities recommended for integrity level 3.

6.2. Practical Approach

In the previous section, a technical approach to graded V&V for I&C systems identifiedrecommended development activities to achieve a target software integrity level. Thepractical approach presented in this section is a step-by-step classification process thatfirst determines a V&V class. Then for each class, V&V methods are described thatcomplement the development activities for the target integrity level. With safety-relatedsystems, the classification is also useful to determine what amount of activity isimportant to satisfy regulatory requirements.

This practical approach considers two major issues of V&V classification: What to Doand How to Do It. Table 6-7 summarizes the concepts of this classification.

Table 6-7Concepts of V&V Classification

Issue Classifying Attributes Products of Classification

What to Do System Safety CategoryRequired IntegrityFunctional ComplexityAdjusting Factors

Defense-in-DepthDevelopment Environment

Software Integrity LevelNecessary Degree of RigorV&V Class

How to Do It Application TypeDesign Characteristics

Constituent ComponentsPlatform TypeProgramming MethodHuman InterfaceHardware/Software Integration

Project CharacteristicsProcurement MethodLife Cycle Stage

Applicable Methods for DeterminingSystem RequirementsSoftware RequirementsSoftware DesignDesign AnalysisReview and InspectionSoftware TestingSystem Testing

Tools That Support These Methods



6-21

6.2.1. What to Do: Determining the V&V Class

The nature of the system (i.e., its safety category, required software integrity, andfunctional complexity) defines What to Do. The classification process establishes theV&V class, which identifies the degree of rigor necessary during the requirements,design, and implementation life cycle stages, as well as the intensity of V&V activities.Recommendations for specific activities and V&V methods then follow.

Figure 6-2 illustrates the classification process.

Most Rigorous

Intermediate

Least Rigorous

High

Low

Med

2

2

3

2

2

2

Low Med High

Required Integrity

Com

plex

ity

Adjusting Factor

3 2

1

3 3 1

1

(Development Environment, Defense in Depth)

+–+–+–

V&V Class V&V Intensity

3

2

1

Figure 6-2Selection of the V&V Class



6-22

To determine the V&V class:

• Assess the software complexity. Software with a higher complexity rating will needmore rigorous V&V. Usually three levels of software complexity are established.Some straightforward procedures for determining the complexity rating are foundin the EPRI reports identified by the i-buttons at the end of this subsection.

• Assess the required software integrity. This can be determined qualitatively byanswering “How important is it that the software not fail?” This is related to thequestion “What are the consequences of software failure?” If the consequences ofsoftware failure are high, then more rigorous V&V is needed. Usually three levels ofsoftware integrity are established. Procedures for specifying the required softwareintegrity level are mentioned in the EPRI reports identified by the i-buttons at theend of this subsection.

• Figure 6-2 represents a matrix of required integrity vs. complexity. Using the twovalues assessed above, find the cell at the intersection of the complexity row and theintegrity column. Notice that each cell has one primary value for V&V class (withlarger font), and some cells have a secondary value (with smaller font).

• The initial value of V&V class considers only the two primary factors (complexityand integrity). Development environment and defense-in-depth are adjustingfactors that lead to final determination of the V&V class. If the softwaredevelopment environment is poor, then more careful V&V will be necessary; on theother hand, an excellent development environment may justify relaxing the finalV&V class. Similarly, a defense-in-depth design philosophy may permit lessrigorous V&V.

The following examples illustrate the V&V classification process.

1. If the software is assessed to have Medium complexity and Medium requiredintegrity, then the primary V&V class is 2. But if the development environment isexcellent or the system design includes defense-in-depth (e.g., diversity,redundancy, or backup systems), then the V&V class can be relaxed to 3, and lessrigorous V&V should be acceptable.

2. If the software must be of High integrity and is expected to have Mediumcomplexity, but the development environment is judged to be poor, then the mostcareful V&V class 1 will be required.

3. For High integrity with Low complexity, the V&V class will always be 2(intermediate). The same is true for Low integrity with High complexity. For theseassessments, the adjusting factors are not considered



6-23

4. If High integrity software also has High complexity (the most difficult scenario),then the V&V class will always be 1.



6.2.2. How to Do It: Selecting Applicable V&V Methods

Having made an assessment of V&V class, the remaining step of the V&V classificationprocess considers additional attributes of the system, such as its constituentcomponents and design characteristics, with the aim of determining How to Do It. Inthis way, the project manager can decide on specific V&V methods and tools that aremost applicable for cost-effectively providing the level of rigor and intensity expectedfor the V&V class.

There are a variety of methods and tools that can be used to perform V&V. Theeffectiveness and applicability of these depend on what type of system is beingconsidered and how it will be designed and built.

The attributes in Table 6-8 can be used to categorize a system. For a given system, it ispossible to assign multiple values to most of these attributes. For example, constituentsoftware components may include an operating system and several component drivers.In addition, different parts of a system may have different values assigned to a givenattribute. For example, a monitoring system may have a data acquisition moduledeveloped using conventional procedural programming, but an operator advisorymodule may use both object-oriented and rule-based programming. For this example, aseparate category should be identified for each subsystem, so that the correct V&Vmethods can be determined for each.

After determining the V&V class and the system attributes, specific V&V methods canbe selected. Many methods are available for performing V&V throughout the softwaredevelopment life cycle. Each of these methods has different characteristics, locatesdifferent kinds of faults, and requires different skills and tools. Therefore, there aredifferent costs associated with these V&V methods. In addition, each method isapplicable to different stages in the life cycle.

There may be several combinations of V&V methods that will adequately address aparticular type of fault. It is more important that the selected set of methods will be ableto detect all the types of faults that may exist in the software being developed. As a



6-24

guideline for selecting specific V&V methods, Table 6-9 lists some methods that areapplicable to systems with custom written software as well as those that usecommercial off-the-shelf (COTS) software. Recommendations are made for each V&Vclass. The recommended methods are applicable to different life cycle stages —requirements, design, and implementation. Note that the list of V&V methods in Table6-9 is merely a representative list and is not intended to be an exhaustive list of all themethods that are available.



6-25

Table 6-8Attributes for Selecting Applicable V&V Methods

Attribute Possible Values

Application Type AnalysisDatabaseMonitoringControlDiagnosisPlanning/SchedulingProtection

Constituent Components Application Source CodeOperating SystemSoftware Development ToolsOffice Automation ToolsComponent Driver(s)Standard Function Library

Platform Type Dedicated Single-Purpose SystemGeneral-Purpose SystemModular Control SystemProgrammable Logic Controller (PLC)

Programming Method Procedural ProgrammingFourth-Generation LanguageObject-Oriented ProgrammingRule-Based System

Human Interface Operator InterfaceInformation DisplayMaintenance PanelNo Human Interface

Hardware/Software Integration Software OnlyEmbedded Firmware

Procurement Method In-House DevelopmentContracted DevelopmentCommercial DedicationTurnkey

Life Cycle Stage New System DevelopmentNew System Replicating Existing FunctionsModification of a Previously Developed System



6-26

Table 6-9Selection of Some Specific V&V Methods

V&V Class Stage

V&V Method 1 2 3 R D I References

Requirements analysis R R R 15Formal languages HR — — 15, 17Formal requirements review HR HR HR 15Modeling and animation with computer tools R — — 16Formal design review HR HR HR 15User interface inspection HR HR R 18, 19Requirements tracing HR HR HR 15, 31v7g1Structural testing HR HR R 12, 27Database analysis R R — 15, 20Software practices review R R — 21, 22System engineering review R — — 23Failure mode effects causality analysis (FMECA) HR — — 30Automated anomaly testing R R — 20Algorithm analysis R — — 15Process trigger and timing analysis R — — 24Data interface inspection R R — 15Process oriented audits HR HR — 21, 31v7g5Formal customer review HR HR HR 22, 31v7g11Functional testing HR HR HR 25, 31v7g3Boundary testing HR HR — 26, 27, 31v7g4Random testing HR R — 28, 31v7g6Robustness testing HR HR — 29, 31v7g8Regression testing HR HR — 15, 31v7g9Validation scenario testing HR HR — 31v6

Legend

Recommendation for V&V Class:HR Highly RecommendedR Recommended— No recommendation for or against

Life Cycle Stage:R RequirementsD DesignI Implementation

References:31vN Volume N of Reference 31. See Section 6.5 for numbered list of references.31vNgP Guideline Procedure P in Volume N of Reference 31.



6-27

The following examples illustrate the selection of V&V methods with Table 6-9.

1. If the software is V&V class 3, then methods that are highly recommended duringthe entire development life cycle are formal requirements review, formal designreview, requirements tracing, formal customer review, and functional testing.

2. For V&V class 2 software, formal requirements review and user interface inspectionare highly recommended methods during the requirements stage of the life cycle.With respect to the latter method, it is expected that only inspection of user interfacerequirements will be possible during the requirements stage. But it is highlyrecommended that the user interface inspection method should be repeated duringthe design and implementation stages, also.

Using Table 6-8 to categorize the system is useful for adjusting some of therecommendations made for V&V methods listed in Table 6-9. For example, if thehuman interface attribute in Table 6-8 is a maintenance panel, then therecommendation for user interface inspection V&V method in Table 6-9 might bereduced. Or if the application type in Table 6-8 is database, then the database analysisV&V method in Table 6-9 might become highly recommended.

Table 6-10 provides a more detailed description of the specific methods listed in Table6-9. These are grouped according to the three broad types of V&V techniques that aredeveloped in Section 3 — reviews and inspections, analysis, and testing. Notice that themethods listed in Table 6-9 are roughly grouped according to life cycle stage.



6-28

Table 6-10Description of Some Specific V&V Methods

Group Method Description Effectiveness During Life Cycle

ReviewsandInspections

FormalReviews(Requirements,Design)

Usually a series of reviews conducted at major milestonesduring the development life cycle (e.g., formal requirementsreview, formal design review). Used to improve developmentvisibility and product quality. Provide a basic means forcommunication between the project team, companymanagement, and users or customers.

Since the cost to correct an error increasesrapidly as the development life cycleprogresses, detecting errors during early stagesby use of formal reviews is an attractiveprospect. Most effective for large projects. Forsmall projects, some reviews may beperformed less formally, combined with otherreviews, or eliminated.

User InterfaceInspection

Inspection of all aspects of human-system interaction tominimize the potential for incorrect operation and to supportrecovery from errors.

Detects requirements specification, design, andimplementation errors related tocommunication, interpretation, andenvironment of system operation.

SoftwarePracticesReview

Review of the software development organization to advisemanagement and staff on improved operation and standardsconformance (e.g., documentation, programming guidelines,communication protocols, etc.)

Detects operational and management defectsplus standards conformance during the designand implementation stages of the life cycle.

SystemEngineeringReview

System engineering experts examine application of therequirements specification to the design description and itsimplementation. A variety of checks and analyses confirmgood system engineering practices.

Detects errors during the design andimplementation stages of the life cycle that mayhave originated during the requirements stage.

ProcessOrientedAudits

Examines the products of software development withemphasis on improving the development process. Performedafter the requirements specification has been developed,during the design and implementation stages.

Identify process improvements and checkcompliance with the project plan during theimplementation stage of the life cycle.

FormalCustomerReview

Formal evaluation of the software product by customerrepresentatives. Critical review of the implementation relativeto requirements.

Increases user (customer) satisfaction. Detectsdefects during the implementation stage of thelife cycle that may have originated during therequirements or design stage.



6-29


AnalysisTechniques

RequirementsAnalysis

Analysis of the requirements to assure they are complete,explicit, consistent, testable, etc. Requirements are normallyspecified using graphical and/or textual means with variouslevels of formality. Methods used for requirements analysiswill depend on how they are expressed.

Detects defects that originated during therequirements stage of the life cycle. For largesystems, this method may be essential to assurean accurate requirements specification.

FormalLanguages

Formal methods and languages enable the definition ofrequirements in a rigorous, consistent, unambiguous way.These provide a basis for proving correctness of thedeveloped software. Examples of such languages are EHDM,Z, VDM, etc.

Detects defects that originated during therequirements stage of the life cycle.

Modeling andAnimationwith ComputerTools

When requirements and/or design are expressed in anunambiguous language, whether it is a semiformal graphicallanguage or a formal mathematical language, it may bepossible to test the description directly, even before anyimplementation activity. An automated tool is stronglyrecommended; e.g., Software Requirements EngineeringMethodology (SREM).

Detects defects that originated during therequirements or design stage of the life cycle.

RequirementsTracing

Verifies that products of the life cycle address eachrequirement of the system and that testing demonstratesadequate and appropriate satisfaction of each requirement. Acommon technique to assist with this method is theRequirements Traceability Matrix (RTM) that relates eachrequirement to corresponding items in the designspecification, test procedures, and test results.

Very effective technique for discovering errorsduring the design and implementation stagesof the life cycle as well as the testing andmaintenance stages. Useful for verifyingcompleteness, consistency, and testability.

DatabaseAnalysis

Analysis of database design, structure, declarations, andattributes. Evaluating interfaces for error handling,consistency checking, etc.

Detects defects related to database operationsand calculations during the design andimplementation stages of the life cycle.

Failure ModeEffectsCausalityAnalysis(FMECA)

Identification of failure modes for each system componentand analysis of the consequences of each type of failure.Information includes failure description, cause(s), defect(s),detection, safety consequences or other hazards, and methodsfor recovery from the failure.

Insures appropriate consideration of possiblefailures. Detects defects during the design andimplementation stages of the life cycle that mayhave originated during the requirements stage.



6-30


AnalysisTechniques(continued)

AlgorithmAnalysis

Analysis of software algorithms without execution of thecode. Examination of the processing logic for adequate andcorrect operation.

Detects defects during the implementationstage of the life cycle that may have originatedduring the design stage. Permits identificationof missing or unintended features.

Process Triggerand TimingAnalysis

Analysis of the conditions that activate a process, with specialconcern for the timing of activation relative to otherprocesses.

Detects defects during the implementationstage of the life cycle that may have originatedduring the design stage. Important for V&V ofreal-time systems.

Data InterfaceInspection

Inspection of data interfaces for satisfaction of requirements,error handling, consistency checking, etc.

Detects defects during the implementationstage of the life cycle that may have originatedduring the design stage, including interfaceconnections, communication protocols, andmismatched formats.


TestingTechniques

AutomatedAnomalyTesting

Tests for irregular style, comments, syntax, logic, complexity,etc. Performed using automated tools.

Detects defects during the implementationstage of the life cycle. Assures good softwaredevelopment practices.

StructuralTesting

A white-box approach to software testing. Based on analysisof the program, a set of input data is chosen to exercise a largepercentage of the program code or a previously specifiedtarget portion. Includes statement testing, branch testing, andpath testing.

Detects the consistency of a component’simplementation relative to its design.

FunctionalTesting

A black-box approach to software testing. The internalstructure and design of the software is ignored duringselection of test cases. The software is viewed as an integratedcollection of functions. Tests are constructed to demonstratethat the software satisfies all of its specified functionalrequirements.

Detects defects during the implementationstage of the life cycle that may have originatedduring any stage. Useful for validation of thefinal system.



6-31


TestingTechniques(continued)

BoundaryTesting

Tests the boundaries and partitions of the input space. A typeof domain testing that selects test case values at and around(just inside, just outside) domain boundaries.

Detects defects during the implementationstage of the life cycle that may have originatedduring any stage.

RandomTesting

Input values for test cases are randomly selected, usually byan automatic test data generator. Input values may berestricted to a specified domain, or not. It is possible to selectfrom many different sample distributions.

Detects defects during the implementationstage of the life cycle that may have originatedduring any stage. Effectiveness is diminishedby the difficulty of evaluating all of the testresults obtained. Empirical tests for seeded andreal errors using various sample distributionsdemonstrated that the uniform distribution,which selects input values with equalprobability, was usually the most powerful.

RobustnessTesting

Exercises programs under challenging conditions. Test casesare selected with extreme inputs and variously degradedsituations.

Detects defects during the implementationstage of the life cycle that may have originatedduring the design or implementation stage.

RegressionTesting

Uses a set of test cases that test all of the system’s functionalrequirements (see functional testing). After each change to thesystem, or periodically during system operation, the system isre-tested using this same set of test cases. Any change in testresults must be explained.

Detects defects during the implementationstage of the spiral (iterative) life cycle.Particularly useful during the maintenancestage. Spurious errors introduced by systemmodifications or corrections may be detected.

ValidationScenarioTesting

Uses test cases with realistic inputs that demonstrate subsetsof important functionality. Usually the final acceptance testintended to provide assurance to the user (customer) andpossibly the regulator.

Detects defects during the implementationstage of the life cycle that may have originatedduring any stage.



6-32

Figure 6-3 shows the distribution of V&V methods listed in Tables 6-9 and 6-10 that areeither recommended or highly recommended for each development stage and for eachV&V class. This gives a quick overview of the V&V effort needed during the life cyclefor a particular V&V class. As expected, more effort is recommended for classes thatrequire higher integrity or have more complexity and, therefore, demand morerigorous V&V. The number of recommended methods increases as the life cycleprogresses because there are more products of development to be reviewed, analyzed,and tested.

Requirements

Design

Implementation

V&

V C

lass 3

2

1

V&

V C

lass 3

2

1

V&

V C

lass 3

2

1

Highly RecommendedV&V Method

RecommendedV&V Method

Figure 6-3Distribution of V&V Methods During Development Life Cycle for Each V&V Class


EPRI TR-103331 Vol. 7 Sec. 2 and 3



6-33

6.2.3. Summary of the Practical Approach

For simplicity of presentation and application, we have proposed that the classificationof V&V can be cleanly separated into two parts that determine What to Do (i.e., thedegree of rigor and intensity of V&V activities) and How to Do It. This separation alsohas the advantage that the question of What to Do is particularly interesting togeneralists such as managers, licensing engineers, or regulators, while the question ofHow to Do It is of more interest to specialists such as system engineers or softwaredevelopers.

Of course, this separation is a simplified model of reality at best. For a real project,estimating the necessary level of V&V effort may first require a determination of anyspecial V&V techniques that must be employed, so that What depends upon How.Nevertheless, the principle of keeping What to Do (e.g., V&V intensity) separate fromHow to Do It (e.g., V&V methods) is still a valid goal, and striving for this separationwill help the project manager to organize plans and to justify decisions.

The next two figures graphically illustrate the main factors involved in this practicalapproach and the relationships between them. Figure 6-4 represents a scale measuringthe V&V class of a specific system. The weights on the pans symbolize characteristics ofthe system. Their different sizes are representative of their relative importance in theV&V class determination. The weights on the right-hand pan force the balance toincrease the V&V class to a more rigorous level. The weights on the left-hand pan dothe opposite. After assessing these factors, we obtain the appropriate V&V class.



6-34

V&V Class

1

2

3

Devel .Environ .

Defensein Depth

SoftwareComplexity Required

Integrity

Figure 6-4Factors Involved in Selection of V&V Class

Figure 6-5 represents a scale measuring the acceptance or rejection of a specific system.The weight on the right-hand pan symbolizes V&V class, obtained from the scale ofFigure 6-4. The weights on the left-hand pan symbolize practices applied duringsoftware development that provide the necessary level of rigor and are importantattributes of software quality. These practices and attributes are:

• V&V (developed throughout this Handbook)

• Quality Assurance (see Sections 1 and 6)

• Configuration Management (see Sections 1 and 6)

• Software Reliability (see Sections 6 and 7)

The increasing weight of a higher V&V class forces the balance towards rejection of thesystem unless adequate development practices are applied. The different sizes of theweights on the left-hand pan are representative of their relative importance indetermining acceptance of the system. They symbolize that V&V is the most importantpart of software QA, CM is the least, and it all goes into improving software reliability.



6-35

Accept

No

V&VClass 2Q

A

V&

V

CM

Rel

iabi

lity

Yes

V&VClass 1

V&VClass 3

Figure 6-5Factors Involved in System Acceptance

6.3. Relationship to Technical and Licensing Issues

This approach to classification and graded recommendations is consistent with thesystem licensing approach presented in Guidelines on Licensing Digital Upgrades EPRITR-102348 [6]. Essentially, this section provides technical and practical details tosupport the general approach of the licensing guidelines.

The classification approach, together with graded recommendations that are tailoredfor each class, provides a framework that can help the nuclear power industry proceedwith analog-to-digital upgrade of I&C systems. The following discussion describes howthe approach helps to address several broad issues of regulatory concern and industryuncertainty.

6.3.1. Software Reliability, Common Mode Failure, and Diversity

A strategy for increasing safety system reliability, with respect to the failure of a singlehardware component, is to introduce redundant hardware trains that execute the samecontrol or protection algorithm in parallel. For digital systems, some regulators haveexpressed concerns that a software design error would be common to each such



6-36

redundant train and would, therefore, introduce a common mode failure. Althoughcounter-arguments can be made on the basis that asynchronous trains would performthe calculations at slightly different times and with slightly different input data,common mode software failure remains an issue. Diversity of software function and/orimplementation is one means for addressing this concern.

A related issue arises in connection with probabilistic analyses, for which severereliability requirements may be demanded of critical systems. Quantitative methods forestimating software reliability are still in early stages of development, and it is difficultto definitively demonstrate that the failure probability of a single softwareimplementation is sufficiently small to meet safety goals.

Referring to Figure 6-1, consideration of defense-in-depth when determining thenecessary software integrity level helps to address both of these issues. The presence ofa diverse method for achieving a safety function implies that the integrity requirementsfor either of the individual methods may be relaxed somewhat (cf. [4,7,8,9]). Thisprovides the possibility for design tradeoffs between a single software system of veryhigh integrity level versus functionally diverse systems (software and/or hardware)with software integrity levels that are lower and, therefore, more readily achievableand demonstrable.

However, Leveson [30] points out that splitting the development budget to support twodifferent software implementations usually means that both receive less than adequatesupport, so the results are not satisfactory. It is probably better to minimize thepossibility of software faults by focusing maximum effort on the quality of a singleimplementation.

For further discussion of the issues related to diversity, defense-in-depth, and commonmode software failure, see Section 7.

6.3.2. Graded Software Development and V&V Practices

For digital system development, there exists considerable uncertainty about which ofthe many specific design and V&V activities are most appropriate. Because itencompasses a wide range of safety systems and situations, IEEE 7-4.3.2 must allowconsiderable flexibility in its implementation, while other references have been vague,incomplete, or inconsistent in their guidance (cf. NUREG/CR-5930 [5]). The issuesinclude:

• Flexibility with activities and documentation during the life cycle

• Independence of the V&V team

• Rigor of specification and/or design methods



6-37

• Level of abnormal conditions and events (ACEs) analysis

• Testing techniques and coverage of testing

By categorizing the software into several integrity levels, an appropriate degree of rigorin design and V&V activities can be prescribed. This graded approach is acceptedpractice in other countries and in other industries [8,9,10,11,12]. By also consideringimportant software attributes (e.g., implementation language, degree of customization,etc.), specific techniques can be recommended without the danger of “ratcheting” torequire higher integrity than necessary. The bases for these recommendations considercurrent and past practices in the US and international nuclear industries, as well aspractices from other industries that can be cost-effectively duplicated in the nuclearindustry.

The graded approach developed in Sections 6.1 and 6.2 first determines target integritylevels for system classification, then it recommends the amount of V&V that isnecessary to achieve each level. Integrity levels are based on consequence of failure andrange from 1 (highest integrity) to 4 (lowest integrity). There are no V&Vrecommendations for level 4.

This is essentially consistent with the latest versions of primary national and inter-national standards that address V&V of digital systems. A brief explanation of thegraded approach presented in three such standards follows:

1. IEEE Std 1012-1998, Standard for Software Verification and Validation [34]

• Software criticality represents what is necessary to keep risk within acceptablelimits.

• Software integrity levels are used to quantify software criticality.

• Software integrity levels range from 1 (lowest integrity) to 4 (highest integrity).

• At every phase of the software development life cycle, a minimum set of V&Vtasks are assigned to each software integrity level.

2. IEC 61508, Functional Safety of Electrical/Electronic/Programmable Electronic Safety-Related Systems, Draft, 1998 [12]

• Safety integrity level (SIL) represents the integrity that is necessary for functionsthat are implemented by safety-related systems.

• A risk-based approach is used to determine SIL requirements.

• Numerical failure measures are linked to the SILs.



6-38

• SILs range from 1 (lowest integrity) to 4 (highest integrity).

• At every phase of the software development life cycle, appropriatetechniques/measures are recommended for selection according to SIL.

• For each SIL, techniques/measures are ranked based on effectiveness, from NR(Not Recommended) to HR (Highly Recommended).

3. DO-178B/ED-12B, Software Considerations in Airborne Systems and EquipmentCertification, FAA, 1992 [11]

• System failure condition categories are established by determining the severityof failure conditions on an aircraft and its occupants (consequences).

• An error in software may cause a fault that contributes to a failure condition;therefore, the software level that is necessary for safe operation is related tosystem failure conditions.

• Software levels range from A (highest integrity) to E (lowest integrity); there areno guidelines for level E.

• At every phase of the software development life cycle, appropriate objectivesand outputs are applicable for each software level.

• To satisfy the objectives of a life cycle process, the independence of processactivities is defined by software level.

• For data produced by life cycle process activities, the control category is definedby software level; a control category is related to configuration managementcontrol.

To compare the three primary standards listed above with this V&V Handbook, therelationship of target integrity level is illustrated in Figure 6-6. Note that some of thegraded approaches are based on consequence and some are based on risk. The risk ofan event is the combination of its likelihood and its consequence. If the failure of asoftware-based system is associated with significant consequence, then the likelihood ofits failure must be lowered to a level such that the overall risk is acceptable.



6-39

borne Systems and Equipment

IEEE 1012, 1998

Standard for Software V&V

44

33

22

11

IEC 61508, 1998 (Draft)

Functional Safety of E/E/PE

11

22

33

This V&V Handbook

AA

BB

CC

DD

DO-178B/ED-12B, 1992

Integrity based on consequence:

Software Considerations in Air-

44

33

22

11

Safety-Related Systems

Certification

Integrity based on risk: Integrity based on risk:

Integrity based on consequence:

Figure 6-6Comparison of Graded Approach to Software Integrity


6.3.3. Acceptance of Commercial Off-the-Shelf Products

Necessity has led the nuclear industry to establish practical guidelines for the use ofcommercial grade hardware in applications formerly reserved for nuclear qualifiedequipment. In the digital system and software areas, there are equally strongmotivations to take advantage of rapidly evolving and widely used commercial off-the-



6-40

shelf (COTS) products such as programmable logic controllers (PLCs). This approach isessential if nuclear power plants are to be modernized in a cost-effective manner and ifimproved features and spare parts are to be readily obtained from the marketplace.Such COTS equipment is usually packaged with vendor supplied software, and theutility company has little control over V&V that is performed by the vendor. In somecases, the vendor’s design and V&V documentation may be unavailable because ofinformal development practices or because the vendor is reluctant to release suchinformation. For some practical guidelines, see [33].

The core consideration in applying a COTS product is what processes were used by thevendor to design, implement, verify, and validate the product and how well thoseprocesses satisfy the requirements of the proposed nuclear application. The objectivesof the graded recommendations presented in this section are identical whether theapplication is implemented using a custom system or a COTS product. Vendorstargeting the nuclear industry should consider these recommendations when planningtheir development activities, and utilities should use these recommendations as an aidwhen evaluating COTS products for a particular application.

But even for a high quality COTS product, realistic commercial considerations may nothave allowed all of the recommended development activities to be performed or all ofthe prescribed documentation to be prepared. Therefore, the present guidance includesflexibility as provided under IEEE 7-4.3.2, which allows consideration of“compensating factors” in lieu of specific design activities. Such compensating factorsmay include:

• Documented operating experience in similar applications

• Additional black-box testing or overall system testing

At the highest levels of required integrity, the flexibility to substitute compensatingfactors should be limited. As the integrity level is reduced, more flexibility may beappropriate. By focusing on one integrity level at a time, broad guidance for acceptingCOTS products can be condensed to more concrete, level-specific recommendations,which will help the utility determine:

• Is an available COTS product appropriate for the required integrity level?

• What additional activities are necessary to demonstrate compliance with overallrequirements and recommendations for the required integrity level?

EPRI TR-106439 Sec. 3 and 4



6-41

6.3.4. Relation to 10 CFR 50.59 Screening Criteria

The software classification approach is considered adequate for developing digitalsystems with quality that is comparable to analog systems because it recommendstechniques that are appropriate for providing the necessary level of rigor in thesoftware development process. NSAC-125 [13] has presented seven screening criteriafor 10 CFR 50.59 safety reviews, and Guidelines on Licensing Digital Upgrades EPRI TR-102348 [6] has identified additional considerations for applying these screening criteriato digital upgrades. Table 6-11 shows that each criterion is related to one or more of theinputs to the software classification process presented in this section. Therefore, theclassification activity should be an aid to 10 CFR 50.59 screening.

Table 6-11Relation to 10 CFR 50.59 (NSAC-125) Screening Criteria

NSAC-125 Screening Criteria Corresponding Input to Classification(Fig. 6-1)

1. Probability of occurrence of an accidentevaluated previously

Event frequency

2. Consequences of an accident evaluatedpreviously

System safety role (failure consequences)

3. Probability of occurrence of a malfunctionof equipment evaluated previously

Probabilistic (reliability) data

4. Consequences of a malfunction ofequipment evaluated previously

Required integrity (software)

5. Create the possibility of an accident of adifferent type

System safety role (failure consequences)

6. Create the possibility of a new type ofmalfunction

Required integrity (software)

7. Margin of safety System safety role (failure consequences) andrequired integrity (software)

6.4. Summary

Because of wide variations in the importance and the application of software for digitalI&C systems, it is essential to decide what software integrity level is necessary to satisfythe objectives of a given project. This provides a basis for deciding what V&V activitiesare recommended to support the requisite software quality. This section discusses:

• A technical approach for classifying software integrity level based upon the safetysignificance of its parent I&C system, with the use of certain adjusting factors.



6-42

• A list of development activities, such as design and V&V, that can be graded toachieve the required level of software integrity.

• A practical approach to find the V&V class that is related to both required softwareintegrity and software complexity, with the use of certain adjusting factors.

• A list of specific V&V methods that are recommended for each V&V class duringdifferent stages of the development life cycle.

• The relationship of this approach to technical and licensing issues of interest to bothutilities and regulators.

6.5. References

1. IEEE Std 603-1991, Standard Criteria for Safety Systems for Nuclear Power GeneratingStations.


3. ANS-58.14, Safety and Pressure Integrity Classification Criteria for Light Water Reactors,1993.

4. EPRI TR-103916, Verification and Validation Guidelines for High Integrity Systems,December 1995.

5. NUREG/CR-5930, High Integrity Software Standards and Guidelines, September 1992.

6. EPRI TR-102348, Guideline on Licensing Digital Upgrades, December 1993.

7. IEC 1226, Nuclear Power Plants — Instrumentation and Control Systems Important forSafety — Classification, 1993.

8. MOD 00-56, Safety Management Requirements for Defence Systems, Part 1: Requirements,Part 2: Guidance, UK Defence Standard 00-56/Issue 2, 13 December 1996.

9. Guideline for Categorization of Software in Ontario Hydro’s Nuclear Facilities with Respectto Nuclear Safety, Ontario Hydro, June 1991.

10. IEC 880, Software for Computers in the Safety Systems of Nuclear Power Stations, 1986.

11. DO-178B/ED-12B, Software Considerations in Airborne Systems and EquipmentCertification, FAA, 1992.



6-43

12. IEC 61508, Functional Safety of Electrical/Electronic/Programmable Electronic Safety-Related Systems, Draft, 1998.

13. NSAC-125, Guidelines for 10 CFR 50.59 Safety Evaluations, June 1989.

14. EPRI TR-104595, Abnormal Conditions and Events Analysis for Instrumentation andControl Systems, January 1996.

15. NBS SP 500-93, Software Validation, Verification, and Testing Technique and ToolReference Guide, National Bureau of Standards, September 1982.

16. Alford, M., SREM at the Age of Eight; The Distributed Computed Design System,Computer, pp. 36–46, 1985.

17. SRI-CSL-91-02, An Introduction to Formal Specification and Verification Using EHDM,SRI International, February 1991.

18. NUREG/CR-4227, Human Engineering Guidelines for the Evaluation and Assessment ofVideo Display Units, July 1985.

19. Rasmussen, J. and K. J. Vicente, Cognitive Control of Human Activities and Errors:Implications for Ecological Interface Design, Presented at the 4th InternationalConference on Event Perception and Action, Trieste, Italy, August 24–28, 1987.

20. Ng, P., and R. Yeh (ed.), Modern Software Engineering: Foundations and CurrentPerspectives, Van Nostrand Reinhold, 1990.

21. Bryan, W. L., and S. G. Siegel, Software Product Assurance: Techniques for ReducingSoftware Risk, Elsevier, 1988.

22. NUREG/CR-4640, Handbook of Software Quality Assurance Techniques Applicable to theNuclear Industry, August 1987.

23. Blanchard, B. S., and W. J. Fabrycky, Systems Engineering and Analysis, Prentice Hall,1990.

24. Hatley, D., and I. Pirbhai, Strategies for Real-Time System Specification, Dorset House,1987.

25. Howden, W. E., Functional Program Testing, IEEE Transactions on SoftwareEngineering, Vol. SE-6, No. 2, pp. 162–169, March 1980.

26. Myers, G. J., The Art of Software Testing, Wiley, 1979.

27. Beizer, B., Software Testing Techniques, Van Nostrand Reinhold, 1990.



6-44

28. Barnes, M., P. Bishop, B. Bjarland, G. Dahll, D. Esp, J. Lahti, H. Valisuo, and P.Humphreys, STEM — A Project on Software Test and Evaluation Methods, Proceedingsof the Safety and Reliability Society Symposium, Manchester UK, November 1987.

29. Miller, L. A., Dynamic Testing of Knowledge Bases Using the Heuristic Testing Approach,Expert Systems with Applications: An International Journal, Special Issue:Verification and Validation of Knowledge-Based Systems, Vol. 1, No. 3, pp. 249–269,1990.

30. Leveson, N. G., Safeware: System Safety and Computers, Addison-Wesley, 1995.

31. EPRI TR-103331, Guidelines for the Verification and Validation of Expert System Softwareand Conventional Software, Vol. 6: Validation Scenarios, Vol. 7: User’s Manual, May1995.

32. EPRI TR-105989, Software Fault Reduction Using Computer-Aided Software Engineering(CASE) Tools, June 1995.

33. EPRI TR-106439, Guideline on Evaluation and Acceptance of Commercial Grade DigitalEquipment for Nuclear Safety Applications, October 1996.



7-1

7 ISSUES RELATED TO V&V

The emphasis of this V&V Handbook has been on software verification and validation.However, the techniques and recommended activities for software V&V should not beviewed in isolation. Section 6 raised issues of system and software classification todetermine what V&V activities are appropriate to perform. This section brieflyindicates the relationship of V&V to other issues and activities that are important forupgrade of instrumentation and control (I&C) systems as well as for other digitalsystem applications in nuclear power plants.

7.1. System and Hardware vs. Software

The question often arises as to whether V&V should be applied to the system or thesoftware. Although most V&V activities are concerned with software, a quick review ofFigures 2-1 and 2-2 will show that system requirements are the driving force behind theentire development life cycle, including V&V. Furthermore, because validation mustrelate back to system requirements, validation testing must include some testing in anenvironment that is representative of the final system to be delivered. Therefore, boththe software and the overall system are subject to V&V.

This V&V Handbook does not discuss much about the hardware life cycle, which is ofequal importance to software in the overview of Figure 2-1. Of course, hardwarecomponents and subsystems should be provided with a level of quality that iscomparable to the software. But the subject of quality assurance for hardwarecomponents is very well established based on over 30 years of experience in the nuclearpower industry, and useful guidance can be found in standards such as ASME NQA-1[1] and IEEE 603 [2]. Digital hardware does require some special considerations withrespect to environmental qualification, such as electromagnetic interference (EMI). EPRIhas sponsored research and provided guidance in this area [3,4].

7.2. System Engineering

The following summary of system engineering and related V&V activities represent astraightforward, common sense approach for development of a quality system. Asdiscussed in Section 6, these activities should be selectively applied according to theamount of stringency or rigor necessary for an acceptable level of V&V depending on


Issues Related to V&V

7-2

required integrity and system complexity. Further details of these activities arepresented in Sections 2 through 5.

1. Prepare thorough documentation including project plans, requirementsspecifications, design descriptions, test procedures, and test reports. Notice thatdetailed specification of requirements is a key element because all verification andvalidation efforts are intended to compare results relative to requirements. Selectfrom the following complete list of documents:

Project Plan Integration Requirements Spec.

QA Plan Software Implementation Report

CM Plan Software Inspection Procedure

V&V Plan Software Test Procedure

Test Plan Software Test Report

System Requirements Spec. Acceptance Test Procedure

System Design Description Acceptance Test Report

Hardware Requirements Spec. V&V Report

Hardware Design Description Installation Manual

Software Requirements Spec. Operation Manual

Software Design Description Maintenance Manual

2. Independently review every revision of each document. Code inspection is a similarprocess for reviewing the software implementation. Peer reviews are the most cost-effective way to insure that the work is complete and correct. Reviews should beperformed by knowledgeable people who are not directly involved in preparationof the work product that is under review.

3. Use design analysis tools that are appropriate for the project. For example,computer-aided software engineering (CASE) tools can be used to prepare data flowdiagrams and perform graphical analysis to confirm completeness of the design oridentify errors. Formal specification and/or design languages are also useful whenthe required integrity level justifies the additional expertise necessary for use ofsuch techniques. There are other tools for calculating metrics of the softwareimplementation (e.g., cyclomatic complexity) to confirm that its structure is withinacceptable guidelines.

4. Use testing tools to instrument the software implementation for the purpose ofstructural (white-box) testing to verify that every statement in the program has beencorrectly executed at least once. Similarly, confirm that every possible result



7-3

(branch) of each decision statement (e.g., IF statement) has been correctly executedat least once. Branch testing is generally reserved for the highest levels of V&V. Pathtesting (every possible combination of multiple branches) is usually so difficult thatit is not attempted.

5. Perform functional (black-box) testing to validate that the completed system satisfiesall of its requirements. Such acceptance tests should be witnessed (if not performed)by independent personnel.

A traceability matrix should be maintained, so that it is easy for an independentreviewer to trace specification, design, implementation, and testing of eachrequirement.

7.3. Failure Analysis

For safety-critical applications, failure analysis is performed to provide assurance that allpossible failure modes have been identified, evaluated for their likelihood andconsequences, and resolved. If a failure is extremely far-fetched and has tolerableconsequences, resolution may be to simply dismiss the failure as an acceptable risk. Fora more troublesome case, it may be necessary to modify the requirements or the designto provide run-time exceptions or other features that mitigate the effect of the failure.

Because engineers cannot necessarily foresee all possible failure modes and incorporatethem into the requirements specification, it is usually necessary to perform failureanalysis at several points in the development life cycle as more details are learnedabout both requirements and design. Examples include:

• System-level failures and environmental disturbances should be considered insystem and software requirements specifications.

• Software failures, including common mode failures, should be addressed in thesoftware requirements.

• Techniques such as software fault tree analysis can be used to locate vulnerabilitiesin the software.

• System and software validation testing should attempt to introduce failures into thehardware, software, or operator actions to challenge the important functionalrequirements.

These activities, commonly known as Abnormal Conditions and Events (ACEs)analysis or hazards analysis, should be performed coincident with or subsequent to thelife cycle activities shown in Figures 2-1 and 2-2. If failure analyses exposevulnerabilities to credible failures, then changes to requirements, design, or



7-4

implementation may be necessary; therefore, it is desirable to consider failure analysisearly in the life cycle and minimize changes later.

7.4. Common Mode Failure

Common mode failure (CMF), by strict interpretation, has a meaning that is somewhatdifferent from common cause failure (CCF). The former occurs when multiplecomponents fail in a similar way; the latter refers to failures that have a similar cause.However, because the discussion here is oriented toward failures that can compromiseredundancy and disable redundant systems, both terms will be used interchangeably.CMF and CCF are characterized by failures that are not truly independent.

In failure analysis of redundant analog systems, CCF is a very well known issue. Theidea is that there are some root causes, difficult to assess, that may affect specificcomponents having the same design, manufacturing, installation, operation, andmaintenance conditions, thus weakening the assumption of independent failures. Thisis especially true among the set of redundant components in a system train. Suchcomponents are usually considered to belong to a common cause failure group, and theirreliability is estimated using specific methodologies. There are different modelsdeveloped to evaluate this particular concern; in the US nuclear industry, the β modelis the most broadly used. Briefly, the β model assumes that a fraction of the totalcomponent failure probability is due to CCF probability. Represented by the β factor, thisvalue is usually between 5 and 10% of the total component failure probability. Thisfactor must be estimated for each specific component. There are many reports thatdescribe this model and contain tables of β factors to be applied.

For digital systems, such models do not apply. Digital system failures are all due todesign or implementation errors, not to random wear processes. In addition, softwaremodules are not standardized like hardware components. So statistical analysis toestablish the fraction of total failures due to CCF is not straightforward for digitalsystems. The fraction of software failures that can become CCFs is likely to be muchhigher than 10%, as is commonly used for analog components. Therefore, we cannot besure if a redundant train is a good way to achieve reliability when digital systems areinvolved.

One solution proposed for software-based systems is n-version programming, which usesdifferent software packages in redundant trains (cf. [5]). Discussion about theadvantages and disadvantages of n-version programming vs. one “extremely good”version is ongoing. It is important to understand that failures frequently result fromspecification errors rather than implementation errors; therefore, different functionalspecifications would be necessary to considerably diminish such problems. The mainfactors to consider in evaluation of n-version programming are the expected reductionin failures vs. the cost and effectiveness of designing, implementing, operating, and



7-5

maintaining different software versions. Whatever the decision, CMF or CCF canchallenge system reliability, specifically in redundant trains that include software-based digital components, and should be carefully evaluated.

In [6], the National Research Council discusses several examples of common modesoftware failure, concluding that it is a credible concern. They also explain why n-version programming is not a recommended solution to the problem. The US NRCemphasizes quality (including V&V), diversity, and defense-in-depth as protectionagainst CMF and CCF (cf. Appendix 7.0-A of [7]).

EPRI TR-103916 Vol. 2 App. B

7.5. Redundancy, Diversity, and Defense-in-Depth

Historically, redundancy and diversity have provided two complementary means forimproving the reliability of a function beyond what is achievable with a singlesubsystem. Redundancy involves two or more identical subsystems (or components)acting in parallel. The tacit assumption is that, since most component failures areindependent, a redundant system is significantly more reliable than any one of itssubsystems. For digital systems, this strategy is also used to reduce sensitivity to thefailure of a single hardware component.

However, redundant computer hardware trains execute the same control or protectionsoftware in parallel. And, as discussed in Section 7.4, software failures do not occurindependently because they originate from design or implementation errors rather thanrandom imperfections or wear. For a given set of inputs, the software will produce thesame response in each train every time; for a particular input combination, thatresponse may result in failure. Therefore, the strategy of redundancy is vulnerable tocriticism that a software fault might introduce a common mode failure.

An alternative strategy is to provide diverse means for accomplishing the same function.Functional diversity and/or diversity of design or implementation may be used. Anexample of the former would be to rely on two distinct physical measures (e.g., powerand pressure) to make a decision. An example of the latter would be to provide twodiverse software modules (e.g., different programmers or programming languages) toperform the same function. Another type of implementation diversity is “nameplate”diversity, which uses components from different manufacturers to accomplish the samefunction. Currently in the regulatory arena, functional diversity is considered aneffective and acceptable means for achieving diversity in safety applications, but theconditions for which diverse designs or diverse implementations are consideredeffective and acceptable are still under discussion (cf. [6]).



7-6

Examples of approaches to diversity for digital systems include:

• Use of a parallel or backup analog system

• Use of multiple computer systems with diverse implementations of software and/orhardware

• Use of operator action and procedures as an alternate means for accomplishing thefunction, providing sufficient time is available

Defense-in-depth uses multiple, layered systems to provide alternate means ofaccomplishing different functions related to common goals. It can also involve acombination of quality and diversity. Systematic processes for design, implementation,and V&V contribute to the demonstrable quality of a digital system. However, toprotect against common mode failures or undetected defects, diversity ensures that anindependent means is available to satisfy the requirements. For certain safetyapplications, a defense-in-depth analysis (cf. [8]) may be required to demonstrate thatthe provided diversity covers each event considered in the plant safety analysis report.

7.6. Probabilistic-Based Approach for Software Reliability Assessment

This section introduces probabilistic methodology as a tool to assess software. Asillustrated in Figure 6-1, when a probabilistic approach is necessary, as for ProbabilisticRisk Assessment (PRA), then reliability data and reliability targets are used inclassification of the software.

PRA techniques assess the relative effects of contributing events on system-level safetyor reliability. Probabilistic methods provide a unifying means for assessing physicalfaults, recovery processes, contributing effects, human actions, and other events. Asmentioned in [6], these techniques are an important element of nuclear power plantsafety analysis and have been used effectively, so it is important to consider how tointegrate digital systems into the PRA.

7.6.1. What Is Software Reliability?

Software reliability is the probability that software, under certain conditions, operateswithout failure during a specific period of time. Reliability is one of the attributes ofsoftware quality; it is considered a key factor, since it quantifies software failures.Software reliability becomes more important when the consequences of those failuresare increasingly unacceptable. Several techniques and tools can be used to estimate orpredict the reliability of software.



7-7

Understanding the reliability characteristics of the software is useful to

• Predict the reliability level that the software will reach after the planned testingprocess.

• Estimate the time needed to test the software in order to achieve a specific reliabilitylevel.

These results may be used to obtain a quantitative basis for assessing the digitalsystems that are included in a PRA and to increase the understanding of softwareattributes that is necessary to accept or reject the software. As explained later in thissection, acceptance of a software module should not be determined by assessment ofthe software reliability attribute only.

For development of reliable software, related topics include:

• Fault prevention, covered by good software design practices

• Fault removal, accomplished by application of V&V methods

• Fault tolerance, obtained with defense-in-depth and diversity

• Fault forecasting, provided by software reliability models

7.6.2. Software Reliability Models

Assessment of software reliability is accomplished by a set of mathematical techniques.The application of these techniques, according to reliability theory, leads to differentsoftware reliability models. These software reliability models establish dependence ofthe failure process on the principal factors that affect it. The goal of these models is toanalyze statistical evidence to predict a curve of the software failure rate, which is theprobability that a failure per unit time occurs in the time interval (t, t+∆t) given that afailure has not occurred before time t.

Currently over 40 software reliability models have been published in the literature.Selection of the model that best applies to each situation is not an easy task andinvolves some subjective analysis. The models listed in Table 7-1 are some of the mosttypical and constitute a good sample of different assumptions, requirements, andapproaches [9].



7-8

Table 7-1Software Reliability Models

Model Concept and Required Data Assumptions Comments

Schneidewind This model supposes that current data may bettermodel present reliability because the failure rateprocess changes over time.

There are three forms of the model, depending onone’s view of the importance of the data over time:

1. All the data points are of equal importance. Useall of the fault counts during the n time periods.

2. Only the later data points are important. Ignorethe fault counts from the first through (s – 1)time periods. Use the data from s through n timeperiods.

3. Use all the data points, but with differentweights depending on whether the fault countswere before or after the s time period. This is anintermediate form of the model (between theother two).

Schneidewind has developed criteria for optimalselection of the s value.

Required Data: Fault counts in each of the testingintervals.

The software is operated in a manner that is similarto that in which reliability predictions are to bemade.

The number of faults detected in one time interval isindependent of the fault count in another interval.

The fault correction rate is proportional to thenumber of faults to be corrected.

The mean number of detected faults decreases fromone interval to the next.

The distribution of the number of failuresexperienced by time t follows a Poisson process.

The failure intensity function λ(t) is assumed to beexponentially decreasing over time.

The testing intervals are equal periods of time.

This model has been used insafety-critical software, suchas IBM’s flight controlsoftware for NASA’s spaceshuttle, with very goodsuccess.

Musa-OkumotoLogarithmicPoisson

The idea of this model is that failures discoveredearlier have a greater impact on reducing the failureintensity function than those encountered later.Therefore, it presents a failure intensity function thatdecreases exponentially as failures occur.

The expected number of failures over time is alogarithmic function.

It is an infinite fault model; this means that thesoftware will never be fault free.

Required Data: Either the times that the softwarefailed or the elapsed time between failures.



The failure intensity function λ(t) decreasesexponentially with the number of failuresexperienced.

The distribution of the number of failuresexperienced by time t follows a Poisson process.

This model is especiallyapplicable when it isanticipated that thesoftware’s operational usewill be decidedly non-uniform, which tends tomake failures encounteredearlier have a greater impactthan later ones. It is thisimpact that the modelcaptures especially well.



7-9

Model Concept and Required Data Assumptions Comments

Littlewood-Verrall Bayesian

This is a Bayesian model, so it takes a subjectivepoint of view; that is, if no failures occur while thesoftware is observed, then its reliability shouldincrease, reflecting growing confidence in thesoftware by the observer.

This model tries to account for new faults beingintroduced during the fault correction process; itallows for the probability that the software couldbecome less reliable than before.

Required Data: The time between failureoccurrences.


Successive execution times between failures areassumed to be independent exponential randomvariables with certain failure rates (λi’s).

The λi’s form a sequence of random variables, eachwith a gamma distribution of parameters α and Ψ(i).

Ψ(i) is an increasing function of i that describes thequality of the programmer and the difficulty of thetask. A good programmer would have a morerapidly increasing function than a poorer one.

GeneralizedExponential

This is a generalized model because it uses a uniqueset of equations to represent several models havingthe common assumption that the failure intensityfunction is exponential.

Required Data: Either the time between failures orthe time of the failures.



The faults that caused a failure are correctedinstantaneously without additional faults beingintroduced by the correction process.

The failure rate is proportional to the number ofcurrent faults remaining in the software.

The failure rate is constant between failures, and it isreduced by the same amount when a fault isremoved.

This model includes someother models of the sameclass that can easily berelated using parameterequivalencies. Its simplicityand generality are some ofits main advantages.



7-10

7.6.3. Selection of a Software Reliability Model

Unfortunately, as mentioned before, there are more than 40 published models in theliterature, and no model is the best in all situations. To successfully apply a reliabilitymodel, it is necessary to find the one that is most appropriate for the specific set of data,the environment in which the data were collected, the characteristics of the softwaredevelopment project, and the scenario where the software will operate. There iscurrently no known method for evaluating these conditions to determine a priori whichmodel will prove optimal for a particular development effort.

Preliminary selection of a model requires a qualitative, subjective evaluation. Besidesthis subjective evaluation, selection of a model implies an understanding of all theparticular model assumptions to be sure that the one selected fits correctly with thespecific case under analysis. After a model has been selected, its performance duringuse can be quantitatively assessed; however, such assessment techniques cannot beapplied to the preliminary selection.

7.6.4. Limitations for Application of Software Reliability Theory

The methods developed to assess software reliability of large-scale commercial systemsare not directly applicable to embedded systems for critical applications. The softwarereliability models have important limitations when used to verify systems of highreliability. This is due to the need for statistically meaningful data and the amount oftesting time necessary to obtain those values.

For instance, suppose we need to certify a software failure probability of 10–6 for amodule that is designed to perform a safety function during a mission time of 10 hours.Assuming exponentially distributed mean-time-to-failure, the time to the next failure isapproximately 107 hours (1141 years). Testing this long appears to be impossible. Ofcourse, this time may be reduced by simultaneously testing more than one copy of thesoftware module, but that implies reproducing several test environments that must allbe similar to the expected operational scenario. Replicating more than 10 software testsis rarely practical because of economic limitations.

The testing time may be reduced if the safety function has a shorter mission time, say 1hour. On the other hand, to get statistically significant data, observing more than onefailure is necessary, which implies even more testing. Considering all these factors, ifwe needed to assure the same failure probability as above, but with a mission time ofonly 1 hour, using 10 software replicates, and searching for at least 5 failures, it wouldstill take 5x105 hours (57 years).



7-11

In conclusion, the testing time necessary to apply reliability theory to high reliabilitysoftware exceeds all practical values. In such cases, other approaches can be used toobtain a statistical approximation. For instance, it is possible to estimate an upperbound on the software failure rate after testing for a long period of time withoutobserving a failure.

Most of the models are useful for predicting software reliability in the later stages of thedevelopment life cycle (during integrated testing and later), but no models areapplicable in the early stages. The models will not have good performance if the datacollected from one day to the next is significantly different. Therefore, the softwaremust be mature, so that frequent or major changes are avoided. This is unfortunatebecause any correction to fix a reliability problem will be significantly more costlywhen it is made later in the development life cycle instead of earlier.

Another reason for limited application of the reliability models during earlier stages ofthe life cycle (say during unit testing) is that reliability measurements obtained fromtesting small units of code (say less than 2,000 lines) will not have a high level ofconfidence.

Finally, the environment in which the software is being tested or surveyed must bevery similar to that in which it will be operated after delivery, and this environmentmust remain uniform during the collection of data. If it changes, or if the software isoperated in a different way (e.g., new functions or capabilities are exercised), thereliability predictions will not be accurate.

7.6.5. Proposed Use of Software Reliability Theory in Nuclear Applications

The use of probability models to forecast software behavior should not be undertakenwithout also applying V&V methods, fault tolerance methods, and QA practices. Thismeans that to license or accept software that is developed by a contractor, goodpredictions from a reliability model should not be a substitute for suitably documentedhistory of software development, including formal reviews, inspections, analyses,testing, characteristics of the development environment, etc.

On the other hand, process oriented standards and QA practices focus on the quality ofthe software development activities. Demonstrating an acceptable process is anecessary approach, but it is not enough. Assessment of the software itself, usingreliability models or other techniques such as software fault injection [5], focuses on theproduct of the development process. This should be applied as a complementaryapproach that is useful to add confidence or improve comprehension of the softwarethat is under analysis.



7-12

The software reliability models have several limitations when assessing software that isdesigned for high integrity levels, as explained in the previous section, but they may bepractical for lower software integrity levels. For critical software, besides using goodprocesses as mentioned above, the best solution is to design software that is verysimple. Complexity has a direct negative impact on software quality, so apply the KISSprinciple (keep it simple, stupid).

To assess probability bounded estimates corresponding to the digital systems includedin a PRA, expert judgment with the aid of software reliability model predictions isprobably better than simply ignoring the digital systems when quantifying fault trees.This methodology may be used to achieve a certain level of confidence that the failureprobability is below a specified upper bound. The statistical and probability values,plus experience and expert judgment, would be complementary for acquiring a morecomplete picture of the software under analysis. For additional discussion on this point,see [6].

7.7. Metrics

According to IEEE 610.12 [10]:

metric. A quantitative measure of the degree to which a system, component, orprocess possesses a given attribute.

The metrics that are of interest with respect to V&V are those related to softwarequality. From IEEE 610.12:

quality metric. (1) A quantitative measure of the degree to which an itempossesses a given quality attribute. (2) A function whose inputs are software dataand whose output is a single numerical value that can be interpreted as thedegree to which the software possesses a given quality attribute.

Metrics numerically characterize simple attributes like length, number of decisions, andnumber of operators (for programs), or number of errors found, cost, and time (forprocesses).

As in other technical disciplines, there is not one unique set of metrics for establishingthe quality of software. Different metrics may be considered, depending on the natureof the problem and the attributes to be measured.

The importance of using metrics for evaluation of digital systems is that a quantitativeassessment of software attributes is usually more objective than a qualitative one.Another indirect benefit is that the process of acquiring useful metric data implies good



7-13

development practices, such as test planning, recording statistical data, storing testresults, maintaining historical documentation, etc.

Improving the software development process can be expected to improve the finalsoftware quality as well. Measuring things is a good starting point for achieving thisgoal. Usually people start controlling whatever they measure; for instance, if weregularly weigh ourselves, we are more likely to try to improve our weight.

There are many metrics with different capabilities for assessment of software attributes.Some of them, especially those related to reliability measurement, are only meaningfulwhen obtained late in the software development life cycle, so they are not useful forearly evaluation of the product. A metric with meaning early in the life cycle is animportant characteristic because corrections are easier to make then rather than later.

It is highly recommended that the application of metrics should be automated usingtools. This reduces the effort to collect data and probably increases the accuracy ofresults.

With proper use of metrics, V&V efforts can be concentrated on the modules orcomponents that are most prone to failure. Some software metrics are listed in Table 7-2. See IEEE 982.1 [11] for additional metrics.



7-14

Table 7-2Some Software Metrics

Name Description Inputs Outputs

Lines of code Measures the size of amodule.

Can be obtained early in thelife cycle.

Useful as a normalizing factor.

Ambiguous because there maybe different ways to calculate.

Usually the number of lines ofprogram statements that arenot comment lines or blanklines

Number of lines ofcode

McCabe’sCyclomaticComplexity

Measures the complexity of amodule’s control flow.

Not related to program size.


From a control flow graph:

E Number of edges

N Number of nodes

P Number of predicatenodes

Cyclomaticcomplexity:C = E – N + 2 = P + 1

Halstead’sSoftwareScienceMeasures

Predicts effort based onsimple measurements.

Sensitive to program size.


n1 Number of distinctoperators

n2 Number of distinctoperands

N1 Total number ofoperators

N2 Total number ofoperands

Program length:N = N1 + N2

Program volume:V = N log2 ( n1 + n2 )

Estimated effort:E = V ( n1N2 / 2n2 )

Estimated length:L = n1 log2 n1

+ n2 log2 n2

Estimated time:T = E / 64800

Estimated bugs:B = V / 3000

ExtendedHalsteadMeasures

Extends Halstead’s measuresto assess, with differentweights, real-time operatorsand operands.

May be used as an indicator ofreliability.

Intended to predict potentialfailures of real-time software.

Same as before, plus:

Number of distinct real-timeoperators and operands

Total number of real-timeoperators and operands

Importance weighting factorsfor the real-time constructs

Same as before, butwith consideration ofreal-time operatorsand operands




7-15

7.8. Tools

The use of tools is desirable in every stage of the development life cycle, especially forthose processes that usually have a higher level of errors when performed manually.Tools can both reduce the effort and increase the accuracy.

Testing, in particular, demands tools (see Section 5.3.2). Methods for describingrequirements and design can benefit significantly from use of tools; for example, theStatecharts method is supported by STATEMATE (see Section 4.2.4.1). Other processesthat can be performed with the aid of tools are syntax checking, structure inspection,path analysis, complexity evaluation, finding memory leaks, requirements compliance,etc.

Software reliability models are more easily applied with the aid of tools, such as:

• AT&T Toolkit (RELTAB, RELPLOT, RELSIM) developed by John Musa of AT&T

• SMERFS (Statistical Modeling and Estimation of Reliability Functions for Software)developed by William Farr of Naval Surface Warfare Center

• SRMP (Software Reliability Modeling Program) developed by Bev Littlewood ofCity University, London

• CASRE (Computer-Aided Software Reliability Estimation Tool) developed by AllenNikora of Jet Propulsion Laboratory and Michael Lyu of Bellcore

• SoRel (Software Reliability) developed by CNRS/LAAS, France

A list of available tools is included in the following EPRI report.

EPRI TR-105989 Vol. 2 App. D

7.9. Licensing Digital I&C Upgrades

EPRI and the Nuclear Management and Resources Council (NUMARC) formed anindustry committee to develop a guideline for licensing digital upgrades to I&Csystems in nuclear plants. The guideline [12] ties together many of the issues discussedin this V&V Handbook and provides a basic approach for licensing digital upgradesthat follows three principles:



7-16

• “The existing licensing process, including 10 CFR 50.59, applies to digital upgrades.” Theguideline goes on to provide guidance in applying 10 CFR 50.59 to digital upgrades,in view of the special characteristics and licensing issues they imply.

• “The issues associated with digital upgrades should be addressed in the context of theirpotential impact on the system being modified.” The importance of an issue (e.g.,software common mode failure) cannot be determined in general. It depends on anychange in event frequency and consequences that is associated with each particularupgrade being considered.

• “This guideline should provide a road map to relevant standards and other guidelines….”For example, the licensing guideline refers to the original EPRI V&V Handbook andto its classification approach for information on how to effectively perform gradedsoftware development and V&V for digital systems.

For information on the regulatory position, see Section 1.4.1. Also, see Section 6.3 foradditional discussion of licensing issues.

7.10. Comparison of EPRI Reports Referring to V&V

EPRI has sponsored various efforts addressing V&V, each of them having specificobjectives and different characteristics; however, some themes overlap, and sometimesthey are developed in different ways. This may cause confusion for the reader.

The following comparison of four such documents (including this one) is intended tofacilitate comprehension of the current set of EPRI reports referring to V&V and toguide use of them in appropriate situations. Links used throughout this V&VHandbook, with graphical “i-buttons” that point to sections in other reports whererelated topics are also developed, help to achieve this same objective and allow thereader to form a more complete understanding of the material that is presented.

Table 7-3 identifies four EPRI documents and indicates the level of discussion forvarious attributes related to V&V. This table is an update of Table 4-1 in [13].


Table 7-3EPRI Documents Concerning V&V of Digital Systems

Title Handbook for V&V of DigitalSystems

Guidelines for the V&V ofExpert System Software andConventional Software

Software Fault ReductionUsing CASE Tools

V&V Guidelines for HighIntegrity Systems

Date December 1998 April 1995 June 1995 December 1995

EPRI Number This Document TR-103331 TR-105989 TR-103916

NUREG Number CR-6316 CR-6293

Scope Digital systems from real-time monitoring and controlto off-line diagnosticapplications. Emphasis onhigh integrity systems.

Expert software systems,frame-based systems, objectoriented systems, andconventional softwaresystems.

All software systems,conventional systems, objectoriented systems, andknowledge based systems.

High integrity softwaresystems, including all safetyand mitigation systems fornuclear power plants.

Strengths Provides a quick overview ofV&V with attention to digitalI&C systems for nuclearplants. Illustrative casestudies are supplied.Includes a practical approachto select graded V&Vmethods.

Prescribes V&V methods forparticular situations.Addresses conventional aswell as expert systems.Provides a thoroughevaluation of the subject.

Defines a useful analyticalmethod to prevent faultsduring the life cycle and toidentify situations that needV&V emphasis. Includesthorough evaluation ofsupporting tools. Bothconventional and expertsystems are addressed.

Describes metrics forquantitative assessment ofsoftware attributes. Presentsuseful information ontesting, fault classification,and applicable standards.

Weaknesses For some items, thediscussion is general.

Presentation is repetitious.Prescribed V&V methodsmay be excessive for projectswith limited resources.Guideline packages arepresented in a cumbersomeformat with manytypographical errors andextensive references tosupporting literature thatmust be available forcomplete application.

Approach may provedifficult, especially analysisof artifacts and developmentof special V&V plans forartifacts with high risk.Several editing errors.

Does not identify particularV&V methods for specificsituations.


Title Handbook for V&V ofDigital Systems

Guidelines for the V&V ofExpert System Softwareand ConventionalSoftware

Software Fault ReductionUsing CASE Tools

V&V Guidelines for HighIntegrity Systems

Stringency Classification 999 999 999 999

Graded Approach 999 999 99

Planning 99 9 99 99

Life Cycle 999 999 999 9

Methods 999 999 999 99

Reviews and Inspections 999 99 99 99

Analytical Techniques 999 99 999 99

Metrics 99 99 9 999

Tools 99 99 999 99

Testing 999 999 99 999

Fault Classification 9 999 999 999

Commercial Dedication 9 9

Configuration Management 9 9 9

Standards 999 99 9 999

Regulatory Issues 99 9

I&C Systems 99 99

Expert Systems 999 99

Probabilistic approach 99

Examples 999 99 99 9

Glossary 99 9 99 99

Bibliography 99 999 99 99

Key9 Mentioned99 Discussed999 Developed



7-19

7.11. References

1. ASME NQA-1-1989, Quality Assurance Program Requirements for Nuclear Facilities.

2. IEEE Std 603-1991, Standard Criteria for Safety Systems for Nuclear Power GeneratingStations.

3. EPRI TR-102400, Handbook for Electromagnetic Compatibility of Digital Equipment, 1994.

4. EPRI TR-102323, Guidelines for Electromagnetic Interference Testing in Power Plants,September 1994.

5. Voas, J., and G. McGraw, Software Fault Injection — Inoculating Programs AgainstErrors, Wiley, 1997.

6. National Research Council, Digital Instrumentation and Control Systems in NuclearPower Plants — Safety and Reliability Issues, Sections 5 and 6, National AcademyPress, 1997.

7. NUREG-0800, US NRC Standard Review Plan, Section 7: Instrumentation and Controls,Rev. 4, 1997.

8. NUREG-0493, A Defense-in-Depth and Diversity Assessment of the RESAR-414Integrated Protection System, March 1979.

9. Lyu, M. (ed.), Handbook of Software Reliability Engineering, McGraw-Hill, 1996.


11. IEEE Std 982.1-1988, Standard Dictionary of Measures to Produce Reliable Software.

12. EPRI TR-102348, Guideline on Licensing Digital Upgrades, December 1993.

13. EPRI TR-107980, I&C Upgrades for Nuclear Plant — Desk Reference, December 1997.


8-1

8 ABSTRACTS OF DIGITAL I&C CASE STUDIES

Volume 2 of the original EPRI V&V Handbook [1] includes four case studies thatprovide detailed information about the design and V&V processes for digital upgradesto instrumentation and control (I&C) systems at various nuclear power stations.20 Briefabstracts of those case studies are presented in this section. For each case study, thefollowing six characteristics are discussed:

1. Description

2. Safety Significance

3. Key Technical Issues

4. Classification (with respect to Table 6-1)

5. V&V Organization

6. V&V Plan

The reader will probably notice that the life cycles and practices from these case studiesmay not closely match the reference life cycle and recommended activities presented inthe other sections of this V&V Handbook. This partly reflects the reality thatengineering judgment has been and should always be used to determine the detailedactivities for each project. In addition, both applicable standards and engineeringpractices are evolving, so some of the activities described in this V&V Handbook wereneither anticipated nor technically feasible just a few years ago.

20 EPRI currently intends to continue availability of Volume 2 of the original V&V Handbook, but Volumes 1 and3 are replaced by the updated V&V Handbook (this document).


Abstracts of Digital I&C Case Studies

8-2

8.1. Case #1Westinghouse Eagle 21 Replacement Hardware for Reactor ProtectionSystems

8.1.1. Description

The Westinghouse Eagle 21 Process Protection System product line consists ofmicroprocessor-based equipment as a digital form, fit, and functional replacement forracks of the original analog process protection system. It initiates reactor trip andactuates engineered safety features, in addition to supplying isolated outputs to otherplant systems. Eagle 21 equipment provides enhanced features for automaticsurveillance testing, self-calibration, and self-diagnostics.

The Eagle 21 Process Protection System is designed to provide three or four instrumentchannels and two trip logic trains for each protective function. These redundantchannels and trains are electrically isolated and physically separated. Eagle 21functions independently from plant control systems, so that its ability to protect theplant is not affected by any fault or malfunction of the control systems. Any onechannel can be maintained in a bypassed condition and, when necessary, tested duringpower operation without initiating protective action at the system level.

8.1.2. Safety Significance

The reactor protection system is relied on to shut down the reactor during design basisevents.

8.1.3. Key Technical Issues

The principal issue is the performance of safety-critical functions during short timeperiods.

Reliability — Because of the high level of reliability and independent scrutinydemanded for safety-critical functions, a derived issue is the transparency toindependent review by vendor, utility, and NRC personnel. A key part of the vendor’sapproach to the transparency issue was to develop strict coding standards that requirethe designers to build source code that is relatively simple and easily understood. Asecond aspect was emphasis on formal traceability throughout the design process. Afunctional decomposition elaborates on the intent of the functional requirements. Inaddition, a system design matrix is used to confirm that each requirement is reflected inthe design and is tested.



8-3

Modularity — A high level of modularity in design and application is necessary for thecommercial viability of the Eagle 21 Reactor Protection System upgrade. Themodularity issue arises because the basic software and hardware components are to beavailable for use in future applications for which they will not be re-verified. Rather,only the integrated system will be validated. To provide the high level of confidence tosupport this approach, the vendor has placed considerable emphasis on unit testing oflow-level software modules. In addition to functional testing, the technique ofstructural testing is used to exercise all possible code branches within a module.

Maintainability — A major motivation to upgrade I&C systems is to enhancemaintainability by including on-line surveillance capabilities. The maintainability ofthis replacement digital hardware was a key issue in its design process. The majorimpact on V&V of surveillance test functions for maintainability is to increase thecomplexity of the system and, correspondingly, the size of the V&V effort.

8.1.4. Classification

The Eagle 21 Reactor Protection System upgrade is a Category A safety-criticalapplication of moderate complexity requiring the highest level of V&V. The product istypically procured as a turnkey system with its software embedded in dedicatedsystems in each of four independent channels. The system includes application code aswell as a simple, once-through executive, and it is programmed in a high-levelprocedural language. Each independent channel of the system sends a signal to theexisting relay logic, which provides fault tolerance through voting on the outputs of theEagle 21 channels.

8.1.5. V&V Organization

V&V is performed by a separate Westinghouse V&V group whose management isindependent from that of the software design group. Several types of review teams areused at different stages of the V&V effort. A Multidisciplinary Review Team is formedfor high-level review at the requirements stage. For review and testing of the detaileddesign and coding, a more specialized Three-Party Review Team is formed, consistingof the designer, a second engineer from the design group who is not involved with thework, and a technical staff member from the V&V group. For verification andvalidation testing activities, the verifiers design the software tests, conduct them on adevelopment computer, and communicate the results in writing to the developmentorganization.



8-4

8.1.6. V&V Plan

Westinghouse developed a V&V Plan based on the requirements of IEEE Std 7-4.3.2-1982. Three life cycle stages were identified: definition, design, andimplementation/test. The plan describes design documents and V&V reviews and teststhat must be performed prior to completion of each stage. Formal reviews by anindependent V&V team are prescribed for design documentation, source code, andfunctional tests. All software modules are subject to functional (black-box) testing toconfirm that all required functions, but no unintended functions, have beenimplemented. In addition, for safety-related software, structural testing of the softwareinternals is required. (In practice, Westinghouse treated all Eagle 21 software as safety-related.)

The emphasis of the V&V Plan is clearly on verification review and testing activities tobe performed throughout the life cycle. Because of the expense and project riskassociated with correcting problems discovered late in the life cycle, the intention is notto rely on validation testing to identify problems.

The validation testing effort includes top-down testing of functional requirements, aprudency review of the design and its implementation, and man-machine interfacetesting. The prudency review goes beyond current standards to probe for theboundaries of effective system operation and to provide a final critical review of thesystem design.

The V&V Plan also defines the roles of the design team (i.e., chief programmer andprogrammers) and the V&V team members in the design and V&V process.

8.2. Case #2Arizona Public Service Palo Verde Diverse Auxiliary FeedwaterActuation System

8.2.1. Description

The Arizona Public Service (APS) Palo Verde Diverse Auxiliary Feedwater ActuationSystem (DAFAS) is designed to mitigate the consequences of an Anticipated TransientWithout Scram (ATWS). An ATWS event occurs when an operational transient (e.g.,loss of feedwater, loss of condenser vacuum, or loss of off-site power) is accompaniedby failure of the reactor protection system to shut down the reactor. To achievediversity, the technology and hardware are distinct from those of the reactor trip andauxiliary feedwater actuation systems.

For each unit at Palo Verde, the two steam generators are assured of adequatefeedwater by two redundant DAFAS trains. Each DAFAS train is included in a separate



8-5

cabinet located in the control room behind the operator’s main control panel. EachDAFAS train includes two redundant programmable logic controllers (PLCs). Selective2-out-of-4 logic is used to assure actuation when required and to prevent unintendedoperation. DAFAS is included for diversity; therefore, it is blocked if auxiliaryfeedwater actuation is successfully performed by the Plant Protection System (PPS) andthe Auxiliary Feedwater Actuation System (AFAS). Note that DAFAS provides onlydiverse logic; analog sensors and final actuation devices are shared with other systems,as represented by the ATWS requirements.

The contract for DAFAS was issued by APS to Combustion Engineering (CE, now ABB-CE) in 1990. The system includes AEG Modicon PLCs, Lightwave Communicationsfiber-optic communication equipment, and a Xycom electro-luminescent flat panelindustrial terminal man-machine interface (MMI). DAFAS was installed at Palo Verdeduring 1992. It was implemented within budget and schedule, and it met all ATWS-related regulatory commitments.


The DAFAS was designed to satisfy the “ATWS Rule” 10 CFR 50.62, Requirements forReduction of Risk from Anticipated Transients Without Scram (ATWS) Events for Light-Water-Cooled Nuclear Power Plants, and Generic Letter 85-06, QA Guidance on ATWSEquipment That Is Not Safety Related. The ATWS Rule requires specific design andoperational improvements to reduce the likelihood of a failure to shut down the reactorfollowing anticipated transients and to mitigate the consequences of an ATWS event.Although not so required by the ATWS Rule, Palo Verde set out to design the system toClass 1E standards in order to ensure a highly reliable system that would notcompromise availability or challenge safety systems by spurious actuation.21


Commercial Grade Equipment — To simplify the design effort, standard commercialprogrammable logic controller (PLC), man-machine interface (MMI), and fiber-opticcommunication equipment was specified.

Micro-984 PLCs were selected from AEG Modicon for their experience base,cooperation, design control, QA procedures, willingness to support equipment for atleast 15 years, and agreement to freeze the revision level for purchase of spare parts.PLC system software manages the processor and handles analog and discreteinput/output, communications, and self-testing. It was purchased as a standard

21 The NRC did not endorse the DAFAS as a Class 1E system; however, DAFAS did satisfy the ATWS rule, whichdoes not require Class 1E.



8-6

commercial product contained in programmable read-only memory (PROM) integratedcircuits (firmware). ABB-CE performed commercial dedication of the Modiconequipment and system software (firmware), with review by APS.

The man-machine interface (MMI) is non-interactive; it is used for status and diagnosticdisplay. The Xycom Operator Interface Language (OIL) implementation was not subjectto specific V&V, since it was not considered part of the safety system. However, theMMI was validated as part of the overall design during system testing.

Similarly, the Lightwave Communications fiber-optic equipment was validated as partof the overall design during system testing.

Diversity — The DAFAS was designed with two channels to increase reliability,decrease the probability of spurious actuation, and permit testing at full power.Diversity between the DAFAS and the Reactor Protection System (RPS) provides anadequate level of common mode failure protection.

Reliability — Application software is deterministic (repetitive, not event driven). Relayladder logic (RLL) programs are used to implement DAFAS functions on the PLCs in amodular fashion. These nuclear safety-related programs are stored in PROM and maynot be changed by the operator.

In addition to detailed application software V&V, the system was tested for seismic,environmental, and EMI qualification to the standards expected of a Class 1E system.

Maintainability — Each PLC performs continuous self-testing to verify RAM writingand reading, the validity of configuration data in RAM, and correct operation ofcommunications links. Power-on startup tests also verify processor function, registeroperation, input/output buffer operation, PROM contents, configuration data in RAM,and communication links.

Although the DAFAS was designed to minimize inadvertent actuation, it is necessaryto assume that it will fail in a mode that will result in actuation at the system level. Thiswould result in increased inventory in the secondary side of the steam generator.Although this condition has been considered in analysis of the plant, it is undesirable.The addition of another auxiliary feedwater initiation system (DAFAS added to AFAS)increases the probability of this condition. However, the Safety Evaluation Report (SER)concluded that the number of system interlocks provides sufficient protection and thatthe design is acceptable.

The DAFAS can be tested on-line and off-line. On-line testing is performed one channelat a time and is manually initiated. It involves verifying proper operation of the DAFAScircuits and initiation relays. Off-line functional testing and calibration were performed



8-7

before operation to verify and validate that the hardware and software conform to thedesign specification.


The DAFAS was designed to be a safety-related actuation system (Category B) ofmoderate complexity using standard commercial programmable logic controller (PLC)equipment. New application software (firmware) was developed under contract usingvery-high-level programming languages, including relay ladder logic (RLL).


Development of the DAFAS was performed by ABB-CE under contract to APS. ABB-CE applied their QA procedures, which required an independent review of all safetysystem projects. Since the DAFAS project involved software development, the QAprocedures required forming a team to perform software V&V review, which wasseparate from and in addition to the activity of the independent reviewer. There wassome difficulty with overlapping responsibilities between the independent reviewerand the V&V team.

The V&V team was required to have expertise comparable to the design team andorganizational independence with respect to the design team. Independence wasaccomplished by having a separate line manager. There was no need for a separateV&V organization. The V&V team performed an audit of documents produced at eachphase of the software life cycle. In addition to reviewing documents, the V&V teamwitnessed testing. The V&V team did not specify or perform testing for the DAFASproject.

8.2.6. V&V Plan

The V&V Plan addressed the following issues:

• Verification of hardware requirements

• Verification of software requirements

• Verification of application software changes

• Verification of previously developed software (firmware)

• Verification of software implementation

• Verification of hardware/software integration



8-8

• Validation testing

• Special hardware validation with respect to seismic and environmental qualification

Reporting of V&V results and problem resolution was specified.

8.3. Case #3FPL Emergency Bus Load Sequencer

8.3.1. Description

The Florida Power & Light (FPL) Turkey Point Emergency Bus Load Sequencer isdesigned to perform two functions bus stripping and bus loading. Each nuclear unitat Turkey Point (Units 3 and 4) has two Emergency Bus Load Sequencers, serving twoEmergency Diesel Generators (EDGs) in a redundant configuration. Each loadsequencer utilizes an Allen-Bradley PLC-2 programmable logic controller plus relayssupplied by ASEA. All inputs and outputs are discrete (relay) signals; there are noanalog signals.

The load sequencers are provided with manual test and automatic self-test capabilities.If the redundant configuration should fail, it is possible to disable the system andperform its functions manually from the switch gear room. The man-machine interfaceconsists of annunciators in the control room with details indicated by status lights on anoperator’s switch panel and diagnostic lights on PLC input/output cards in the loadsequencer cabinet.

The digital load sequencer systems were intended to duplicate the functions of theoriginal analog equipment, with some new features for sequence timing and self-testingfeatures. They were installed during a two-unit outage in 1990 and 1991. The systemsfunctioned well during Hurricane Andrew in the summer of 1992; there was no off-sitepower for seven days following the hurricane.


The Emergency Bus Load Sequencer was designed as a safety-related Class 1E system.


Commercial Grade Equipment — To simplify the design effort, standard commercialprogrammable logic controller (PLC) equipment was specified. The Allen-Bradley PLC-2 was chosen based on years of operational experience in industrial facilities.



8-9

PLC system software manages the processor and handles discrete input/output andinternal diagnostics. It was purchased as a standard commercial product contained inprogrammable read-only memory (PROM) integrated circuits (firmware). Commercialdedication of the Allen-Bradley equipment and system software (firmware) wasperformed by an audit team including one representative each from United Controls,Inc. (contractor and team leader), FPL, Combustion Engineering, and CardinalIndustrial Products.

Redundancy — The Emergency Bus Load Sequencer system consists of two redundanttrains, where each train includes one load sequencer and one Emergency DieselGenerator (EDG). The logic in each train is similar, so this configuration does notprovide diversity. In the unlikely event of common mode failure, the systems may bedisabled and operated manually.

Reliability — Application software is deterministic (i.e., repetitive, not event driven).Ladder logic programs are used to implement load sequencer functions on the PLCs ina modular fashion. These nuclear safety-related programs are stored in PROM and maynot be changed by the operator. It is possible to make configuration changes using aportable programming terminal with procedural, keylock, and password securitycontrols.

Maintainability — Each load sequencer design includes manual and automatic on-linetest functions. Manual testing is separated into two independent tasks bus strippingand bus loading — to prevent inadvertent operation of equipment in the event of testmalfunction. The system continually and automatically tests process logic and discreteoutput signals, without actually operating any output devices. Testing willimmediately terminate if a valid input signal is received.22 If a malfunction is detectedduring testing, an annunciator will alert the control room.

Configuration Control — The PLC program has twelve files, which are periodicallycompared with known master files using a portable programming terminal and manualprocedures. The programs are the same for each of the four load sequencers. Masterfiles are maintained on disks, which are stored in a magnetic protective safe.

Coding Conventions — Standard Allen-Bradley PLC-2 on-line/off-line ProgramDevelopment and Documentation Software was used for programming, documenting,trouble-shooting, and printing ladder logic programs. The programs were divided intothree major functions process, manual test, and automatic test. Logic was developed

22 As described in [2], this automatic test feature failed on November 3, 1994, because of faulty design and imple-mentation. In its review, the NRC concluded with respect to V&V, “The plan was weak in that it relied almostcompletely on testing as the V&V methodology. More emphasis on the analysis of the requirements and designwould have increased the likelihood of discovering the design flaw.”



8-10

in segments, where each segment contained input statements, logic operations, outputstatements, and sometimes intermediate statements. Output from one segment mayserve as input to another segment.


The Emergency Bus Load Sequencer system was designed to be a safety-criticalactuation system (Category B) of low complexity using standard commercialprogrammable logic controller (PLC) equipment. New application software (firmware)was developed under contract using ladder logic, a very-high-level programminglanguage.


The original Emergency Bus Load Sequencer specification did not address V&V. NRCreview of FPL’s design report lead to a Request for Additional Information (RAI)regarding V&V and other issues. A V&V Plan, along with related documents, wasprepared and transmitted to NRC as part of FPL’s response to the RAI.

The V&V Plan specified that such documents would be reviewed and other verificationand validation would be performed by “individuals independent of the developmentorganization with skills similar to those individuals performing the development[emphasis added].” It further specified that V&V activities would be auditable,problems would be immediately resolved, and written records would be included inthe final V&V Report. Also to be included in the V&V Report was a requirementstraceability matrix identifying where each system function was “defined, documented,implemented, and tested.”

The documents available for this case study gave no indication of how manyindividuals participated in V&V, their level of independence, nor their qualifications.The available documentation also gave no examples of how verifiers used checklistsand certification forms for commercial dedication.

8.3.6. V&V Plan

The Emergency Bus Load Sequencer V&V Plan was based on the following principlesof effective software development:

• Well defined system requirements

• Comprehensive software development methodology

• Comprehensive testing procedures



8-11

• Independence of the V&V reviewer with respect to the development organization

The V&V Plan addressed the following issues:

• Verification of hardware requirements

• Verification of software requirements

• Verification of application software

• Verification (by commercial dedication audit) of previously developed software

• Verification of software implementation

• Verification of hardware/software integration

• Validation testing

• Special hardware validation with respect to seismic and environmental qualification

• Review and audit procedures, including reporting

• Content of the V&V report, including requirements traceability matrix anddocumentation requirements

8.4. Case #4B&W Owners Group Advanced Control System

8.4.1. Background

In the late 1980’s, as part of a trip reduction program, the B&W Owners’ Group(BWOG) began to consider alternatives for improving the reliability and performanceof the Integrated Control System. After extensive study, the BWOG decided to launch amajor effort to completely redesign the control strategy and to implement it usingmodern fault-tolerant digital technology. The BWOG commissioned an AdvancedControl System Task Force (ACSTF) to provide technical direction, and in 1991, EPRIjoined with the ACSTF in a tailored collaboration for funding and management of thisproject. Babcock Wilcox Nuclear Technologies (BWNT) served as the system integrator,and several contractors and equipment vendors eventually became involved in the ACSproject.



8-12

8.4.2. Description

The overall concept for achieving high reliability and, in particular, insensitivity to thefailure of individual signals and digital modules relies primarily on fault tolerance, asopposed to fault detection. Three parallel signals and computations of the algorithm areperformed, and the resulting output signals are voted before being sent to drive plantactuators. The ACS architecture employing triplication and voting is a first-of-a-kindsynthesis of standard Foxboro Intelligent Automation (I/A) channels with a highreliability downstream voter manufactured by Triconex.

The objectives of the ACS advanced control algorithm are to avoid trips for a widerange of transient events and to increase operational flexibility by removingunnecessary conservatism from operating margins. Meeting these objectives requiresthe tight coordination of controlled responses in order to maintain energy and massbalances during transient events, and thus to stabilize key temperatures and pressures.This tight coordination of several systems requires the use of both feedback logic andfeed-forward logic. The feed-forward logic incorporates models of plant dynamics andrelies on a static energy balance to compute target values for individual controllers.


While the Plant Control System is not a safety system, it is essential for plantavailability, and its failure or degraded operation has potential for challenging safetysystems.


Complexity of Plant Response — The underlying challenge in designing the PCS, incomparison with other nuclear plant control systems, is the highly coupled and fastresponse to transients exhibited by B&W plants.

Operator Interface — Out of concern for acceptance of the ACS by plant operators, theACSTF took an extremely conservative point of view and specified several levels ofinterfaces. The most comprehensive interface is based on a pair of color monitors withthe ability to select displays showing the status of plant process variables as well asdiagnostics and internal details of the ACS itself.

Need for High Fidelity Testing — The first two issues (complex plant response andnew operator interface) taken together implied the need for testing on a high-fidelity,full-plant training simulator, so that operators can simultaneously test the effectivenessof the control logic and the interface to provide suitable plant control.



8-13

Complexity of the Digital Control System — The ACS is highly complex incomparison to other digital upgrades in domestic nuclear power plants. Thefunctionality is complex, because the control algorithm is necessarily intricate to obtainthe performance required from the B&W plant. In addition, the implementation of thisfunctionality is complex because of the need for redundant hardware to obtain highreliability and because multiple control processors are needed to handle thecomputational burden. The basic approach to the complexity issue is to add rigor to thedesign and V&V processes in the more complex and important software areas.

Time Delays — The time delays introduced by control processing and buscommunications in the digital architecture turned out to be a major technical barrier toachieving stable, high-performance control. These time delays were significant enoughthat they invalidated time response assumptions made in the algorithm prototypetesting and destabilized the control algorithm, so that the algorithm had to be re-tuned.

Commercial Dedication — A project decision was made early on to employ acommercial distributed control system (DCS) with a built-in library of softwarefunctions, so that the control algorithm could be synthesized from previously verifiedsoftware elements.


Although it is not a safety system, it is crucial for plant availability. Therefore, the ACSsystem is categorized as Important to Operations (Category C).


Reviews of requirements and control algorithm concept were provided by a PeerReview Group (PRG) consisting of representatives from the participating utilities. ThePRG included Senior Reactor Operators (SROs) who assessed the impact of controlfunction changes and interface changes on plant operations and training.

During the simulator validation testing period, a huge volume of test results and nearly400 change requests to the ACS were generated. Given the pace of testing and changes,it was impractical for a large, geographically dispersed committee to perform detailed,timely reviews, so an independent V&V Team was assembled to perform review taskson behalf of the ACSTF.

During the several steps of translation of the algorithm from the ACSL language toSAMA diagrams and (formally) to Foxboro functional diagrams, there was verificationof each step. One knowledgeable simulation and control engineer from BWNT was“held back” from doing any design work, specifically so that he could provideindependent verification of each translation.



8-14

8.4.6. V&V Plan

The V&V Plan closely follows the guidance of IEEE 1012 and this V&V Handbook. Thelife cycle V&V discussion, which contains much of the substantive direction, includes aseparate sub-project life cycle for the effort to upgrade and validate the simulator as aPCS test facility; however, the present case study summarizes life cycle activities onlyfor the mainstream ACS itself. It should be mentioned that the formal V&V Plan wasprepared midway through the project when it was recognized that the design andquality processes should be linked to important standards.

8.5. References


2. National Research Council, Digital Instrumentation and Control Systems in NuclearPower Plants — Safety and Reliability Issues, pp. 39–40, National Academy Press, 1997.


9-1

9 COMBINED LIST OF REFERENCES

This section combines individual references from all previous sections into analphabetized list.

Alford, M., SREM at the Age of Eight; The Distributed Computed Design System, Computer,pp. 36–46, 1985.

ANS-58.14, Safety and Pressure Integrity Classification Criteria for Light Water Reactors,1993.

ASME NQA-1-1989, Quality Assurance Program Requirements for Nuclear Facilities.

ASME NQA-2a-1990, Part 2.7, Quality Assurance Requirements of Computer Software forNuclear Facility Applications.

Barnes, M., P. Bishop, B. Bjarland, G. Dahll, D. Esp, J. Lahti, H. Valisuo, and P.Humphreys, STEM — A Project on Software Test and Evaluation Methods, Proceedings ofthe Safety and Reliability Society Symposium, Manchester UK, November 1987.

Beizer, B., Software Testing Techniques, Van Nostrand Reinhold, 1990.

Blanchard, B. S., and W. J. Fabrycky, Systems Engineering and Analysis, Prentice Hall,1990.

Boehm, B. W., A Spiral Model of Software Development and Enhancement, IEEE Computer,Vol. 21, No. 5, pp. 61–72, May 1988.

Boehm, B. W., et al., Some Experience with Automated Aids to the Design of Large-ScaleReliable Software, IEEE Transactions on Software Engineering, Vol. 1, pp. 125–133,March 1975.

Bryan, W. L., and S. G. Siegel, Software Product Assurance: Techniques for ReducingSoftware Risk, Elsevier, 1988.

Coad, P., and E. Yourdon, Object-Oriented Analysis, Yourdon Press, 1991.

Davis, A. M., Software Requirements, Prentice Hall, 1993.


Combined List of References

9-2

Delisle, N., and D. Garfan, A Formal Specification of an Oscilloscope, IEEE SoftwareSpecial Issue on Formal Methods, pp. 29–36, September 1990.

DeMarco, T., Structured Analysis and System Specification, Prentice Hall, 1978.

DO-178B/ED-12B, Software Considerations in Airborne Systems and Equipment Certification,FAA, 1992.

DOD-STD-2167A (superseded by MIL-STD-498), Defense System Software Development.

DRIC-BR-115966, Translating Data Flow Diagrams into Z (and Vice Versa), Royal RadarEstablishment, United Kingdom, October 1990.

Dyer, M., The Cleanroom Approach to Quality Software Development, Wiley, 1992.

EPRI TR-102323, Guidelines for Electromagnetic Interference Testing in Power Plants,September 1994.

EPRI TR-102348, Guideline on Licensing Digital Upgrades, December 1993.

EPRI TR-102400, Handbook for Electromagnetic Compatibility of Digital Equipment, 1994.

EPRI TR-103291, Handbook for Verification and Validation of Digital Systems, Vol. 2: CaseHistories, December 1994.

EPRI TR-103331, Guidelines for the Verification and Validation of Expert System Software andConventional Software, Vol. 2: Survey and Assessment of Conventional Software Verificationand Validation Methods (Revision 1), Vol. 6: Validation Scenarios, Vol. 7: User’s Manual,May 1995.

EPRI TR-103916, Verification and Validation Guidelines for High Integrity Systems,December 1995.

EPRI TR-104595, Abnormal Conditions and Events Analysis for Instrumentation and ControlSystems, January 1996.

EPRI TR-105989, Software Fault Reduction Using Computer-Aided Software Engineering(CASE) Tools, June 1995.

EPRI TR-106439, Guideline on Evaluation and Acceptance of Commercial Grade DigitalEquipment for Nuclear Safety Applications, October 1996.

EPRI TR-107980, I&C Upgrades for Nuclear Plant — Desk Reference, December 1997.



9-3

EPRI TR-108831, Requirements Engineering for Digital Upgrades: Specification, Analysis,Tracking, December 1997.

Forte, G., Tools Fair: Out of the Lab, Onto the Shelf, IEEE Software, May 1992.

Guideline for Categorization of Software in Ontario Hydro’s Nuclear Facilities with Respect toNuclear Safety, Ontario Hydro, June 1991.

Harel, D., Statecharts: A Visual Formalism for Complex Systems, Scientific ComputerProgramming, Vol. 8, pp. 231–274, 1987.

Harrison, R., Effective Techniques for Software Testing Seminar Workbook, Data-TechInstitute, 1992.

Hatley, D., and I. Pirbhai, Strategies for Real-Time System Specification, Dorset House,1987.

Howden, W. E., Functional Program Testing, IEEE Transactions on SoftwareEngineering, Vol. SE-6, No. 2, pp. 162–169, March 1980.

Ichiyen, N. M., V&V Experience — AECL/Ontario Hydro, Presentation to EPRI/UtilityWorking Group Meeting on V&V and Related Issues, Chattanooga, TN, May 4–5, 1993.

IEC 880, Software for Computers in the Safety Systems of Nuclear Power Stations, 1986.

IEC 987, Programmed Digital Computers Important to Safety for Nuclear Power Stations,1989.

IEC 1226, Nuclear Power Plants — Instrumentation and Control Systems Important for Safety— Classification, 1993.

IEC 61508, Functional Safety of Electrical/Electronic/Programmable Electronic Safety-RelatedSystems, Draft, 1998.

IEEE Std 7-4.3.2-1993, Standard Criteria for Digital Computers in Safety Systems of NuclearPower Generating Stations.

IEEE Std 603-1991, Standard Criteria for Safety Systems for Nuclear Power GeneratingStations.

IEEE Std 610.12-1990, Standard Glossary of Software Engineering Terminology.

IEEE Std 730-1989, Standard for Software Quality Assurance Plans.

IEEE Std 828-1990, Standard for Software Configuration Management Plans.



9-4

IEEE Std 829-1983, Standard for Software Test Documentation.

IEEE Std 830-1993, Recommended Practice for Software Requirements Specifications.

IEEE Std 982.1-1988, Standard Dictionary of Measures to Produce Reliable Software.

IEEE Std 1008-1987, Standard for Software Unit Testing.

IEEE Std 1012-1998, Standard for Software Verification and Validation.

IEEE Std 1016-1987, Recommended Practice for Software Design Descriptions.

IEEE Std 1016.1-1993, Guide to Software Design Descriptions.

IEEE Std 1028-1997, Standard for Software Reviews.

IEEE Std 1042-1987, Guide to Software Configuration Management.

IEEE Std 1044-1993, Standard Classification for Software Anomalies.

IEEE Std 1044.1-1995, Guide to Classification for Software Anomalies.

IEEE Std 1059-1993, Guide for Software Verification and Validation Plans.

Joannou, P. K., et al., Software Requirements Specification for Critical Applications,Proceedings: Methodologies, Tools, and Standards for Cost-Effective, Reliable SoftwareVerification and Validation, EPRI TR-100294, January 1992.

Jones, C. B., Systematic Software Development using VDM, Prentice Hall,1990.

Leveson, N. G., Safeware: System Safety and Computers, Addison-Wesley, 1995.

Lyu, M. (ed.), Handbook of Software Reliability Engineering, McGraw-Hill, 1996.

Mahony, B. P., and I. J. Hayes, A Case-Study in Timed Refinement: A Mine Pump, IEEETransactions on Software Engineering, Vol. 18, No. 9, pp. 817–825, September 1992.

Marketing Literature, Requirements Driven Developer, Ascent Logic Corporation,December 1989.

Marketing Literature, The STATEMATE Approach to Complex Systems, i-LogixCorporation.



9-5

May, R. S., Formal Methods and Tools for Nuclear Power Plant Control Systems: AnExploratory Study, Report to EPRI Nuclear Power Division Exploratory ResearchCommittee, August 1993.

Miller, L. A., Dynamic Testing of Knowledge Bases Using the Heuristic Testing Approach,Expert Systems with Applications: An International Journal, Special Issue: Verificationand Validation of Knowledge-Based Systems, Vol. 1, No. 3, pp. 249–269, 1990.

MOD 00-55, Requirements for Safety Related Software in Defence Equipment, Part 1:Requirements, Part 2: Guidance, UK Defence Standard 00-55/Issue 2, 1 August 1997.

MOD 00-56, Safety Management Requirements for Defence Systems, Part 1: Requirements,Part 2: Guidance, UK Defence Standard 00-56/Issue 2, 13 December 1996.

Myers, G. J., The Art of Software Testing, Wiley, 1979.

National Research Council, Digital Instrumentation and Control Systems in Nuclear PowerPlants — Safety and Reliability Issues, National Academy Press, 1997.

NBS SP 500-93, Software Validation, Verification, and Testing Technique and Tool ReferenceGuide, National Bureau of Standards, September 1982.

Ng, P., and R. Yeh (ed.), Modern Software Engineering: Foundations and CurrentPerspectives, Van Nostrand Reinhold, 1990.

Nicholl, R. A., Unreachable States in Model-Oriented Specifications, IEEE Transactions onSoftware Engineering, Vol. 16, No. 4, April 1990.

NSAC-125, Guidelines for 10 CFR 50.59 Safety Evaluations, June 1989.

NUREG-0493, A Defense-in-Depth and Diversity Assessment of the RESAR-414 IntegratedProtection System, March 1979.

NUREG-0800, US NRC Standard Review Plan, Section 7: Instrumentation and Controls,Rev. 4, 1997.

NUREG/CR-4227, Human Engineering Guidelines for the Evaluation and Assessment ofVideo Display Units, July 1985.

NUREG/CR-4640, Handbook of Software Quality Assurance Techniques Applicable to theNuclear Industry, August 1987.

NUREG/CR-5930, High Integrity Software Standards and Guidelines, September 1992.

Page-Jones, M., Practical Guide to Structured Systems Design, Yourdon Press, 1988.



9-6

Parnas, D. L., et al., Assessment of Safety-Critical Software in Nuclear Power Plants, NuclearSafety, Vol. 32, No. 2, pp. 189–198, April–June 1991.

Potter, B., et al., Introduction to Formal Specification and Z, Prentice Hall, 1991.

Pressman, R. S., Software Engineering: A Practitioner’s Approach, McGraw-Hill, 1992.

Rasmussen, J. and K. J. Vicente, Cognitive Control of Human Activities and Errors:Implications for Ecological Interface Design, Presented at the 4th International Conferenceon Event Perception and Action, Trieste, Italy, August 24–28, 1987.

SAMA Std PMC 22.1-1981, Functional Diagramming of Instrument and Control Systems.

Schneidewind, N., Reliability Modeling for Safety-Critical Software, IEEE Transactions onReliability, Vol. 46, No.1, March 1997.

Sommerville, I., Software Design for Reliability, Software Reliability Handbook, P. Rook,Ed., pp. 21–50, Elsevier, 1990.

SRI-CSL-91-02, An Introduction to Formal Specification and Verification Using EHDM, SRIInternational, February 1991.

US NRC Regulatory Guide 1.152, Criteria for Digital Computers in Safety Systems ofNuclear Power Plants, Rev. 1, January 1996.

US NRC Regulatory Guide 1.168, Verification, Validation, Reviews, and Audits for DigitalComputer Software Used in Safety Systems of Nuclear Power Plants, September 1997.

US NRC Regulatory Guide 1.169, Configuration Management Plans for Digital ComputerSoftware Used in Safety Systems of Nuclear Power Plants, September 1997.

US NRC Regulatory Guide 1.170, Software Test Documentation for Digital ComputerSoftware Used in Safety Systems of Nuclear Power Plants, September 1997.

US NRC Regulatory Guide 1.171, Software Unit Testing for Digital Computer Software Usedin Safety Systems of Nuclear Power Plants, September 1997.

US NRC Regulatory Guide 1.172, Software Requirements Specifications for Digital ComputerSoftware Used in Safety Systems of Nuclear Power Plants, September 1997.

US NRC Regulatory Guide 1.173, Developing Software Life Cycle Processes for DigitalComputer Software Used in Safety Systems of Nuclear Power Plants, September 1997.

Voas, J., and G. McGraw, Software Fault Injection — Inoculating Programs Against Errors,Wiley, 1997.



9-7

Ward, P., and S. Mellor, Structured Development for Real-Time Systems, Prentice Hall,1985.

Wilson, T. L., PCS Design Description Report, EPRI and B&W Owners’ Group AdvancedControl System Task Force, February 1992.

Wing, J. M., A Specifier’s Introduction to Formal Methods, IEEE Computer, pp. 8–22,September 1990.

handbook for v&v of digital systems

Documents

epri representative

member of epri

epri distribution center

r1final report

iiicitationsthis report

palo alto

power plants

package issuitable