doe-epri on-line monitoring implementation guidelines/67531/metadc... · doe-epri on-line...

DOE-EPRI On-Line Monitoring Implementation Guidelines This report describes research sponsored by EPRl and the U.S. Department of Energy

under the Nuclear Energy Plant Optimization (NEPO) Program.

Technical Report I

DOE-EPRI On-Line Monitoring Implementation Guidelines

1007622

Interim Report, January 2003

Cosponsor U.S. Department of Energy Washington, D.C.

EPRl Project Manager R. Shankar

EPRl * 3412 Hillview Avenue, Palo Alto, California 94304 - PO Box 10412, Palo Alto, California 94303 - USA 800.313.3774 - 650.855.2121 - [email protected] - www.epri.com

mailto:[email protected]

http://www.epri.com

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

THIS DOCUMENT WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN ACCOUNT OF WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) BELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM:

(A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, ( I ) WITH RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (11) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (111) THAT THIS DOCUMENT IS SUITABLE TO ANY PARTICULAR USERS CIRCUMSTANCE: OR

(8) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRl OR ANY EPRl REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT.

ORGANIZATIONS THAT PREPARED THIS DOCUMENT

Edan Engineering Corporation

Expert Microrystems, Inc.

ORDERING INFORMATION

Requests for copies of this report should be directed to EPRl Orders and Conferences, 1355 Willow Way, Suite 278, Concord, CA 94520, (800) 313-3774, press 2 or internally x5379, (925) 609-9169,

Electric Power Research Institute and EPRl are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc.

Copyright (8 2003 Electric Power Research Institute, Inc. All rights reserved.

(925) 609-1310 (fax).

CITATIONS

This report was prepared by

Edan Engineering Corporation 900 Washington Street, Suite 830 Vancouver, WA 98660

Principal Investigator E. Davis

Expert Microsystems, Inc. 7932 Country Trail Drive, Suite 1 Orangevale, CA 95662

Principal Investigator R. Bickford

This report describes research sponsored by EPRI and the U.S. Department of Energy under the Nuclear Energy Plant Optimization (NEPO) Program, NEPO Task: FY02 5-1 13.

The report is a corporate document that should be cited in the literature in the following manner:

DOE-EPRI On-Line Monitoring Implementation Guidelines, EPFU, Palo Alto, CA, and the U.S. Department of Energy, Washington, D.C.: 2003. 1007622.

... 111

REPORT SUMMARY

Background The On-Line Monitoring (OLM) Group operates under the Instrumentation and Control (I&C) Nuclear Target. There are, at present, two programs under this group, separately funded under individual subscription programs that address the needs of power plants with regard to instrument monitoring, instrument calibration reductiodextension, and sensor validation; These are the Instrument Monitoring and Calibration (IMC) Users Group formed in 2000 and the On- Line Monitoring Implementation Users Group formed in 2001.

EPRI’s strategic role in on-line monitoring is to facilitate its implementation and cost-effective use in numerous applications at power plants. To this end, EPRI has sponsored an on-line monitoring implementation project at multiple nuclear plants specifically intended to install and use on-line monitoring technology. The goal is to apply on-line monitoring to all types of power plant applications and document all aspects of the implementation process in a series of EPRI deliverables. These deliverables will cover installation, modeling, optimization, and proven cost- benefit.

EPRI will continue to foster the development of on-line monitoring technology and its application through the efforts of the IMC Users Group. Through the IMC Users Group, on-line monitoring, its a key technology, will continue to be supported technically as its use grows throughout the industry. The EPRI IMC Users Group will also continue to support generic technical issues associated with on-line monitoring, such as providing implementation guidance for calibration reduction of safety-related instrumentation.

0 bjectives To provide technical information regarding on-line monitoring as a calibration-extension and performance-monitoring tool

To provide s o h a r e verification and validation information in support of on-line monitoring by the multivariate state estimation technique (MSET)

To provide technical information describing software installation and setup for on-line monitoring

To address data management and interface issues

To provide guidance for evaluating on-line monitoring system results

To explain the steps and actions necessary to implement an on-line monitoring system

V

Approach This report provides detailed information regarding the application and implementation of on- line monitoring to nuclear plant instrument systems. The MSET is the principal method covered because it is the basis for the EPRI on-line monitoring implementation project.

Results Industry and EPRI experience at several plants has shown on-line monitoring to be very effective in identifying out-of-calibration instrument channels or indications of equipment-degradation problems.

The EPRI implementation project for on-line monitoring has demonstrated the feasibility of on- line monitoring at several participating nuclear plants. The results have been very encouraging, and substantial progress is anticipated in the coming years.

EPRI Perspective EPRI’s strategic role in on-line monitoring is to facilitate its implementation and use in numerous applications at power plants. On-line monitoring of instrument channels provides better information about the condition of monitored channels through accurate, more frequent evaluation of each channel’s performance over time. This type of performance monitoring offers an alternative approach to traditional, time-directed calibration. EPRI is committed to the development and implementation of on-line monitoring as a tool for extending calibration intervals and evaluating instrument performance.

Keywords Cali bration Condition monitoring Instrumentation and control Maintenance Nuclear plant operations and maintenance Signal validation

vi

EPRI and Edan Engineering recognize the following individuals for their contribution to this project. Their time and attention in support of this project are greatly appreciated.

David Carroll

Pat Colgan

Dave Hooten

Jerry Humphreys

Aaron Hussey

Robert Kennedy

Vo Lee

Hubert Ley

David Lillis

Edwina Liu

Connie Love

Adrian Miron

Karl Nesmith

Mike Norman

Steve Orme

Jeff Richardson

Richard Rusaw

South Carolina Electric and Gas

Exelon Corporation

Carolina Power and Light

CANUS Corporation

EPRI

Exelon Corporation

Expert Microsystems, Inc.

Argonne National Laboratory

British Energy

Expert Microsystems, Inc.

Tennessee Valley Authority




British Energy

British Energy


vii

Larry Straub

Bill Turkett

Tom Wei

Chenggang Yu

Nela Zavaljevski

Exelon Corporation





... Vll l

CONTENTS

1 INTRODUCTION .................................................................................................................. 1-1

1 . 1 Report Purpose ............................................................................................................ 1-1

1 . 2 Report Applicability ....................................................................................................... 1-1

1.3 EPRl's Role in On-Line Monitoring ............................................................................... 1-2

1.4 Terminology Used in This Report ................................................................................. 1-3

. .

1.4.1 Channel. Sensor. and Signal ................................................................................. 1-3

1.4.2 Modeling Terms .................................................................................................... 1-4

2 DATA MANAGEMENT ........................................................................................................ 2-1

2.1 On-Line Monitoring System Architecture ...................................................................... 2-1

2.1 . 1 Off-Line Batch Mode Using Historical Data ........................................................... 2-2

2.1.2 On-Line Batch Mode Using Current Data .............................................................. 2-3

2.1.3 Real-Time Mode .................................................................................................... 2-4

2.2 Data File Naming .......................................................................................................... 2-5

2.3 Data Storage Format .................................................................................................... 2-6

2.4 Data File Configuration Management ........................................................................... 2-7

3 MULTIVARIATE STATE ESTIMATION TECHNIQUE . SOFTWARE ACCEPTANCE ........ 3-1

3.1 Overview of Multivariate State Estimation Technique ................................................... 3-1

3.2 Software Functions of the Multivariate State Estimation Technique .............................. 3-1

3.3 Functional Overview and General Capabilities of SureSense ....................................... 3-2

3.4 Verification and Validation With Multivariate State Estimation Technique ..................... 3-3

3.4.1 Verification and Validation of the MSET Base Code by Argonne National Laboratory ...................................................................................................................... 3-4

3.4.1 . 1 Background ................................................................................................... 3-4

3.4.1.2 Verification and Validation Documents ........................................................... 3-4

3.4.2 Verification and Validation of Expert Microsysterns' SureSense Software ............. 3-5 3.4.3 EPRI-Independent Verification and Validation of the Suresense Software ............ 3-6

ix

3.5 Additional Verification and Validation Considerations ................................................... 3-6

3.5.1 Data Acquisition .................................................................................................... 3-6

3.5.2 Model Configuration Control .................................................................................. 3-7

3.6 Multivariate State Estimation Technique - Software Acceptance Test .......................... 3-7

4 ON-LINE MONITORING PROCEDURES ............................................................................. 4-1

4.1 Procedure Support Documents Provided by EPRl ........................................................ 4-1

4.2 Software Use ................................................................................................................ 4-1

4.2.1 Procedure ............................................................................................................. 4-1

4.2.2 Personnel Training ................................................................................................ 4-2

4.3 Model Development and Evaluation Procedures .......................................................... 4-2

4.3.1 Model Documentation ........................................................................................... 4-2 4.3.2 Periodic Model Evaluation ..................................................................................... 4-3

4.3.2.1 Example Model Evaluation Procedure ........................................................... 4-3

5 FAILURE DETECTION AND ALARM RESPONSE ............................................................. 5-1

5.1 Summary of How Failures Are Identified by On-Line Monitoring ................................... 5-1

5.2 Recommended Response to an Identified Failure ........................................................ 5-1

5.3 Failures Identified Because of Data-Acquisition Problems ............................................ 5-3

5.4 Operation Outside the Training Space .......................................................................... 5-5

5.4.2 F’ermanent Change in Operating Space - System-Operation Change .................. 5-7

5.4.3 F’ermanent Change in Operating Space - Equipment Repair ................................ 5-8

5.4.1 Permanent Change in Operating Space - Normal Process Changes .................... 5-5

. 5.5 Incorrect Initial Training .............................................................................................. 5-10

5.6 Excluding Transient or Abnormal Operating States .................................................... 5-12

5.7 Occasional Outliers .................................................................................................... 5-16

5.8 Equipment Operating States Not Covered by Available Training Data ........................ 5-17

5.9 Inadequate Initial Training Within the Defined Operating Space ................................. 5-18

5.1 0 Using Threshold Settings for Overly Sensitive Alarms .............................................. 5-20

5 . 10 . 1 Overview of Threshold Settings ......................................................................... 5-20

5.10.2 Establishing Threshold Limits ............................................................................ 5-21

6 REFERENCES ..................................................................................................................... 6-1

A GLOSSARY ........................................................................................................................ A-I

X

LIST OF FIGURES

~~ ~ ~~~~~ ~~ ~ ~~~~~~~~~ _ _ _ _ _ ~

Figure 1-1 Instrument Channel in Terms of On-Line Monitoring .............................................. 1-3 Figure 2-1 On-Line Monitoring System .................................................................................... 2-3 Figure 2-2 Typical Plant Data Archiving and Retrieval System ................................................ 2-6 Figure 3-1 SureSense Operation ............................................................................................. 3-3 Figure 5-1 Typical Drift Behavior ............................................................................................. 5-2 Figure 5-2 Residual Plot Showing Drift Significance Upon Failure Alarm ................................. 5-3 Figure 5-3 Extreme Example of Data-Acquisition Error . Stuck Data ...................................... 5-4

Figure 5-4 Extreme Example of Data-Acquisition Error . Data Stuck for Almost One Year ..... 5-4 Figure 5-5 Test Data Outside Training Range . SureSense Result ......................................... 5-6

Figure 5-6 Test Data Outside Training Range -Actual Data ................................................... 5-6 Figure 5-7 Test Data Outside Training Range . System Operation ......................................... 5-7 Figure 5-8 Tank-Level Variation .............................................................................................. 5-8 Figure 5-9 Test Data Outside the Training Range . Equipment Repair ................................... 5-9

Figure 5-10 Test Data Outside the Training Range . MSET Results ....................................... 5-9 Figure 5-1 1 Test Data Outside the Training Range . Replacement of Data-Acquisition

Card ... , .......................................................................................................................... 5-10 Figure 5-12 Recalibrated Sensor . Model Trained on Out-of-Calibration Data ...................... 5-11 Figure 5-13 Recalibrated Sensor . Initially Out of Calibration ................................................ 5-12 Figure 5-14 Signal Behavior During a Transient .................................................................... 5-13 Figure 5-15 One Transient During an Extended Period of Operation ..................................... 5-14 Figure 5-16 Example of Routine Transients Exceeding the Training Space ........................... 5-15 Figure 5-17 Short-Term Outlier .............................................................................................. 5-16 Figure 5-1 8 Many Possible Operating States ........................................................................ 5-17 Figure 5-19 Example of a Pump That Almost Always Runs ................................................... 5-18 Figure 5-20 Unexpected Change in Estimate ........................................................................ 5-19

Figure 5-22 Threshold Settings Applied to Residual Plot ..................................... ...,.. ............. 5-21 Figure 5-21 Inadequate Training for Pump On and Off Conditions ........................................ 5-20

xi

LIST OF TABLES

Table 5-1 Instrument Channel Uncertainty Sources . ... ... ... . .. .. ... .. ..... . ... .. .. ... . . ...... ... ..... . .. .... .. . 5-22

... XI11

1 .I Report Purpose

This topical report discusses the application and implementation of on-line monitoring to nuclear plant instrument systems. A considerable amount of application and development has been completed by the EPRI on-line monitoring implementation project and the experience gained has been documented for the benefit of EPRI members. The purpose of this report is to accomplish the following:

Provide technical information regarding on-line monitoring as a calibration-extension and performance-monitoring tool.

Provide software verification and validation information in support of on-line monitoring by the multivariate state estimation technique (MSET).

Provide technical information describing software installation and setup for on-line monitoring.

Address data management and interface issues.

Provide bwidance for evaluating on-line monitoring system results.

Explain ihe steps and actions necessary to implement an on-line monitoring system.

This interim report is based on the experience of the EPRI on-line monitoring implementation project. This report will be revised in 2003 to incorporate the implementation experience of the participating plants.

1.2 Report Applicability

This topical report addresses specific issues and considerations applicable to any on-line monitoring system. The EPRI on-line monitoring implementation project is applying the MSET to nuclear plant instrument systems and the implementation guidelines directly describe an MSET approach. Argonne National Laboratory (ANL) originally developed the MSET approach for nuclear power applications. The SureSense software, supplied by Expert Microsystems, is a commercial implementation of MSET that was selected for use by the EPRI on-line monitoring implementation project. In some cases, SureSense software features are used to illustrate certain aspects related to implementation.

This report is not intended to serve as a software user’s guide. Instead, it addresses implementation issues at a higher level and rarely discusses modeling issues. The EPRI on-line

1-1

Introduction

monitoring implementation project has produced a separate report, Suresense Diagnostic Monitoring Studio Users Guide, which should be reviewed for Suresense-specific considerations. For other software systems, refer to the supplier’s user’s guide.

Modeling guidelines are provided in EPRI Report 1003661, Plant System Modeling Guidelines to Implement On-Line Monitoring. These modeling guidelines complement the implementation guidelines provided here.

In summary, the implementation guidelines provided in this topical report apply directly to MSET and the examples have been illustrated with the SureSense software. The software verification and validation information provided in this report apply specifically to the ANL MSET and the Expert Microsystems’ Suresense. But, many of the principles described here can be applied to other on-line monitoring methods. In particular, issues associated with plant data system integration and alarm assessment considerations apply to most on-line monitoring methods.

1.3 EPRl’s Role in On-Line Monitoring

EPRI’s strategic role in on-line monitoring is to facilitate its implementation and cost-effective use in numerous applications at power plants. To this end, EPRI has sponsored an on-line monitoring iimplementation project at multiple nuclear plants to install, demonstrate, and use on-line monitoring technology. The purpose of the EPRI on-line monitoring implementation project is to: 1) apply on-line monitoring to all types of power plant applications, and 2) document all aspects of the implementation process in a series of EPRI deliverable reports. These reports will cover installation, modeling, optimization, and proven cost-benefit. The planned EPR.1 reports are:

SureSense Multivariate State Estimation Studio Users Guide - provides detailed guidance for the application of SureSense MSET for nuclear plant systems. This report is periodically updated as a result of user feedback or software revisions.

Plant Syr:tem Modeling Guidelines to Implement On-Line Monitoring - addresses all aspects of modeling for on-line monitoring applications. This report describes model development, data quality issues, training requirements, retraining criteria, response to failure alarms, and declaration of a model ready for use. The models developed by the project participants are described in the appendices for the benefit of future users of on-line monitoring systems.

On-Line Monitoring Implementation Guidelines (this report) - discusses implementation of on-line monitoring. This report addresses plant procedures for on-line monitoring, software verification and validation, data management, and the transition from batch mode to on-line mode of model operation.

Cost Benefits of On-Line Monitoring - discusses the expected costs and benefits of on-line monitoring. Direct, indirect, and potential benefits are covered. The experience of the EPRI on-line monitoring implementation project participants is included.

Implementation of On-Line Monitoring for Technical SpeciJication Instruments - addresses on-line monitoring for safety-related applications, including the Safety Evaluation of the

1-2

Introduction

Nuclear Regulatory Commission (NRC) for on-line monitoring. Topics include technical specifications, uncertainty analysis, procedures and surveillances, MSET application considerations, and miscellaneous technical considerations. Nuclear Energy Plant Optimization (NEPO) projects related to uncertainty analysis and software verification and validation provide input to this report.

EPRI fosters development of on-line monitoring technology and its application through the efforts of the IMC Users Group. Through the IMC Users Group, on-line monitoring will continue to be supported technically as a key technology while its use grows throughout the industry. The EPRI IMC Users Group also supports generic technical issues associated with on-line monitoring, such as providing implementation guidance for calibration reduction of safety-related instrumentation.

1.4 Terminology Used in This Report

Appendix A provides a glossary of terms used in this report. But, some terms require additional clarification in support of using this report. The following sections explain key terms.

1.4.1 Channel, Sensor, and Signal

The terms channel, sensor, and signal are often used almost interchangeably in this report, but there is an important distinction between the three terms. The sensor is the device that measures the process value. The sensor and associated signal-conditioning equipment are referred to collectively as the instrument channel, or channel. The electrical output from the channel is the signal. Figure 1-1 shows the relationship between the three terms for a safety-related channel; a non-safety-related channel might not have the isolator or bistable as shown.

Figure 1-1 Instrument Channel in Terms of On-Line Monitoring

1-3

Introduction

For a non-safety-related channel, there might be little in the way of signal conditioning, with the sensor being the only real monitored device. More complex measurements might contain several signal-conditioning modules. This report will usually refer to the channel rather than the sensor in terms of what is monitored. Although other industry documents and published papers often discuss on-line monitoring using the term sensor, it is the channel (or some portion of the channel) that is actually monitored. The discussion provided in the following sections will frequently refer to sensor drift because the sensor is usually the most common source of drift, but any portion of the channel might actually be the cause of drift.

The on-line monitoring system does not know the layout of the channel; it only receives a digitized signal from the plant computer, a historical file, or other data-acquisition system. Although the instrument channel is typically producing a milliampere or voltage output, the signal acquired by the on-line monitoring system is often scaled into the expected process units, such as pressure, temperature, or percent. When this report refers to signals, it means the scaled or unscaled digitized output signals from the monitored channels.

7.4.2 Modeling Terms

The term model is used to describe the group of signals that have been collected together for the purpose of signal validation and analysis. Depending on the context, model might refer just to the selected group of signals, or it might also include the various settings defined by the on-line monitoring system that are necessary to optimize the performance of the signal validation procedure. In the context of on-line monitoring, model does not necessarily refer to some functional relationship between model elements, defined by a set of equations.

The term Observation vector is used to describe the observed values for all of the signals in the model at a p;irticular instant in time. For example, if the signal data were contained in a spreadsheet, a single row of data representing a particular point in time would be a vector.

The term stafe space is used to describe the operating states that form the basis for training a model. The state space contains the expected range for each signal in the model and it also defines different operating state$ within that range. For example, a state space for a pressure sensor might cover a range of 800 to 1200 psig (561 1 to 8369 Wa). Within this range, there might be several distinct operating states associated with different equipment lineups or plant power levels.

The term estimate is used to describe the best estimate or prediction of the actual process or signal value. For each observation, the on-line monitoring system produces an estimate of the corresponding expected value for each monitored signal. The term residual refers to the mathematical difference between an observed value and the corresponding estimate for that observation. Fault detection is often based on the behavior of the residual.

1-4

2 DATA MANAGEMENT

On-line monitoring is a data intensive process and the amount of data to be managed should be understood in advance. Section 2 discusses various aspects of data file management. This information applies to either a batch mode or an on-line mode of operation.

2.1 On-Line Monitoring System Architecture

On-line monitoring can be applied in various modes, ranging from batch mode using data files up to a real-time, on-line mode receiving a continuous data stream. The term on-line has not been well defined in the industry. Originally, the term on-line was used to indicate that signal validation was performed while the plant was operating at power, without regard to how the data were acquired. In most cases, the on-line monitoring method has actually been off-line in terms of data acquisition, meaning that data are accumulated in batch files for processing.

The analysis procedures within the monitoring system treat all signals as if they originate from an on-line data collection system. The on-line monitoring system is not concerned with the layout of the channel; it simply receives a digitized time series of signals from the plant computer, a historical file, or other data-acquisition system. Although the instrument channel is typically producing a milliampere or voltage output, the signals acquired by the on-line monitoring system are often scaled into the expected process units, such as pressure, temperature, or percent.

On-Line Monitoring of Instrument Channel Performance (EPRI Report TR- 104965-R1 NRC SER) defines the following possible options for an on-line monitoring system:

1. An automated system that performs data acquisition and analysis essentially continuously in real time at a specified sample rate.

2. An automated system that performs data acquisition and analysis at discrete, specified intervals.

3. An automated system that is normally off and is manually activated to perform data acquisition and analysis at a set interval (at least quarterly).

4. A manual system in which data are acquired manually on at least a quarterly interval and entered manually into a computer program for the purpose of analysis.

The differences between each of the above options mostly involve the degree of automation in the data-acquisition step, including the method of data collection and the frequency of the data

2- 1

Data Management

analysis. Most of the above options are actually operating in a batch mode in which data are accumulated in stored files; the differences relate to the locations of the data files, and to how frequently data files are generated and evaluated. For typical nuclear plant applications, either option 2 or option 3 above will usually be used.

The typical on-line monitoring system consists of the following building blocks:

0 Separate off-line computer hardware on which the system resides.

Communications hardware and software to obtain data from the plant process computer, plant data historian or other source, if the data are automatically acquired. Manual data acquisitilon is also possible.

The on-line monitoring software that analyzes, displays, and archives the data and presents results interactively in graphs and reports.

2.1.1 Off-Line Batch Mode Using Historical Data

The term buifch mode means that data files are stored in some location and are accessed by the on-line moniitoring system. The term ofl-Zine butch mode applies if the data files are manually extracted from archive or must be manually specified by the user. Depending on the frequency of data collection, batch mode might be used to evaluate the previous quarter, month, week, or day. Notice that a system operating in batch mode is evaluating specified data files that cover some period of time; batch mode is not receiving a real-time data stream.

Even if the intent is to apply on-line monitoring in a true on-line mode, certain modeling and analysis functions are generally best performed in a batch operation, including the following:

Training -the data used for training require careful review and cleanup. Training data must be error-free and properly characterize the normal operating states that will be monitored. The data used for training must be kept available for retraining if certain model settings are modified.

Model dtwelopment -historical data files are used to evaluate model performance as part of the mode 1 development process. Some of the historical data are used for training, and some data are i ypically used for verification and performance assessment.

Look back - after model development is complete and the model has been placed in service, it can be useful to review the historical performance of a signal or group of signals when evaluating current signal performance. It is often easier to maintain historical data in pre-configured files for ready access and comparison. Extracting data for an extended time period from a plant data historian or other archive can be time-consuming.

The above effort for a model is about the same regardless of the selected on-line monitoring approach; all models require historical data for training, retraining, model development, and historical performance analysis. The differences in on-line monitoring mode of operation show up after the model has been developed, tested, and placed in service, and primarily relate to how subsequent data are acquired and tested. In the case of an off-line batch model, the user has to periodically extract data for each model, typically at monthly intervals, which represents a

2-2

Data Management

recurring maintenance cost. In its simplest implementation, in which no changes were made to the existing data archive software, this typically involves the following steps:

Extract the previous month of data into a separate file for each model.

0 If the data are extracted in text format, optionally convert each extracted file into a binary format for improved performance during analysis.

Link the data file to the model and run the model.

If there are many models, this can be a time-consuming process, which is why some level of automation is preferable. The next section describes an automated approach to data-acquisition and file management.

2.1.2 On-Line Batch Mode Using Current Data

All users initially start with an off-line batch mode of operation as they develop models and learn how to use the on-line monitoring system software. As the models are placed in service, the transition from off-line to on-line operation mode of operation should be considered. Figure 2-1 shows a typical system architecture for a true on-line monitoring system that includes a more efficient approach to data management.

Computer

Sewer

3 Engineering

Server

Monitoring station

Modeling Station

Figure 2-1 On-Line Monitoring System

The above system operates as follows:

No changes are made to the plant computer software and its associated data historian. Plant process data are acquired from the plant computer and archived just as it was prior to installing an on-line monitoring system.

Data are acquired from the data historian using an extraction routine (referred to in Figure 2- 1 as a data bridge) that might be unique to the plant-specific computer configuration. At periodic intervals, data are downloaded from the data historian and stored in the appropriate format for each model. For example, an on-line monitoring system might acquire the previous day’s data for each model and store it in a designated location.

2-3

Data Management

The on-line monitoring software combines the new data with previously acquired data, automatically runs each model with the new data included, and stores the run results. The latest run results are available on demand by plant personnel.

In terms of power plant requirements, the above description represents a true on-line monitoring system. Notice that the models are still technically operating in a batch mode in that the models run on historical files that include up to the previous day’s data. This is not real-time operation, but it is close enough for most purposes. The key differences between this periodic, on-line approach and the off-line batch mode described in Section 2.1.1 are in the following automated steps:

Data acquisition - an engineer or technician does not manually extract data files.

Data file formatting - file conversions are handled automatically.

Data file linking - the model automatically links to the latest data set.

Model run - the model runs automatically and stores the run results in a specified location for ready retrieval by any user.

Making the lransition from off-line batch mode to on-line mode usually requires some unique programming, which can be provided by either the software supplier or by plant personnel. The benefits of on-line mode are considered important enough that it is the recommended approach for most useirs;

2.7.3 Real-Time Mode

Real-time, on-line monitoring is unnecessary for most, if not all, nuclear plant applications, especially considering the dedicated computing overhead required to operate numerous models simultaneously in a real-time environment. The National Aeronautics and Space Administration (NASA) space shuttle is one example of a true real-time, on-line monitoring application where the software used in this project has been employed. In this case, a continuous data stream is sampled and processed through the signal-validation software throughout the period from just before launch until the completion of launch. For the few minutes that it takes to reach orbit, a large quantity of data is evaluated as it is received to ensure that the mission-critical sensors are providing valid data to the flight computers.

Nuclear plant requirements are quite different from the space shuttle in terms of data-acquisition requirements. Rather than operate for only a few minutes during launch, a power plant might operate at near 1 00-percent power for an entire operating cycle of 18 to 24 months. During this extended period of operation, many sensors will experience only small process changes, essentially monitoring the process about a single point. Setting up on-line monitoring software to operate in a real-time mode can be done, but it involves an additional level of complexity that is not needed for typical plant applications because the software’s output will not be used in real time.

2-4

Data Management

2.2 Data File Naming

When on-line monitoring is applied to its fullest potential, there will be many data files stored in the computer system and linked to the various models. When many data files are stored in a common location, the file naming convention becomes important. The following naming convention works well and is recommended:

Unit Number (if needed) - Model Name - Year - Month - Portion of Month

Unit number refers to the power plant designation (or designation for some monitored asset). Model name refers to the specific diagnostic software model used to monitor some asset or system. The dates refer to the period of operation during which the data were acquired. The following three examples illustrate typical naming conventions:

1. Unit 1 Vessel Level-2002-01

2. RPS-2001-10

3. OTSG-A.-200 1 - 12

All files for a single model will be located as a group within a directory, sorted by year and month, if the: above naming convention is used. Missing months or time periods are easily recognized. If data covers portions of a month-such as the first 10 days in one file, the next 10 days in a second file, and the last 10 days in a third file-it is recommended that the three files be combined into a single file for the month.

Notice that the above naming conventions imply that the data for all signals in a given model will be acquired collectively as a group. This is the preferred approach. While readily supported in some commercial software packages, such as Suresense, separating signal data for a single model for a specified period of time into multiple files complicates data cleanup and review and can limit the applicability of the data for use with other software applications. Also, the removal of bad data within separate files must be carefully managed to prevent files with time stamp simultaneity inconsistencies.

Notice also in the above naming convention that data for several models is not contained in a single file. For example, it might be easier during data retrieval to extract the data for five models at once. It is strongly recommended that a single data file contain only the data for a single model. While combined data files are readily supported in some commercial software packages, such as Suresense, combining the data can actually complicate other aspects of data management. For example, suppose the data for six models, each containing 25 signals, is simultaneously extracted and stored in a single file. If data are acquired at a 1-minute rate, this file will contain up to 44,600 rows of data for 150 columns of signals, a total of 6,690,000 data points. Files will likely be in a spreadsheet format to allow review and editing, as necessary. The size of each tile will be approximately 60 megabytes, which is time-consuming to manage and very difficult to review. These files are also treated as text files by spreadsheet programs, which takes longer to read-each data access has to read through the entire file to read the data of interest. Based on the experience to date, keep each file as small and simple as possible.

2-5

Data Management

2.3 Data Storage Format

On-line monitoring is a data-intensive activity. The overall objective is to acquire plant data and process the acquired data to determine whether it is indicative of a healthy, normal state of the system or alternatively some degraded or unhealthy state of the system. Power plants are characterized by large numbers of subsystems and signals, many of which are important targets of on-line monitoring. Data-acquisition rates for these large plants are reasonably high. As a result, very large data sets will be managed by either the on-line monitoring system or by its operators. In most cases, a high level of on-line-monitoring-system automation would be preferred.

When implementing an on-line monitoring system, plant-specific data issues must be addressed and resolved. One such issue will be the format of the acquired data. Computers preferentially process data in binary format while human operators prefer text or graphical formats.

Human operators often process measurement data in the form of text. As a result, many plants have established data extraction utilities to extract operating data in a text-based format, often to ASCII text files. The signal data are typically acquired by the plant computer, stored in a data archive, and extracted on demand by a user as shown in Figure 2-2. These existing procedures will typically provide the initial data-extraction capability for on-line monitoring. What is often required is a final automation step to perform the extraction from the data archive. Some on-line monitoring systems, such as Suresense, will provide a ready means to implement this data archive connection either on-demand by the user or through a regularly scheduled data extraction service program (also known as a data bridge).

Plant Data Acquisition System

Plant Data Sewer

Figure 2-2 Typical Plant Data Archiving and Retrieval System

Engineering Station

Although text files are human readable, they are extremely inefficient for use with a computer program. The program must read each character individually and interpret the series of characters to determine the number represented. Further, the space required for each value is unknown until the value has been read, which reduces file-access speed when attempting to read data in the middle of the file. In contrast, computers can read binary-format files with greater efficiency and speed and can readily access any data value in the file immediately and directly.

2-6

Data Management

Therefore, it is desirable to store plant data in a binary format for efficient use by the on-line monitoring system.

The SureSense software used by the EPRI on-line monitoring implementation project provides a general-purpose data interface that can be readily adapted to any input data format. It can be used to convert a set of data in a plant-specific format into one of several standard formats that can be readily shared between the project participants without further translation or processing.

It is recommended that initial implementation work be performed using one of these standard formats. After developing an initial set of models, a plant-specific, data-acquisition interface can be implemented if supported by the capabilities of the user’s on-line monitoring software. The SureSense software uses a plug-in data reader capability to enable a highly adaptable and low-cost interface to any plant data system.

Most users will already have text-format data available from their plant data system. One common format for text-based data is the Microsoft Excel comma-delimited-file (CSV) format. An advantage to this format is that Microsoft Excel can be used to examine and characterize the data. The Suresense software includes a plug-in data reader for CSV-format files. SureSense users should consult their users manual for further details.

Standards for binary-format files are typically industry and software specific. The EPRI on-line monitoring implementation project has selected a simple, non-proprietary format known as the signal-data-file (SDF) format. SDF files provide fast and efficient data access and the SureSense software conies with a plug-in data reader for the SDF file format. SureSense users should consult their users manual for further details.

2.4 Data File Configuration Management

Data configuration management is an important element of any on-line monitoring implementation program. The following items require configuration management for an effective on-line monitoring implementation in a nuclear power plant environment.

On-line monitoring, model-definition files, including any supporting phase-determiner plug- ins and time-formatter plug-ins.

On-line monitoring, model-training-data files, including any supporting data-reader plug-ins.

On-line monitoring, software-acceptance-test files and acceptance-test results.

On-line monitoring configuration management should follow procedures currently in place for document and data management at the plant. Section 3.5 discusses model configuration management with respect to software quality assurance.

2-7

MULTIVARIATE STATE ESTIMATION TECHNIQUE - SOFTWARE ACCEPTANCE

The EPRI on-line monitoring implementation project selected the MSET as its on-line monitoring method. For this reason, this section addresses software acceptance in support of a plant-specific implementation.

3.1 Overview of Multivariate State Estimation Technique

MSET is a software-based tool for on-line monitoring and was specifically developed by ANL for nuclear power applications. MSET is general in scope and can also improve performance, prevent downtime, and reduce operating costs for fossil power plants and many other industrial applications. MSET is a statistical modeling technique that learns a high-fidelity model of a process or apparatus from a sample of its normal operating data. Once built, the model produces an accurate estimate for each observed signal, given a new data observation from the process or equipment. Each estimated signal is compared to its actual signal counterpart using a highly sensitive fault-detection procedure to statistically determine whether the actual signal agrees with the learned model.

To use MSET, the user starts by collecting sensor-generated data from the process under consideration that bounds all normally expected operational states. These data are used by the MSET system to establish the state space of normal process operation (train MSET to recognize normal behavior) and will be used in the monitoring phase to identify abnormal behavior. During monitoring, sensor data are read by MSET; an estimate of the current state of the process is determined by comparing the measured sensor data with that obtained during training; and the difference between this state estimate and the measurement is calculated. This difference or residual error is then analyzed by a statistically based hypothesis test that determines if the process is operating normally or abnormally. If an abnormal condition is detected, the initial diagnostic step identifies the cause as either a degraded sensor or an operational change in the process. When a degraded sensor is identified, MSET uses the estimated value of the signal from this sensor to provide a highly accurate “virtual sensor” that can be used to replace the function of the faulted sensor in the MSET estimation process.

3.2 Software Functions of the Multivariate State Estimation Technique

MSET consists of 1) a pattern-recognition system that provides analytically derived values of the monitored signals and 2) a statistically based hypothesis test that compares the analytically estimated signal values with the measured values to detect faults in the signals or monitored

3- 1

Multivariate State Estimation Technique - Software Acceptance

equipment. MSET consists of three essential modules and a number of supporting modules. The essential modules are:

A training algorithm for selecting and characterizing a subset of representative data from sensors during normal operation of the system - the training module is used to produce a training matrix of operating data that ideally encompasses all expected normal operating states of the system.

A system state estimation module for parameter estimation - the system state estimation module is used to estimate the expected values of the monitored signals.

A statistically based fault-detection algorithm -the fault-detection module is used to detect abnormal disturbances in the monitored signals by examination of the difference between the estimated and measured signal values.

MSET provides a high-fidelity estimate of the expected response of an asset’s data signals by w i n g advanced pattern-recognition techniques to measure the similarity between the signals within a learned state space. The learned patterns or relationships among the signals are used to identify the operating state that most closely corresponds with the current measured set of signals. By quantifying the relationship between the current and learned states, MSET estimates the current expected response of the signals.

The difference between a signal’s estimated value and its directly sensed value is used as the indicator for sensor and equipment faults. The sequential probability ratio test (SPRT) technique provides ANL’s basis for detecting statistical changes in the sensor signals at the earliest possible time, including usable information regarding the type and location of a disturbance. The SPRT technique provides a superior surveillance tool because it is sensitive not only to disturbances in signal mean, but also to very subtle changes in the statistical quality (that is, variance, skewness, bias) of the monitored signals. Instead of threshold or control limits, the SPRT technique uses user-specified, false-alarm and missed-alarm probabilities, allowing the user to control the likelihood of missed or false alarms. For sudden, gross failures of sensors or system components, the SPRT can annunciate the disturbance as fast as a conventional threshold limit check. However, for slow degradation that evolves over a long time period (such as gradual decalibration in a sensor, wearout or buildup of a radial rub in rotating machinery, loss of time constant degradation in a pressure transmitter, or change-of-gain failure without a change in signal mean), the SPRT can indicate the onset of a disturbance long before it would be apparent by visual inspection of strip chart or cathode ray tube (CRT) signal traces, and well before conventional, threshold-limit checks would be tripped.

3.3 Functional Overview and General Capabilities of Suresense

The Suresense software is a commercial implementation of the MSET software licensed from ANL for use in the power industry. SureSense provides additional capabilities that build upon the ANL algorithms to enhance the usability and diagnostic performance of the MSET procedures. The EPRl on-line monitoring implementation project is using SureSense for its on-line monitoring efforts.

3-2


SureSense performs on-line diagnostic monitoring of a process or item of equipment. The SureSense software combines the MSET procedure for on-line parameter estimation with sophisticated decision logic for a solution that provides excellent, real-time performance, well-defined error rates, and is scaleable to validate any number of system signals. The software’s fault-detection procedure employs a proprietary adaptive sequential probability (ASP) hypothesis test technique to determine whether the signal is uncharacteristic of the learned model and therefore indicative of a sensor or equipment fault. ASP fault events are filtered by a Bayesian conditional probability (BCP) fault decision procedure to reduce the possibility of false alarms by accumulating evidence over a short series of observations.

Figure 3- 1 illustrates the SureSense estimation and fault detection procedure. The Suresense Multivariate State Estimation Studio Users Guide provides detailed information regarding the SureSense software.

Yes

Figure 3-1 Suresense Operation

3.4 Verification and Validation With Multivariate State Estimation Technique

The V&V of the MSET software has been performed at several levels:

The MSET base code was developed by ANL and has been licensed to Expert Microsystems Inc. and Smartsignal Corporation. The V&V of this base code is described in Section 3.4.1.

3-3

Multivariate Slate Estimation Technique - Software Acceptance

The Expert Microsystems’ SureSense software has been applied by the EPRI on-line monitoring implementation project. SureSense uses the MSET base code licensed by ANL and overlays additional features and capabilities for signal validation and monitoring. The SureSense software V&V documents are described in Section 3.4.2.

EPRI has sponsored an independent V&V of the SureSense software (version 1.4) in support of the EPRI on-line monitoring implementation project. This effort is described in Section 3.4.3.

Section 3.6 provides an example software-acceptance test. This example has been provided specifically in support of the EPRI on-line monitoring implementation project and is based on a Suresense implementation.

3.4.7 Verification and Validation of the MSET Base Code by Argonne National Laboratory

3.4.1 .I Background

The ANL VtkV effort was funded by the NEPO program, which is a U.S. Department of Energy (DOE) research and development (R&D) program focused on performance optimization of currently operating US. nuclear power plants. The primary research areas for the R&D program are plant aging and optimization of electrical production. The NEPO program is a public-private R&D partnership with equal or greater matching funds coming from industry. The NEPO program was initiated in fiscal year 2000 and is explained in detail on the DOE web site, 1.1 ttp://nepo.ne .doc .gov/.

The IMC Users Group coordinated NEPO projects that support the continued development of on-line monitoring and directly support tasks associated with the EPRI on-line monitoring implementation project. ANL was awarded the initial scope of work for 2001 to complete the V&V of the MSET kernel software. This task completed in July 2002 and directly supports the on-site acceptance of the associated software.

3.4.1.2 Verification and Validation Documents

In July 2002., the ANL Reactor Analysis and Engineering Division completed the software quality assurance documentation in support of the MSET base code. The purpose of the ANL V&V effort was to develop the documentation necessary for participating nuclear plants to demonstrate that the software analysis modules and state estimation kernels are reliable and of high quality.

As part of the ANL V&V effort, a code configuration management plan (CMP) was implemented! and the code was placed in a location controlled by the CMP. A V&V plan was written that incorporated the required activities and acceptance criteria. Implementation of the V&V plan included a formal test plan.

3 -4


The following ANL documents were produced in support of the V&V effort:

Quality Assurance Documentation Package for the Multivariate State Estimation Technique Softwire

Software

Software Software

Software Software

Software Software

Software (MSET),

System (MSET) Software System Kernel, Revision 0, July 25, 2002.

Quality Assurance Program, Revision 0, January 23,2002.

Configuration Control Procedure for the Multivariate State Estimation Technique System (MSET), Revision 1, November 26,200 1.

Requirements and Specifications for the Multivariate State Estimation Technique System (MSET), Revision 2, May 2 1,2002.

Verification and Validation Plan for the Multivariate State Estimation Technique System (MSET), Revision 1, February 15, 2002.

Test Plan for the Multivariate State Estimation Technique Software System Revision 0, June 25,2002.

Verification of the MSET Base Code Version 3.5, July 19, 2002.

MSET Kernel Validation Report, Version 3.5 of the MSET Base Code, July 17,2002.

The above documents were developed and completed as a NEPO project and are available in support of software acceptance and use for any MSET application that employs the ANL base code. Contact the EPRI project manager for electronic copies of the above documents.

3.4.2 Verification and Validation of Expert Microsystems’ SureSense Soft ware

Expert Microsystems’ SureSense software is a commercially supported implementation of the ANL MSET base code. The software is also referred to as the SureSense Diagnostic Monitoring Studio, with version 1.4 being the most recent release as of the issuance of this report.

The following documents describe the quality assurance processes associated with Suresense:

0

0

0

0

Expert Microsystems’ Software Quality Assurance Plan, Revision 1 .O, Document No. 2002- 4470 -this document sets forth the process, methods, standards, and procedures used to perform .the software quality assurance function for the SureSense software.

Expert Microsystems’ Software Configuration Management Plan, Revision 1 .O, Document No. 2001 -4471 - this document sets forth the policy, procedures and processes used to accomplish configuration management for the SureSense software.

SureSense Diagnostic Monitoring Studio Software Requirements Specijkation, Revision 1.4, Document No. 200 1-4489 - this document specifies software requirements for version 1.4 of the SureSense Diagnostic Monitoring Studio software.

SureSense Diagnostic Monitoring Studio SofhYare Verification Test Plan, Revision 1.4, Document No: 2001-4479 - this document defines test procedures verifying that the software meets all requirements defined in the Software Requirements Specification. Guidelines are provided for unit testing, integration testing, system testing, regression testing and acceptance

3-5

Multivariate Slate Estimation Technique - Software Acceptance

testing. This document provides step-by-step test instructions. Variables and data are specified as well as expected results. A traceability matrix is included in this document to verify that testing of all items listed in the requirements document is performed.

3.4.3 EPRI-Independent Verification and Validation of the SureSense Software

EPRI sponsored an independent V&V of the SureSense soflware as part of its release for version 1.4. Although Expert Microsystems performed internal V&V testing of its software before release, the rationale for an independent V&V included the following considerations:

e

e

An independent V&V could assist nuclear plants participating in the EPRI on-line monitoring implementation project with ensuring adequate documentation is in place to support plant-specific software acceptance.

An independent review and documentation of the software functions is generally considered good software engineering practice and was considered a beneficial activity supporting the EPRI on-line monitoring implementation project.

The independent V&V testing procedure could form the building block for the plant-specific acceptance test and a periodic, full-features test, if required by plant-specific, software- acceptance procedures. This independent V&V test was conducted on known simulated data designed to test the various software estimation and fault-detection methods. Future periodic software test results will be traceable to the independent V&V documentation.

The term independent V& V is used here to indicate that the personnel performing the review were not invsolved in SureSense software development or development testing. The EPRI independent V&V was completed in July 2002 and was issued as an unpublished EPRI report to the participants in the EPRI on-line monitoring implementation project. The supporting documentation for the report included the documents listed in Sections 3.4.1 and 3.4.2.

3.5 Additional Verification and Validation Considerations

3.5.1 Data Acquisition

The method by which data are acquired for on-line monitoring should be verified and validated. The evaluation should confirm that data obtained from the plant computer or an associated data archive is correct for the application. This evaluation depends on plant-specific data-acquisition, archiving, and retrieval procedures. Plant personnel should perform this verification and validation periodically. Test requirements, procedures, plans, and results should be placed under configuration management.

3 -6


3.5.2 Model Configuration Control

In the context of configuration control, the term model refers to the following:

The selected group of signals that have been collected together for the purpose of signal validation and analysis.

The various settings defined by the on-line monitoring method that are necessary to optimize the performance of the signal-validation procedure.

The data used for training, including any filtering of the data.

As part of developing a model, the above signals, settings, and data are defined. After the model has been tuned for optimal performance, the completed model is placed in service and will be used as the basis for future signal validation. A completed SureSense project file contains the specifications for the items delineated above. In addition to the project file (*.svm file), a completed model includes all associated software plug-ins and training data files used by the project. This completed model should be placed under configuration control to ensure the following:

The correct version of the model is in service.

Changes have not been made to the model or associated plug-ins without undergoing a formal review process.

The method by which a model is placed under configuration control will vary with each plant and its plant-specific, document-control procedures.

3.6 Multivariate State Estimation Technique - Software Acceptance Test

The V&V documents described in the previous sections include a procedure with expected results for an acceptance test. The test files referenced in this procedure are provided directly to SureSense software users. These test procedures and test files can be used to support the plant- specific, software-acceptance process.

3-7

ON-LINE MONITORING PROCEDURES

Section 4 provides general on-line monitoring procedures to assist with the plant-specific, on- line-monitoring-implementation process. These procedures should be used as guidance for items to consider during plant-specific procedure development.

4.1 Procedure Support Documents Provided by EPRI

In addition to this report, other EPRI documents have been produced specifically to support the on-site implementation process. The following documents should be reviewed:

Plant System Modeling Guidelines to Implement On-Line Monitoring, EPRI, Palo Alto, CA: 2002. 1003661. This modeling guidelines report is intended as a companion document for this implementation guide. Together, these two reports address many issues and considerations associated with the setup of models and their implementation at a nuclear plant.

Cost Benefits of On-line Monitoring, EPRI, Palo Alto, CA: 2002. 1003572. This cost-benefit report addresses cost benefits associated with different on-line monitoring applications and does not specifically support on-site implementation.

SureSense Diagnostic Monitoring Studio Users Guide, Version I . 4, EPRI, Palo Alto, CA: 2002. This user’s manual was developed expressly in support of the EPRI on-line monitoring implementation project.

Safety-related applications have additional considerations that are documented in the following EPRI reports:

On-Line Monitoring of Instrument Channel Performance, EPRI, Palo Alto, CA: 2000.

Implementation of On-Line Monitoring for Technical Specijkation Instruments, EPRI, Palo Alto, CA: 2002. 1006833.

TR-104965-R1 NRC SER.

4.2 Software Use

4.2.1 Procedure

On-line monitoring software tends to be quite complex and the underlying algorithms can be especially difficult to understand. For this reason, the software user’s manual should provide adequate background information to facilitate a basic understanding of the principle of operation

4- 1

On-Line Monitoring Procedures

as well as provide detailed user’s instructions. The SureSense Diagnostic Monitoring Studio Users Guide, Version 1.4, was developed in support of the EPRI on-line monitoring implementation project and will be periodically updated as required to reflect software revisions or user’s requests for the duration of this project. This user’s manual has been designed to serve as the software use procedure at a nuclear plant.

4.2.2 Personnel Training

The software used for on-line monitoring typically implements a complex set of algorithms, and the principles behind modeling and software use are not intuitive to most users. Furthermore, the application of on-line monitoring ultimately affects the on-site, instrument-calibration program, which also requires some training and preparation of plant personnel. The plant should conduct various training classes as part of the implementation process. Typical training classes include:

Detailed, software-use training for those personnel who will develop and maintain models.

General, software-use training for those personnel who might review on-line monitoring system results.

I&C training to explain how on-line monitoring affects the instrument-calibration program, including how to assess the need for calibration and how to evaluate alarms as they occur.

4.3 Model Development and Evaluation Procedures

EPRI Interim Report 100366 1 , Plant System Modeling Guidelines to Implement On-Line Monitoring, provides considerable guidance regarding the model development and evaluation process. The following sections provide additional guidance regarding how to develop, maintain, and evaluate the in-service models.

4.3.1 Model Documentation

In the context used here, the term model refers to the following:

The selected group of signals that have been collected together for the purpose of signal validation and analysis.

The various settings defined by the on-line monitoring method that are necessary to optimize the performance of the signal-validation procedure.

The data used for training, including any filtering of the data.

The term model documentation refers to the configuration management of the completed model. For each model, the following should be documented:

The file name and file date of the completed model as approved for use, including any revision information.

4-2


A list of the instrument channels covered by the model, including which channels have been designated as calibrate on demand, based on the model-run results by the instrument- calibration program.

Model settings that form the basis for model operation. These model settings include the definition of signal, data set, phase determiner, and estimation settings.

The file name and date of any plug-ins used by the model.

Training files and file dates for those files that form the training basis of the model.

4.3.2 Periodic Model Evaluation

After a model has been placed in service, it will be periodically evaluated by plant personnel. This evaluation is intended to accomplish the following:

Confirm that instrument channels included in the model are performing properly.

Identify any channels that appear to have drifted beyond acceptable limits.

Identify any concerns about equipment-condition monitoring, depending on whether or not the model was designed to function as a condition-monitoring tool.

Determine whether the model appears to be adequately trained for the most recent data.

Confirm that model settings continue to be adequate for the model and plant operating state.

4.3.2.1 Example Model Evaluation Procedure

The following provides a typical procedure for periodic model evaluation. This procedure is based on a single model and the instrument channels evaluated by the model. The model used to illustrate the procedure is HP FW HEATERS, which contains various instrument channels in the high-pressure, feedwater heater system.

1.0 PURPOSE

1.1 Confirm that instrument channels included within the scope of the on-line monitoring program are operating within acceptable limits.

1.2 Verify that the on-line monitoring system for the evaluated model is operating normally and does not require modification to the model settings.

4-3


1.3 The procedure applies to the HP FW HEATERS model, which con1 instrument channels in the high-pressure, feedwater heater system:

Computer Point Instrument Channel Description Tag Number

I MFW pump A pressure 1 PT3-66 I P2214A I I MFW pump B pressure I PT3-80 I P2215A I

MFW pump A outlet temperature TE 3-68 T2362A

MFW pump B outlet temperature TE 3-82 T2363A

MFW pump outlet header temperature TE 3-2 T2364A

MFW pump A outlet flow FT 3-70 F2250A

MFW pump B outlet flow FT 3-84 F2251A

FW heater AI inlet temperature TE 3-6 T2240A

FW heater B1 inlet temperature TE 3-8 T2260A

inlet temperature TE 3-16 T224 1 A

FW heater A1 outlet temperature TE 3-18 T2261A

outlet temperature TE 3-26 T2242A

FW heater C1 outlet temperature

FW heater 1 outlet pressure

SG 1A inlet flow FT 3-35A

I SG IB inlet flow I FT3-358 I F0404A I SG 2A inlet flow 1 FT3-48A I F0423A

I SG 28 inlet flow I FT3-488 I F0424A I SG 3A inlet flow FT 3-90A F0443A

FT 3-908 F0444A

SG 4A inlet flow FT 3-103A F0463A

SG 46 inlet flow FT 3-1038 F0464A

SG 1 inlet temperature TE 3-36 TO4 1 8A

SG 2 inlet temperature TE 3-49 T0438A

Lains the following

4-4


lnstrume'nt Channel Description Computer Point Tag Number

SG 3 inlet temperature I TE3-91 ' 1 T0458A

SG 1 inlet pressure

SG 2 inlet pressure

~ ~~ ~~

SG 4 inlet temperature

PT 3-37 P0403A

PT 3-50 P0423A

I TE3-104 I T0478A

SG 3 inlet pressure

SG 4 inlet pressure

PT 3-92 P0443A

PT 3-1 05 P0463A

NOTE: The period of time evaluated by this procedure is plant specific. Some plants might choose to operate in an on-line mode in which data is available on . a near real-time basis. Other plants might choose to operate in a batch mode and evaluate the last quarter, last month, or some other period of data.

2.0

3.0

4.0

REFERENCES

2.1 Plant System Modeling Guidelines to Implement On-Line Monitoring, EPRI, Palo Alto, CA: 2002. Interim Report 1003661

2.2 Suresense Diagnostic Monitoring Studio Users Guide, Version 1.4, EPRI, Palo Alto, CA: 2002. Interim Report

PLANT STATUS

3.1 Normally operating.

PREREQUISITES

4.1 The on-line monitoring system software is operational.

4.3 The performer of this procedure has been trained to use the on-line monitoring software and evaluate the test results.

4-5


5.0 PROCEDURE

5.1 Start the on-line monitoring software and open the HP FW HEATERS model.

5.2 Ensure that the model used for evaluation is the approved model. Verify the following initial conditions:

The model file name, date, size, and revision number is correct.

The model is trained for use and has been trained on the file@) specified by the model documentation.

Model settings are correct in accordance with the model documentation.

Acceptance limits have been specified for the instrument channels in the model.

5.3 Locate the test file containing the most recent data. If necessary, link this file to the model and run the model using this latest test data.

5.4 tJpon completion of the monitoring run, review the run results. Identify any channels that were identified as failed during the run.

5.5 Review the observation-estimate plot and the residual'plot for each channel identified as failed during the run. Classify the identified failure into one of the following categories:

The channel has drifted beyond acceptable limits. A channel recalibration will be necessary.

The channel shows evidence of some drift, but a review of the residual plot confirms that the drift is not significant. A channel recalibration should not be necessary.

Alarms were generated because of a plant- or system-operating transient for which the model was inadequately trained. If channel performance is acceptable before and after the transient, recalibration should not be necessary.

The model is not adequately trained for the plant-operating state. Model settings or model retraining with additional data might be necessary.

4-6


5.6 Initiate a work order to recalibrate any instrument channels that have drifted beyond acceptable limits.

5.7 Initiate a request to update the model if alarms were generated because of inadequate training for the plant or system operating state.

4-7

5 FAILURE DETECTION AND ALARM RESPONSE

Section 5 provides information regarding failure detection by an on-line monitoring system, as well as the recommended response to different failure types. This information is intended to assist plants with developing a consistent response to alarms. It also provides a basis for evaluating an on-line monitoring model and assessing its need for additional training.

Section 5 also provides examples to illustrate how on-line monitoring models can react to different system operating conditions. These examples have been obtained from models developed as part of the EPRI on-line monitoring implementation project.

5.1 Summary of How Failures Are Identified by On-Line Monitoring

As data observations are processed through a model, estimates for the observations are calculated. Elach residual (the difference between an observation and its corresponding estimate) is then evaluated by a series of statistical tests, and if appropriate, an alarm is generated. But, an alarm is typically only the first part of a failure determination; failure declaration is usually a two-step procedure:

If the residual is larger than expected by the model settings, an alarm will be generated for that signal.

Depending on the model settings, a failure declaration will be made based on the number of alarms generated within a sequence of observations. For example, it might take five alarms out of a series of 10 observations to achieve a failure declaration.

Even if everything is working well, occasional alarms will occur because of random variations in the data or minor process variations. This is why a single alarm is not considered a failure; it takes a certain number of alarms within a short sequence before a failure will be declared.

5.2 Recommended Response to an Identified Failure

Based on the experience to date, instruments used in nuclear plants are generally well behaved with only occasional occurrences of unacceptable drift or failure. Accordingly, most failures identified by the on-line monitoring system are not failures; instead, alarms will often result from operating states that have not been adequately defined by the training data. Sometimes, the failure alarms are caused by overly sensitive fault-detection settings. The following summarizes the most likely causes of alarms or failure declaration:

Data-acquisition problems resulting in bad input data.

5-1

Failure Detection and Alarm Response

Overly sensitive fault detection.

The EPRI on-line monitoring implementation project has developed dozens of models for hundreds of sensors at a variety of nuclear plants and significant instrument drift or failure has rarely been observed in the models developed to date. Even if an instrument channel is drifting, the fault detection method can be very sensitive, often producing failure alarms long before the drift is significant. Figure 5-1 shows an example of typical drift behavior (observations are shown as blue crosses and estimates are shown as red triangles); it is apparent that this pressure transmitter is slowly drifting low and it has drifted low by approximately 1 .O% over a period of three months. Figure 5-2 provides the residual plot for this signal. A residual plot shows the difference between the observations and estimates and is a useful tool for assessing the significance of the drift. The failure was first identified when the drift was approximately -3.0 psig (73.8 F a ) , or approximately 0.35% of span.

Operating outside the training space - permanent or temporary changes in process values.

Operating outside the training space - short-term transients.

Inadequate model training for the operating state.

Equipment changes, such as system repairs or instrument calibrations.

Figure 5-1 Typical Drift Behavior

5-2

-~ ~


- 1 1 20 I ! ! I I ,.

-14.00 I I 07109101 wnsni 08105101 08/191oi wmimi w / i m w m m i 10112101 ion5mi iim8mi

TLN P0398A. &dd n Timr

Figure 5-2 Residual Plot Showing Drift Significance Upon Failure Alarm

The first response to a failure declaration is to determine if it is a false alarm. The termfalse alarm is used in this context as any cause for a failure alarm that is not associated with a true instrument drift or equipment problem. The experience to date is that most failures identified by on-line monitoring are not caused by an actual instrument-drift event. Before assuming that a failure alarm represents a real out-of-calibration condition for the identified channel, first evaluate the model and its performance for the operating state. The following sections discuss different types of spurious or false alarms, and provide recommendations for how to respond to these alarms.

5.3 Failures Identified Because of Data-Acquisition Problems

Data-acquisition problems will cause occasional failure alarms. Data-quality problems are often readily identifiable, particularly if there are redundant signals to allow direct comparison of the data. In extreme cases, a signal data-quality problem might persist for months in archived data; Figure 5-3 and Figure 5-4 show examples of data that was stuck for several months. In these examples, the actual process signals are not stuck; instead, the data stored in the data archive are incorrect. As a result, the data provided to the on-line monitoring system are incorrect and will be identified as such by a failure alarm.

5-3


I 15.301 I I I # I 1 oimmi 03/16mi 0 5 n m i 07n8101 iommi im8ni 02113102 wnom2 06136102 o g n i a

Tim

PT-003-102A Obrm4onn. T m * PT003-102A: Estinutionn. Time

Figure 5-3 Extreme Example of Data-Acquisition Error - Stuck Data

Figure 5-4 Extreme Example of Data-Acquisition Error - Data Stuck for Almost.One Year

In the cases shown above, these stuck signals were left in the model and it was recognized that the archived data were in error. The signals were not removed from the model so that signal validation could be performed again once acceptable data were obtained. Notice that the data-storage problem in both examples above was eventually corrected.

5-4


Not all failure alarms will be caused by long-term data problems as shown above. Many data- related problems will involve random spurious data values, occasional data dropouts, or other short-term data errors. Occasional short-term failure alarms caused by data-acquisition problems can be handled in various ways:

Annotate the model with a note explaining that the identified failures are caused by bad data or a data-acquisition problem. This approach leaves the bad data in historical data sets.

Remove the bad data from the data set so that it does not continue to cause failure alarms whenever the data are rerun. This is the preferred approach for data files that are developed or archived for use as model-training or verification-testing data sets. Bad data must be removed from training data sets; otherwise, the model will be trained to recognize the bad data as normal.

Annotate the database with data-quality information that specifies the bad data as invalid. This is the preferred approach when the plant’s operating database is used as the permanent repository for the data in question.

5.4 Operation Outside the Training Space

The estimation procedure depends on the quality and range of the training data, and MSET cannot produce an estimate significantly outside of the range of data stored in the training matrix. However, not every system operating state will be known or available in the data when developing a model, and it is not unusual for some operating states to shift signals to outside the initial-training boundaries. In these cases, these signals will be appropriately identified as failed by the model because they are outside the region defined as “normal and expected behavior.” The model likely needs additional training data to describe these new operating states. Some models will rarely have this happen; other models will have multiple operating states and it might be difficult to initially train for all possible states. The following sections show instances in which model retraining might be necessary.

5.4.1 Permanent Change in Operating Space - Normal Process Changes

If the operating space shifts significantly, multiple failure alarms might be generated as shown in Figure 5-5 . Notice that almost all steam-pressure channels are identified as failed; this is a SureSense display in which the failed signals are highlighted in yellow rather than red, where yellow indicates that a group of signals are outside the training space. This type of fault-detection behavior indicates that the model is operating outside its training space, which is clearly shown for the actual data (refer to Figure 5-6). Notice that the model is trained for a steam pressure of approximately 980 psig (6852 Wa), but average pressure for all signals permanently increased to 985 psig (6886 kPa) during the evaluated time period. The signals are not faulty; the model is inadequately trained for the new operating condition. The model should be retrained with additional data that reflect this operating state.

5-5


995

mo

985 Pressure

(Pig) 980

975

970

965

Simple Steam System Model from Steam Generator to Feedllne

Valid Test Data Training Data

Steam Generator C

Reactor Power

-‘-w 1-F-I1

Steam Line A 1-F-12

p5Ei 1-P-11 -Q 1- 1-P-12

Turbine

1-F-01

Feed A 1-F-02

-‘m 1-F-03

F ~ ~ ~ u ~ ~ B 1-F-04

Figure 5-5 Test Data Outside Training Range - SureSense Result

Test Data Outside Training Range - Actual Data

5-6


5.4.2 Permanent Change in Operating Space - System-Operation Change

The initial training of a model is usually based on the best available understanding of how the system varies over time. Hopefully, the training data set bounds all normally expected operating states. However, there can be periods of plant operation during which system process parameters are constantly changing, even if power is nearly constant. One example is the end-of-cycle coast-down period for a boiling water reactor (BWR). Figure 5-7 shows a typical example; the flow for this system is essentially constant until the end-of-cycle coastdown. If the model was only trained to recognize the earlier period of system operation as normal, failure alarms will start as soon as the system operation changes.

Figure 5-7 Test Data Outside Training Range - System Operation

Despite the best efforts to acquire training data that represents normal system operation, system-operating changes can occur that invalidate all previous training data. As an example, consider the water-storage-tank level shown in Figure 5-8. This plot shows normal behavior in which the tank level exhibits daily level variations as the tank heats up during the day and cools down at night. Over the period of one to two weeks, tank level slowly falls to just less than 98%, at which time operators refill the tank to almost 98.5%.

5-7


f 1--1-

98 06

97.99 I _ - _ _ - II

97 93 P F - 7 - I

TitlU LT-63-50: 0 b - h w. Tinu

Figure 5-8 Tank-Level Variation

Suppose operators change the way in which the tank level is controlled. For example, they might choose to let tank level fall to 97% before refilling and then refill to more than 99%. If this happens, the range of operation will have almost quadrupled from the range initially used for training. Once it is recognized that this has happened, it will be necessary to retrain the model with data that describe the new operating space. This type of change can occur with any system in which operators have some control in how the system is operated, including any adjustable control system.

5.4.3 Permanent Change in Operating Space - Equipment Repair

The process values that define a normal operating space can change because of an equipment repair or replacement. Figure 5-9 shows an example in which a reactor’s coolant-pump impeller was repaired. Before the repair, measured flow was approximately 78% of span; after the repair, measured flow increased to approximately 84% of span. As can be seen, there is an immediate step change in the measured flow. Figure 5- 10 shows that MSET cannot produce an estimate anywhere near the new operating state (observations shown as blue crosses and estimates as red triangles) and the estimates are flat-topped at the limit of the data used for training. The only approach that can be taken here is to retrain with data reflecting the new operating condition.

5-8


92

88

Flow 84 (percent)

80

76

Before Repair

1- After Repair

72 ' Mar May Jul B P Nov

Time (month)

Figure 5-9 Test Data Outside the Training Range - Equipment Repair

Figure 5-10 Test Data Outside the Training Range - MSET Results

Figure 5-1 1 shows a more subtle type of equipment repair or replacement (observations are shown as blue crosses and the estimates are shown as red triangles). Engineers wanted to monitor a signal at a higher data-acquisition frequency and replaced the data-acquisition card in the plant computer. The old card sampled at a 2-second rate and the new card sampled at a 0.1 -second rate. Several other signals are also passed through this module. Unfortunately, there is an obvious change in signal noise before and after the card replacement that cannot be explained

5-9


alone by the higher sample frequency. All of a sudden, relatively clean data appear to be very noisy and routinely wanders outside the training range. The model recognizes this behavior as abnormal in comparison to its training data. In this example, the data-acquisition card should probably be restored to its original configuration.

Befare Input Matvlechange

Figure 5-1 1 Test Data Outside the Training Range - Replacement of Data-Acquisition Card

5.5 Incorrect Initial Training

Figure 5-12 illustrates another type of potential problem in which a model was trained with one steam-flow sensor already out of calibration (observations are shown as blue crosses and the estimates are shown as red triangles). The model has been trained to recognize the erroneously low signal of this steam-flow sensor as normal. Eventually, the out-of-calibration condition was identified by conventional methods and the transmitter was recalibrated. After calibration, the sensor is now identified as failed because the model has been trained with bad data. The model should be retrained with new data after the calibration. The prior training data must be excluded because they incorrectly identify the out-of-calibration condition as normal.

5-10


mo Tknc

R-1-286: Estimation VI. T k x. R-1-28A: Obstrvrtionvs. Tknc

Figure 5-1 2 Recalibrated Sensor - Model Trained on Out-of-Calibration Data

Figure 5-13 shows a minor example of an incorrectly trained model (observations are shown as blue crosses and the estimates are shown as red triangles). This temperature sensor varied from other redundant sensors by approximately 4 degrees, but the model was trained to recognize this behavior as normal. After recalibration, this sensor matched the signals from the other redundant channels, but the model routinely declared this channel as failed because it was not trained for this new operating condition.

5-1 I


102

Timr

TE-004-104C: C h m t m n w . Tvlw TE-009-104C: Estinutionw. T u

Figure 5-13 Recalibrated Sensor - Initially Out of Calibration

5.6 Excluding Transient or Abnormal Operating States

EPRI Report 100366 1, Plant System Modeling Guidelines to Implement On-Line Monitoring, explains the advantages of using sub-models that are established by partitioning the overall model into discrete operating states. The procedure used to define and partition these operating states is referred to as aphuse determiner and its setup defines the phase in which the system is operating at any given time. For example, reactor power is commonly used to segregate high-power operation (near 100% power) from lower-power operation. Even when a phase determiner is applied to a model, abnormal conditions might occur near the phase boundary between the sub-models. Figure 5-14 shows an example in which turbine first-stage pressure operates for an extended period at approximately 735 psig (5 162 kPa), but drops to almost 725 psig (5093 kPa) during a short transient, which is outside the range of the data used for training.

5-12

~~


Time P0399A: Observation YT. Tkne W399A: Estimation YT. Time

01

Figure 5-14 Signal Behavior During a Transient

Figure 5-15 shows another example in which a single transient occurs during an entire year of operation. The model was not trained to recognize this transient as normal behavior, thereby declaring the channel as failed during the transient. Immediately after the transient, the channel returns to its expected operating state. In this instance, there is little benefit in training the model to recognize this transient behavior.

5-13

~


433 00

5 431.M) 42900

e* 427.00 v

- - _ a - a 42500 t - I

e- 42300 , - _ _ _ _ _ _ I-

8 421.00

.9 1 11700

$ 415.00

413.00 1211

9 41900 - ___

Tinu TE-GU6-108A Obmvatmnw. Tinu A TE-006-108A: EstLartiouvr. T k

Figure 5-1 5 One Transient During an Extended Period of Operation

Short-term transients can occur that cause signal values to vary outside the training space. As these transients occur, signal failures will appropriately be identified because the model does not recognize this operating state. Figure 5-1 6 shows another example of short-term transients. Notice that there are periodic spikes in the flow rate as pumps are adjusted, These flow spikes are normal and expected events. The training data included the first two spikes because this is a known-and-expected operating state, but notice that the next three spikes exceed the limits of the training data, thereby resulting in alarms.

5-14


Maximum values in training data Transients exceed

training data limits

-.-"- 11105100 12mm I

Figure 5-16 Example of Routine Transients Exceeding the Training Space

One of three approaches can be taken here:

Retrain the model with additional data from the largest observed transient so that future smaller transients are not treated as signal failures.

Do not retrain the model with additional data and accept that failure declarations might occur during these transients. This is generally the preferred approach for short-term transients such as shown in Figure 5-16. Notice that the observations (blue crosses) and estimates (red triangles) immediately before and after each transient appear normal with little deviation. This channel appears to be operating normally with no problems.

Define a separate phase (using a phase determiner) for the transient behavior. This would cause transient data to be partitioned from the other data, effectively turning off all fault-detection processing while the system was in the transient phase and eliminating the false alarms.

If the transient rarely occurs, no action is the preferred approach. If the transient routinely occurs, retraining to include the transient or excluding the transient using the phase determiner is probably the preferred approach. Review transients and the signal behavior at the phase boundaries. In some cases, the phase determiner should be modified to exclude untrained transients such as this.

5-15


5.7 Occasional Outliers

Occasional data spikes or dips will occur that defy any explanation. Some of these occurrences are probably data-acquisition problems, although the specific cause is often difficult to prove. Figure 5-1 7 shows an example in which two redundant, steam-pressure transmitters simultaneously dip from the normal operating range of approximately 830 psig to approximately 810 psig (5817 to 5679 Wa). This is enough of a change to cause both channels to be simultaneously declared as failed. This dip only occurred once during the operating cycle. A review of other correlated data shows that no transient occurred during this period and the data are most likely a partial data dropout caused by the data-acquisition system.

Figure 5-17 Short-Term Outlier

If an identified failure “recovers,” the problem was probably not a sensor or instrumentation problem. What is important in terms of drift detection is whether or not the channel behavior permanently changes over time. Referring to Figure 5-17, it is apparent that a short transient of some sort or a partial data dropout simultaneously affected both redundant channels, with a full recovery to normal conditions within a short period. Any failures identified during this period are not actual instrument failures; they are a consequence of either inadequate training for this event or data-acquisition problems.

The signal behavior shown in Figure 5-17 is acceptable before and after the event. Rather than attempting to train the model to recognize an unexplained transient that only happened once; it is recommended that no action be taken. Rare and unexplained events will occur in real-world data.

5-16


The model's recognition of these events as abnormal is an expected and desirable result. It is preferable to evaluate such events on a case-by-case basis rather than including these rare cases in the training data and thereby incurring the risk of desensitizing the model to failure events in general.

5.8 Equipment Operating States Not Covered by Available Training Data

There are likely to be some models for which it will be difficult to train for all possible operating states. Figure 5-1 8 provides a simple exampleaepending on the system requirements, either one, two, or three pumps might be running.

Figure 5.,18 Many Possible Operating States

Referring to Figure 5-1 8, consider the following possible system flow conditions:

Low flow - only one pump is running. Three different operating states are possible.

Medium flow - two pumps are running. Three different operating states are possible.

High flow - all three pumps are running. This is probably the easiest operating state to model.

As shown above, there are at least seven different operating states if the three pumps are individually instrumented. This becomes even more complex if the pumps can operate at variable speed.

Some plants preferentially operate two specific pumps with a third pump off. Figure 5- 19 shows an example of a condensate booster pump that almost always runs. While running, a pump's bearing-temperature measurement usually indicates approximately 150°F (65.6"C). But, when the pump motor is turned off, the temperature drops to approximately 75°F (23.9"C). During an 1 8-month period, this pump was stopped once for only two days. For the non-operating state, it is unlikely that adequate training data will be available for such an infrequent operating configuration.

5-17


Figure 5-1 9 Example of a Pump That Almost Always Runs

Uncommon operating states are a real issue for training and operating a model. Which operating states will be validated and which operating states will not be validated? For many models, it is unlikely that adequate training data can be acquired to train the model on all possible operating states. For that reason, phase determiners should always be considered so that untrained operating states are excluded from signal validation and subsequent failure alarms.

5.9 Inadequate Initial Training Within the Defined Operating Space

The MSET training vector selection method is not perfect. Even when adequate training data are available to define an operating space, estimation can be influenced by the combination of observations compared to the corresponding vectors contained in the training matrix. Figure 5-20 shows an example in which the observations (blue crosses) and the estimates (red triangles) track together very well for an extended period, followed by the estimations jumping while the observations remain almost unchanged. This change in the estimate was prompted by a small dip in the observations, which took some time before the estimates again tracked with the observations. The vectors selected for training do not adequately cover the operating space in this particular region and additional vectors should be specified for the training matrix.

5-18


5.03

5.00

.' FY-006-1K606C: Obscwation YF. Titne Timc

F!?-006-1K606C: LstMionvs. Time

Figure 5-20 Unexpected Change in Estimate

Figure 5-2 1 shows a more dramatic example of a pump's bearing-temperature sensor (observations are shown as blue crosses and estimates are shown as red triangles). The model was trained to recognize both running and off conditions. As trained, the model has trouble differentiating between the two states because of the influence of other signals in the model. The model needs to be retrained with additional training vectors. Or, if the pump is seldom off, it should probably be trained with either 1) a phase determiner based on the pump state or 2) without the pump off data so that it does not try to perform fault detection in both of the two possible states using a single model.

5-19


Figure 5-21 Inadequate Training for Pump On and Off Conditions

5.10 Using Threshold Settings for Overly Sensitive Alarms

5.70.7 Overview of Threshold Seftings

Fault detection is based on a comparison of the difference or deviation between each observation and its corresponding estimate, referred to as the residual. Depending on 1) the signal's fault-detection settings and 2) the variance of the training-data residual values, fault detection can be very sensitive, resulting in alarm generation for deviations of less than 0.1%. For many applications, this is more sensitive than necessary and it might be preferable to apply simple threshold settings as a fault-detection tool.

Threshold settings are applied directly to the residuals for an analysis. Figure 5-22 shows an example of threshold settings applied to a signal. Two levels of drift have been specified-an allowable drift of *l% and a maximum drift of &1.5%. As the sensor output starts drifting low, the residuals become increasingly more negative, eventually reaching the allowable drift limit.

5-20

~ ~~


1 70 _.__ .-_.. .~ " __.

0.68

G 034 3

1 Oo0 -034

-0 68

- 1 02

-1 36

-1 70 02101101 02m7mi 0 2 ~ o i 02121101 0.2128101 03m710i 03/i4101 03nimi 03a810i 04m4mi

T m LT-02 Res&aIw T m

Figure 5-22 Threshold Settings Applied to Residual Plot

The channel shown above was identified as failed by the MSET statistically based fault-detection method when the residual was only approximately -0.2%. But, this small of a drift is not an immediate concern. By providing clear channel-acceptance limits, a residual plot provides another tool to assess the urgency of any required corrective action. As shown in Figure 5-22, drift is occurring at a slow rate, which can allow for pre-planning of any calibration activity.

The maximum acceptable drift should be specified for each signal validated by the model. Whenever failure alarms are received, a residual plot, such as the one shown in Figure 5-22, can provide additional insight into the severity of the detected drift.

5.10.2 Establishing Threshold Limits

Figure 5-22 shows an example in which threshold limits are specified at kl-percent-allowable and f 1.5-percent-maximum drift. These particular limits are useful for evaluating fault detection during model development, but should be adjusted as necessary before the model is placed in service. As part of model completion, threshold limits should be established based on the drift allowances of the monitored instrument channels. For safety-related channels, the threshold limits depend on the allowances specified in the corresponding setpoint study, if applicable. For non-safety-related channels, the threshold limits depend on the allowed channel variation for acceptable system performance.

Table 5- 1 shows typical elements of uncertainty or accuracy that are considered significant for most instrument loops. The uncertainty terms associated with the sensor are usually of most interest. Rack-mounted, signal-conditioning modules usually perform quite well and often have a

5-2 1


corresponding lower contribution to the channel uncertainty, but still should be considered whenever they are included in the instrument circuit.

Table 5-1 Instrument Channel Uncertainty Sources

Present in On-Line Present in Safety- Included in Sensor Monitoring Path? Related Trip Path? Calibration? Uncertainty Term

Process measurement effect (P M E) X X

Process element accuracy (PEA) X X

Sensor reference accuracy (SRA) X X X

Sensor drift (SD) X X X

Sensor temperature effect (SD) (normal variation) X X X (partial)

Sensor pressure effect (SPE) X X

Sensor vibration (SV) X X

Sensor M&TE accuracy (SMTE) X X X

Isolator reference accuracy (IRA) X

Isolator drift (ID) X

Isolator temperature effect (ITE) X

Isolator M&TE accuracy (IMTE) X

Computer input AID accuracy (AID)

Bistable reference accuracy (BRA) X

Bistable drift (BD) X

Bistable temperature effect (BTE) X

Bistable M&TE accuracy (BMTE) X

With regard to on-line monitoring, the sensor uncertainty elements can be grouped according to whether they are associated with 1) process/environmental effects or 2) calibration effects. The uncertainty elements associated with process/environmental effects explain why redundant channels might not display the same value; there is some random variation in the measurements caused by these uncertainty elements. The uncertainty elements associated with calibration effects are specifically what is being evaluated by an on-line monitoring program and it is these terms that should relate directly to the specified threshold limits.

X

5-22


Suppose the following uncertainty values are provided (or are considered allowable) for a non- safety-related channel:

SA - i0.5%, sensor reference accuracy

SD - i1.5%, sensor drift

SMTE - i0.25%, measurement and test equipment uncertainty

The combined uncertainty of the above can be calculated as follows:

SU = *,/SA2 + SD2 + SMTE' Eq. 5-1

SU = f ,/0.52 + 1.52 + 0.252 = 1.6%

The uncertainty of the MSET estimate is also a consideration, because it can affect the drift allowance for the channel also. For example, if the estimate uncertainty were *0.25%, the above uncertainty calculation would be adjusted as follows:

Eq. 5-2 2 SU = f dSA2 + SD2 + SMTE' -MSET,,,

SU = +.,/0.52 + 1.52 + 0.252 -0.252 = 1.58%

For this channel, threshold limits might be established as follows:

Allowable - 1.2% (arbitrarily selected)

Maximum - 1.58%

The EPRI on-line monitoring implementation project is coordinating with two DOE NEPO projects to review the estimation uncertainty for various plant models and data types. The next update to these guidelines will include the results from these projects. Safety-related instruments have additional considerations for uncertainty; refer to EPRI Report TR-104965-R1 NRC SER, On-Line Monitoring of Instrument Channel Performance and EPRI Report 1006833, Implementation of On-Line Monitoring for Technical SpeciJication Instruments.

5-23

REFERENCES

Cost Benefits of On-Line Monitoring, EPRI, Palo Alto, CA: 2002. 1003572.

Implementation of On-Line Monitoring for Technical Specification Instruments, EPRI, Palo Alto, CA: 2002. 1006833.

On-Line Monitoring of Instrument Channel Performance, EPRI, Palo Alto, CA: 2000. TR-104965-R1 NRC SER.

Plant System Modeling Guidelines to Implement On-Line Monitoring, EPRI, Palo Alto, CA: 2002. 1003661.

Suresense@ Diagnostic Monitoring Studio=* Users Guide, Version 1.4, EPRI, Palo Alto, CA: 2002.

6- 1

GLOSSARY

This glossary provides definitions for technical terms used in the report or otherwise applied to on-line monitoring. Abbreviations used in the body of the report are also included in the glossary.

-A-

Accuracy (Reference) - In process instrumentation, a number or quantity that defines a limit that error should not exceed when a device is used under specified operating conditions. Error represents the difference between the measured value and the standard or ideal value.

Adjustment - The activity of physically adjusting a device to leave it in a state in which its performance characteristics are within acceptable limits.

ANL - Argonne National Laboratory.

ANN - Artificial neural network.

API - Application programming interface. A set of software functions or methods externally callable by another software program.

As-Found - The condition in which a channel, or portion of a channel, is found after a period of operation and before any calibration.

As-Left - The condition in which a channel, or portion of a channel, is left after calibration or surveillance check.

-B-

Bayesian Belief Network - A mathematical method of specifying the probabilistic relationships among events. In the context of on-line monitoring, a belief network is an expression of the probabilistic knowledge of a system and its operation.

Bayesian Conditional Probability - An inference procedure for determining signal failure based on a preceding number of alarms.

BCP - Bayesian conditional probability.

Belief Network - refer to Bayesian belief network.

A- 1

Glossary

BWR - Boiling water reactor.

-C-

Calibration - The process of adjustment, as necessary, of the output of a device such that it responds within a specified tolerance to known values of input.

Calibration Interval - The elapsed time between the initiation or successful completion of calibrations or calibration checks on the same instrument, channel, instrument loop, or other specified system or device.

Calibration (Time-Directed) - The calibration of an instrument at specified time intervals, without regard of the existing calibrated state of the instrument.

Channel - An arrangement of components and modules as required to generate a single protective action signal when required by a generating-station condition, a control signal, or an indication function.

Channel Check - The qualitative assessment, by operator observation, of channel behavior during operation and includes, where possible, comparison of the channel indication to other indications from other redundant channels measuring the same parameter.

CMP - Configuration management plan.

CSV - Comma-delimited file format.

-D- DDD - Design description document.

Desired Value - A measurement value with no error existing.

Deviation - The difference between the parameter estimate and the monitored signal (more commonly referred to as the residual).

DOE - U.S. Department of Energy.

Drift - An undesired change in output over a period of time that is unrelated to the input, environment, or load.

-E-

Error - The undesired algebraic difference between a value that results from measurement and a corresponding true value.

Estimate - The best estimate of the actual process value; used interchangeably with parameter estimate.

A-2

Glossary

-F-

Field calibration - Performing the activities of surveillance and adjustment using an external reference source.

FRD - Functional requirements document.

-1-

ICMP - Instrument Calibration and Monitoring Program.

IMC - Instrument monitoring and calibration.

Initial Training - see Training.

Instrument Channel - An arrangement of components and modules as required to generate a single protective action or indication signal that is required by a generating station condition. A channel loses its identity when single protective action signals are combined.

-L- Linear - A straight-line relationship between one variable and another. When used to describe the output of an instrument, it means that the output is proportional to the input.

Loop - see Channel.

-M-

M&TE - Measuring (or measurement) and test equipment.

Margin - An additional allowance added to the instrument channel uncertainty to allow for unknown uncertainty components. The addition of margin moves the setpoint further away from the analytical limit or nominal process limits.

Model - The group of signals that have been collected for an analysis.

Module - Any assembly of interconnecting components that constitutes an identifiable device, instrument, or piece of equipment. A module can be removed as a unit and replaced with a spare. It has definable performance characteristics that permit it to be tested as a unit. A module can be a card, a drawout circuit breaker or other subassembly of a larger device, provided it meets the requirements of this definition.

Monitoring - The activity of evaluating instrument channel performance to determine that it is performing within acceptable performance limits.

MSET - Multivariate state estimation technique.

A-3

Glossaty

-N- NEPO - Nuclear Energy Plant Optimization.

Noise - An unwanted component of a signal or variable. It causes a fluctuation in a signal that tends to obscure its information content.

Nonlinear - A relationship between two or more variables that cannot be described as a straight line. When used to describe the output of an instrument, it means that the output is of a different magnitude than the input.

NRC - Nuclear Regulatory Commission.

-e \

ODBC - Open database connectivity.

OLM - On-line monitoring.

OLMS - On-line monitoring system.

On-Line Monitoring - An automated method of monitoring instrument performance and assessing instrument calibration while the plant is operating.

Operating Space - A defined region of operation.

Operating State - A defined region of operation, often established by power level or equipment lineup. Often used interchangeably with operating space.

-P-

Parameter Estimate - The best estimate of the actual process value.

Pattern Recognition - The ability of a system to match large amounts of input information simultaneously and generate a categorical or generalized output.

PDF - Probability density function.

PWR - Pressurized water reactor.

-R- Random - Describing a variable whose value at a particular future instant cannot be predicted exactly, but can only be estimated by a probability distribution function.

RCS - Reactor coolant system.

A-4

Glossary

Reference Accuracy - A number or quantity that defines the limit that errors will not exceed when the device is used under reference operating conditions.

Residual - The difference between the observation and the corresponding estimate for that observation. Also known as the residual error.

Retraining - Any change made to the set of data originally selected as representative of system normal and expected behavior.

-S- SDF - Signal data file format.

SDMS - SureSense Diagnostic Monitoring System.

Sensor - The portion of a channel that responds to changes in a plant variable or condition and converts the measured process variable into an electric or pneumatic signal.

Setpoint - see Trip Setpoint.

Signal - The output data from a channel.

Signal Conditioning - One or more modules that perform further signal conversion, buffering, isolation or mathematical operations on the signal as needed.

Span - The region for which a device is calibrated and verified to be operable.

SPRT - Sequential probability ratio test.

SQL - Structured query language.

State Space - The operating states that form the basis for training a model.

SureSense -. A commercially supported implementation of the MSET software originally developed by Argonne National Laboratory.

Surveillance - The activity of checking a device to determine if it is operating within acceptable limits.

Surveillance Interval - The elapsed time between the initiation or successful completion of a surveillance or surveillance check on the same instrument, channel, instrument loop, or other specified system or device.

A-5

Glossary

-T-

Test Interval - see Calibration Interval.

Time-Directed Calibration - see Calibration (Time-Directed)

Training - For a pattern-recognition system such as MSET, the selected vectors that describe the operating state for normal and expected behavior.

Training Matrix - the matrix of vectors selected by the MSET training process. These vectors represent the model in terms of its recognition of “normal” system behavior. Also, referred to as the D-matrix or the process memory matrix.

Trip Setpoint - A predetermined value at which a bistable device changes state to indicate that the quantity under surveillance has reached the selected value.

-U-

Uncertainty - The amount to which an instrument channel’s output is in doubt (or the allowance made therefore)because of possible errors, either random or systematic, that have not been corrected for. The uncertainty is generally identified within a probability and confidence level.

-V-

V&V - Verification and validation.

Vector (of Signals) - All data observations for a single time step. For example, if the data are contained in a spreadsheet, a single row of data is a vector.

Vector Ordering - An algorithm that adds representative vectors from the inner regions of a vector space to produce a more accurate process model. Vector ordering is differentiated from the MinMax algorithm in that it describes the interior of a space whereas the MinMax algorithm bounds the vector space.

A-6

About EPRl Zrget: Nuclear Power EPRl creates science and technology solutions for

the global energy and energy services industry. U.S.

electric utilities established the Electric Power

Research institute in 1973 as a nonprofit research

consortium for the benefit of utility members, their

customers, and society. Now known simply as EPRI,

the company provides a wide range of innovative

products and services to more than IO00 energy-

related organizations in 40 countries. EPRl’s

multidisciplinary team of scientists and engineers draws on a worldwide network of technical and business expertise t o help solve today’s toughest

energy and environmental problems.

EPRI. Electrify the World

0 2003 Electric Power Research Institute (EPRI), Inc.All rights reserved. Electric Power Research institute and EPRl are registered service marks of thc Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Instiwte. Inc.

@ Printed on recycled paper in the United Stater ofAmerica

I007622

EPRl 4 34 I 2 Hillview Avenue, Palo Alto, California 94304 PO Box IO4 12. Palo Alto, California 94303 USA 800.3 13.3774 650.855.2121 - [email protected] - w.epri .com

mailto:[email protected]

http://w.epri.com