boiler reliabiliti optimization 2001

EPRI Boiler Reliability OptimizationProgram

Case Studies from 1998-2001

Technical Report

LI

CE

NS E D

M A T E

RI

AL

WARNING:Please read the License Agreementon the back cover before removingthe Wrapping Material.

EPRI Project Manager P. Abbott

EPRI • 3412 Hillview Avenue, Palo Alto, California 94304 • PO Box 10412, Palo Alto, California 94303 • USA 800.313.3774 • 650.855.2121 • [email protected] • www.epri.com

EPRI Boiler Reliability Optimization Program Case Studies from 1998-2001

1006537

Final Report, December 2001

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

THIS DOCUMENT WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN ACCOUNT OF WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) BELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM:

(A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (III) THAT THIS DOCUMENT IS SUITABLE TO ANY PARTICULAR USER'S CIRCUMSTANCE; OR

(B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT.

ORGANIZATION(S) THAT PREPARED THIS DOCUMENT

Eskom

ORDERING INFORMATION

Requests for copies of this report should be directed to EPRI Customer Fulfillment, 1355 Willow Way, Suite 278, Concord, CA 94520, (800) 313-3774, press 2.

Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc.

Copyright © 2001 Electric Power Research Institute, Inc. All rights reserved.

iii

CITATIONS

This report was prepared by

Eskom Megawatt Park P.O. Box 1091 Johannesburg, 2000 South Africa

Principal Investigator D. McGhee

This report describes research sponsored by EPRI.

The report is a corporate document that should be cited in the literature in the following manner:

Boiler Reliability Optimization Project Case Studies from 1998 – 2001, EPRI, Palo Alto, CA: 2001. 1006537.

v

REPORT SUMMARY

The prospect of competing in a deregulated environment is driving utilities to explore more cost-effective means of operating and maintaining their plants. Recognizing this, a number of companies —including American Electric Power, Great River Energy, Detroit Edison, Arizona Public Services Company, and Hawaiian Electric Company— have participated in EPRI’s Boiler Reliability Optimization Program. This report provides seven case studies demonstrating key aspects of the program.

Background EPRI’s Boiler Reliability Optimization Program consists of several customized projects. Each project includes four phases that involve EPRI technical staff, with participation from the plant’s operating, maintenance, and engineering support staff, as follows: Phase 1—technical and organizational assessments, Phase 2—development of a boiler failure defense plan, Phase 3—implementation of a failure defense plan, and Phase 4—continuous improvement and process automation. Each phase, in turn, builds on the strengths of the preceding phase. EPRI compiled these case studies to demonstrate the scope of the program and its applicability to a wide range of power production issues.

Objective To provide case studies from membership participation in EPRI’s Boiler Reliability Optimization Program between 1998 and 2001.

Approach The principal investigators on this report selected case studies that would demonstrate key facets of the program, including development of a maintenance strategy, implementation of Streamlined Reliability Centered Maintenance (RCM) procedures, outage task prioritization, Boiler Predictive Maintenance (BPdM), performance of root cause analysis, and application of EPRI’s Boiler Maintenance Workstation (BMW) software. Each case study includes a scope of work description, findings and observations, conclusions, and recommendations.

Results Two case studies from American Electric Power—Big Sandy Plant-Unit 2 demonstrate phases 1, 2, and 4 of the program, with specific application to water- and steam-touched tubing. This project focused on the long-term integrity of the water- and steam-touched tubing within the boiler envelope. It involved training on EPRI’s Streamlined RCM software using the fuel system as an example, additional training on EPRI’s tube failure reduction program, development of a

vi

BPdM process, and the provision of EPRI’s BMW software package for trending and tracking tube failure data and information. An outage task prioritization model—Risk Evaluation and Prioritization (REAP)—was used to prioritize the tasks identified for the Unit 2 scheduled outage in April 2000.

A case study from Great River Energy’s Coal Creek Station-Unit 2 discusses the decision to change the boiler maintenance overhaul frequency from two to three years. The project emphasizes development of a comprehensive and effective boiler inspection plan to take the place of the currently reactive plan, which focuses almost entirely on sootblower erosion and does not cover all boiler components in sufficient detail.

A case study from Detroit Edison’s St Clair Plant-Unit 7 focuses on technical and programmatic opportunities for improving boiler reliability. The project identifies the causes of boiler tube failures and highlights a number of opportunities for improvement with respect to managing the short- and long-term health of the boiler.

Two case studies at Arizona Public Services Company’s Four Corners and Cholla plants emphasize root cause analysis. A review of the 1999 Lost Generation Action Plans from both plants evaluates whether the “true” root causes of lost generation have been identified, and whether appropriate action plans are in place to address these causes.

A case study from Hawaiian Electric Company (HECO) addresses EPRI’s independent review of HECO’s Remaining Useful Life and Generation Asset Management Program. The EPRI team identified a number of opportunities where current project management and root cause analysis efforts could be enhanced to provide additional assurance that availability and reliability goals would be achieved.

EPRI Perspective A Boiler Reliability Optimization Program is only as effective as the people, management systems, component manufacturers, technology specialists, and predictive maintenance tools or equipment that comprise it. For a condition-based, highly planned, maintenance optimization program to be effective, it must integrate the work process, management, work culture, technologies, and people into the total plant operation. EPRI believes this documentation of case studies demonstrates innovative applications of EPRI’s Boiler Reliability Optimization Program and methods for achieving such integration under a wide variety of circumstances. Utilities will find that implementation of the Boiler Reliability Optimization Program will help them compete cost-effectively in the deregulated power generation market.

Keywords Boiler reliability Boiler Reliability Optimization Program Competitive power generation

EPRI Licensed Material

vii

CONTENTS

1 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1, 2 & 4 AT AMERICAN ELECTRIC POWER’S BIG SANDY PLANT – UNIT 2............................................................ 1-1

1.1 EXECUTIVE SUMMARY........................................................................................... 1-1

1.2 INTRODUCTION ...................................................................................................... 1-2

1.3 SCOPE OF WORK ................................................................................................... 1-3

1.4 FINDINGS AND OBSERVATIONS............................................................................ 1-3

1.4.1 Phase I and II - Technical and Organisational Assessment.................................. 1-3

1.4.1.1 Work Process................................................................................................ 1-5

Maintenance Strategy........................................................................................... 1-7

Findings:............................................................................................................... 1-8

Recommendations:............................................................................................... 1-8

Work Identification ................................................................................................ 1-8

Findings:............................................................................................................... 1-9

Recommendations................................................................................................ 1-9

Work Control .......................................................................................................1-10

Findings...............................................................................................................1-10

Recommendations...............................................................................................1-10

Work Execution ...................................................................................................1-11

Findings...............................................................................................................1-11


1.4.1.2 Technologies................................................................................................1-11

Findings...............................................................................................................1-12

Recommendation ................................................................................................1-12

Work Management System .................................................................................1-12

Findings...............................................................................................................1-12


Maintenance & Diagnostic Technologies .............................................................1-12

Findings...............................................................................................................1-12


viii


Information Integration.........................................................................................1-13

Findings...............................................................................................................1-13


1.4.1.3 Management and Work Culture....................................................................1-13


1.4.1.4 People..........................................................................................................1-15

Findings:..............................................................................................................1-15


1.4.2 Failure Defence Plan – Water and Steam Touched Tubing..................................1-15

Findings ...................................................................................................................1-16

Economiser .........................................................................................................1-16

Waterwalls...........................................................................................................1-16

Platen Superheat.................................................................................................1-17

Finishing Superheat.............................................................................................1-17

First Reheat (High Pressure) ...............................................................................1-17

Second reheat (low pressure)..............................................................................1-18

Spray Attemperators............................................................................................1-18


1.4.3 Streamlined Reliability-Centered Maintenance.....................................................1-30

1.4.4 Unit 2 Outage Task Prioritisation..........................................................................1-30

1.4.5 Boiler Tube Failure Reduction and Cycle Chemistry Improvement Training .........1-33

1.4.6 Boiler Maintenance Workstation (BMW) installation and training..........................1-33

AEP Big Sandy Unit 2, Boiler Reliability Optimization - Phase 4, July 2001.......................1-34

Executive Summary ..........................................................................................................1-34

Introduction .......................................................................................................................1-34

Description of Phase 4 of a Typical Boiler Reliability Optimization Project.........................1-35

Findings........................................................................................................................1-35

Conclusions.......................................................................................................................1-40

Recommendations ............................................................................................................1-41

Appendix 1: Interview List..................................................................................................1-44

Appendix 2: List of Typical headings that appear in a policy and or a procedure ...............1-45


ix

2 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 2 – BOILER INSPECTION PLAN REVIEW AT GREAT RIVER ENERGY’S COAL CREEK STATION – UNIT 2 ................................................................................................................................. 2-1

2.1 Executive Summary..................................................................................................... 2-1

2.2 Introduction.................................................................................................................. 2-1

2.3 Findings and Observations .......................................................................................... 2-2

2.4 Recommendations....................................................................................................... 2-3

ANNEXURE I: Extract from a Boiler Inspection Report....................................................... 2-7

Inspection Areas ....................................................................................................... 2-7

1200 Waterwall Cleaning Devices, operational condition ............................................... 2-7

Inspection results........................................................................................................... 2-7

Recommendations for 2004: .......................................................................................... 2-8

1501 Superheat Division Panels, attachment problems ................................................ 2-8

Inspection results:.......................................................................................................... 2-8

Recommendations for 2004: .......................................................................................... 2-8

ANNEXURE II: Unit 2 Outage Inspection Plan as of August 21, 2000 ................................ 2-9

1. Burner Front Team Member...................................................................................... 2-9

1.1 Prior to the outage (within 2 weeks of the start of the outage) ............................. 2-9

1.2 At the start of the outage (During the installation of the boiler scaffolding)..........2-10

1.3 Burner Front inspection (Upon completion of the boiler scaffolding) ...................2-10

2. Waterwall Team Member .........................................................................................2-12

2.1 Prior to the outage (within 2 weeks of start of the outage) ..................................2-12

2.2 Prior to the installation of the boiler scaffolding (after the throat scaffolding is installed) ..................................................................................................................2-12

2.3 Upon completion of the boiler scaffold................................................................2-12

2.4 Upon completion of the bottom ash scaffolding ..................................................2-13

3. Superheat and Reheat Team Member .....................................................................2-14


3.2 During the outage...............................................................................................2-14

4. Back-pass Team Member ........................................................................................2-16



5. External Boiler Team Member .................................................................................2-18

5.1 Prior to the outage (within 2 weeks of the start of the outage) ............................2-18



x

3 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1 AT DETROIT EDISON’S ST CLAIR PLANT – UNIT 7 AMERICAN ELECTRIC POWER’S BIG SANDY PLANT – UNIT 2 ....................................................................................................... 3-1

Executive Summary ........................................................................................................... 3-1

3.1 Introduction.................................................................................................................. 3-3

3.2 Findings and Observations .......................................................................................... 3-3

3.2.1 Programmatic and Data Sources Assessment ..................................................... 3-3

3.2.1.1 Maintenance Strategy................................................................................... 3-6

3.2.1.1.1 Findings ................................................................................................ 3-6

3.2.1.1.2 Recommendations................................................................................ 3-7

3.2.1.2 Work Process............................................................................................... 3-7

3.2.1.2.1 Findings ................................................................................................ 3-8

3.2.1.2.2 Recommendations ................................................................................ 3-8

3.2.1.3 Predictive Maintenance Program and Technology.......................................3-10

3.2.1.3.1 Findings ...............................................................................................3-10

3.2.1.3.2 Recommendations...............................................................................3-11

3.2.1.4 People.........................................................................................................3-11

3.2.1.4.1 Findings ...............................................................................................3-11

3.2.1.4.2 Recommendations...............................................................................3-12

3.2.2 Failures and Root Causes...................................................................................3-12

3.2.2.1 Boiler Tubes ................................................................................................3-12

3.2.2.1.1 Failure Mechanisms, Actions taken and Recommendations ................3-14

3.2.2.2 Boiler Headers.............................................................................................3-18

3.2.2.2.1 Boiler Test and Inspection Plan Guideline............................................3-19

Purpose...............................................................................................................3-19

Definitions............................................................................................................3-19

Scope ..................................................................................................................3-19

Inspection and Test Plan Development ...............................................................3-20

3.3 Conclusions................................................................................................................3-21

3.4 Recommendations......................................................................................................3-21

Appendix 1: List of interviewees ........................................................................................3-23

Appendix 2: Example of a Typical Equipment Condition and Technology Matrix ...............3-24

Appendix 3: Remnant Life Assessment Flow Chart for Thick Section................................3-25


xi

4 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1- ROOT CAUSE ANALYSIS AT ARIZONA PUBLIC SERVICES COMPANY’S FOUR CORNERS AND CHOLLA PLANTS.................................................................................................................. 4-1

FOUR CORNERS PLANT.................................................................................................. 4-1

4.1 Executive Summary...................................................................................................... 4-1

4.2 Introduction................................................................................................................... 4-2

4.3 Plant Description .......................................................................................................... 4-3

4.4 Incidents investigated ................................................................................................... 4-3

4.4.1 Unit 5 #4 Turbine Control Valve chatter - 197,471 Mwh lost................................ 4-4

Incident Description................................................................................................... 4-4

Root Cause Analysis Chart ....................................................................................... 4-4

4.4.2 Unit 3 Scrubber unit failure - 33,446 Mwh lost...................................................... 4-5


Root Cause Analysis Charts...................................................................................... 4-6

4.4.3 Unit 4 Loss of Booster Fans 38,413 Mwh.......................................................... 4-7



4.4.4 Unit 3 Pulverizer Pinion & Bull Gear Failure - 34,881 MW Hrs. ............................ 4-7



4.4.5 Unit 3 Back-pass Flyash Erosion 39,378 Mwh ................................................... 4-8

Problem Description.................................................................................................. 4-8


4.4.6 Unit 4 Air Pre-heater Failure 166,744 Mwh. ....................................................... 4-9



4.4.7 Units 4 Flyash Erosion 39,288 MW/Hrs.............................................................. 4-9


Root Cause Analysis Chart ......................................................................................4-10

4.4.8 Unit 3 Turbine H.P. Nozzle Pluggage 100,628 Mwh..........................................4-10

Problem Description.................................................................................................4-10


4.4.9 Unit 4 LP Exciter Field Failure 45,229 Mwh........................................................4-11

Incident Description..................................................................................................4-11

Root Cause Analysis................................................................................................4-11


xii

4.4.10 Unit 4 HP Exciter Field Failure 33,500 Mwh .....................................................4-11


Root Cause Analysis................................................................................................4-11

4.5 Findings ...................................................................................................................4-12

4.6 Conclusions.................................................................................................................4-12

4.7 Recommendations.......................................................................................................4-13

4.8 Appendix 1: Root Cause Analysis Charts ...................................................................4-13

CHOLLA PLANT ...............................................................................................................4-26

4.9 Executive Summary.....................................................................................................4-26

4.10 Introduction................................................................................................................4-27

4.11 Plant Description .......................................................................................................4-28

4.12 Incidents Investigated................................................................................................4-28

4.12.1 Mechanical Dust Collector Retrofit 41,146 Mwh. ..............................................4-29



4.12.2 Unit One ID Booster Fan Bearing Failures 7,544 Mwh. ....................................4-30


Root cause Analysis Chart .......................................................................................4-30

4.12.3 Unit one Turbine Thrust Trips 9,971 Mwh. .......................................................4-31



4.12.4 Unit Two “A” Booster ID Fan Motor Failure 30,464 Mwh. ................................4-32



4.12.5 Unit Four Air Pre-heater Pluggage 48,740 Mwh. ..............................................4-33



4.12.6 Unit Four Scrubber Problems 76,917 Mwh.......................................................4-34



4.12.7 Unit Four High Opacity Shutdown 42,731 ........................................................4-35



4.12.8 Unit Two Waterwall Tubes Hydrogen Damage 188,944 Mwh...........................4-35


xiii



4.12.9 Unit Two Cable Tray Fire 28,899 Mwh. .............................................................4-36



4.12.10 Unit Four Air Pre-heater Guide Bearing Fire 6,099 Mwh. ...............................4-38


Root cause Analysis Chart .......................................................................................4-38

4.13 Findings...............................................................................................................4-39

4.14 Conclusions.........................................................................................................4-40

4.15 Recommendations...............................................................................................4-40

4.16 Appendix 2 ...............................................................................................................4-40

Root Cause Analysis Charts (Excel file)........................................................................4-40

5 BOILER RELIABILITY OPTIMIZATION PROJECT PROGRAM AUDIT AT HAWAIIAN ELECTRIC COMPANY........................................................................................ 5-1

5.1 EXECUTIVE SUMMARY............................................................................................. 5-1

5.2 Introduction.................................................................................................................. 5-1

5.3 Findings....................................................................................................................... 5-2

5.3.1 Boiler Reliability Optimization Project Plan........................................................... 5-2

5.3.2 Project Activities................................................................................................... 5-2

5.3.2.1 BTFR/CCI Program ...................................................................................... 5-2

5.3.2.2 Streamlined Reliability Centered Maintenance ............................................. 5-3

5.3.2.3 Root Cause Analysis .................................................................................... 5-3

5.3.3 Condition based data collection ........................................................................... 5-4

5.4 Conclusions................................................................................................................. 5-4

5.5 RECOMMENDATIONS ............................................................................................... 5-4

Appendix 1: Boiler Reliability Optimization Storage Plan .................................................... 5-7

Appendix 2 ........................................................................................................................5-10

Appendix 3 ........................................................................................................................5-11

List of Typical Headings that Appear in a Policy or a Procedure ...................................5-11


xv

LIST OF FIGURES

Figure 1-1 Maintenance Optimization Program ....................................................................... 1-4 Figure 1-2 Big Sandy benchmarked against Industry Best Practice ........................................ 1-5

Figure 1-3 Work Process ........................................................................................................ 1-6 Figure 1-4 Work Process Model.............................................................................................. 1-7 Figure 1-5 Task Scatter Diagram - Value to Cost ...................................................................1-31

Figure 1-6 Accumulated Value verse Cost .............................................................................1-31 Figure 3-1 Spider Chart........................................................................................................... 3-4 Figure 3-2 ..............................................................................................................................3-13 Figure 4-1 Unit 5, #4 Control Valve 197,471 Mwh..................................................................4-14

Figure 4-2 Unit 3 Scrubber Problems 33,446MW/Hrs.............................................................4-15 Figure 4-3 Unit 3, Scrubber Problems 33,446MW/Hrs............................................................4-16 Figure 4-4 Unit 3 Scrubber Recycle Pump Problems 33,446MW/Hrs.....................................4-17

Figure 4-5 Unit 4, Loss of Booster Fans, 38, 413 MW/Hrs .....................................................4-18 Figure 4-6 Unit 3 Pulverizer Pinion and Bull Gear Failures, 38,881 MW/Hrs. .........................4-19 Figure 4-7 Unit 3 Backpass Flyash Erosion 39,378 MW/Hrs. .................................................4-20

Figure 4-8 Unit 4 Air Preheater Failure 166,744 MW/Hrs. ......................................................4-21 Figure 4-9 Units 4 and 5 Backpass Flyash Erosion ................................................................4-22 Figure 4-10 Unit 3 HP Turbine Nozzle Plugging, 100,628 MW/Hrs.........................................4-23 Figure 4-11 Unit One, Mechanical Dust Collection Retrofit, 41,146 Mwhs..............................4-41

Figure 4-12 Unit One, ID Booster Fan Bearing Failures, 7,544 Mwhs ....................................4-42 Figure 4-13 Unit One Turbine Thrust Trips, 9,971 Mwhs........................................................4-43 Figure 4-14 Unit Two, A Booster ID Fan Motor Failure, 30,464 Mwhs....................................4-44

Figure 4-15 Unit Four, Air Preheater Pluggage, 48,740 Mwh .................................................4-45 Figure 4-16 Unit Four Scrubber Outage, 76,917 Mwh............................................................4-46 Figure 4-17 Unit Four, High Capacity Shutdown, 42,731 Mwh ...............................................4-47

Figure 4-18 Unit Two Waterwall Tubes Hydrogen Damage, 188,944 Mwh.............................4-48 Figure 4-19 Unit Two Cable Tray Fire, 28,899 Mwh ...............................................................4-49 Figure 4-20 Unit Two Cable Tray Fire March 1999, 28,899 Mwh............................................4-50 Figure 4-21 Unit Four Air Preheater Guide Bearing Fire, 6,099 Mwh .....................................4-51


xvii

LIST OF TABLES

Table 1-1 Component Description and Engineering Identification Number.............................1-18 Table 1-2 Phase 2 Recommendations and Actions Taken .....................................................1-36

Table 3-1 Boiler tube Failures, Causes and Recommended Action........................................3-15 Table 3-2 Typical Header and Steam Drum Problem Areas...................................................3-18 Table 4-1 Cholla - Megawatt-hour loss summary report-fourth quarter 1999..........................4-28


1-1

1 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1, 2 & 4 AT AMERICAN ELECTRIC POWER’S BIG SANDY PLANT – UNIT 2

1.1 EXECUTIVE SUMMARY

Preparing for competition American Electric Power (AEP) requested EPRI to undertake a Boiler Reliability Optimisation project at their Big Sandy Plant. An availability of 95 % during peak periods – May through August and 90 % for the remainder of the year was set as a corporate target. Meeting this target would give Senior Management the assurance that they could compete successfully in the power generation market.

The project focussed on the long-term integrity of the water and steam touched tubing within the boiler envelope, training on EPRI’s tube failure reduction program and the provision of a EPRI’s Boiler Maintenance Workstation (BMW) software package for trending and tracking tube failure data and information. The project scope also included training on EPRI’s Streamlined Reliability Centered Maintenance software using the fuel system as an example. An outage task prioritisation model – Risk Evaluation and Prioritisation (REAP) was used to prioritise the tasks identified for the Unit 2 scheduled outage in April 2000.

This project, Phase II of EPRI’s Boiler Reliability Optimisation project, commenced in March 2000 and was finalised with the installation of the BMW software and user training in July 2000. Phase I was completed with the issuing of the final report in January 1999. Between March and July 2000 a number of visits to the plant were made to interview and collect data for analysis.

This report focuses on the development of a failure defense plan for the water and steam touched boiler tubing and contains a number of recommendations.

The main recommendation are listed below:

1. In the Technical and Organisational Assessment section there are a number of specific recommendation. Collectively these are related to improving planning and scheduling of boiler maintenance work. Thus, a high level recommendation would be to align the boiler maintenance strategy i.e. frequency and duration of scheduled outages and inspection scopes of work to the overall station performance goals of 95% availability during peak periods. Specific boiler availability targets should be developed relating to the number of tube leaks per year. One tube leak per unit per year, no repeat failures, no failures because of sootblower erosion are considered reasonable targets. The strategy and performance targets

EPRI Licensed Material Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Power’s Big Sandy Plant – Unit 2

1-2

should be compiled into separate plant directives, which are reviewed annually for effectiveness and appropriateness.

2. To repair and calibrate all boiler instrumentation critical to the efficient and effective operation of the boiler. This includes air, gas, water and steam temperatures and pressures and reheater and superheater metal temperatures.

3. The installation of on-line water and steam chemistry instrumentation and the establishment chemical performance index. This index can the be used in a proactive way to ensure compliance to EPRI water and steam chemistry guidelines.

4. To prepare detailed inspection plans based on the tasks identified in the Boiler Failure Defence Plans presented in this report. These inspection plans should be loaded into the CMMS. This would facilitate the development of a scheduled and or forced outage plan at short notice.

5. Once the installation and the user training on the Boiler Maintenance Workstation (BMW) have been completed, the BMW database needs to be updated with all the history of previous tube failures. This will enable tube leaks and repairs to be tracked and trends to be established. This will also assist in formulating focussed boiler inspections plans.

The plant has knowledgeable and skilful people. Given time, management support and coaching to implement the recommendations highlighted in this report the plant team will achieve the goals identified by Senior Management.

1.2 INTRODUCTION

During the period 1997 to 1999 AEP – Big Sandy Plant’s availability loss was 7.9 % of which 4.2 % can be attributed to boiler tube failures. The causes of these failures varied however the dominate repeat failures were due to fly ash erosion, oxygen pitting, long term over heating and firing side corrosion. During 1999, in excess of $2.0 million dollars of lost revenue can be attributed to outages on Unit 2. Concerned about this loss and a possible future increase in the Equivalent Forced Outage Rate (EFOR) lead senior management at Big Sandy Plant to request assistance from EPRI.

Participation in EPRI’s Boiler Reliability Optimization program was proposed. The program consists of several customised Boiler Reliability Optimisation Projects. Each project consists of four phases that involve EPRI technical staff, with participation from the plant’s operating, maintenance and engineering support staff. Each phase builds on the strengths of the preceding phase. These phases are:

• Phase I - Technical and Organisational Assessments

• Phase II - Development of a Boiler failure defence plan

• Phase III - Implementation of a failure defence plan

• Phase IV - Continuous Improvement and Process automation


Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Power’s Big Sandy Plant – Unit 2

1-3

AEP – Big Sandy Plant’s management chose to limit their participation to elements of Phase I. This was completed in January 1999. Satisfied with the results, AEP – Big Sandy management requested EPRI to continue with the program, i.e. complete Phase I – Technical and Organisational Assessment, and implement a tailored Phase II. Phase II was limited to only the water- and steam- touched boiler tubing. This report discuss the findings of this limited scope project.

1.3 SCOPE OF WORK

This consisted of the following activities:

1.1 Complete Phase I – Technical and Organisational Assessment.

1.2 Develop a Failure Defence Plan for water and steam touched tubing.

1.3 Provide Streamlined Reliability Centered Maintenance (SRCM) software and training, and analyse two systems.

1.4 Evaluate and prioritise all outage tasks on an upcoming outage, using EPRI’s – Risk Evaluation and Prioritise Program.

1.5 Provide Boiler Tube Failure Reduction and Cycle Chemistry Improvement training.

1.6 Supply, install and provide training on EPRI’s Boiler Maintenance Workstation (BMW) software for tracking boiler tube failure information.

1.4 FINDINGS AND OBSERVATIONS

1.4.1 Phase I and II - Technical and Organisational Assessment

A maintenance process assessment was conducted as a part of the work performed in the Boiler Reliability Optimization Project. The purpose is to understand the current processes, technologies, organizational strengths and weaknesses, and plant people to design a manageable change within the implementation plan. Data was collected and evaluated and benchmarked against best practices identified and tailored by EPRI. The assessment team spent one week on site interviewing a cross section of plant staff to determine how boiler maintenance work is identified and performed. The team focused on four main topics namely Work Process, Work Culture/Management, Technology and People. The data and information obtained was compared to industry best practice as identified in the EPRI “Best Practice”, to identify opportunities for improvement.

This section of the report presents the findings and recommendations to align AEP - Big Sandy Plant’s work process with Best Practice i.e. to move boiler maintenance from a reactive mode to one that is planned. The benefits of moving towards a more planned approach will be seen in reduced O&M cost and improving boiler reliability.


1-4

With this information and available resources AEP - Big Sandy Plant can become the “Best Practice Plant” within the AEP organisation for which it has been striving to become. Given time, management commitment and coaching, the plant team has the vision and can successfully lead this plant into full implementation of the boiler reliability optimization processes. From the data and interviews, strengths and weaknesses were plotted on a spider chart and recommendations formulated

For a condition-based, highly planned, maintenance optimization program to be effective, it must integrate the Work Process, Management and Work Culture, Technologies, and the People into the total operation of the pant. Each category is equally important to the success of the program, as illustrated in Figure 1-1.

PDMPEOPLE

PROCESS

MANAGEMENT TECHNOLOGY

Figure 1-1 Maintenance Optimization Program

To derive maximum value, all four categories and their sub-categories must be integrated to form one cohesive program. The key sub-categories are listed below:

Work Process Work culture/Management

Work Identification Setting Goals Leadership

Work Planning Organisation Accountability

Work Execution Communications

Work Close-out Global Metrics

Technology People/Skills

Work Management System Training Qualifications

Diagnostic Technologies Utilization Communication

Integration Tools/Techniques



1-5

Figure 1-2 is the benchmarked Spider chart utilizing the Power Industry as the source database. The outer circle at 8.0 represents best practice elements found in industry. The inner circle at 5.8 is the industry norm. Big Sandy Plant was evaluated and found to be at or near the industry norm. Big Sandy Plant has second quartile performance when compared to industry benchmarks. Information Integration, Work Execution, Metrics, and Utilization elements are slightly below industry norms.

It should be noted that overall, Big Sandy has relatively equal strengths throughout most elements. Information Integration can be improved in accordance with some of the recommendations coming from this project. Metrics are also contained in this project. Utilization will improve as more effective planning and a scheduling processes are put into place. Big Sandy staff given the direction has the strengths that are necessary to succeed.

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00PM Basis

W ork Identification

W ork Control

W ork Execution

W ork M anagem ent Sys tems

CB M Technologies

Inform ation Integration Tools

Continuous Improvem ent

AccountabilityOrganizationLeadership

M etrics

Comm unication Inter

Setting Goals

Benchm ark ing

Training

Utilization

Com m unication Intra

Qualification

Figure 1-2 Big Sandy benchmarked against Industry Best Practice

1.4.1.1 Work Process

For a maintenance program to function cost effectively, all equipment maintenance work should proceed through four distinct steps – Maintenance Strategy, Work Identification, Work Control (Planning and Scheduling) and Work Execution (including feedback) as shown in Figure 1-3.


1-6

Work Flow Process

Maint. Strategy Work ID Work

ControlWork

Execution

PMBasis CBM

PlanningScheduling

WorkActivities

Feedback CBM

Figure 1-3 Work Process

Using the Work Process Model - Figure 1-3 the assessment team produced a detailed model that highlights the areas of responsibilities. These are shown in Figure 1-4.



1-7

PM BasisChange Control

PM's

RCA

CBM Data Collection &Analysis

Boiler Specialist

Planners

CBM Coordinator

Technology Owners

PlanningPlanners

Boiler Specialist

InformationIntegration &

Decision

CD's, PR's

CAP

Boiler Specialist

Technology Owners, CBMCoordinator, RCA Process Owner

Boiler Specialist

Work OrderGeneration

SchedulingPlanners

Operators

InitiatorCM's

Post MaintenanceTesting Boiler Specialist, CBM

Coordinator

Work Execution

Close OutPlanners, Craft

Craft

WorkExecution

Work Control

WorkIdentification

PM Basis

Work Mgmt System

Impr

ovem

ent

Boiler Specialist

Boiler Specialist

Boiler Specialist

Boiler Specialist

Figure 1-4 Work Process Model

Maintenance Strategy

The starting point for developing a maintenance strategy is to first determine a maintenance objective. An objective should be quantified in terms of availability and performance. These requirements should be determined jointly between the Maintenance and Operating and or the


1-8

Production Departments. Maintenance Management is then in a position to determine the best mix of maintenance techniques to used, resources requirements and the costs thereof.

A maintenance strategy is then developed from these agreed objectives. It defines the process for identifying and achieving sound operational objectives, selecting the maintenance approaches, monitoring performance and providing effective management control. The strategy therefore consists of a management and technical strategies. The Management strategy defines how the business management skills are used to integrate people, policies, equipment and practises to identify improvement opportunities. The technical strategy states how the technical knowledge and experience are used to identify and implement the best proactive maintenance, repair, service and replacement of all equipment in line with AEP’s business performance objectives.

The maintenance strategy is built primarily from the preventive maintenance (PM) basis that exists at the plant. The aim is to develop an optimized set of PM tasks that will ensure Big Sandy Plant - Unit #2 Boiler will achieve its reliability targets.

Findings:

• The Boiler Specialist owns the Boiler System PM Basis.

• The Boiler System PM contains most of the inspection requirements, physical improvement PM’s, etc for work to be performed at various intervals on the various boiler components.

• Boiler Specialist ensures that the technology owners complete their inspection PM’s in accordance with the PM Basis for the Boiler.

• The Boiler Specialist actively participates in the Root Cause Analysis performed on the boiler events.

• Operating procedures have not been reviewed for some time and are out of date with current practice.

Recommendations:

1. The PM Basis must be reviewed and updated to take into account the failure mechanisms identified during the past outage inspection.

2. The Boiler Specialist should have an Operating Department counterpart who would be responsible to review and update and align operating procedures to current practice.

3. Boiler tube failures/Root Cause Analysis should be tracked and trended.

4. The boiler inspection plans need to be updated to take cognise of the change in the frequency of boiler periodic outages.

Work Identification

Work on equipment is identified from one of the following three sources of information:



1-9

Corrective Maintenance Repair tasks (CM) - tasks as a result of loss-of-performance, component breakdown, or catastrophic equipment failure that must be attended to immediately. These tasks are generally not scheduled.

Preventive Maintenance Tasks (PM) - tasks that are time based recurring work that is deemed necessary to keep equipment in optimum running condition. Implementation of this work should be both planned and scheduled.

Predictive Maintenance tasks (CD) - tasks that are condition directed based on information derived from a variety of condition information sources. These sources are identified and controlled on an E&CI matrix. All condition directed efforts to maintain equipment should be both planned and scheduled.

Findings:

• Prioritization seems to be “fix it now” or “fix it later”. When dealing with boiler issues, which are mainly outage tasks, “value of work” concepts should replace “prioritization”.

• The work process is split into two separate planning windows. One for emergency work i.e. works requiring immediate attention and the other for normal work, which passes through a daily scheduling activity.

• Emergency maintenance work is discussed between operations and maintenance leaders, while all other work orders (CM, PM and CD) are reviewed, selected and assigned to be executed by the PSL’s.

• The various Crafts persons walk the job down and plan job requirements.

• There is no formal root cause analysis process in place to ensure failures are thoroughly investigated to prevent recurrence.

Recommendations

1. A formal RCA program will result in additional information valuable in managing the Boiler.

2. The roles and responsibilities of the PdM Co-ordinator should be clearly defined and should include the following aspects:

– Ensure data collection routes and analyses are performed on schedule.

– Conduct pre-outage inspections.

– Post-maintenance testing.

– Working with system owners to integrate condition based data/information

– Develop and maintain a condition status report.

– Document case histories and cost-benefit calculations.

– Ensure PDM results are used to moderate the PM program – extend periodicity, PM activity modifications, removing redundant PM etc. Thus taking full advantage of the CBM program.


1-10

3. Infrared thermography should be used on the boiler to identify casing leaks, valve passing etc.

4. Develop an Equipment and Condition Indicator matrix (E&CI) for all critical boiler equipment/components such as in Figure 1-5. This will assist the equipment owner to assessing the condition of equipment under his control.

Work Control

The work control function shown in Figure 1-3 is an important function when focussing on reducing O&M costs and simultaneously increasing reliability

One of the activities when optimizing work control is to review and purge if necessary the corrective maintenance (CM) component of the workload. CM negatively impacts the ability to meet performance goals of reducing O&M cost performance and Equivalent Forced Outage Rate (EFOR). Decreasing the amount of CM does not mean that equipment failures will no longer occur. Rather, it means that management will be actively involved in deciding whether the best economic decision for the station is to allow a specific component, with “known problem conditions” to run-to-failure; or, should plans be established to effect the repair at a convenient time.

Findings

• The frequency of major outages has been changed to every 3 years with an inspection/minor repair outage of short duration at approximately 18 months.

• Outage work scopes are agreed to prior to the outage. However, a considerable amount of outage work is uncovered during the outage with the inspections. Review of the data available prior to the outage indicates some of the inspection results could have been predicted and repairs planned.

• Planners prepare work packages for critical work orders.

• Boiler Specialist serves the role of QA/QC for all boiler activities whether performed by AEP forces or contractors.

• Forced outages are planned, reviewed, and published once per month. Not all items on the forced outage list are fully planned and trigger ready.

• Forced outage plans are coded in the PIMS and maintained by planners.

Recommendations

1. In order for the 3 yearly boiler outages to be successful a detailed inspection plan needs to be developed by the boiler specialist to all known and potential problem areas are adequately inspected and repaired to ensure a high availability between periodic outages.

2. Develop a four-week rolling scheduling process for all maintenance to assure high utilization of resources.



1-11

3. Include post maintenance testing requirements into the planning process. This is a must for most CD work orders.

Work Execution

This is the final activity and includes work close-out.

Findings

• AEP support personnel and contractors do a reasonable job judged by the small number of QA/QC issues raised and re-work done.

• Pad welding and temporary repairs are tracked and noted to enable final repairs to be done during a suitable planned outage.

• Limited post maintenance testing is done.

• Work close out is performed by the PSL’s and at times without the knowledge of the Boiler Specialist.

Recommendations

1. System owners and the Boiler Specialist should review close-out information for effectiveness.

2. Compile a management directive identifying the minimum acceptable standards to be complied with when performing boiler maintenance or repair work. This would include such statements as every tube leak/failure should be investigated, pad welding should be discouraged etc.

1.4.1.2 Technologies

An Optimisation program is made up of people, management systems, component owners, technology specialists and predictive maintenance tools or equipment. The key predictive maintenance tools used in a Boiler Reliability Optimisation project are infrared thermography, vibration, oil analysis, ultra-Sonics and plant process data and information such as air, gas water and steam temperatures and pressures, metal temperatures and water and steam chemistry data. Equipment owners and technology specialists collect and collated data on various boiler components. This data is organised and presented in an Equipment and Condition Indicator Matrix. The matrix specifies the predictive maintenance technology to be used, the equipment to be monitored and when the data should be collected.

The “how”, “what”, and “when” of condition data analysis is based on exceeding pre-established threshold levels and this information is presented in an “Event Report” and in an “Asset Condition Status Report”. Analysis of the data and diagnosis of the cause incorporates all available information including diagnostic data from all related technologies, operating conditions from process data, operating log information, maintenance history, design information


1-12

and knowledge of the experienced personnel. Condition Assessment Reports should list, by asset, any marginal or critical determinations by each technology, and include recommendations for action based on the integrated analysis of the data.

The “what”, “how” and “when” of action is based on analysis of the data, and this establishes what action is to be taken to correct the anomaly, how the work should be accomplished, and when the work needs to be done to prevent an unplanned failure.

Determining the threshold levels to declare an event is based on technology standards and deviations from baselines for the monitored equipment. Therefore, it is necessary to establish baselines for all equipment in the PDM Program, and re-baseline the equipment following any related repairs by performing post-repair acceptance testing.

Findings

• Draught gauges and metal temperature thermocouples were found to be defective.

• No on line sodium analysers for continue monitoring of steam and water quality was available.

Recommendation

1. Repair and calibrate all essential/critical boiler instrumentation.

Work Management System

Findings

• PIMS meets the basic needs of the organization and has the capability to capture boiler inspection data.

Recommendations

1. PIMS should be expanded to capture all boiler inspection data

Maintenance & Diagnostic Technologies

Findings

• NDE Inspections are consistent with industry practice

• Limited boiler performance testing and trending of results is done.

Recommendations

1. Establish technology owners with responsibilities to include condition analysis.



1-13

2. Need to have a more comprehensive PDM reports as health report not exception reports.

Information Integration

The key to utilizing the condition-based information in an effective manner is the ability to quickly access the information and to trend the data over time to enable a prediction of when work should be performed. EPRI’s Boiler Maintenance Workstation (BMW) is one such system, which can be used to track and trend boiler tube failure data and information. Another useful tool is the WEB based “Condition Status Report” software. The software has the capability to store equipment data and information, capture recommended actions, and report on the overall condition of the critical plant equipment in detail. From this information, management can determine whether the overall condition of the plant is deteriorating or improving by tracking the history and costs of the equipment. Decisions made and or action taken and work order numbers are also recorded. Post maintenance test results are recorded to determine whether the diagnosis and maintenance was appropriate.

Findings

• There is no integrated condition status reports produced on the boiler or in fact any other part of the plant.

Recommendations

1. Formalise monthly Boiler Condition Based Maintenance (CBM) meetings. Attendance by Equipment/Component and Technology owners, PdM Co-ordinator, Operating and Maintenance staff and the Boiler Specialist to make maintenance decisions based on the condition of the equipment/component. Once a year, perhaps prior to any budgeting cycle, the Boiler Specialist should present a Boiler Failure Defence Plan – indicating the current condition, short and long term activities/tasks that need to be undertaken to ensure boiler reliability.

2. Case histories and cost benefits analysis of using PDM technologies should be document either in the BMW or PIMS databases.

1.4.1.3 Management and Work Culture

The objective is to develop a work environment in which an effective combination of people, work processes and technologies are used to achieve the overall business goals. To achieve this management must exhibit leadership i.e. the ability to delegate responsibility and authority, and to hold people accountable for their performance. Leadership also includes defining organisational direction, setting realistic measures and targets and communicating these through out the organisation. Measures (metric) can be lagging – measure the output of the process or leading a measure of the effectiveness of the elements of an organisation.


1-14

Recommendations

1. Roles and responsibilities must be clearly defined and understood by the individuals in the plant.

2. Measures (metrics) and targets need to be developed for the boiler. These metrics could be the following:

– # of tube leaks per year –suggest one tube per boiler per year,

– no tube leaks caused by sootblower erosion,

– no repeat boiler tube leak failures,

– boiler availability,

– boiler forced outage rate as a contribution to the overall forced outage rate and

– long term plant health measures like a thermal index which is a measure of the equivalent number of operating hours or extra life consumption experienced by boiler headers as a result of operation at metal temperatures in excess of design.

These measures and targets should be specific, measurable, achievable within a specified period of time and reasonable and should be communicated through out the organisation.

3. An annual review of the maintenance program should be undertaken to determine its effectiveness i.e. the right technologies are being applied to the right assets at the right frequency to meet the overall business objectives and maintenance strategy. These assessments should validate the following questions:

– Were there unexpected or premature failures that were not detected by at least one technology? This could result in applying PDM to an asset that was not previously being monitored, or increasing the frequency of monitoring on a previously established route, or indicate the need for a new PDM technology, or readjustment of a CM Task.

– Were failures occurring that were detected but without sufficient time to take action? This could result in adjustment of alert threshold levels.

– Were there long periods of operation without any developing problems that contained numerous condition data sets? This could result in decreasing the frequency of monitoring.

– Was a technology ineffective in detecting an emerging problem? This could result in removal of an ineffective technology from the program.

– Did the costs, measured in dollars, exceed the benefits of the program? This could signal the need to redesign the program.

– Was all of the data collected analyzed?

– Was action taken on detected events? This could result in a refocus of the program leaders and a re-aligning of sponsors.

– Did the program fail to meet the established goals? This could result in a redesign of the program.



1-15

4. Consideration should be given to having an annual Boiler Specialist Meeting and forming a Boiler Specialist’s Users Group. This will aid in the transferring of information and knowledge amongst the Boiler Specialists within the AEP organisation.

1.4.1.4 People

Human beings are an important asset. Their reliability – “making and keeping commitments” has a huge impact on plant reliability. All equipment can in some way be traced back to human beings. If human beings are reliable, your equipment will be reliable.

Findings:

• Training meets the basic needs of the plant. New equipment training for the craft is lacking.

• Communications among PSL’s is good.

• Good team work.

• Utilization is lower than expected. This is as a result of inadequate planning and staff has to wait for other disciplines.

Recommendations

1. Consider providing Level of Awareness training (LOA) training for the entire staff on Plant Maintenance Optimization.

2. Re-train all PIMS users on the details of PIMS and how best to it.

1.4.2 Failure Defence Plan – Water and Steam Touched Tubing

To have an effective defence plan, it is important to uniquely identify – Equipment Identification Number (EID) all the various sections and/or components to be included in the Defence Plan. This number is then used in the Computerised Maintenance Management System (CMMS) to track inspection details and trends. Table 1-1 shows all the various tubing sections of the boiler.

Table 1-2 represents the Failure Defence Plans for the various sections of tubing listed in Table 1-1. These plans were developed from a detailed boiler inspection and discussions with the Boiler Specialist. They identify the failure mechanisms, direct causes, short and long-term actions to be taken to prevent or at least minimise the failure and or damage. These action plans also give the various NDE and NDT techniques to be used to identify and quantify damage by the various failure mechanisms and the criteria to be used when using performing boiler maintenance and or repair work.

The following failure mechanisms were identified as being active in the various tubing sections:

• Sootblower erosion

• Fly ash erosion


1-16

• Fire side corrosion

• Corrosion fatigue

• Dissimilar metal welds

• Oxygen pitting

• Short term overheating

• Long term overheating

Findings

Economiser

• No significant tube failures have been experienced in the economizer.

• The economizer is of the ringed tube type and suffers from pluggage especially on the North side. Baffles have been installed to equalize gas flow and minimize ash pluggage.

• Clean air velocity tests have not been done to verify the correct position of the baffles.

• Plans are in place to open up gas passes to minimize pluggage.

• One leak was encountered last year due to flyash erosion. Tube shields were installed 2 years ago to minimize flyash erosion.

• No header cracking has been noted.

Waterwalls

• The ash hopper slope has channelling between the membrane and the tube in the tube material. About 82 tubes have been found with some amount of material loss due to slag erosion. No leaks have been attributed to this problem. Slagging of these tubes has not been a problem since changing to low NOx burners in 1994.

• The sidewalls have experienced severe wall loss since conversion to low NOx burners. Portions of the sidewalls were overlaid in the fall of 1999 with Type 312 SS to mitigate this problem. No further testing has been performed since this inspection. The unit has only experienced one trip since the outage and did not stay off line long enough to allow for testing. This overlay will be inspected in March and an additional 500 sq. ft. will be overlaid.

• Since the overlay outage, testing has been performed using 3/8” tubing inserted through the wall membrane during the outage on the left side. Initial measurements showed no oxygen during operation. Burner modifications and adjustment have raised the oxygen content to a positive level. Plans are to install several more taps into the firebox during the next outage to help facilitate future burner adjustments. Currently measurements are performed manually on the sample positions. Automatic sampling should be developed and tied into the database with feedback to controls or maintenance.

• The upper furnace has experienced no problems.



1-17

• The center division wall has experienced no problems.

• One failure has been reported in the roof due to erosion from a casing leak. The attic has since been pressurized and no more leaks have been observed.

• Several areas of casing/membrane leaks were noted in the upper firebox. These should be repaired at the next outage to prevent erosion from ash or casing damage from heat.

• Screen tubes have had some erosion from flyash and sootblowers. Tube shields have been tried with no success. The screen tubes are carbon steel with a wall thickness of 0.300”.

• Some pad repair or weld overlay work is expected next outage. More information is needed on this problem including UT thickness readings. This wear is probably from water in the sootblowing system but could be aggravated by large coal size, late combustion and LOI. Plant staff indicated coal mill fineness tests were performed twice a year.

• No severe problems were noted with the waterwalls other than the wastage due to low NOx burners. This is being addressed with the weld overlay and burner adjustments.

Platen Superheat

• Approximately 300 DMW are located in the platen superheat section and are scheduled for replacement using EPRI developed technology.

Finishing Superheat

• Some sootblower erosion has occurred in this section. Some tubing has been pad welded to restore minimum wall. No leaks have occurred due to erosion. Alignment handcuffs in this area are in need of repair. Current plans are to install heavier handcuffs next outage to keep tubing in alignment and avoid flyash erosion.

• No plugging problems have been noted in this area except when sootblowers are out of service. The tubing in this area was too long when installed and hit the slope when hot. This tubing has been shortened. No operational problems are noted at this time.

First Reheat (High Pressure)

• This area probably contains the most damage of any part of the boiler. Past lay-up procedures allowing moisture to condense in this area has led to oxygen pitting in the horizontal sections.

• Current lay-up procedures of using nitrogen blanketing, drying out with dried air, or heating up to dry after coming off line are believed to have stopped the damage.

• The second or third tube in the bundle has suffered from long term overheat – fish-mouth type failures. Other AEP 800 MW plants have replaced this tubing with P91 material. A life assessment using EPRI TUBELIFE and other NDE should be performed on this tubing. If tubing requires replacement, T91 or other new upgraded material such as T23 should be used.

• Damage is also occurring in this section from vibration fretting of the “cane” tube spacers. Egg shaped spacers are placed vertically between tubing bundles and turned until tight. Then


1-18

straps are welded between adjacent spacers to hold in place. No attachment is made to the boiler tubing. During operation these spacers become loose and allow the tubing to vibrate, fretting the tubing and causing leaks. AEP is currently not replacing spacers when they fall out.

Second reheat (low pressure)

• The outlet header and top two sections of tubing have been replaced with SA 213 T-91 material in the highest temperature areas. No other leaks or failures have occurred in this area.

Spray Attemperators

• The nozzles have been inspected and replaced. The liner has been replaced on the Main Steam Attemperator. This equipment is currently on a seven-year inspection cycle. This inspection frequency should be monitored and adjusted as required. Liners should be inspected with boroscope. Nozzles should be removed and inspected. Water stop valves should be inspected to prevent thermal shocks to the superheat and reheat piping.

Recommendations

1. Develop a series of inspection tasks using the data and information identified in the Failure Defence Plans in Table 1-2. These Defence Plans need to be reviewed and converted into specific action tasks with allocated responsibilities for completion. The inspection type tasks need to be formatted and standardised and put into the Computerise Maintenance Management System database for reference and future boiler inspections.

Table 1-1 Component Description and Engineering Identification Number

# EID DESCRIPTION

1 5512230200 ECON 1 Economiser Tubes U2 – 1st Reheater side

2 5512230200 ECON 2 Economiser Tubes U2 – 2nd Reheater side

3 5512240200 DIVWL Waterwall Tubes U2 – Division wall

4 5512240200 LFFWL Waterwall Tubes U2 – Lower furnace front wall

5 5512240200 LFSWL Waterwall Tubes U2 – Lower furnace side walls (incl. 40 tubes front and rear)

6 5512240200 LFRWL Waterwall Tubes U2 – Lower furnace rear wall

7 5512240200 UFRWL Waterwall Tubes U2 – Upper furnace rear wall

8 5512240200 UFFWL Waterwall Tubes U2 – Upper furnace front wall

9 5512240200 UFSWL Waterwall Tubes U2 – Upper furnace side walls



1-19

10 5512243200 Screen Tubes U2 – Furnace

11 5512241200 HRAT Heat Recovery Area (HRA) U2 – Incl. Side, front and rear walls, partition wall and roof

12 5512241200 Aperture Tubes U2 – Incl. Side walls, floor and screen

13 5512240200 FROOF Waterwall Tubes U2 – Furnace roof tubes

14 5512210200 Platen Superheater

15 5512212200 Finishing Superheater

16 5512220200 1st Reheater

17 5512222200 2nd Reheater


1-20

Possible Causes

Cor

rosi

on C

orro

sion

The

rmal

Fat

igue

Fly

Ash

Ero

sion

Pitt

ing

Aci

d D

ewpo

int C

orro

sion

Fat

igue

Cor

rosi

on F

atig

ue

Low

Tem

pera

ture

Cre

ep

Hyd

roge

n D

amag

e

Sup

ercr

itica

l Wat

erw

all C

rack

ing

Aci

d P

hosp

hate

Cor

rosi

on

Cau

stic

Gou

ging

Sho

rt T

erm

Ove

rhea

ting

Coa

l Par

ticle

Ero

sion

Fal

ling

Sla

g E

rosi

on

Fire

Sid

e C

orro

sion

Soo

tblo

wer

Ero

sion

Preventive Measures Short/Long Term

ND

E D

etec

tion

Tec

hniq

ue

CRITERIA

# A B C D E F G H I J K L M N O P Q #

1

• • Confirm mechanism by inspecting tubes around and in the immediate vicinity of the blower.

VT /EMAT

• Check for tube wastage - flat and/or 'shiny' surfaces or if the boiler has been washed check for a fresh layer of rust andgouges from steam cutting.

• Determine the extent of the erosion by mapping the tube thickness in the affected ares. Compare with previous readings and calculate an erosion rate and trend results -if applicable.

UT • Use AEP's Standard for repairing or replacing tubes. Typically 70% of wall thickness is used. Do not pad weld to restore thickness for < 70% because of the potential to introduce other detrimental failure mechanisms - copper embrittlement and hydrogen damage. If root cause of erosion has been determined and adequately addressed no additional erosion should place.

• Check, correct and verify the operation of moisture traps and remove remaining orifices.

VT

• Check, correct and verify blower or system operation to ensure correct sequence, frequency, angel of rotation, travel, misalignment, operating steam temperature and pressure.

VT • Use the OEM's manual or site specific procedure.

Com

pone

nts

1 th

roug

h 16

(al

l com

pone

nts

with

in s

ootb

low

er r

ange

)

Malfunction of a sootblower or the sootblowing system because of incorrect operation and/or inadequate maintenance (control logic - sequence, operating temperature and pressure, mechanical misalignment, drainage pipe slopes etc)

• Check, correct and verify steam supply (pressure and temperature), drainage slopes and piping insulation.

VT



1-21

Possible Causes

Cor

rosi

on C

orro

sion

The

rmal

Fat

igue

Fly

Ash

Ero

sion

Pitt

ing

Aci

d D

ewpo

int C

orro

sion

Fat

igue

Cor

rosi

on F

atig

ue

Low

Tem

pera

ture

Cre

ep

Hyd

roge

n D

amag

e

Sup

ercr

itica

l Wat

erw

all C

rack

ing

Aci

d P

hosp

hate

Cor

rosi

on

Cau

stic

Gou

ging

Sho

rt T

erm

Ove

rhea

ting

Coa

l Par

ticle

Ero

sion

Fal

ling

Sla

g E

rosi

on

Fire

Sid

e C

orro

sion

Soo

tblo

wer

Ero

sion


ND

E D

etec

tion

Tec

hniq

ue

CRITERIA


• Emergency repairs such as pad welding should be replaced at the next opportunity.

• Develop a map identifying the exact location of each pad weld to ensure all locations are recorded.

• Optimize blowing sequence to ensure an effective boiler sootblowing program.

VT

• Establish and implement a routine maintenance schedule to inspect and test blower operation.


1-22

Possible Causes

Cor

rosi

on C

orro

sion

The

rmal

Fat

igue

Fly

Ash

Ero

sion

Pitt

ing

Aci

d D

ewpo

int C

orro

sion

Fat

igue

Cor

rosi

on F

atig

ue

Low

Tem

pera

ture

Cre

ep

Hyd

roge

n D

amag

e

Sup

ercr

itica

l Wat

erw

all C

rack

ing

Aci

d P

hosp

hate

Cor

rosi

on

Cau

stic

Gou

ging

Sho

rt T

erm

Ove

rhea

ting

Coa

l Par

ticle

Ero

sion

Fal

ling

Sla

g E

rosi

on

Fire

Sid

e C

orro

sion

Soo

tblo

wer

Ero

sion


ND

E D

etec

tion

Tec

hniq

ue

CRITERIA


2

• • Confirm mechanism and determine the extent by completing a dirty and a clean inspection. The dirty inspection will highlight areas where excessive local velocities occur. These can be comfirmed by a detailed clean inspection. The areas mostly likely to show signs are at the base of the fins of the first economiser bank, the element ends between tube and wall and adjacent to any flow staighteners and/or baffles.

UT VT

• Perform a CAVT to confirm high velocity areas. If high velocities are detected - design and install baffles or flow straighteners. Perform a second CAVT to confirm actual velocity profile.

• Use AEP's standard for tube replacement (<70% wall for waterwall tubing and <85% for steam tubing) in conjunction with EPRI's remaining life assessment calculations results from Level I or II as judgement criteria - TR-111559. If trend data available, replace tubing that will drop below wall thickness criteria before next outage. If erosion correction action taken and confirmed to eliminate future wall loss, can leave tubing in service with >70 % but <100%.

Com

pone

nt #

1, 2

and

12

thro

ugh

17

Primary causes are excessive local velocities and/or dust burden. Excessive local velocity - design/configuration - distorted/misalignment tubes - Gas flows above design rating Dust burden - Increase in erosive particles ie increase % ash - Sootblower operation Other - Incorrectly install shields and/or baffles - Inappropriate and/or incorrectly applied coatings - Broken or misaligned "canes" and "handcuffs.

• If baffles and/or flow straighteners have been installed perform an annual inspection to determine the effectiveness of the modification and adjust if necessary.



1-23

Possible Causes

Cor

rosi

on C

orro

sion

The

rmal

Fat

igue

Fly

Ash

Ero

sion

Pitt

ing

Aci

d D

ewpo

int C

orro

sion

Fat

igue

Cor

rosi

on F

atig

ue

Low

Tem

pera

ture

Cre

ep

Hyd

roge

n D

amag

e

Sup

ercr

itica

l Wat

erw

all C

rack

ing

Aci

d P

hosp

hate

Cor

rosi

on

Cau

stic

Gou

ging

Sho

rt T

erm

Ove

rhea

ting

Coa

l Par

ticle

Ero

sion

Fal

ling

Sla

g E

rosi

on

Fire

Sid

e C

orro

sion

Soo

tblo

wer

Ero

sion


ND

E D

etec

tion

Tec

hniq

ue

CRITERIA


• Repair, replace and/or align damaged tubes and/or elements.

• For major damage, plan for element/bank replacement.

• Use AEP's Standard for repairing or replacing Tubes

• If damage is at the base of the fins on the first bank and replacement is necessary then consider designing out the problem by installing straight tubes.

• Repair, replace and/or align damaged "canes" and "handcuffs".

• See EPRI document for economizers GS-5949

• Check for and remove any ash build up/plugage in the economizer tube banks.

• 70 % through 200 mesh


1-24

Possible Causes

Cor

rosi

on C

orro

sion

The

rmal

Fat

igue

Fly

Ash

Ero

sion

Pitt

ing

Aci

d D

ewpo

int C

orro

sion

Fat

igue

Cor

rosi

on F

atig

ue

Low

Tem

pera

ture

Cre

ep

Hyd

roge

n D

amag

e

Sup

ercr

itica

l Wat

erw

all C

rack

ing

Aci

d P

hosp

hate

Cor

rosi

on

Cau

stic

Gou

ging

Sho

rt T

erm

Ove

rhea

ting

Coa

l Par

ticle

Ero

sion

Fal

ling

Sla

g E

rosi

on

Fire

Sid

e C

orro

sion

Soo

tblo

wer

Ero

sion


ND

E D

etec

tion

Tec

hniq

ue

CRITERIA # A B C D E F G H I J K L M N O P Q #

3

• • Confirm mechanism and extent of wall thinning by inspecting the area around the burners (approx 2 diameters) and on the side walls at approx. midway between front and rear walls. these surveys should be conducted in conjunction with remain ing life assessment techniques to determine wastage rates. Also monitor/measure the levels of Oxygen and CO near the tubes on the side walls.

UT /EMAT

• For damage, repair by the use of high chromium weld overlay material in affected areas to resist corrosion attack. Optimize combustion to minimize the affect of substoichiometric conditions and institute long-term monitoring - inspections and oxygen and CO at or near the tube walls.

• Use AEP's standard for replacement ie < 70% wall. • CO level of < 1% and/or Oxygen levels of < 0.1% • Wastage rate of 12 mills/yr is consideredexcessive. • • Use AEP's standard for repairing or replacing tubes

Com

pone

nts

3, 4

, 5 a

nd 6

Primary cause is a substoichiometric (reducing) enviroment as a result of poor combustion - burner air and fuel distribution and incorrectly set burner registers. Secondary causes are related to fuel composition in particlar the amount of chlorine and aggressive ash and also flame impingement.

• Optimize combustion to minimize the affect of substoichiometric conditions and institute long-term monitoring of oxygen and CO at or near the tube walls.



1-25

Possible Causes

Cor

rosi

on C

orro

sion

The

rmal

Fat

igue

Fly

Ash

Ero

sion

Pitt

ing

Aci

d D

ewpo

int C

orro

sion

Fat

igue

Cor

rosi

on F

atig

ue

Low

Tem

pera

ture

Cre

ep

Hyd

roge

n D

amag

e

Sup

ercr

itica

l Wat

erw

all C

rack

ing

Aci

d P

hosp

hate

Cor

rosi

on

Cau

stic

Gou

ging

Sho

rt T

erm

Ove

rhea

ting

Coa

l Par

ticle

Ero

sion

Fal

ling

Sla

g E

rosi

on

Fire

Sid

e C

orro

sion

Soo

tblo

wer

Ero

sion


ND

E D

etec

tion

Tec

hniq

ue


• Inspect, repair or replace damaged burner quarls.

• Develop a spreadsheet to be use to collect and trend thickness measurement data.

4 •

• Confirm mechanism and determine the extent by inspection and sampling. Remove sample attachment tubes and areas of high stress.

VT UT

• Replace damaged tubes.

• Using specialised UT/EMAT test for ID initiated cracks. If cracks are > 50% through wall then replace tubing.

• Modify tube attachments to relieve obvious stress raisers.

• Ensure compliance to AEP's water chemisrty practices.


• Ensure compliance to start-up and shut down procedures.

• Ensure compliance to AEP's chemical cleaning procedures.

• Correct tube spacing • Set inspection levels to

determine the effectiveness of any modification and monitor damage accumulation.

Com

pone

nts

4 an

d 6

(hop

per

slop

e tu

bes

and

area

s of

hig

h he

at fl

ux)

• Cyclic stresses brought about by improper start-up and shut down practices and • Poor water chemistry control especially the control of ph during shut down and start-up.


1-26

Possible Causes

Cre

ep

Fire

side

Cor

rosi

on

Dis

sim

ilar

Met

al

Shor

t Ter

m O

verh

eat

Stre

ss C

orro

sion

Cra

ckin

g

Soot

blow

er E

rosi

on

Fati

gue

Rub

bing

/Fre

ttin

g

Pitt

ing

Gra

phit

izat

ion

Che

mic

al c

lean

ing

Dam

age

Fly

Ash

Ero

sion

Lon

g T

erm

Ove

rhea

t


ND

E D

etec

tion

Tec

hniq

ue


5

•

• Confirm mechanism and determine the extent. Map location and inspect all DMWs.

VT UT

• Replace DMW with a "dutchman"of upgraded material. Depending on the specific circumstamces re-positioning the DMW may be more cost effective.

• Use AEP's stanard for tube insert- 18 inches preferably 36 inches. • Use EPRI's standard for weld geometry.

• Perform a remaining life assessment calculation (Level I, II or III).

• Inspect and correct tube supports, tube bindings etc leading to axial stresses at welds.

DMW life code from SIA (based on EPRI PODIS code)

• Ensure functionality of metal and steam temperature thermocouples.

Com

pone

nt #

14 Higher than expected metal

temperatures (brought on by overfiring conditions) and stresses - both thermal and differential expansion



1-27

Possible Causes

Cre

ep

Fire

side

Cor

rosi

on

Dis

sim

ilar

Met

al

Shor

t Ter

m O

verh

eat

Stre

ss C

orro

sion

Cra

ckin

g

Soot

blow

er E

rosi

on

Fati

gue

Rub

bing

/Fre

ttin

g

Pitt

ing

Gra

phit

izat

ion

Che

mic

al c

lean

ing

Dam

age

Fly

Ash

Ero

sion

Lon

g T

erm

Ove

rhea

t


ND

E D

etec

tion

Tec

hniq

ue


6

• • Confirm mechanism and determine the extent by inspection and sampling in particular areas where the tubing has sagged - allowing condensate to accumulate.

VT UT

• If root cause has been identified and rectified, minor pitting damage can be left alone ie there is no need replace tubing. Continue to monitor and take samples for analysis to check for deterioration.

• Use the remaining life assessment calculations results from Level I or II as judgement criteria.

• For major damage, plan to replace, modify operating procedure and institute long-term monitoring.

• Review and correct if necessary the effectiveness of the shut down and lay up procedures.


• Enforce compliance to shut down and lay up - nitrogen blanketing procedures.

Com

pone

nt #

17

• Oxygen-saturated stagnant condensate as a result of a unit shutdown - forced cooling and/or imprper draining and venting and aggrevated by the • Carryover of sodium sulfate from the reheat steam (Sodium Sulfate tends to form at typical reheat pressures)

• If the lay up procedure - nitrogen blanketing cannot be used because of boiler work, some other means of drying must be used to ensure a moisture free enviroment in the reheater.


1-28

• Check drainage slopes and tube bundle sagging. Correct if possible, if not possible then to check for any deterioration by inspection and sampling.

Possible Causes C

reep

Fire

side

Cor

rosi

on

Dis

sim

ilar

Met

al

Shor

t Ter

m O

verh

eat

Stre

ss C

orro

sion

Cra

ckin

g

Soot

blow

er E

rosi

on

Fati

gue

Rub

bing

/Fre

ttin

g

Pitt

ing

Gra

phit

izat

ion

Che

mic

al c

lean

ing

Dam

age

Fly

Ash

Ero

sion

Lon

g T

erm

Ove

rhea

t


ND

E D

etec

tion

Tec

hniq

ue


7

•

• Confirm cause by either an analysis of a sample of the debris or start-up and/or shut down records. Confirm extent through inspection of tube in the immediate vicinity. Check for swelling and cracks.

VT MT

• Check to ensure blockage has been removed by using compressed air or water.

• Repair by inserting a 36 inch "dutchman".

• Use AEP's Standard for chemical cleaning

• Ensure compliance to start-up and shut down practices.

• Ensure compliance to start-up and shut down practices.

• Use AEP's Standard for repairing or replacing tubes

• Check combustion and or burners are optimized to ensure no flame impingment is taking place.

Com

pone

nt #

3 to

17

(wat

er a

nd s

team

cir

cuit

s)

• Exfoiliation induced: Caused by the natural process of growth and exfoiliation of oxides. These oxide flakes tend to collect at sharp bends. • Mainenance induced: Caused by poor quality control during welding, improper/incorrect chemical cleaning or poor combustion resulting in flame impingment on water walls. • Operatging induced: Caused by improper start-up and shut down practice (not boiling out condensate collected in the loops/bends) and overfiring to compensate for low feedwater temperature.



1-29

Possible Causes

Cre

ep

Fire

side

Cor

rosi

on

Dis

sim

ilar

Met

al

Shor

t Ter

m O

verh

eat

Stre

ss C

orro

sion

Cra

ckin

g

Soot

blow

er E

rosi

on

Fati

gue

Rub

bing

/Fre

ttin

g

Pitt

ing

Gra

phit

izat

ion

Che

mic

al c

lean

ing

Dam

age

Fly

Ash

Ero

sion

Lon

g T

erm

Ove

rhea

t


ND

E D

etec

tion

Tec

hniq

ue


8

• • Confirm mechanism and determine the extent by inspection. Perform UT metal and ID oxide thickness measurements.

VT UT

• Remove several samples both across and within the bank for analysis and replace withn a 36 inch "Dutchman"


• Perform a remaining life assessment calculation (Level I, II or III).

• For major damage, plan to replace, modify operating procedure and or design and institute long-term monitoring.

• Use the remaining life assessment calculations results from Level I or II as judgement criteria- EPRI's TUBELIFE program. • Replacement should be T91 or equivalent material such as T23.

• Perform hydraulic test as an integrity check.

• Compliance to ASME code requirements fortest pressures and temperatures. Increasing pressures above code to force failure is a poor practice and fruitless as crack > 50% through wall can tolerate design pressures without failure.

• Calibrate and ensure functionality of all installed metal temperature thermocouples.

Com

pone

nt #

17

Prolonged operation at high temperatures (design operating temperature and above)

• Set inspection levels to determine the effectiveness of any modification and monitor damage accumulation.


1-30

1.4.3 Streamlined Reliability-Centered Maintenance

Streamlined Reliability Centered Maintenance is a logical, systematic, functionally based methodology used in the evaluation of a facility’s system or unit. The evaluation includes a step-by-step consideration of the integral functions related to the operation of the system or unit. The failure mechanisms of each of these functions, the effects of the failure and the selection of appropriate and effective maintenance tasks to mitigate these failures are identified. The goal is to develop a cost-effective maintenance program, based on system functionality, which will enhance system reliability. This makes optimum use of available maintenance resources and provides a documented base for future additions/revisions to the maintenance program.

Two systems were reviewed – steam- and water-touched tubing and the fuel system. Appendix 1 gives the results of this analysis.

These results still need to be compared with those in Computerised Maintenance Management System. The purpose of this comparison is to identify changes to the existing program, to optimise the Fuel system preventive maintenance program. The comparison also provides another check of the analysis to ensure completeness and validity of assumptions.

Once the comparison is complete and has been approved the “new” tasks can be incorporated in the CMMS.

1.4.4 Unit 2 Outage Task Prioritisation

An EPRI Risk Evaluation and Prioritisation (REAP) model was used to demonstrate how outage tasks can be evaluated and prioritised. Thus providing a documented basis for decision making. The model is based on determining the risk of performing or not performing a preventive or corrective maintenance task. This is achieved by scoring a number of answers to questions about the task at hand. Labour hours and cost are also used to determine the value and cost of the task.

The following is the results of a REAP analysis performed using the Unit 2 outage task list. A total of 119 maintenance outage tasks were reviewed. These tasks had been determined from a much longer list, which had been subjected to a “gut” feel prioritisation method. Unfortunately the total list was not available to compare the “gut” feel method verse the approach taken in the EPRI model.

Figure 1-5 represents the resultant scatter diagram of all 119 tasks analysed.



1-31

T a s k C o s t V a lue S c a t t e r

1.E+00

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1.E+07

1.E+08

1.E+09

$100 $1,000 $10,000

C os t

V alue/C o s t Op timiz atio n Lines

Figure 1-5 Task Scatter Diagram - Value to Cost

The angled lines in figure 1-5 are generated by recognising that outage work scopes are built by starting with the most valuable least cost work and continuing to add more expensive valuable work until all to work has been included. Integrating all this work results in the Accumulative Value and Cost curve – Figure 1-6.

No. of Tasks & Value vs Cost

0.E+005.E+081.E+092.E+092.E+093.E+093.E+094.E+094.E+095.E+09

0 20000 40000 60000 80000 100000

Costs

Val

ue

0

20

40

60

80

100

120

140N

o. o

f Tas

ks

Tasks

Value

Figure 1-6 Accumulated Value verse Cost


1-32

Figure 1-6 represents the accumulated value versus cost versus number of tasks and shows that at the 32,700 cost level (cost is number of man-hours), 3.74 E09 value is captured with 95 of the 119 outage tasks. In other words – with 40% of the outage task hours, 95% of the outage task value can be captured with 80% of the tasks being performed. The vertical lines show where there is a significant slope change. In this case they show perhaps where one can “draw the line” on an outage work scope i.e. where 80 % of the value is captured with 20 % of the cost.

At that threshold, the following tasks would not be performed:

AIR HEATER SEALS 3356430




4KV ROOM WALL 3374302

INSPECT GAS OUTLET DUCTS 3422425

CIRC WATER PUMP DISCHARGE EXP. JOINTS 3356636

DEAERATOR INSPECTION 3422484

OVERLAY SIDEWALLS 3357443

NDE ON FACTORY WELDS 1ST RH PIPING 3374254

PA & FD FANS - CLEAN & INSPECT 3422941

BOILER DIVISION VALVE-OPEN, INSPECT & REPAIR 3420561

COAL CHUTE RENEW WEAR PLATES 3422915

COAL CHUTE RENEW WEAR PLATES 3422904

REPLACE COAL CHUTES 3415274

REPAIR PENTHOUSE CASING LEAKS 3421165

REPLACE 80 TUBES REAR ASH HOPPER SLOPE 0577695

REPAIR APERATURE MEMBRANE 3365526

801 ASH GATE 3358014

BURNER REGISTERS - INSPECT & REPAIR 3420966

BURNER REGISTERS - INSPECT & REPAIR 3420970

GENERATOR INSPECTION 3374350

MAIN TURBINE OIL COOLERS 3423195

INSPECT #1 BEARING ON TURBINE 3424116

Although the outage tasks have all been approved and scheduled, this REAP analysis shows how outage scope of work in terms of cost and value can be determined, evaluated and prioritised.



1-33

1.4.5 Boiler Tube Failure Reduction and Cycle Chemistry Improvement Training

From 8–10 February 2000, an EPRI team presented the above training programs to a number of plant and corporate staff. Appendix 1 lists the names of all attendees. A vertical and horizontal cross section of staff attended, representing all disciplines – operating, maintenance engineering and management. The two-and-half day course was well received by all attendees, and they were eager to proceed with the development of a Boiler Tube Failure Reduction program. A plant directive – Boiler Reliability Optimisation Project has been signed by the Plant Manager in support of the program. The directive (see Appendix 2) outlines the minimum requirements to manage a successful Boiler Reliability Optimisation program.

1.4.6 Boiler Maintenance Workstation (BMW) installation and training

During July 2000 the BMW software was successfully installed. Software user training was given to a number of site staff.

BMW is a windows based software program used to track and trend boiler tube failure data and information – tube failures, correction and prevention, and control actions. It consists of two modules - Boiler Works and Tube Condition. The boiler works module is a database and graphics program designed to track maintenance failure data/information – failure location and mechanism, repair costs and methods, etc. This collated information is used to focus attention on specific activities, for both planned and forced outages, to prevent future repeat failures. The Tube Condition program stores and analyses data collected from various inspections, e.g. tube wall thickness, oxide thickness data. It uses two or more data sets to determine wastage rates. These wastage rates are used to determine remaining tube life and predict tube wall thickness. This helps plant personnel plan future boiler tube inspections, maintenance activities and tube replacements.


1-34

AEP Big Sandy Unit 2, Boiler Reliability Optimization - Phase 4, July 2001

Executive Summary

Utilities throughout North America are facing the challenges of deregulation and competition. The prospect of competing in a non-regulated environment is driving utilities to find more cost effective means to operate and maintain their plants. Recognizing this, Big Sandy Plant participated in EPRI’s Boiler Reliability Optimization Program. The project started during the fourth quarter of 1999. The project aim was to identify improvement opportunities to attain and sustain an availability of 95% during peak periods – May through August and at least 90% during the remainder of the year. During 2000 phase 1,2 and a portion of phase 3 were successfully implemented resulting in sustained plant performance. As a result Big Sandy Staff were keen to continue with the Boiler Reliability Optimization Project – Phase 4, albeit with limited funding. Phase 4 is the continuous improvement phase – the proactive maintenance aspects of the program. The emphasis is on the operations aspect that is what the operator can do to minimize maintenance.

During July 2001 an EPRI team spent a week on site reviewing actions taken as a result of the Phase 1 to 3, interviewing operating and maintenance staff and published a draft report. This report documents all the findings, conclusions and recommendations accepted by Big Sandy Plant staff.

Introduction

Utilities throughout North America are facing the challenges of deregulation and competition. The prospect of competing in a non-regulated environment is driving utilities to find more cost effective means to operate and maintain their plants.

Recognizing this, Big Sandy Plant participated in EPRI’s Boiler Reliability Optimization Program. The project started during the fourth quarter of 1999. During 2000 phase 2 and a portion of phase 3 was completed with the issuing of a final report.

The staff at Big Sandy were keen to continue with the Boiler Reliability Optimization Project – Phase 4, albeit with limited funding. Phase 4 is the continuous improvement phase – the living aspects of the program.

The scope of work of this project covered the implementation of a portion of Phases 4 of EPRI’s Boiler Reliability Optimization Project and was limited to the boiler steam and water system – tubes and headers. The project consisted of the following tasks:

1. Review the progress made in implementing recommendations from previous Boiler Reliability Optimization phases.

2. Develop a framework for specific boiler long-term plant health indicators.

3. Review and optimize/enhance the daily operation of Unit 2 to result in proactive maintenance.



1-35

Description of Phase 4 of a Typical Boiler Reliability Optimization Project

Phase 4 is the continuous improvement - the “living aspects” of a Boiler Reliability Optimization Project. It builds on the successful implementation of the previous phases and provides proactive action and feedback on the overall boiler performance – reliability and maintainability aspects. Phase 4 focuses on the human reliability aspects as there is a direct relationship between plant reliability and the people who operator and maintain the boiler. If the people are reliable then the plant will be reliable. In this context human reliability is making and keeping commitments. Phase 4 also introduces the concepts of proactive maintenance into the world of the operator.

Proactive maintenance is an activity performed to detect and correct root cause abnormalities of failure. It is a first line of defense against material degradation and subsequent performance degradation, failures that ultimately lead to precipitous and catastrophic forms of failure and plant breakdown. The operator can correct a conditional failure mode that is the cause of an unstable condition and ensure that degradation type failures never occur. Thus proactive maintenance is not an activity that reacts to material and/or performance type failure conditions of a plant. Rather, it prevents such plant degradation from occurring. In reality, proactive maintenance is a preemptive first strike against failure – a true avoidance activity.

Thus, the focus of Phase 4 is on the:

• role operations personnel play in the effective and efficient operation of the boiler,

• tools, both hardware and software at their disposal

• instrumentation to control and monitor critical component parameters and

• Standard Operating instructions and procedures.

The ultimate success of a Phase 4 Boiler Reliability Optimization project is largely dependent on the following:

• Operator skills and knowledge of the boiler and the steam raising process,

• Operator understanding of the effects operating practices have on both the short and long- term plant health and

• The extent to which operating is involved in both the long and short-term maintenance decision making process.

Findings

1. All of the recommendations made in the Phase 2 report have been accepted. More than 80% have been implemented while the remaining are in the budgetary and approval processes. Table 1-1 below summarizes the status of the actions taken and for each recommendation given in Phase 2. A recommendation that has still to be implemented is a formalized root cause analysis process. This is key to the successful implementation of a boiler reliability optimization project.


1-36

Table 1-2 Phase 2 Recommendations and Actions Taken

# Recommendations Tabled Action taken

1 Specific boiler availability targets should be developed relating to the number of tube leaks per year. The strategy and performance targets should be compiled into separate plant directives, which are reviewed annually for effectiveness and appropriateness.

The following the best in class (BIC) goals for unit 2 have been established:

01 ’02 ’03

FOF 5.38 1.69 0.82

MOF 0.41 0.41 0.41

POF 0.0 26.85 1.92

To meet these goals a Strategic Plan for the boiler has been compiled

2 To repair and calibrate all boiler instrumentation critical to the efficient and effective operation of the boiler. This includes air, gas, water and steam temperatures and pressures and reheater and superheater metal temperatures.

During the planned outage in 2000 the boiler tube temperature recorders and draft gauges were repaired. However since then some are out of service awaiting spare parts

3 The installation of on-line water and steam chemistry instrumentation and the establishment chemical performance index. This index can be used in a proactive way to ensure compliance to EPRI water and steam chemistry guidelines.

Improved cycle chemistry monitoring with the installation of sodium analyzer in sample room and “ChemExpert” software in the control room.

Chemical indices still need to be developed.

4 To prepare detailed inspection plans based on the tasks identified in the Boiler Failure Defense Plans presented in this report. These inspection plans should be loaded into the CMMS. This would facilitate the development of a scheduled and or forced outage plan at short notice.

Prepared detailed inspection plans on noted problems and drew up inspection PM’s.

Developed a “short notice forced outage plan”.

Tasks are discussed at a weekly meeting – every Thursday morning.

5 Once the installation and the user training on the Boiler Maintenance Workstation (BMW) have been completed, the BMW database needs to be updated with all the history of previous tube failures. This will enable tube leaks and repairs to be tracked and trends to be established. This will also assist in formulating focussed boiler inspections plans.

BMW was installed and the history data file has been updated with data from 1994 to 2000. Currently working on updating the history file with data from 1986 to 1994.

Training classes for welders is scheduled for August 2001

6 Fireside corrosion – Confirm mechanism and extent of wall thinning by inspecting the area around the burners and side wall. Also monitor O2

and CO near the tubes on the side walls

Monitoring NOx strategy has taken top priority to measuring O2 on the side walls however inspections and NDE is done during forced outage.



1-37

7 Fly ash erosion – Perform a CAVT to confirm high velocity areas. If high velocity areas are detected – design and install baffles or flow strengtheners. Perform a second CAVT to confirm results and velocity profile

Only CAVT completed on the economizer and re-positioned baffles and re-tested. Results not as expected. Re-testing will continue as outage work permits

8 DMW failures – Replace all DMW with Dutchmen of upgraded material. Depending on the specific circumstances re-positioning of the DMW may be more cost effective

All DMW replaced.

9 Internal pitting – Revise and update the boiler lay-up procedure

1st Reheater lay-up procedure revised and updated. Performed remaining life estimation on 1st reheater, replaced tubes below minimum wall thickness. Plan to replace reheater in 2002. NDE results taken in 2000 show that there was no need to do a chemical clean.

10 Perform a level II assessment of all final outlet headers in accordance with EPRI guidelines.

Completed during 2000

11 The PM Basis must be reviewed and updated to take into account the failure mechanisms identified during the past outage inspection.

Developed new inspection PM to inspect repeat mechanisms

12 The Boiler Specialist should have an Operating Department counterpart who would be responsible to review and update and align operating procedures to current practice.

As a result of the adopting the Process Centering model, this task is now the responsibility of the Sub-process owner.

13 Boiler tube failures/Root Cause Analysis should be tracked and trended

BMW software will assist in tracking and trending tube failures and their associated causes

14 The boiler inspection plans need to be updated to take cognize of the change in the frequency of boiler periodic outages

The weekly outage-planning meeting discusses and develops a forced outage plan. Inspections plans (PM) are included in these forced outage plans

15 A formal RCA program will result in additional information valuable in managing the Boiler.

RCA still not formalized.

Tube failures are analyzed and tracked through BMW

16 The roles and responsibilities of the PdM Coordinator should be clearly defined

The PdM Coordinator position has been made obsolete by the re-organization – Process Centering. However, the PdM activities still remain with the previous incumbent.

17 Infrared thermography should be used on the boiler to identify casing leaks, valve passing etc.

Infrared thermography survey are performed quarterly as described in EPRI Boiler reliability Optimization IR TAD document


1-38

18 Develop an Equipment and Condition Indicator matrix (E&CI) for all critical boiler components. This will assist the equipment owner to assessing the condition of equipment under his control.

Trending boiler process data, using the E&CI matrix as a base, is done using the limited available instrumentation. Addition instrumentation will be installed, if, and when a new DCS is approved. It is intended to install FEGT indication during the next outage.

19 To ensure a high availability between periodic outages, 3 yearly, a detailed a inspection, test and repair plan of all known and potential problem areas

20 Develop a four-week rolling scheduling process for all maintenance to assure high utilization of resources.

It is intended to incorporate a four week rolling schedule in the new CMMS – Indus-passport system

21 Include post maintenance testing requirements into the planning process. This is a must for most CM work orders.

Post maintenance testing will be incorporated into the new CMMS

22 System owners and the Boiler Specialist should review closeout information for effectiveness.

The newly appointed Process Centering Sub-owners review closeout information.

23 Compile a management directive identifying the minimum acceptable standards to be complied with when performing boiler maintenance or repair work. This would include such statements as every tube leak/failure should be investigated, pad welding should be discouraged etc.

A Boiler Reliability Optimization Directive has been signed by the Plant Manager and is in effect.

24 Establish technology owners with responsibilities to include condition analysis.

Process Centering sub-owners have been nominated in place of technology owners.

25 Need to have a more comprehensive PDM reports as health report not exception reports

An existing PdM health report is considered adequate

26 Formalize monthly Boiler Condition Based Maintenance (CBM) meetings. Once a year, perhaps prior to any budgeting cycle, the Boiler Specialist should present a Boiler Failure Defense Plan – indicating the current condition, short and long term activities/tasks that need to be undertaken to ensure boiler reliability.

Plan to include monthly Boiler CBM meetings into the Boiler Reliability Optimization Directive



1-39

27 Case histories and cost benefits analysis of using PDM technologies should be document either in the BMW or PIMS databases.

Case histories are being established in the BMW database. Metric are being developed through the centering process

28 An annual review of the maintenance program should be undertaken to determine its effectiveness i.e. the right technologies are being applied to the right assets at the right frequency to meet the overall business objectives and maintenance strategy.

Included in the business strategy feedback loop.

29 Consideration should be given to having an annual Boiler Specialist Meeting and forming a Boiler Specialist’s Users Group. This will aid in the transferring of information and knowledge amongst the Boiler Specialists within the AEP organization.

AEP has formed a Boiler Reliability Optimization Team whose responsibility is to support the implementation of the program

30 Consider providing Level of Awareness training (LOA) training for the entire staff on Plant Maintenance Optimization.

LOA training has been done previously. Not considered necessary at this stage.

31 Re-train all PIMS users on the details of PIMS and how best to it

PIMS is to be replaced by Indus – passport by January 2002. Training on the new system starts in the last quarter of 2001

32 Develop a series of inspection tasks using the data and information identified and update the CMMS.

Inspection tasks for 2000 have been completed.

2. Five operators with a wide range of experience were interviewed (four Unit operators and one Equipment Operator). The interviews focussed on the knowledge of the boiler and the steam and water cycle. Responses to questions from some of the operators regarding boiler operation and operating parameters on critical equipment were incorrect. Critical components within the boiler were not known. Under base or transient loads the focus is on steam temperature only, metal temperatures are not considered. These same operators have been given latitude to develop their personal operating practices and to experiment with set points on critical equipment. The combined effect of the above can lead to an increased risk with the consequent reduction in unit performance, both efficiency and reliability.

3. Each unit operator interviewed uses his or her own set of procedures - start-up procedure, marked up with his or her own notes. The start-up procedure reviewed in the control room was dated 1970, where as, the set given at the beginning of this project was dated October 1998. Not all operators were aware there was a new/revised start-up procedure. Those who had a revised copy preferred to use their own marked up copy. This results in inconsistent start-up times, and possible inappropriate actions during unit start-ups.


1-40

Operating instructions in the form of unsigned, undated, hand written notes were found taped to the control panel. The operator bases identification of the author of these instructions on trying to recognize the handwriting. Operators have no way of knowing what new operating instructions have been issued unless they see a new piece of paper taped on the board or page through the log book assuming it was written. There is no way of ensuring that all operating personnel are being made aware of changes in critical operating parameters.

4. There is no documentation standard or control of Operating Procedures and Instructions. Procedures reviewed did not follow a common format, approval signatures, date issued, revision number etc. The new Unit Lay-up procedure given for review was in the form of an email.

5. Operators are not always informed of equipment, including chart recorders, being taken out of service for maintenance work. Unanticipated requests to have equipment taken out of service and clearances issued results in inefficiencies and delays for both maintenance and operators. The introduction of the “Forced Outage Maintenance Work” matrix posted in the control room has eased this situation. The matrix is an excellent communication tool. It keeps operating personnel informed of what work will be done under several forced outage scenarios. This puts the operators in a proactive mode regarding clearances and equipment shut down priorities.

6. Experienced and qualified Laboratory personnel control unit chemistry within EPRI guidelines. Operating personnel have limited knowledge of unit chemistry or the effects of operating outside EPRI guidelines on unit reliability. Operators are in a reactionary mode regarding chemistry excursions, depending on laboratory personnel for direction and corrective action.

There are no feedwater or boiler pH indications in the control room. On a once-through, super critical unit using an oxygenated treatment (OT), maintaining a proper pH level is of the highest importance, as there is only a short time window of opportunity to react to low pH levels. Lack of pH readings in the control, coupled with the limited knowledge of operating, presents an increased risk to the reliability of the unit.

7. Operators spend approximately 25% of their time taking manual readings for staff personnel. A number of these reside in the computers in the control. The need for and the information from these readings has not been explained and or justified. This leads to resentment and frustration between operators and other staff members.

Conclusions

1. Big Sandy staff needs to be congratulated on the effort they have expended in successfully implementing the recommendations from Phase 2 Boiler Reliability Optimization Report. This effort shows the commitment to improve and sustain high performance.

2. There is a lack of management control in the operations area. Operators are not held accountable for their individual actions, nor the results of their actions on Unit efficiency or reliability.



1-41

3. Operators are not being given the tools or resources they need to operate the Unit in a consistently safe, reliable, and efficient manner. Operating procedures and instructions are written and distributed in a uncontrolled manner. Training of operators is inadequate and inconsistent. As a result, there are no primary standards for operation or control of critical Unit processes.

4. The loss of many older, more experienced operators and supervisors has diluted the knowledge base of the operations group. This has resulted in a wide range of operating philosophies among the Unit Operators.

5. Chemistry control is good due primarily to the efforts of the laboratory group personnel. Operations contribution to the success of this program is minimal due to their lack of knowledge of the chemical processes involved.

6. Control room instrumentation is an extension of the Unit Operators senses. The sense of urgency related to instrument repair and/or replacement needs to be raised to a new level. This should be accomplished in the near future if the practice of assigning only one operator to the control room is to be successful.

7. If Big Sandy is to successfully implement a “Pro-Active Maintenance (PAM) program, it is essential that the Operations group plays an active role with its implementation. Without their full commitment, success will be difficult. Serious consideration should be given to the preceding recommendations before plant management attempts to implement such a program.

Recommendations

1. Continue with the efforts to implement the outstanding recommendations from Phase 1 (above table) especially with installation of a FEGT monitoring system. This will allow the operator to load the unit to its maximum without placing excessive thermal stresses on the super heater.

2. Develop a series of long term plant health indices for the boiler: thermal excursion, chemical and trip index.

– Thermal excursion index is the number of additional equivalent operating hours experienced by the most sensitive header in the boiler as a result of metal temperature excursions in excess of design temperature with a boiler pressure > 80% of normal operating pressure. In addition, all excursions in excess of design + 100o F shall be included irrespective of boiler pressure.

The Additional Operating hours = Constant (k) x Σ (∆ T x ∆ t)

Where:

∆ T = difference between the peak temperature reached in an excursion and the design operating metal temperature

∆ t = total duration of the excursion in hours


1-42

Constant (k) = approximation, assuming a straight-line relationship between elevated temperatures and equivalent operating hours.

Thermal index = Additional Operating hours x Total hours in month divided Σ of unit operating hours for the month.

– Chemical Excursion Index is a set of indices in which measurements taken on the plant of chemical conditions of steam, feedwater, etc are normalized to a base of one. The actual values are manipulated mathematically to give a result that is of the order of one. A result of 0.5 is excellent and greater than 0.9 is considered unacceptable. Results of between 0.6 and 0.9 indicate chemical condition under control and no risk to long term health. Results between 0.9 and I are acceptable for short duration only. Results greater than 1 require immediate attention. These results can be represented in a bar chart and displayed to the operator

Chemical index = Σ of all incidents of unit parameters > 0.9 for current month x Total hours in month divided by Σ of operating hours for the current month

– Trip index is the accumulative total number of automatic and manual trips of the unit. This will give a feel for the number of stress cycles the plant has been subjected to no matter what was the cause

Trip Index = Σ of all trips for current month x Total hours in the month divided by sum of unit operating hours for the month

These indices can be developed for all units and used as a means to compare unit performance

Note the numbers quoted above are guidelines, they should be adjusted to suite the particular unit circumstances and conditions

3. Analyze the effects of the wide range of conditions under which the Unit is operated. Select the “Best Practice” and develop a set of procedures and hold all Unit Operators to those standards. The start-up procedure should have target times for mile stone activities and check boxes for the operator to insert the actual time and reasons for any deviation from target. After the start-up these times can be analyzed for improvement opportunities.

4. Operations group should be involved in planning and scheduling of maintenance on any equipment that needs to be taken out of service. Operators should be informed on the status of any instrumentation that is out of service. They should know why, and the expected return to service date.

5. Develop a document control system and conduct periodic compliance audits to ensure the correct document/procedures are being used. This system should include all operating procedures and instructions. As a minimum, these should include signed approval levels, revision date, distribution list, standardized format, and collection and disposal of outdated documents. A list of typical headings found in a procedure can be found in Appendix 2. One complete set of operating procedures and a “temporary operating instruction” book should reside in the control room. All entries in this book should be signed and dated. Operators at the beginning of their shift should review this book for updated changes.



1-43

6. Provide training to all operating personnel on basic unit water and steam chemistry and the alarm and action level guidelines.

7. Provide pH indication for condensate, economizer inlet, and main steam in the control room.

8. Review the need and frequency for the manually recorded readings. Values, which reside in the computer, should be collected automatically by creating a specific computer log. The use of and value obtained from the readings collected manually should be feed back to the operators.

9. Develop and formalize an improvement program - root cause analysis process. Use the following as a guide.

– Obtain Management support such a program – financial and human resources

– Select and nominate a champion – owner of the process

– Review the industry and select a root cause analyses system using the following seven factors:

Easy to use in the field by non-experts

Effective in consistently identifying root causes

Well documented

Accompanied by effective user training

Credible with the workforce

Helpful in presenting results to management

Designed to allow collection, comparison, and measurement of root cause trends.

– Train investigators/ Coaches – representation from Operating and Maintenance

– Give awareness training to operators and craftsperson

– Compile an administrative procedure – purpose, objective, scope responsibilities, investigation and reporting matrix etc

– Develop a database for causes and corrective actions

10. Provide training to ALL operating personnel on Unit chemistry guidelines and emergency responses. Of special importance is the relationship between specific conductivity, ammonia levels, and pH values in the condensate/feedwater systems. The ammonia feed-rate is controlled using the specific conductivity of the feedwater, which maintains the target pH values. The relationship between conductivity values and ammonia is a straight-line relationship, whereas the pH is logarithmic. Changes in conductivity are the operator’s first indication of approaching problems. Operators should also understand the difference between cation and specific conductivity, as well as the importance of sodium levels. Provide Ph reporting instrumentation to the control room. This should pH values on the condensate, feedwater, boiler, and main steam.


1-44

Appendix 1: Interview List

Name Designation

1 D. Mell Production Support Manager

2 J. Burton Region 3 Engineer

3 E. Dillow Region 3 Engineer

4 A. Robinson Production Services Leader

5 S. Jenks Production Services Leader

6 J. Skaggs Lab Technician

7 D. Pinson Unit Operator

8 R. Lewis Unit Operator

9 J. Adkins Unit Operator

10 R. Peck Unit Operator

11 M. Keene Equipment Operator

12 M. R. Mckenzie Production Services Leader



1-45

Appendix 2: List of Typical headings that appear in a policy and or a procedure

1. Group name

2. Document type: policy, standard, guideline, procedure etc

3. Reference number

4. Revision number

5. Description of revisions made

6. Review date

7. Date authorized

8. Date when document came into effect

9. Compiled by

10. List of designations who have approved the document

11. Authorized by

12. Title

13. Background/preamble/introduction

14. Purpose

15. Scope of document

16. References

17. Definitions and or abbreviations

18. Responsibilities

19. Responsibility matrix

20. Requirements: step by step of what is required to be done

21. Work process flow and responsibility matrix

22. Distribution list

23. Appendices


2-1

2 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 2 – BOILER INSPECTION PLAN REVIEW AT GREAT RIVER ENERGY’S COAL CREEK STATION – UNIT 2

2.1 Executive Summary

Senior Management made the decision to change the boiler maintenance overhaul frequency from two to three years, and wanted assurance prior to the Spring 2001 outage that the existing boiler inspection plans were comprehensive and complete. Confidence in the newly adopted maintenance strategy is essential, as the next general overall outage after the Spring 2001 outage is in 2004. EPRI was requested to review the current Unit 2 outage inspection plan and recommend changes and/or additions, to provide this assurance.

During the week of August 21, 2000, two EPRI representatives interviewed a number of Coal Creek Power Plant staff members from the operations, maintenance, engineering and management groups. The unit history, maintenance, operating practices, and the boiler inspection plan were reviewed and analyzed.

The current boiler inspection plan is reactive, i.e. it focuses almost entirely on known problem area – sootblower erosion - and does not cover all the components within the boiler in sufficient detail. There is a limited predictive element to the plan. It does not address any future or potential failure mechanisms, and is therefore not comprehensive and will not give the necessary assurance that all risk areas have been adequately covered.

This report highlights a number of recommendations that would assist in developing a comprehensive and effective boiler inspection plan. These recommendations, together with comments/suggestions added to the Unit 2 Outage Inspection Plan given to the EPRI team on 21 August 2000, will provide a quality inspection plan. Annexure II

An extract from another utility’s boiler inspection plan and is given in Annexure I to illustrate the detail that is required in a typical Boiler Inspection Plan.

2.2 Introduction

Faced with de-regulation and competition, Senior Management at Great River Energy – Coal Creek Station, challenged the existing plant maintenance strategy and sought opportunities to improve plant availability and reliability. One of the changes made was to the overall unit

EPRI Licensed Material Boiler Reliability Optimization Project Phase 2 – Boiler Inspection Plan Review at Great River Energy’s Coal Creek Station – Unit 2

2-2

maintenance strategy, i.e. the frequency and duration of outages was changed from 2 to 3 year intervals and a shortened duration. This change would improve the overall unit availability. To improve unit reliability the focus shifted to understanding and eliminating the causes of boiler forced outages; the predominant cause being boiler tube leaks. To ensure “no surprises” a comprehensive boiler inspection plan needs to be developed and implemented during the next outage – Spring 2001 .

A boiler internal inspection is one of the most important inspections to be carried out during an outage. It must be done timeously, systematically, accurately and by experienced inspectors. The quality of this inspection will ensure that a reliable boiler will be put back into service. The only approach is “do it right the first time all the time”

The selection and number of inspectors should not be based on quantity but rather on quality. The staff chosen must be committed to the assignment and have sufficient experience and background to identify any abnormalities. The staff should have a good knowledge of the gas flow patterns, temperature profiles, material composition and movement/expansion of the boiler.

EPRI was requested to review the existing plans and suggest changes and or additions that will give this assurance, to provide assurance that an adequate and comprehensive plan was in place.

During the week August 21, 2000 two EPRI representatives visited the plant, reviewed and discussed the current boiler inspection plan and tube leak history data with the boiler engineer, and interviewed a number of staff from the operating, maintenance and engineering groups.

2.3 Findings and Observations

The findings below are based on the Unit 2 outage inspection plan as presented during the review (see Annexure II) and on the personal interviews.

2.3.1 The inspection plan is divided into five sections: burner front, waterwall, Superheater and reheater, back -pass and boiler external. Although the boiler as a whole is covered, the specific inspection details of the areas within each section are not covered sufficiently e.g. superheater, DMW etc.

2.3.2 Each section has been assigned to a responsible person who will manage the inspection and data collection. The boiler engineer has overall responsibility to ensure adequacy of inspections, data collection and coordination between the various sections.

2.3.3 The plan is reactive and focuses almost entirely on inspecting known problem areas e.g. identifying and correcting soot blower erosion damage.

2.3.4 Areas within each of the five sections are not prioritized and no “go” – “no go” criteria given e.g. minimum tube wall thickness and crack/groove size/length and shape.


Boiler Reliability Optimization Project Phase 2 – Boiler Inspection Plan Review at Great River Energy’s Coal Creek Station – Unit 2

2-3

2.3.5 Documentation of the pre-outage external inspection and walk-down of the boiler is comprehensive

2.3.6 Infrared survey of the boiler casing and valves is not included in the pre-outage inspection.

2.3.7 No provision is made to complete a ‘dirty boiler’ inspection.

2.3.8 The present sootblower maintenance strategy is not in line with current or future operating mode of the unit. Sootblower preventive maintenance has not been adequate, as shown by the number of corrective maintenance activities that have needed to be performed. The top five SB problem areas are: Feed tube wear, Sprocket and chain wear, Bearing failures, Packing failure and Carriage wear

2.3.9 Thermal drains are being replaced with steam traps without a thorough root causes analysis being done to ensure that this is the correct action to take. There is a limited history regarding failures and repairs.

2.3.10 Result trending and predictions cannot be adequately completed, as there is no central database to record and keep boiler test and inspection data and information. Boiler inspection reports that identify actions taken, future actions needed, or lessons learned, are inadequate. All history data on the boiler’s tube leaks is owned and maintained by the Boiler Engineer and is in hard copy format.

2.3.11 Root Cause Analysis is not formally done. The Boiler Engineer maintains tube leak reports, using “CE Availability” data sheets. Not all tube leaks are investigated. History summary is available from 1989. The summary includes date, suspected failure cause, corrective action taken, and recommended outage follow up actions.

2.3.12 Startup and shutdown procedures are combined. They are too general and give little guidance as far as criteria to be met – metal temperatures have been exceeded in the past because of this. No precautionary measures are given to the operator, e.g. when firing the boiler at over pressure.

2.4 Recommendations

2.4.1 A more detailed proactive inspection plan is needed to support a three-year outage cycle plan, while meeting plant goals. The boiler components are time and temperature dependent. Over-stressing these components will accelerate aging. By adopting a proactive stance, knowing the remaining life of these components, replacement plans can be developed and implemented before aging/failure affects unit reliability.

2.4.2 A schedule should be developed for all inspection areas in the boiler. Prioritization should be based on potential findings that would affect outage duration, safety, budget restraints, and unit reliability. All areas that have a high priority should be completed by the end of the first week. This requires a pre-outage study of the duration and cost of inspection techniques to be used. It should include component disassembly, scaffolding,


2-4

insulation removal, surface preparation, ventilation, lighting, and environmental compliance.

2.4.3 Criteria need to be developed for the inspection of each area, based on historical data, possible damage mechanisms, typical locations, inspection techniques, required preparation and support, and estimated inspection times. It is important to establish a definition of what are permissible flaws, and what action should be taken regarding a repair/replace decision when an unacceptable flaw is found. Non-critical flaws should be duly noted, entered into the database, and trended over at least two outage cycles.

2.4.4 All data gathered during the inspection, as well as all repair/replace decisions, should be entered into a database maintained by the Boiler Engineer. This database should include ‘as found ‘ and ‘as left’ conditions, repairs made, and recommendations for future outages. Examples of both inspection and repair record formats are included in the annexure I

2.4.5 Expand inspections of the superheater and reheater sections beyond soot blower damage, broken clips, rubbing damage, and ash erosion to include remaining life assessments. UT measurements should be taken to determine the oxide thickness on the superheater. These values can be used to determine the high temperature areas in the SH and will validate the temperature profile as measured by the installed tube metal temperature instrumentation. Tube samples should be taken from hot areas of the SH for metallurgical analysis to determine remaining life. These analyses may have an impact on operating practices.

2.4.6 Horizontal and pendant SH sections are difficult to drain completely, if at all. As a result, they often suffer internal damage due to pitting from condensation. Tube samples should be taken and analyzed for this.

2.4.7 Header base line data, internal and external, needs to be obtained, as it would indicate the effect of past operating modes, and provide information to determine future operating modes. Superheat and reheat headers have never been inspected. Fiber optics can be used to determine if any erosion or cracks have developed.

2.4.8 The current inspection plan calls for inspection of the remaining wall blower refractory tabs. Some of these tabs have been removed, as they have been a source of tube leaks. The remaining tabs should be removed, as they are neither used nor needed and are potential areas for tube leaks.

2.4.9 High stress raisers are potential sources for tube leaks. Inspect and itemize all internal boiler tube attachments for cracks, erosion or damage.

2.4.10 An infrared survey of the boiler casing, valves, and other potential sources of heat loss, should be performed as part of a pre-outage inspection.

2.4.11 A ‘dirty boiler’ inspection should be included in the boiler inspection plan. Many will argue that this inspection is uneconomical. However, there is much to be gained by



2-5

observation and correct interpretation of what is observed. An experienced boiler engineer can benefit enormously and gather insight of what is going on inside the furnace from this inspection. Visually examining ash deposits – color, size, build up and profiles - and tube surface conditions, will highlight areas of erosion and high or low gas flow regions.

2.4.12 Although no Dissimilar Metal Weld (DMW) failures have been reported, welds should be inspected visually at every outage using UT. DMW tube samples should be taken and analyzed for remaining life.

2.4.13 The sootblowing system’s design should be reviewed. This should include an evaluation of reliability, availability, maintenance and operating issues. This review should also include an evaluation of the effectiveness of steam traps in eliminating water in the system. From this design review an optimized sootblower operation and maintenance strategy can be developed, implemented, and issued regularly via the Maximo system.

2.4.14 Perform a functional check on all sootblowers. Each sootblower should be operated in turn and checked visually from the inside of the furnace for correct nozzle position, blower insertion length and rotation.

2.4.15 Full implementation of the Maximo system should include the following.

– All identified boiler inspection tasks should be kept in the Maximo system as preventive maintenance (PM).

– These PM should include, but not be limited to, duration, location, man-hours, equipment needed, and collaboration needed with other groups.

2.4.16 Implement EPRI’s Boiler Maintenance Workstation (BMW). BMW is a Windows-based software program used to track and trend boiler tube failure data and information – tube failures, correction and prevention, and control actions. It consists of two modules - Boiler Works and Tube Condition. The boiler works module is a database and graphics program designed to track maintenance failure data/information – failure location and mechanism, repair costs and methods, etc. This collated information is used to focus attention on specific activities, for both planned and forced outages, to prevent future repeat failures. The Tube Condition module stores and analyzes data collected from various inspections, e.g. tube wall thickness, oxide thickness data. It uses two or more data sets to determine wastage rates. These wastage rates are used to determine remaining tube life and predict tube wall thickness. This helps plant personnel plan future boiler tube inspections, maintenance activities and tube replacements.

2.4.17 Following every planned outage, a detailed report should be issued. This report should identify inspection results, repairs and modifications, lessons learned, and activities that need to be planned for the next outage.

2.4.18 All NDE data on boiler pressure parts should be trended and used for failure prediction and long term planning.


2-6

2.4.19 Senior Management should support a formalized ‘Boiler Tube Failure Reduction’ team. The team, headed by the Boiler Engineer, should have members from the operations, maintenance and engineering groups. The team should follow a recognized root cause analysis process when analyzing all boiler tube leaks. Records should be kept in a central database and failure statistics and tube condition – minimum wall thickness, oxide thickness and remaining life results, etc., should be tracked and trended.

2.4.20 Separate Unit Start-up and Shut Down procedures should be written, as these are two distinctly different activities each with specific criteria and operator requirements. Each procedure should identify the precautions to be taken before or during each step. This should be followed by a step by step sequence of actions with criteria to be met, e.g. metal temperature rates of change, pressure ramp rate, differential temperatures, etc. Each action should have a check box and a remark column where the operator can check off the action and explain/give reasons for any deviation from the specified criteria.



2-7

ANNEXURE I: Extract from a Boiler Inspection Report

This report covers all boiler internal inspections, including tubes, burner tips, and dead air spaces performed by Plant personnel. Inspections of the mud drums, the main steam drum, and penthouse are included.

1st digit (location): 1 Firebox

2 Upper Arch 3 Down draft 4 Dead-air Spaces, Drums 5 Penthouse

2nd digit (tube type): 1 Economizer Tubes 2 Waterwall Circuit Tubes 3 Steam Cooled Tubes 4 Reheater Circuit Tubes 5 Superheater Circuit Tubes

Inspection Areas

Most of these are known areas where failures have occurred or repairs have been made during past Periodic Outages and Forced Outages. DM designates D-meter inspections for tube thickness.

1200 Waterwall Cleaning Devices, operational condition

This inspection involves both pre-outage and outage responsibilities. Operations must check blower and lance operating conditions prior to Unit shutdown. This requires that all water-lances be tested for proper speed settings and visually verified for operating effectiveness, noting condition of the booster pumps and blowing pressures. All "partial-arc" wall-blowers must be verified for proper operation. Note any existing conditions that may effect tube life and closely inspect theses areas (see item Nos. 1209 and 1210). A furnace side inspection is to be performed by the Boiler Inspection Team from a furnace scaffold, including nozzle head conditions and insertion depth settings on all wall blowers.

Inspection results

All wall blowers were inserted for inspection and checked for appropriate depth setting, yielding 20 required adjustments (vs. 43 total in 1996). Lists of all damage and needed adjustments were given to the Unit Planner. Depth setting adjustments were made during the outage, while the lance changes were scheduled for after the outage due to parts availability. Over half of the original water-lances (WL) were found with damaged lance tips (cracking or holes).. Lists of all damage and needed adjustments were given to the No. 7 Unit Planner


2-8

Recommendations for 2004:

Several of the lances were binding on start-up, and at least three of the lance tubes will need replacement. Air cooling to the control boxes will be added to keep the box temperatures below 1220F. Access to WL nozzles inside the windbox will need to be made during a furnace cleaning outage. For 2004, wall blower and WL repairs and adjustments are expected to be minimal.

1501 Superheat Division Panels, attachment problems

Look for misalign tubes and possible resulting damage. Look for failed "wrapper tube" ties. Look for abrasion damage from the attachment “ears” rubbing into adjacent tubes. Recommend repairs.

Inspection results:

There were 10 locations found and repaired (vs. 31 found and 23 completed in 1996). Of these, two were failed attachments, and 8 were from attachment rubbing damage. Some of the rub damage locations were severe, causing abrasion damage in excess of 1/2 wall thickness (see Fig. 6). The recommended repair for the ear rubbing was to remove that portion of the attachments.

Recommendations for 2004:

Continue inspections for failed attachments and attachment ear rubbing. Expect nearly the same amount of work as in 2001.

Figure No. 6 Attachment “ear”

division tub

attachment

rub damagearea(s)



2-9

ANNEXURE II: Unit 2 Outage Inspection Plan as of August 21, 2000

This plan contains numerous visual checks and does not give guidance on what to look for, how to look for it, and what to do with the data/information obtained? Some guidance needs to be documented for each team on what to do if cracks or dents are identified, e.g. measure wall thickness, cut out and insert a “Dutchman”, weld over lay, etc.

Comments/additions have been made in each section below. For ease of identification they have been underlined.

1. Burner Front Team Member

Will perform burner front inspection and coordinate all repairs of the burner from inside and outside the boiler. The following inspections will be performed:

1.1 Prior to the outage (within 2 weeks of the start of the outage)

Conduct and external walk down of the burner fronts from 5th floor to 10th. The items to check shall include:

• Look for any coal or ash leaks around the coal piping and nozzles.

• Check the condition of all coal piping spring hangers.

A. Record the hot position of the hangers.

B. Check if any hangers are bottomed out.

C. Visually inspect the hanger rods and attachment pins.

D. Inspect the hanger lugs on the support steel and on the coal piping.

• Note if any hagen drives are disconnected.

• Inspect the air supply and signal lines to the hagen drives. Check for any broken or disconnected lines and to see if any lines have broken loose from their supports. Check the condition of the multi tube lines from the burner front to the LIE cabinets.

• Check with Operations and the E&I techs to see if any of the tilt drives are short stroked.

• Visually inspect the external tilt linkage for any members that are bent, broken or disconnected.

• Note from the Honeywell any flame scanner with low intensity readings. These scanners will be inspected in detail after the unit is off line.

• Check with the E&I techs to find out if there are any flame scanners that they have had trouble inserting or removing. Also check to see if there are any work orders on the flame scanners.

• Check for any oil leaks around the burner front oil guns and supply piping.


2-10

• Inspect all of the oil guns and spark igniters and oil gun air lines.

• Visually inspect the new SAS air ducts on 9th and 10th floor.

A. Visually inspect the SAS duct hanger rods and attachment pins.

B. Check for any interference's between the SAS registers and the structural steel.

C. Inspect the windbox hanger rods on 9th floor.

• Visually inspect the fabric expansion joints.

• Inspect all external pipe work of the various systems for steam and water leaks, sagging, abnormal internal noise, signs of shock, pipe guides, hangar supports – free movement of constant and spring supports, missing bolts and nuts and dust covers, signs of corrosion and integrity of the supports.

• Inspect boiler skin casing cladding for damage, missing or corroded sheets.

• Inspect all ducts for air and gas leaks, abnormal movement or vibration, defective damper linkages or fouling with moving parts.

• Inspect the boiler main and sub structure for deformation, missing bolts and nuts, flame cutting damage, severe corrosion, signs of unauthorized welding on main load bearing beams, beams/buckstays for sagging and or distortion.

1.2 At the start of the outage (During the installation of the boiler scaffolding)

• Inspect the windbox and windbox expansion joints.

• Stroke all of the windbox dampers manually, checking for any that are binding and that the connector link between the damper blades is intact.

• Remove all the flame scanners and insert an old one in each guide tube noting any problems with inserting the flame scanner.

• Manually stroke the tilts on the over and underfire air compartments and the underfire air compartment dampers.

• Stroke all aux air and SAS yaws.

• Inspect the inside of the SAS ducts. Check condition of the fabric expansion joints and air dampers.

1.3 Burner Front inspection (Upon completion of the boiler scaffolding)

• Insert all oil guns and check the dimension from the face of the diffuser to the oil gun tip. See the CE drawing for details.

• Insert all of the ignitors and check the insertion depth and position of the ignitor tip.

• Inspect the oil nozzles looking for the following items.



2-11

A. Condition of the tip. Look for distortion and burning of the tip. (Oil guns should be tested in a test rig to ensure proper oil spray angles. This help determine the performance of each gun)

B. Check condition of the oil gun and the flame scanner guide tubes.

C. Look for broken or bent tilt linkage. Note if any lever arms are working off of the stationary pivot pin.

D. Note if any of the diffuser discs are missing on the elevation 5/6 and 7/8 oil nozzles.

• Inspect the coal nozzles looking for the following items.

A. Condition of the two piece tip. Determine which tips will be repaired or replaced.

B. Check for thinning of the removable transition piece on the front of the coal nozzle body.

C. Check for missing or loose ceramic tile in the coal nozzle body and gate body.

D. Look for broken or bent tilt linkage. Note if any lever arms are working off of the stationary pivot pin.

• Inspect the aux air and SAS nozzles looking for the following items.

A. Condition of the tip. Look for distortion and burning of the tip

B. Check condition of the yaw linkage, looking for any bent or broken pieces.

C. Check condition of the scanner guide tubes in the aux air nozzles.

D. Look for broken or bent tilt linkage. Note if any lever arms are working off of the stationary pivot pin.

Use the form shown as Attachment 1 to record all of the repairs that the contractor shall perform. All repairs such as nozzle replacement, linkage repairs and nozzle ceramic repairs should be detailed on the form.


2-12

2. Waterwall Team Member

Will perform inspections and coordinate repairs of the waterwalls, center wall and slopes. The following inspections will be performed.

2.1 Prior to the outage (within 2 weeks of start of the outage)

• Review the tube leak records for the last three years and identify any areas that need further repairs. This should include any areas where window were installed or any areas where tubes were just welded shut.

2.2 Prior to the installation of the boiler scaffolding (after the throat scaffolding is installed)

• Perform a general visual inspection of the slope before the scaffold is installed. Check for large dents in the slope, dented tubes and worn areas.

The wear areas should include the following.

A. The slope along the side walls under corners 2 and 7.

B. Both the front and rear slope along both sides of the center wall.

C. The side walls at the throat opening.

D. The center wall where it penetrates the slope.

2.3 Upon completion of the boiler scaffold

• Clean the waterwalls and perform an UT inspection of the waterwalls between the upper burner front and the SAS registers. See the attached drawing, Attachment 2 for details of the area to be inspected.

• Inspect all of the wall blower openings. The following items should be checked.

A. Visually inspect for any worn or eroded tubes. Mark any tubes that may need a follow up UT inspection. Mark any tubes that require replacement. (Criteria need to be developed – 70 % wall thickness is commonly used)

B. Check the condition of the refractory and mark any openings that require refractory repairs. A list of the openings requiring refractory repairs should be forwarded to the External Boiler Team Member.

C. Visually inspect for any cracking occurring at the tabs welded around the opening. (Remove unnecessary tabs by carefully grinding)

• Inspect the crotch tubes at the top and bottom of the burner front and SAS register openings.



2-13

• Perform an UT inspection of the boiler nose. (“Go” and ‘no go” criteria need to be provided). See the attached drawing, Attachment 2 for details of the area to be inspected.

• Inspect the bends in the waterwall tubing below the nose. Perform a random UT inspection of the tube bends. (record the measurement position and tube location as a reference)

• Inspect the bends at the bottom of the radiant reheat section. Perform a random UT inspection of the bends. Also check for any tubes out of alignment or bent tubes.

• Visually inspect the joints between the center wall panels. Check for rubbing damage on the tubes at the panel joints.

• Perform a visual and UT inspection of the slope looking at the following areas.

A. The front and rear slope along both sides of the center wall. The area to be inspected should be the full length of the slope for a 20 tube wide panel.

B. The slope along the side walls under corners 2 and 7. The area to be inspected should be the full length of the slope for a 20 tube wide panel.

C. The slope just below the upper bends directly under burner fronts 2, 4, 5 and 7.

• Inspect the front wall between the burner front and radiant reheat for any tubes that are smashed or dented.

• Replace the lower radiant reheat bend and section of tubing in furnace ‘A’ that previously failed.

• Remove a two tube samples from the waterwalls. The sample should be removed from the front or rear wall on ninth floor between the burner fronts. (consider taking more samples to get a representative set)

2.4 Upon completion of the bottom ash scaffolding

• Visually inspect for any smashed or dented tubes at the throat opening.

• Check the side wall tubes at the throat opening. Check the horizontal tubes at the center joint of the side wall.

• Inspect the lower side of the throat tubes.


2-14

3. Superheat and Reheat Team Member

Will perform inspections and coordinate repairs of the superheat and reheat pendants. This work will be completed on night shift. The following inspections will be performed.


• Review the tube leak records for the last three years and identify any areas that need further repairs. This should include any areas where windows were installed or any areas where tubes were just welded shut.

3.2 During the outage

• Visually inspect all of the sootblower lanes on the leading edge of the superheat pendants. This work will be performed using the boiler scaffolding.

• Visually inspect the fluid cooled spacer tube that runs through the front of the superheat pendant. Check for broken clips or rubbing damage.

• Scaffold the sootblower lanes in the center of the superheat pendants and visually inspect all sootblower lanes.

A. Special attention should be given to inspecting the double offset bends in the crossover area.

B. Visually inspect the fluid cooled spacer tube that runs along the center wall. Check for rubbing damage.

C. Check for any broken or missing flex connectors in the pendants

D. Inspect the bent tubes located on the A side of the furnace. Determine what repairs will be required to straighten the tubes.

• Scaffold the sootblower lanes between the rear waterwall hanger tubes and the reheat pendants. The following inspections should be performed.

A. Perform a UT inspection of the trailing edge of the rear waterwall hanger tubes at each sootblower lane.

B. Inspect the rear waterwall hanger tubes where they penetrate the slope. Check for ash erosion on the trailing edge of the tube from ash sliding down the slope and check the leading edge for flyash erosion.

C. Visually inspect the leading edge of the reheat pendants and the trailing edge of the superheat pendants at each sootblower lane, checking for sootblower erosion.

D. Visually inspect the fluid cooled spacer tubes located in the center of the boiler on the back side of the superheat pendants. Check for rubbing damage between the spacer tube and the rear waterwall hanger tubes and superheat pendants.



2-15

E. Inspect the fluid cooled spacer tube that runs through the front of the reheat pendants. Check for broken clips or rubbing damage.

F. Inspect the front lower bends of the reheat pendants. Check for sootblower erosion on the inside and outside of the bends and for any missing or damaged tube shields.

• Perform oxide thickness measurement on the “hottest” across the superheater to determine the temperature profile.

• Remove samples for metallurgical analysis – creep damage and remaining life.

• Inspect DMW for cracks. If cracks are found cut out a section and replace with a “Dutchman”. If no cracks found cut out a section from the hottest area for metallurgical analysis.

• Scaffold the sootblower lane on the back side of the reheat pendants. The followings inspections should be performed.

A. Visually inspect the sootblower lane looking for sootblower erosion. Special attention should be directed to the double offset bends in the crossover and to the bottom side of the horizontal crossover tubes.

B. Visually inspect the spacer strap in the reheat finishing section, looking for broken or missing U straps.

C. Check for any broken or missing flex connectors

D. Inspect the tube shields at the bottom sootblower on the trailing edge of the reheat pendants. Check for missing or damaged shields.

• Inspect all sootblower openings along the side wall. Check for tube erosion and for cracking along the tabs. Also visually check the condition of the lance nozzles.

• Inspect the roof and slope tubes, checking for bent or warped tubes.

• When performing pendant inspections special attention should be given to the first five pendants from each side wall. Historically the first five pendants have suffered more severe sootblower erosion.


2-16

4. Back-pass Team Member

Will perform inspections and coordinate repairs of the back-pass from the furnace screen tubes to the economizer. The following inspections will be performed.


• Review the tube leak records for the last three years and identify any areas that need further repairs. This should include any areas where windows were installed or any areas where tubes were just welded shut.


• Perform an UT inspection of the inner radius of the lower bend of the furnace screen tubes just above the refractory dam.

• Visually inspect the spacer straps in the front of the superheat finishing section looking for missing or broken U clips. Also inspect the spacer straps that wrap around each pendant. Check for any broken or cracked welds on the front of the wrap around straps. 30 new straps have been ordered based on a previous inspection. Inspect for sootblower erosion along the lower sootblower lane on the front of the superheat finishing section and the rear of the furnace screen tubes.

• Scaffold the sootblower lanes on the front side of the superheat finishing section. Visually inspect the condition of the tube shields on the trailing edge of the furnace screen tubes. Visually inspect the condition of the superheat pendants checking for sootblower erosion and broken or missing flex connectors.

• Visually inspect the rear extended backpass floor tube located directly in front of the superheat backpass front tubes. Check for sootblower erosion on the floor tube that occurs in the spaces between the backpass front tubes.

• Scaffold the upper sootblower lane on the back side of the superheat finishing section. Check for sootblower erosion on the trailing edge of the superheat pendants and on the leading edge of the backpass front wall tubes.

• Visually inspect the spacer straps located on the backpass front wall tubes.

• Check for any U clips that are broken or have burned off. Also check for any missing or damaged tube shields in this area.

• The sootblower lane for retracts 19 and 48 which is located at the rear of the superheat front wall tubes will require several inspections.

A. The trailing edge of the superheat front wall tubes and the top side of the horizontal superheat tubes should have a UT inspection performed on them along this sootblower lane.

B. The front side of the superheat low temperature pendants should be visually inspected for sootblower erosion and for missing or damaged tube shields. Special attention should be given to the lower bends on the low temp pendant section.



2-17

C. There was a previous tube failure on the West Side on the top of the superheat horizontal section. Tubes number 2 and 3 from the West Side wall should be replaced.

• Visually inspect the top of the horizontal superheat section. Check for any tubes that have broken loose from the hanger tubes and check the bends on each end for any bend that may be displaced and rubbing on an adjacent tube row. Also check the condition of the refractory and flyash screens located at the rear bends of the horizontal superheat. Inspect the rear bends for any indication of flyash erosion.

• Inspect the area between the two horizontal superheat sections. Check for any tubes that have broken loose from the hanger tubes. Visually inspect the spacer straps located on the front hanger tubes. Check for any broken or missing U clips and to ensure that the straps are still raised up off of the horizontal tubes in the center half of the boiler. The spacer straps were raised because they were deflecting the sootblower steam down against the horizontal tubes.

• Special attention should be given to inspecting the sootblower lanes for retracts 10 and 39. This area has been a very high wear area.

A. Check the condition of the tube shields located on the lower side of the upper horizontal section. The condition of these tubes on the lower side of the upper horizontal section should be checked at least 5 tubes deep up into the bundle.

B. Check the condition of the deflectors installed on the hanger tubes. The horizontal tubes should be checked for sootblower erosion where they are attached to the hanger tube.

C. Visually inspect the condition of the pad welding on the top side of the lower horizontal section. A UT inspection of these tubes should be performed on any tubes that are not covered with tube shields.

• Visually inspect the area on the topside of the economizer. Check for any broken economizer hangers or bent tubes. Perform an UT inspection on the topside of the bends located at the superheat horizontal inlet header. These tubes have shown signs of flyash erosion on the outer radius of the bends.

• Inspect the area located between the economizer sections. Check for any broken hangers or bent tubes. Check the condition of the offset bends located along the superheat backpass front inlet header located along the front wall. These offset bends have been subjected to some flyash erosion.

• Check the condition of the refractory dam at the furnace screen tubes. Check for any broken or missing pieces.

• When performing pendant inspections special attention should be given to the first five pendants from each side wall. Historically the first five pendants have suffered more severe sootblower erosion.


2-18

5. External Boiler Team Member

Will perform inspections and coordinate the repairs external to the boiler. The following inspections will be performed.

5.1 Prior to the outage (within 2 weeks of the start of the outage)

Conduct an external walk down of the boiler from 21st to 2 1/2 floor.

The items to check shall include:

• Condition of the buckstays. Checking for any binding or bent buckstays. Look for any missing or loose pins in the connector links for the boiler trusses.

• Check for any external steam leaks at the sootblowers, sootblower piping and boiler vent and drain valves.

• Note any lagging or flashing that may require repair.

Check the Maximo system for workorders for any boiler vent and drain valves that may require a code repair. Review all boiler valves that are going to be replaced to determine if the repairs should be code repairs.


• A visual inspection should be performed in the front and rear lower dead air spaces. The following items should be checked.

A. Inspect all of the hanger rods checking for broken or missing pins. Also check for cracking on the hanger rod lugs.

B. Check for any missing or cracked tube clips on the horizontal buckstays.

C. Perform a dye penetrant inspection where the side wall capture angle irons are welded to the back side of the slope tubes.

D. Visually inspect the scallop bars located at the bottom of the side walls. Check for any cracking where the scallop bar was repaired and for any cracking were the scallop bar is welded to the tube.

E. Check for any cracking in the tube membrane located towards the inside of the side wall scallop bars.

F. In the rear dead air space, check the condition of the centerwall tube nipple that was cut off and plugged.

G. Check for any bent or cracked buckstays and structural steel.

H. Visually inspect for any tube membrane cracking at the throat bends and where the slope is welded to the side walls.



2-19

• Visually inspect both upper dead air spaces. Check for any membrane or skin casing leaks and check the condition of the structural steel.

• After the boiler has cooled, walk down the entire external boiler checking for any bent or warped buckstays.

• Inspect the upper boiler drum, check the following items.

A. The turbo separators, checking for alignment, loose hold down clamps and the condition of the chevrons.

B. Missing or loose screens at the downcommer nozzles.

C. Loose or missing feed pipe hold down clamps.

D. The condition of the secondary separators.

E. Plugged lines for the boiler level devices.

• Inspect the lower boiler drums, check the following items.

A. Check for any missing orifices or orifice clamps.

B. Check the condition of the screens. Look for any loose or missing screen bolts.

C. Visually inspect the drum crossover links.

• Remove the hand hole covers on the superheat desuperheaters and use the borescope to check the condition of the liner, set screws and the nozzle.

• Visually inspect all of the boiler access and inspection doors recording any that require refractory or other repairs.

• Perform an internal inspection of the penthouse checking for the following items.

A. Check the condition of all the hanger rods. Check for broken rods or missing pins.

B. Look for any tubes that exhibit signs of overheating. The overheated tubes will have a dark discoloration and may have some scale exfoliation.

C. Look for any cracks and flyash leaks in the skin casing. Also check for any flyash leaks at the crown seals. Special attention should be given the skin casing at the center of the boiler. The boiler has a history of skin casing cracks along the center of the boiler

D. Check the condition of the insulation at the superheater outlet header and links.

E. Visually inspect all of the trapeze hangers located on the connector links.


2-20

F. Visually examine all of the headers. Look for any signs of cracking at the nipples or ligaments.(Suggest you use some NDE method than just looking – dye penetrant)

Perform and inspection on top of the penthouse. Check the condition of the upper hanger rod attachment looking for any missing pins or cracked lugs.


3-1

3 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1 AT DETROIT EDISON’S ST CLAIR PLANT – UNIT 7 AMERICAN ELECTRIC POWER’S BIG SANDY PLANT – UNIT 2

Executive Summary

In an effort to improve boiler reliability at the Detroit Edison St. Clair Plant, EPRI was contracted to identify technical and programmatic opportunities for improving Unit 7 boiler reliability. The scope of work was limited to a portion of Phase 1 of EPRI’s Boiler Reliability Optimization project, namely boiler tubes and headers. The study was requested by the then Production Manager – Phil Thigpen from St Clair Plant. A team from EPRI spent two weeks on site reviewing plant record, boiler inspection plans, procedures etc and conducted interviews with key operations, maintenance and engineering staff. A list of those interviewed can be found in appendix 1.

The report focuses on two areas - the programmatic and data collection and tube failures and their causes and highlights a number of opportunities for improvement with respect to managing the short and long term health of the boiler.

From the review of the data/information obtained and the interviews revealed that the current maintenance strategy, of a periodic every three years, is in line with other utilities however the duration is out of step with best practice in the industry. Utilities with comparable units have 4 to 5 week outages with a Forced Outage Rate of between 4 and 6 percent. . Unit # 7 has a comprehensive boiler tube inspection plan. Tube leaks, their causes and action plans to prevent recurrence are recorded and well managed as evidenced by the reduction in the number of tube leaks over the past five years. The inspection plan is reactive as it only includes known boiler tube problem areas. Inspection of the headers is limited to the superheater and reheater outlet header. This is of concern as there is little to no information is available about the condition of the remaining headers within the boiler.

The recent down sizing and re-organization has resulted in some uncertainty regarding the roles and responsibilities of the newly formed service groups. Key staff have been moved to perform other project activities, thus losing the advantage gained from past inspections. Organizational measures and measurements were also not well understood.

To improve and sustain a high boiler reliability/availability and to minimize the future risk of down time and subsequent production loss it is recommended that Detroit Edison – St Clair Plant

EPRI Licensed Material Boiler Reliability Optimization Project Phase 1 at Detroit Edison’s St Clair Plant – Unit 7 American Electric Power’s Big Sandy Plant – Unit 2

3-2

• Invest into a comprehensive Boiler Reliability Optimization program to develop a detailed Boiler Failure Defense Plan. This plan would assist in the development of a specific Boiler Test and Inspection Plan that would cover all the major components within the boiler envelop. This plan can then be integrated into an expanded – 15 years plus Long term Reliability Plan. This can be accomplished through EPRI’s Boiler Reliability Optimization Program

• Review the basis of their current outage management philosophy and approach. The current nine week outage on unit # 7 is out of step with other USA and international utilities even with the current turbine problems being experienced. Huge operating and maintenance cost saving and increased revenue can be made by reducing outage times by bring them inline with best practice.

• Implement the recommendations identified in the various section of this report.

Participating in Phase II of EPRI’s Boiler Reliability Optimization Program would not only enhance the efforts being put into the program but will give Senior Management the assurance that the goals being set will be met. Membership details and costs related to a Phase II can be obtained from the author.

Armed with the information in this report and available resources, St Clair plant can become the “Best Practice “ plant within the Detriot Edison organization for which it has been striving to become. Given time, management commitment and coaching in EPRI’s Boiler Reliability Optimization Program, the plant team has the vision and capability, as evidence in the Boiler Tube Inspection Reports, to successfully lead the plant into full implementation of a Boiler Reliability Optimization Project.


Boiler Reliability Optimization Project Phase 1 at Detroit Edison’s St Clair Plant – Unit 7 American Electric Power’s Big Sandy Plant – Unit 2

3-3

3.1 Introduction

Detriot Edison St Clair Unit 7 had a history of tube failures causing considerable down time with corresponding loss of production. Phil Thigpen, the St. Clair Production Manager, requested that EPRI be contracted to review their boiler technical plans and identify opportunities for improvement. The scope of work was limited to the high pressure, high temperature boiler components – tubes and headers.

Four EPRI team members - Dave McGhee, Pat Abbott, Jack MacElroy, and Jim Cossey spent varying amounts of time on site studying documentation and conducting interviews. Interviews were conducted with both Plant and Support staff responsible for activities on the boiler. These interviews were conducted during the weeks of May 15 and August 29. 2000. The names of those interviewed are listed in the appendix 1. All relevant and available data and information on Boiler 7 was reviewed. In addition, data and information relating to the overall management of the boiler, from an operating, maintenance and engineering point of view was reviewed. This enable the team to gain an insight into the overall approaches taken to manage the technical aspects of the boiler.

3.2 Findings and Observations

Based on the scope of work, the following represents the findings and observations in the two areas, viz. programmatic/data sources, and failures and root causes.

3.2.1 Programmatic and Data Sources Assessment

A maintenance process assessment was conducted as part of the work performed in this project. The purpose is to understand how the management processes and technologies used in the plant affect or influence boiler operation and maintenance and therefore the long-term health. Data was collected and evaluated and benchmarked against known best practices. The assessment team spent a week on site reviewing this data and interviewing key site and support staff. From the data analyzed, together with the information obtained from the interviews a “spider” chart was drawn.


3-4

Score

0

2

4

6

8Maintenance Strategy

Work Identification

Work Control

Work CloseoutDocumentation

Project Management

Change Control

Operating Practices

Boiler Inspection PlanTechnical & Action Plans

Performance Monitoring

Roles & Responsibilities

Communication

Metrics

Technical ResourceDevelopment

Skills/Experience Profile

Utilization

Score

World Class

Figure 3-1 Spider Chart



3-5

Attribute Key Findings

Boiler Maintenance Strategy Conscience change from 2 to 3 years

Inspection and repair plan is aligned with outage frequency

Work Process 50% of the backlog are preventive maintenance activities.

“as found” and “as left” conditions were not being documented

Project/Outage Management Capital projects are detailed and well planned

Performance/Reliability of a unit returning from a periodic outage does not meet expectations i.e. not optimal

Unreliability and unavailability occurrences are not investigated to determine their cause and root cause.

Operating Practices Critical boiler metal temperatures are not monitored or trended

Metal temperature excursions are not analyzed or recorded

No objective evidence to indicate that boiler start-up or shut down procedures are followed

Sootblowing procedure and criteria are not being adhered to.

Boiler Test and Inspection Plan Well documented boiler tube inspection and repair plan

No detailed condition assessment of all boiler headers – only superheater and reheater outlet are inspected

Metrics and targets Overall plant performance targets exists

Boiler EFOR is not measured

Boiler tube leak target is place

Roles and responsibilities Many of the staff interviewed were unsure of their roles and responsibilities and that of the support group. This was illustrated by the performance of unit 4 after its periodic outage

Performance Monitoring Superheater metal temperature excursions are not investigated or recorded

No objective evidence that excess oxygen is controlled or monitored

Skills/experience profile Experienced people are being lost due down sizing and retirements. This affects the transfer of technical “know how” to new incumbents - operators


3-6

3.2.1.1 Maintenance Strategy

The starting point for developing a maintenance strategy is to first determine a maintenance objective. An objective should be quantified in terms of availability and performance. These requirements should be determined jointly between the Maintenance and Operating and or the Production Departments. Maintenance Management is then in a position to determine the best mix of maintenance techniques to use, resources requirements and the costs thereof.

A Maintenance strategy is then developed from these agreed objectives. It defines the process for identifying and achieving sound operational objectives, selecting the maintenance approaches, monitoring performance and providing effective management control. The strategy therefore consists of a management and technical strategies.

The Management strategy defines how the business management skills are used to integrate people, policies, equipment and practices to identify improvement opportunities.

The Technical strategy states the what, who and how the technical knowledge and experience are used to identify and implement the best proactive maintenance, repair, service and replacement of all equipment in line with the Plant’s business performance objectives. The technical strategy therefore consists of a combination of corrective, preventive, predictive and proactive maintenance activities where:

Corrective – “Fix it when it breaks”

Preventive – “Fix it before it breaks”

Predictive – “Fix it when it starts to break”

Proactive – “Fix it before it starts to fail”

The maintenance strategy is built primarily from the preventive and predictive maintenance basis that exists at the plant.

3.2.1.1.1 Findings

• The Preventive Maintenance (PM) base has not been updated for some years. Preventive maintenance is not being done judged on the age and extent of PM in the backlog.

• Random outage rate (ROR) – defined “as the sum of all forced outages and de-rates and all unforeseen outages and de-rates divided by the available generation excluding periodic outages and de-rates and expressed as a percentage” is the availability/reliability measure used.

• Nearly all the staff interviewed could not define ROR nor did they know what the target was for St Clair.

• The planned outage frequency has been increased from two years to three years. The next periodic is scheduled for Spring 2001



3-7

• During the 1998 periodic an extensive inspection and repairs were carried out on the boiler tubes. Since the return to service to August 2000, only two boiler tube leaks outages have occurred. Thus meeting management’s expectations/target, of one tube leak per unit per year. This achievement illustrates the effort that has been put in by the boiler engineer to reduce the risk of tube leaks.

• Boiler tube inspections are detailed and thoroughly done, the data and information obtained from these inspections and subsequent repair work is stored on the Boiler Engineer’s computer.

• Ownership of the boiler preventive and predictive maintenance basis is unclear.

• EPRI’s Boiler Maintenance Workstation (BMW) has been purchased however implementation has been restricted because of the lack of allocated financial and human resources.

3.2.1.1.2 Recommendations

• Clearly define and allocate the roles and responsibilities of the various plant system engineers.

• Clearly define plant performance measures and targets to all staff. These should be tracked, trended and displayed where everyone can see them.

• Review and update the PM basis on the boiler and its auxiliaries. This can be achieved through the use of a reliability centered maintenance package. The aim is to develop an optimized set of PM tasks that will ensure the plant will achieve its reliability targets.

• Continue with the detailed boiler tube inspection program and develop a common database, accessible to other key people, that enables trending of the data and information and predictions to be made.

• Allocate resources to complete the implementation of the BMW throughout all Detroit Edison plants

3.2.1.2 Work Process

A typical work process consists of four separate, but equally important steps.

• Work Identification - Work is identified as corrective, preventive, predictive, or pro-active.

• Work Control - Establishing a balance between the four types of identified maintenance which maximizes the use of manpower while ensuring that the proper maintenance is performed on equipment at the appropriate time.

• Work Execution - Work is planned and scheduled. Parts and tools are staged and the proper level of manpower is dedicated to the job.

• Work Closeout – Recording of “as found” and “as left” information on all work orders. This information is maintained in an electronic database and used to plan future maintenance activities.


3-8

3.2.1.2.1 Findings

• Periodic outage duration is between 7 and 9 weeks. Boiler inspection is an on going activity that can take up to three weeks to complete while boiler cleaning and scaffolding is accomplished during the first week of the outage..

• Boiler tube repairs made during planned periodic outages are well documented albeit in an independent system to the CMMS. The inspection report and repair records are well written, detailed, and complete. The repair record identifies the location of all repairs and the type of repair made on the tubes, as well as the suspected failure mechanism.

• All boiler tube repairs during a periodic are prioritized using three levels, #1 being the highest priority.

• Numerous priority # 2 and # 3 are not completed because of lack of time and resources.

• Quality control plans for boiler tube repairs are used and checked by the boiler engineer.

• Recommendations made by the Boiler Engineer as a result of a periodic outage inspection are not acted on until a few months before the next periodic outage. With the current outage frequency, this could be a three-year gap. The responsibility for acting on these recommendations is not clearly understood or defined.

• There is no backlog of planned boiler work – preventive, predictive etc to be worked during forced outages, nor is there a team dedicated to boiler inspections during forced outages.

• Boiler performance data does not play a role in planning boiler outage work. The large amount of data generated by the DCS system or stored in the PI system to assist in directing or planning boiler work is not being used. Parameters such as tube metal temperatures, temperature changes across reheat and superheat sections, furnace gas temperatures, heat rate, and changes in gas pressure and flows can be used to direct inspection and repair activities.

• A long-term Reliability Management plan has been developed. The plan establishes five year ROR, LOP, and generation targets for the St. Clair Power Plant. The plan details work to be done at the next periodic outage, and there are no specific plans for the high pressure sections which are approaching or exceeded the expected 100,000 hours life expectancy.

• Although net heat rate is mentioned in the long-term Reliability Plan, there are no goals established for heat rate improvement, or heat rate improvement projects listed


• Conduct a technical review of outage management. This review should be and based on sound project management principles and include a technical review of the scope of work from each discipline. Many USA and International utilities are achieving 4 to 5 week outages every three years. These plants are similar to St Clair Unit # 7 in size and capacity burning Powder River Basin Coal. Shortening a 9 week outage by two weeks would have huge impacts on outage costs and revenue.

• The boiler inspection plan should be integrated into an all-inclusive Predictive Maintenance (PdM) plan. A PdM program consists of a series of activities that are performed prior to breakdown in order to forestall the immediate occurrence of impending failures. The present



3-9

boiler inspection program focuses on identifying and correcting existing problems. Application of PdM methods for boiler inspections and inclusion of findings into an integrated plant PdM database would enhance the scheduling and planning of boiler maintenance work and contribute to reduced ROR goals. The inspection report and repair records prepared by the boiler engineer are key indicators of the boilers condition. These indicators should lay the groundwork of both long- and short-term planning for the boiler maintenance program and improvement projects.

• Responsibility for compiling work orders, and following through on recommendations made as a result of the boiler inspections, should be assigned and acted on in a timely manner. The reports should be made available in electronic form to all Detroit Edison employees involved with the boiler maintenance program.

• Advantage should be taken of forced outages to inspect available areas of the boiler. “Trigger ready” forced outage plans - work orders/work packages need to be developed for inspection of identified problem areas in the boiler immediately following both periodic and forced outages. Thus, in the event of a forced outage advantage can be taken to perform boiler inspection and or repair work.

• The present long-term plan should be expanded to a minimum of ten years and a maximum of twenty years. Mile stone years should be based on the outage frequency i.e. the plan should have significant detail for years one, three, six etc. The steps to consider in formulating this long-term plan are:

• History

1. Design review

2. Construction review

3. Operating and maintenance history review

4. Inspection reports

5. Industry experiences

• Evaluation of Present Condition

1. Metallurgical evaluations

2. Visual inspections

3. Non-destructive examination (NDE) prioritization

4. Destructive examination prioritization

5. Operating conditions and practices

6. Maintenance strategy

7. Database compilation

• Plan Formulation

1. Remaining life assessment

2. Long-term inspection plan

3. Risk assessment


3-10

4. Short and long-term recommendations

• Plan Re-evaluations

1. Review and updating of inspection plan

2. Reviews of new NDE technology

3. Inspection planning and scheduling

4. Risk assessment re-evaluation

5. Confirmation of remaining life assessment

6. Updating of plan database

• Boiler performance and process data (metal temperatures, excess oxygen, steam and water temperatures and pressures and air and gas temperatures) is available via the DCS and PI system and should be tracked and trended. Deviations from specified should be investigated analyzed and action taken to prevent recurrence.

• Enforce the use of the CMMS historian. No Work Order should be closed out until the craftsman who has done the work fills out this section of the Work Order. Every Work Order should be reviewed for repeat failures, and a proper action plan developed. This is a key factor in the continuous improvement cycle.

3.2.1.3 Predictive Maintenance Program and Technology

An optimization boiler maintenance program is made up of people, management systems, component owners, technology specialists and predictive maintenance tools or equipment. The key predictive maintenance tools used in a Boiler Reliability Optimization project are infrared thermographs, vibration, oil analysis, ultra-sonic etc. Equipment owners and technology specialists collect and collate data on various boiler components. This data is organized and presented in an Equipment and Condition Indicator (E&CI) Matrix. The matrix specifies the predictive maintenance technology to be used, the equipment to be monitored and when the data should be collected, threshold/action limits/levels and actions to be taken. An example of such E&CI matrix can be found in Appendix 2

The “how”, the “what”, and the “when” of condition data analysis is based on exceeding pre-established threshold levels and this information is presented in an “Event Report” and in an “Asset Condition Status Report”. Direct causes of deviations from threshold are determined and corrective actions are communicated to the relevant people. Threshold levels are based on industry, design criteria and baseline data.

3.2.1.3.1 Findings

• During periodic outage, visual inspections and ultra sonic measurement methods are used extensively by the boiler engineer. The focus is on the boiler tubes.

• Infrared thermography is used on electrical equipment, the boiler casing, and identification of leaking valves.

• Metallurgical analysis of failed boiler tubes is performed on a limited basis.



3-11

• An oil analysis program on the fans and pulverizes is in place.

• The unit water chemistry program is well controlled and the units operated within EPRI guidelines. The unit has not been chemically cleaned for nine years, and there have been no boiler tube failures due to waterside problems.

• Report 97E08-81, dated 10/20/97, gives the results of deposit analysis on three Unit 7 waterwall tubes. The north wall sample had the highest grams/ft2 and 19.15% copper, as Cu. The two other tube samples showed significantly less grams/ft2 and copper values

• Boiler efficiency/performance tests are no longer performed.


• The boiler inspection report, tube leak reports, DCS data, and PdM data need to be integrated into a single database. The key to utilizing the condition-based information in an effective manner is the ability to quickly access the information and to trend the data over time to enable a prediction of when work should be performed. EPRI’s Boiler Maintenance Workstation (BMW) is one such system, which can be used to track and trend boiler tube failure data and information. Another useful tool is the WEB based Plant view software.

• Remove additional tube samples from areas where high copper had been detected (see Report 97E08-81 dated 10/20/97) and check and confirm the presence of copper.

• Use the data collected by the DCS and PI proactively eg trend and track metal temperatures, excess oxygen etc, in order to identify incipient failures.

3.2.1.4 People

Human beings are an important asset. Their reliability – “making and keeping commitments” has a huge impact on plant reliability. The design, manufacture, installation, operation and maintenance of all equipment is in some way or other related to the performance of human beings. Therefore, “if human beings are reliable your equipment will be reliable”.

3.2.1.4.1 Findings

• The staff at St. Clair has recently undergone reorganization, with a reduction in staff numbers. The present organization is highly matrixed, and the roles and responsibilities of individuals are not clearly understood. Communication within the matrix is poor, with many of the plant staff reporting they are out of the loop regarding decisions on the boiler. As an example, there is no clear understanding, of who is responsible for following up on recommendations made by the boiler engineer in the boiler inspection report. Is it the boiler engineer or maintenance supervisor? At the time of writing this report, the current boiler engineer had been assigned other duties which would take him away from doing thorough inspections.

• There is no succession planning for plant staff. Technical expertise and experience is concentrated among a small number of staff within AMO and ESO. Plant staff are therefore dependent on these organizations for support and boiler expertise.


3-12

• An extensive on-job-training program for new operators has recently been put in place.


• A ‘Boiler Tube Failure Reduction’ team consisting of plant, ESO, and AMO staff should be formed. Every tube failure should be investigated and an in-depth report issued. This report should include a ‘Root Cause Analysis’, short and long-term recommendations, and a copy of Work Orders written by the team. Semi-annual progress reviews should be issued giving the status of action being taken on recommendations.

• Management needs to clearly define and communicate roles and responsibilities of the Plant, AMO, and ESO staff, and the relationship of individuals within these groups.

• With the current age profile of the plant work force, it is essential to develop and put in place a succession plan that will provide qualified people.

3.2.2 Failures and Root Causes

The section deals with the causes of failures, actions taken and the actions to be taken to mitigate future failure.

3.2.2.1 Boiler Tubes

From 1995 to April 2000, on Unit 7 there have been 21 forced outages due to boiler tube failures. Analysis of these reports shows 54 tube failures, involving eight failure mechanisms. See Figure 3-2 below.

The documenting of tube leaks is done via the “Short Form” report. The report also includes details of the other repairs due to the consequence of the first failure.



3-13

0

2

4

6

8

1 0

1 2

1 4

1 6

A tta c hm e ntsC link e r D a m a g eF a tig ue C ra c k ingO xid a tio n o f R H T ub e sQ ua li ty C o ntro l/W e ld sS la g E ro s io nS o o t B lo w e r E ro s io nW a te r L a nc e Q ue nc hing

A tta c hm e nts 2 1 1

C link e r D a m a g e 1 4 2

F a tig ueC ra c k ing

1

O xid a tio n o f R HT ub e s

1 1 5

Q ua li tyC o ntro l/W e ld s

3 3 2

S la g E ro s io n 2

S o o t B lo w e rE ro s io n

5 1 1

W a te r L a nc eQ ue nc hing

4 4 1

1 9 9 5 1 9 9 6 1 9 9 7 1 9 9 8 1 9 9 9 2 0 0 0

Figure 3-2 Tube Leak Failures 1995 - 2000

Analysis of Figure 3-2 reveals the following trends:

• Failures due to water lance quenching peaked in 1995 and 1997, with four tube failures in each year. Since replacement of the water lances, there has only been one incident, which was due to damage prior to replacement. This pro-active approach has been successful in eliminating this problem.

• Failures due to soot blower erosion peaked in 1995 with four failures, and only one reported in 1998 and 1999. This is due to extensive pad welding done during the 1998 outage. This


3-14

approach seems to have solved this problem for the short-term, but the large number of pad welds threatens the unit’s future reliability.

• Failures due to clinker damage were the highest in 1997 with four failures, followed by two reports in 1998. Installation of reinforced wear bars during the 1998 outage to date has eliminated this mechanism.

• Failures caused by the lack of adequate welding quality control has increased. Of the eight tube failures reported in 1999 and 2000, five, or 63%, were due to poor workmanship or using the wrong material. There were also three incidents in 1997. These failures peak following periodic outages.

• Analysis of a failed reheat tubes in 1997, showed the cause of the failure to be excessive external oxidation. Inspections performed during the 1998 periodic outage revealed additional fifteen tubes with excessive wall thinning due to excessive oxidation. Although there was only one forced outage due to this mechanism, it emphasizes the importance of monitoring, trending, and controlling tube metal temperatures.

3.2.2.1.1 Failure Mechanisms, Actions taken and Recommendations

The 1998 Boiler Inspection and Action Plan was reviewed and the following table represents the Findings and Recommendations. The bold text represents the additional recommendations as a result of the review process.



3-15

Table 3-1 Boiler tube Failures, Causes and Recommended Action

Attachment Failures

Findings Cause/effect Action taken/Recommendations

In 1997 two seal trough failures occurred

The cause of the stress cracks was not determined.

During 1998 the boiler seal system was replaced with an upgraded design to minimize the design.

In February 2001 an inspection revealed several rigging/support lugs welded onto the waterwall and back pass/economizer.

In many instances these lugs if not carefully removed can cause future tube leaks

Welding lugs on pressure parts – waterwall, economizer etc is not a good practice and should be discouraged.

Burned or eroded superheater alignment clips are rlaced and or repaired.

Continual welding as in repair/replace can result in damage to the external and internal tube surface of the tube, resulting in a tube leak

”Hancuffs” should be used after carefully removing broken or damaged clips.

In September 1998 and April 1999 two failures occurred in the horizontal portion of the steam cooled spacer tube attachments. This failure mechanism was first noticed in 1994 and 1996 periodic outages

The cause, as recorded in the “Short Form Report” is erosion which lead to thinning of the material and increased mechanical stress.

During the 2001 periodic outage replace the water-cooled spacers utilizing the new PIB-74 attachment system as designed by ABB-CE

In future, when an attachment failure occurs - tube samples should be removed to check for possible corrosion fatigue damage.

Clinker Damage/Slag erosion


Six failures have occurred, one in 1996, four in 1997 and two in 1998.

Localized slagging – three to four feet from each end has been the major cause.

Investigate the cause of this localized slagging.

During 1998 periodic outage, all tubes on the lower slope have been replaced with a wear barring tube – extra material on top of the tube.

Continue to inspect these tube as and when opportunities arise - during forced and planned outages, to determine a wear rate from which predictions can be made


3-16

Fatigue/Stress Cracking


Nine fatigue/stress failures have occurred. Four in both 1995 and 1997 and one in 1999.

These failures were all caused water quenching from defective/deficient water lances.

These water lances have been replaced with an upgraded design

In February 1997 a superheater rear wall hanger tube

Loose/broken hanger resulting in the tube vibrating loose and crack caused this failure.

Compile a hangar inspection PM with all the necessary data/ information to be recorded so it could easily be issued during a forced outage. This same PM would also be issued during a periodic outage.

Long Term Overheating/Creep


In 1997 and 1998 there were one and fifteen failure on the reheater respectively. Analysis of tube samples by ESO (Report # 99V008-0001) indicated tube failures were due to thinning by oxidation. The protective oxide coating had spalled off because of mechanical stress causing thinning,

Two causes were given – high temperature and mechanical stress because of hangar support failures. A hanger support pin was missing.

The action taken during the 1998 periodic outage consisted of repairing the hangers and insert 60 “Dutchman”

Regarding recommendation for hanger inspection – see recommendation under fatigue/stress cracking.

Use UT to measure oxide thickness across the superheater, correlate with temperature profile – thermocouples and remove tube samples metallurgical analysis and conformation of oxide thickness measurement

The ESO report failed to identify or mention if any short or long term overheating had taken place. There was no mention of the metallurgical aspects/condition of the tube. The report states “thermocouple data does higher temperature in the center” however no conclusions or recommendations are made regarding the high temperatures.

The ESO report therefore did not provide adequate information/guidance on long term heath/life aspects of the reheater tubing.

All analysis reports of tube samples should give recommendation regarding future inspection strategies and or examination requirements.

Remove during the Spring 2001 a representative sample of tubes from across the reheater for remaining life assessment.



3-17

During start-up and under normal operation little attention is given to adhering to metal temperature limits and or recording the extent and duration of excursions.

Creep damage is dependent on stress level and tube metal temperature. Thus conditions. Therefore conditions that deviate from design which lead to increase stress and temperature will result in reduced tube life.

The importance of operating within design parameters must be impressed upon operating staff.

Metal temperatures should be continuously recorded and monitored by the operator. Rates of change and maximum limits should be determined and adhered to. Deviations from expected should be recorded and investigated. Actions to prevent recurrence should be reviewed and followed up to ensure compliance.

Quality Control


Since 1997 a number of pad welds and or welding have failed

An inspection and or investigation revealed that the welders failed to comply with the welding norms and standards. Quality control of these welds was also inadequate.

Pad welding should be discouraged not only because it destroys the evidence but it also can create the potential for other mechanisms to occur – hydrogen damage. However if it is used, criteria should given to when and when not to use pad welding.

Welding procedures should be compile with and all welding should be inspected – quality control

Soot Blower Erosion


Since 1995 several tube failures have occurred because of soot-blower erosion. Forty nine tube were repaired during the 1998 periodic outage

Soot-blower erosion is controllable. The causes are related to excessive use – operational and improper maintenance.

Continue with the soot blower replacement project – new helix design.

Optimize the soot blowing sequence and frequency by using thermal condition in the boiler to determine when and how long to blow.

Waterwall Fireside Corrosion


During the 2001 periodic outage low Nox burners will be fitted

Apart from the environmental benefits low Nox burners can cause substoichiometric conditions on the side and rear wall near the burners. These conditions nearly almost result localized corrosion taking place.

Obtain a baseline by measuring tube thickness local to the burners - near the bottom burner level to about 10 feet above the top burners and the side walls midway between the front and rear walls at the same height of the burners.

Ensure that the newly installed burners are correctly optimized – measure CO and Oxygen near tube walls if possible. High levels of CO (>1%) and low levels of oxygen (<0.1%) near the walls would be of concern.


3-18

3.2.2.2 Boiler Headers

Detroit Edison has a high-pressure piping and header inspection program in place for St Clair Unit # 7. The plan reviewed did not give any details beyond the next outage. The program includes the final superheater and reheater outlet headers and excludes all other boiler headers. The condition of all remaining headers is unknown even though the unit has operated for well over 100,000 hours. It is recommended that a Level II Remnant Life Assessment be undertaken. See Appendix 3 for a typical flow chart. The findings from Level II may lead to further examination – Level III. From these assessments a Boiler Test and Inspection Plan should be developed. The plan should cover at least 15 years. The various planning windows would be dependant amongst other things on the particular maintenance strategy adopted – outage frequency, duration, extent of inspections etc. Table 2-2 shows the typical failure modes and failure mechanisms found in steam drums and headers that the inspections plan should cover.

Table 3-2 Typical Header and Steam Drum Problem Areas

Header or Drum Failure mode Failure Mechanism

Final Superheater and Reheater outllet

Internal Cracking

Swelling

Local bulging

Creep-Fatigue

High Temperature Creep

Creep or Thermal expansion

Reheater inlet Sagging Support problem and or over temperature

Final Superheater inlet Internal Cracking Thermal Fatigue

All headers External laminations

External Cracking

Tube nipple Cracking

Fabrication Defects

Thermal Shock or expansion

Thermal Expansion Fatigue

Drums Internal surface pitting

Ligament and bore hole cracking

External nozzle weld cracking

Oxygen control in feedwater

Thermal and corrosion Fatigue

Thermal expansion Fatigue

Economizer Inlet Internal Cracking Thermal and Corrosion Fatigue

Below is a brief guideline of a typical Boiler Test and Inspection Plan



3-19

3.2.2.2.1 Boiler Test and Inspection Plan Guideline

An effective inspection and test plan based on design review, historical operational data and selective physical examination allows accurate assessment of plant condition and future life cycle management.

Inspection of critical items of plant on design reviews together with component reviews of results and historical data will enable effective planning of repairs/ replacement programs to be formulated with accurate budgets. His will enhance plant availability and safe operation.

Purpose

To provide guidance in the preparation of inspection programs aimed at providing a comprehensive plant condition database, allowing accurate assessment of current plant condition and predicting future replacement and refurbishment requirements.

Definitions

• Major Component - This is a component with an internal diameter exceeding 4 inches. It includes headers, attemperators, internal boiler pipework etc. Heat exchanger tubing is excluded.

• Component – A pressure containment operating in the creep range or subject to fatigue together with its weldments comprising the weld metal and the heat affected zones.

• Sub Components – Are those components when welded together form a major component.

• Component Inspection Plan – This is a detailed listing for each generating unit giving a specific schedule of components (major and sub components) to be inspected, the type and extent of the inspection.

• Inspection Report – Inspection reports are generated following the physical inspection based on the component inspection plan.

• Plant Condition Review – This is a rigorous review of relevant inputs that will indicate the requirements for inspection. These include operating (including temperature excursions and major events) and maintenance history and previous condition assessment reports.

• Critical Component – A component that has been identified by calculation or inspection as having a limited life and is expected to suffer the greatest degree of life exhaustion. Calculated values should be based on actual dimensions where possible.

Scope

The scope of the plan should cover all components (major components, components and sub components) within the boiler envelope. The intent of the plan should be that by the time the plant has operated for 50% of the nominal design life, at least 50% of the components have been physically inspected and their condition known and documented.


3-20

Inspection and Test Plan Development

• Compile a list of uniquely identifiable Test and Damage code.

Inspection and Test Plan Codes

Test Codes Damage Codes

1. Visual examination A. No Damage/ defects

2. MT B. Defects removed < 0.1” deep

3. UT C. Defects removed < 0.2” deep

4. ----- D. -------

• Using the manufacturer flow diagrams and isometric sketches compile a list and uniquely

identify each major component, components, sub components (tee pieces, spool pieces, end caps etc) and welds (girth, nozzle, seam etc).

• Using the information from 1 and 2 above, compile a master inspection plan – spreadsheet. Use the vertical axis (rows) to list the components (major, component and sub components) on the horizontal axis (columns) list the following information.

1. Operating parameters

2. Date of last inspection

3. Frequency of test

4. What damage/indication to identify

5. Responsible person/ system owner

6. Reference drawings

7. Dimensions

8. Material specifications

9. Test procedure

10. Database reference

11. Comment/remarks

• Conduct a plant condition review and identify the critical components.

• Using the master plan compile a component specific inspection plan. This plan should be linked to a CMMS work-scheduling package. This will allow the test and inspection plan to be included into a short or periodic outage.

• Conduct the inspection and record all the relevant data.

• Assess the component specific data to determine the action to be taken – operate at design operating temperature or below, refurbish, repair or replace.

• Document and record actions and decisions.



3-21

3.3 Conclusions

The down sizing and re-organization has had an impact on the effectiveness of various management processes on site as staff are getting to grips with their new roles and responsibilities. The work flow process from work order initiation to work close out has been affected. Planning and scheduling of short and long term work has been most affected.

The current outage on unit # 7 of 9 weeks is long in comparison to other utilities. Many USA and International utilities have outages of between 4 and 5 weeks every 3 years. It is understood that the outage scope of work may well be totally different however these differences need to be analyzed and improvement opportunities determined.

The periodic outage boiler tube inspection plan and outage reports are well thought out, well organized, and well executed. An important key to achieving a three- year outage cycle is the continued excellence of this program. This information is stored on the boiler engineers computer even though a Boiler Maintenance Workstation exists.

The inspection and repair efforts during the last periodic have paid dividends – a reduction in the number of boiler tube failures due extensive repairs made during the last periodic however the number of forced outages due to tube leaks has remained constant.

The current boiler inspection plan although detailed and thorough is reactive in that it focuses on repairing the immediate problems identified during the inspection. Areas that are not problematic now are not inspected. These areas may well result in forced outages in the future. .

There is single central boiler tube failure database that integrates inspection results, root causes, recommendations, failure history and metallurgical analysis

Extensive pad welding is done in lieu of resolving the sootblower replacement and or modification/maintenance issues. Pad welding not only destroys evidence of a failure mechanism but can be the source of a potential failure.

The “Fossil Generation Long Term Reliability Plan” for St Clair only includes work to be undertaken up to the next outage. This is considered to be too short time frame to obtain the real benefits of long term planning.

3.4 Recommendations

Consideration should be given to having a Plant Maintenance Optimization Workshop. The workshop focuses on the broader issue surrounding maintenance. It helps identify opportunities for improvement in all facets of maintenance, where as this report really focussed on the maintenance issues affecting boiler maintenance. The two-day workshop is available to members of the Plant Maintenance Optimization Target.

Compile an action plan, with allocated responsibilities and completion date, to implement the recommendations made in the above sections of this report.


3-22

Review and benchmark the current outage management philosophy and approach with the intent to reduce the overall outage time frame in line with best practice.

Management needs to clearly communicate the roles and responsibilities of the newly formed service groups to all personnel involved. Expectations need to be clearly defined and individuals held accountable for their successes and failures.

Compile a root cause analysis process guideline to be complied with by all Detroit Edison plants. Train all staff in the basic concepts of root cause analysis and a number of engineers at each site on the details of applying root cause analysis to incident and problem resolution.

Assemble a Boiler Tube Failure Reduction team to investigate ALL boiler tube failures. Their responsibilities would be, determine root causes, recommend solutions, follow up on recommendations, issue annual reports on the boilers long term health, maintain an integrated database, and, put together and continuously update the long term plan for the boiler.

Complete the soot blower upgrade project, assign a higher priority to soot blower and water lance PMs, and analyze unit operating data to optimize the combustion process and decrease soot blowing frequency. These actions would greatly decrease the number of pad welds needed to counteract soot blower damage.



3-23

Appendix 1: List of interviewees

Melanie McCoy Plant Manager

Phil Thigpen Reliability Engineer

Bob Sausser Supervisor Plant Operations

Dan Kuhlman Supervisor Plant Operations

Dan Lavere Predictive Maintenance Coordinator

Tom Seaton General Foreman

Mike Mallon Boiler Reliability Engineer

John Shinske Unit #7 Foreman

John Renehan Unit # 7 Planner

Harry Mueler Unit # 7 Engineer

Bill Fielhauer Unit # 7 Engineer

Mike Bennet Water Chemistry Engineer

Bill Harvey Instrument Shop Foreman

Willie Myers Water Laboratory Technician

Fred Cutter Operations Coordinator

Dave Hurt Shift Supervisor

Jack Miketich Shift Supervisor

Bill Hornby Shift Supervisor

Barry Wagner Supervising Operator

Jeff Hat Supervising Operator

Charles Burmann Supervising Operator

Jim Jordan Supervising Operator

Ken Brinker Supervising Operator

Dave Stephan Training


3-24

Appendix 2: Example of a Typical Equipment Condition and Technology Matrix



3-25

Appendix 3: Remnant Life Assessment Flow Chart for Thick Section

Assemble design and service informationDesign: * Dimensions * Pressure

* Materials * Temperature* Minimum * Stresses

creep-rupture

Service: * Boiler running hours* Past repairs and replacements* Dimension and composition checks

Answer key questions

Has unit significantly exceeded design P and/or T?

Yes

Will future service involve P and/or T above original design?

Yes

Has failure history of boiler been excessive?

Yes

Steam temperature records not available?

Level I Assessment

Using design and basic operating data, calculate hoop stress, life fraction expended and remaining life

Is remaining life greater than the projected life extension?

Set inspection interval; maintain accurate operating records

Level II Assessment

Calculate remaining lifeRL = (1 – LFE)tR

No

NoYes

Level II Assessment

Yes

Yes

Level II Assessment: Thick Section Parts Level III Assessment: Thick Section Parts

Conduct detailed inspection by best NDE methods available. Measure dimensions of all crack-like defects.

Attach thermocouples; determine temperature distribution. Monitor temperature at hot spots to obtain representative sample.

Using actual temperature and pressure, calculate life fraction expended and remaining life



Level III Assessment

Perform transient and steady-state thermal analyses by finite element or finite difference methods.

Perform fatigue, creep and fracture mechanics analyses. Calculate remaining life to crack initiation and to fracture after crack growth.


Perform detailed stress analysis by finite element or finite difference methods.


Make repair or replace decision

Estimate costs and schedule

Input to plant and corporate plans

YesNo

No

YesNo

Conduct careful visual inspection. Are there cracks at ligaments or at penetration welds?

Level III Assessment


4-1

4 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1- ROOT CAUSE ANALYSIS AT ARIZONA PUBLIC SERVICES COMPANY’S FOUR CORNERS AND CHOLLA PLANTS

FOUR CORNERS PLANT

1999 Lost Generation Incident Review


In an increasingly competitive environment, the ability to reliably produce the lowest cost power is essential for the long-term survival of a utility. With this vision in mind Arizona Public Services Company (APSC) requested EPRI to review the 1999 Lost Generation Action Plans from the Four Corners Plant. The intent of this review was to provide assurance or not that the “true” root causes of lost generation have been identified, and appropriate action plans are in place to address these causes.

The review took place between 30 May to 9 June 2000 on site at the Four Corners Plant. Ten incidents, resulting in a loss of 728,978 Mwh were reviewed. Of the ten incidents reviewed only one was adequately documented - backing up the plant’s root cause analysis efforts. The review team, two from EPRI, one from Cholla Plant and one from Four Corners Plant, tried, through a series of interviews, to re-construct the incident. More than twenty staff members were interviewed.

Two methods, Behavior Justification and Event and Causal Factor Charting were used to re-construct the incidents. Diagrams/charts were constructed from information that depended, almost entirely, on the interviewees' long-term recollection of incidents. Verification of information was difficult, as there was a lack of formal incident-specific documentation and because of the time that has elapsed between the incidents and the analyses. In many incidences, perceptions and memory of what happened and in what sequence varied, emphasizing the importance of performing Root Cause Analysis in a timely manner, and then documenting the results.

Based on the information obtained during the two weeks at the plant, the EPRI team uncovered many opportunities for improvements in operations, equipment maintenance, inter-departmental communications at all levels of management, human performance, root cause analysis documentation and engineering response. The Lost Generation Action Plans as developed by the

EPRI Licensed Material Boiler Reliability Optimization Project Phase 1- Root Cause Analysis at Arizona Public Services Company’s Four Corners and Cholla Plants

4-2

plant need these improvements to ensure that they have a sound base. Documentation of incident investigations, no matter how minor, provides an audit trail to ensure a historical record of what has occurred at the plant, how it was analyzed, what the cause of the problem was and how it was solved. Lack of this audit trail often leads to repeat incidents because staff are not aware of what has happened previously/how problems were solved, often leading to short-term, inadequate solutions, rather than structured, long-term and cost-effective solutions.

No formal structured root cause analysis process exists. Some plant employees have been trained in a root cause analysis technique, but do not use it as it was intended to be used. The staff members were cooperative and mostly forthcoming and interested in using a Root Cause Analysis tool to solve long-term problems at the plant. Root Cause Analysis training/re-training needs to be implemented plant to ensure that staff members are confident in their abilities to solve problems that arise.

Each incident reviewed has a number of recommendations. These recommendations need to be evaluated and implemented. The main recommendation is that a formal, structured approach to incident investigation needs be developed and implemented at each plant. A corporate directive should be developed to give guidance on the requirements. As a start, all unit trips/forced outages, load loss, and failures to meet expected outputs, should be subjected to thorough root cause analysis. Incidents where plant damage, exceeding a predetermined amount, has occurred should also be investigated and any significant capital expenditure needs to be backed up with a justification that is based on a thorough root cause analysis.

4.2 Introduction

In an ongoing effort to improve reliability and reduce maintenance costs in the APS system, EPRI was contracted to review the 1999 Loss Generation Action Plans at Four Corners Plant. The main objectives of this review were:

4.2.1 To provide assurances that the ‘true’ root causes of lost generation have been identified, and appropriate action plans are in place to address these causes:

4.2.1.1 determine and verify the true root causes of technical and/or production problems,

and

4.2.1.2 identify preventive actions to effectively manage the identified problems and minimize the affect on availability.

The team consisted of, two from EPRI and two from APSC, one from Cholla – Jack Stant and one from Four Corners Plant – Mike Rosner. The reviews took place between May 30 through June 9 on site at Four Corners Plant. More than twenty staff members were interviewed regarding the 10 incidents reviewed. These incidents represented a loss of 728,978-Mwh at the Four Corners Plant. The incidents selected represented the major Mwh losses and were prioritized by the plant management team. The selection was based their significance, available documentation, and availability of staff having knowledge of the incidents.


Boiler Reliability Optimization Project Phase 1- Root Cause Analysis at Arizona Public Services Company’s Four Corners and Cholla Plants

4-3

The Behavioural Justification method of analysis was used to analyze some of the incidents that occurred – some APS personnel have been trained in this method. Events and Casual Factors (E&CF) Charting was used as an alternative method. In this method the sequence of events is charted chronologically, from left to right, and information/conditions relating to events in the sequence are charted directly underneath the events. Events are shown as text within rectangles, joined by arrows, ending in a circled unwanted event (reason for the analysis). The information/conditions are shown in ovals, joined by lines that show their association. Dotted ovals or rectangles indicate information that still needs to be verified. Causal factors are identified by asking the question “ Which of these factors, if eliminated, would have either prevented the incident from occurring or at least minimizing its impact?” From these causal factors Root Cause are determined. Root Cause is the cause/s which, if eliminated, would prevent this event recurring or minimize its impact.

4.3 Plant Description

Four Corners Power Plant is owned by Arizona Public Service and is situated in the Four Corners region of New Mexico. Units 1 & 2 are Riley Stoker front fired, natural circulation boilers, rated at 170 Mw each. They were placed in service in 1962 and 1963 respectively. Unit 3 is a Foster Wheeler front fired, natural circulation boiler rated at 220 Mw and went into service in 1964. Units 4& 5 are supercritical B&W units rated at 755 Mw each. They went into service in 1969 and 1970 respectively. All units burn high ash Western coal.

4.4 Incidents investigated

The incidents selected and the Mwh lost are listed in the following tables:

Four Corners - Megawatt-hour loss summary report-fourth quarter 1999

Mw Hour Lost Area Mwh. Lost

Unit 5 #4 Control Valve 197,471

Unit 4 Air Pre-heater Failure 166,744

Unit 3 Turbine Copper Fouling 100,628

Unit 4 LP Exciter Field Ground 45,229

Unit 3 Boiler Tube Leaks 39,378

Unit 4 Flyash Erosion 39,288

Unit 4 Battery Failure 38,413

Units3 Pulverizers 34,881

Unit 4 HP Exciter Field Ground 33,500

Unit 3 Scrubber, and Recycle Pumps 33,446

Total 728,978


4-4

4.4.1 Unit 5 #4 Turbine Control Valve chatter - 197,471 Mwh lost

Limited documentation was made available on this incident. To construct a sequence of events the following people were interviewed - Jack Rogers, Willard Billey, Dave Saliba, Joe Rogers, John Bates, and Randy Wilde

Incident Description

From June 1 - 20, 1999, and from July 1 - Dec. 27, 1999, Unit 5 was de-rated by 6 % due to # 4 turbine control valve problems. On June 17, 1999 during a tube leak outage the valve linkages were repair and the valve operated satisfactorily for 11 days. On July 1 an operator reported that the valve was once again chattering and later slammed shut, and could not be re-opened. The unit was operated with the load limiter set at 94% until a boiler tube leak outage on Nov. 26. During this outage the control valve was checked and found to be operational and the governor pilot valve was rebuilt using re-conditioned parts and other parts, which were not from the original equipment manufacturer (OEM). The unit was brought back on line, and continued to run with the load limiter set at 94%. On Dec.25 the unit trip because of a switchyard failure, the unit was returned to service the same day with no load limitations due to the # 4 control valve.

Root Cause Analysis Chart

Figure 4-1 was drawn based on interviews and operator logs. Knowledge of the problem and actions taken, or not taken, differed between individuals. This made it difficult to accurately record the sequence of events. However, the information as reflected in the chart was accepted by all those interviewed as a fair reflection of what happened.

An analysis of the chart identifies a number of opportunities, that if investigated, at the time, would have eliminated the event or at least minimized the impact of the event. These are:

• There was no attempt to inspect or repair the valve during the June 11 boiler tube leak outage.

• On July 1 it was reported that the valve was chattering and then had slammed shut and could not be reopened. This was accepted without an investigation and the unit ran de-rated until November 26. Reason for not investigating at time could not be established.

• During the November 26 outage the value was checked and verified as operational. However the unit ran de-rated until the December 25 outage. The reason for this continued de-rating could not be established

• No valve work was performed during the December 25 outage, however after the outage Operations were informed that the valve could be operated. The unit was returned to service with no de-rating attributed to the control valve.

To determine the root cause of why so many Mwh were lost, the above points need to be investigated further. This could not be accurately done because of the uncertainty surrounding the sequence of the events from June though December 1999. However, given that the facts in the chart are a fair reflection of what happened, two causes come to mind that if corrected would



4-5

have minimize the impact of operational failure of the valve. These relate to problem ownership – no person took responsibility to solve the problem and communication between individuals and between departments.

An operational test of the value undertaken at the time of the review showed that the valve continues to chatter. The cause of the chattering is still unknown and waiting for suitable outage to determine the cause.

4.4.2 Unit 3 Scrubber unit failure - 33,446 Mwh lost

No documentation was made available on the two incidents that made up this Mwh loss, however Phil Valencia provided some background information.


These two incidents involved the following deficiencies.

• Increasing rate of failure of scrubber recycle pumps. Scrubber debris was being caught in the pump impeller, causing pump imbalance. This imbalance resulted in pump bearing failures.

• Repeated blockage of the overflow and emergency overflow pipes with rubber lining and debris. Rubber-lined piping is used throughout the scrubber system to protect the piping from the highly erosive fluids, which it handles. Portions of this lining break off and block the overflow pipes.

• Repeated unit outages because of high levels in the scrubber unit. Slurry flowing out of the emergency and or overflow pipe is used as an indication of level. These overflow pipes can get blocked from excessive amounts of debris is in the slurry system.

• Structural failure the “Dentist Bowl”. The bowl is attached to a telescoping assembly used to maintain bowl position. Debris collects in the assembly and prevents free movement. This caused high stresses on the bowl when operated which result in eventual failure.

The above resulted in two trips of Unit 3.

• The first incident was the result of a high/high level caused by blocked overflow pipe. The emergency overflow was also plugged with debris. The scrubber flooded preventing flow. The delta pressure across the scrubber increased resulting in a unit trip - high furnace pressure

• The second incident involved a recycle pump failure. Refractory had fallen off a metal skirt in the scrubber and jammed in the recycle pump impeller causing an imbalance. This imbalance caused the pump bearings to fail resulting in a pump tripped. This caused the unit to trip.


4-6

Root Cause Analysis Charts

Figure 4-2 shows the correctable opportunities identified for all four deficiencies highlighted the Behavioral Justification methodology. Clearances had been opened on the “dentist bowl” reducing the risk of any jamming of the telescopic section.

Figure 4-3 represents the E&CF chart was used to analyze the lining failure and blockage problems. Since the piping failures and scrubber level problems were interrelated, they were analyzed together. At low temperatures the gum rubber separates from the metal surface – “cold wall effect” forming blisters. These blisters eroded by the flowing slurry, break open and tear resulting in pieces of the lining coming off the pipe. These pieces of rubber block both the normal and emergency overflow pipes. Operators use the flow through the normal overflow pipe as an indication of correct level in the tank. Figure 43 highlights the correctable opportunities.

• Using a lining, which is more resistant to “cold wall effect”. Experience at Cholla Plant has proved that the use of a different rubber - chlorobutyl, which has a higher resistance to the “cold wall effect”, significantly reduces blistering.

• A more sophisticate method to determine the level in the tank should be installed rather than use the normal overflow pipe.

Figure 4-4 is an analysis of scrubber recycle pump failures and highlights some correctable opportunities.

The pumps have urethane-coated impellers in which debris can easily become lodged, causing rotor imbalance. This imbalance causes the seals fail resulting in slurry entering the bearings, resulting in bearing failure. The major source of debris is refractory, which had been applied to a metal skirt in the upper portion of the scrubber in an attempt to protect the skirt. Moisture gets in between the skirt and the refractory. The resulting corrosion causes large pieces of the refractory to break off and fall into the sump. These pieces get sucked into the pump suction and jam in the pump impeller.

Further investigation is required – design review, to determine whether the skirt is required for the correct operation of the system. There was some doubt by members of the review team of its purpose. If, in deed it is required, then it should be replaced with a corrosive resistant material rather than a material with refractory.

As and when opportunities present themselves, the urethane impellers should be replaced with white iron impellers. These impellers are more robust and fit for purpose in the scrubber environment. This has proved to be successfully at Cholla

In addition suitably sized suction screens need to be installed. This is an inexpensive activity if undertaken during a suitable outage. These screens would be the first lines of defense to prevent over sized material entering the pump.

These items are not specifically addressed in the 2000 Action Plan but can be accommodated in the $500k is budgeted for scrubber work.



4-7

4.4.3 Unit 4 Loss of Booster Fans 38,413 Mwh.

Mel Begayle and Richard Bagienski provided some background information, as there was no documentation available.


Regular preventive maintenance tasks performed on the twelve years old Booster fan control backup batteries had shown that the batteries needed to be replaced.

For some unknown reason, AC power to the DC converter was interrupted; this required the controls to be supplied by the batteries. The DC voltage dipped below 108 volts. This dip in voltage caused the BETA alarm panel to reset, since the batteries were unable to maintain the required voltage. The booster fans tripped, tripping Unit 4. Batteries purchased for another purpose were used for replacement and another set of batteries ordered. The alarm panel was also changed.


The E&CF chart - Fig. 4.3 shows the sequence of events. Lack of any documentation and relying on the memory of the individuals involved made it difficult to re-construct the sequence of events and hence determine the root causes. The direct cause of this incident was the poor condition of the batteries. However the following was determined from the interviews:

• electrical maintenance staff did not know how to correctly check the batteries

• Incorrect meter was used to perform the battery check.

• No action had been taken of the battery deterioration as noted on the preventive maintenance (PM) schedule. The crystal growth in the batteries had been visually verified and noted on the PM.

Additionally, a crossover switch between the Unit 4 batteries and Unit 5 batteries has been installed. The proper meter is now being used and the PM has been rewritten. All the actions discussed are included in the Lost Generation Action Plan.

4.4.4 Unit 3 Pulverizer Pinion & Bull Gear Failure - 34,881 MW Hrs.

Bill Burnett and Calvin Charlie provided the information to re-construct this incident.


This incident involved the following:

• The foundation of 3B pulverizer had failed due to excessive vibration caused by incorrect alignment of the bull and pinion gears. It failed at the same place where a similar repair was made ten years previously.


4-8

• The pinion gears, which had been purchased from different suppliers, did not meet the original specification and were installed by maintenance staff.

• A bull gear known to be out of specification was accepted and installed.

• Failure of the pulverizer lube oil pump.


The Behavioral Justification methodology was used to analyze this incident. This is shown graphically in Fig.4.4. Four correctable opportunities were identified.

• A procedure needs to be developed with criteria on the acceptance/rejection of equipment/components that do not meet specification/requirements. Plant management had accepted a bull gear, which had been improperly manufactured and had installed it. Reason for this decision could not be established.

• A Quality Assurance/Quality Control (QA/QC) process needs to be developed and followed to ensure that equipment maintained or manufactured by a supplier conforms to minimum requirements and is fit for purpose. This is especially essential with components like pinions and gears. Pinion gears are purchased from several suppliers and no QA or QC is performed.

• Operator rounds need to be enforced. The failed lubrication oil pump went unnoticed for an unknown length of time. The lack of lubrication caused the gears to overheat and contributed to the failure.

The Lost Megawatt Action Plan does not address this issue. Replacement of the pulverizers is being considered. However, the problems discussed above will not be solved with new pulverizers. New equipment will not resolve procedural, management, or communication problems.

4.4.5 Unit 3 Back-pass Flyash Erosion 39,378 Mwh

David Hughes, Drexel Pruitt and Duane Pilcher shared with the review team their insight and knowledge of the erosion in the back-pass of Unit 3

Problem Description

Flyash erosion and the resulting tube failures have been the major cause of forced outages on Unit 3. Changes in coal quality – higher ash content combined with a furnace size, which was marginal for the originally specified coal, and continued operation at loads equal to or greater than design output have all contributed to fly ash erosion. Extensive shielding and numerous modifications have further increased the gas/particle velocities, which has compounded the fly erosion problem.



4-9


Fig.4.5 identifies a number of correctable opportunities. These all need to be evaluated – separately and collectively. The purpose is to reduce gas velocities. Some of them are related to poor pulverizer maintenance. However, a significant contributor is the number of unqualified modifications. In the past, a number of modifications have been done to solve an immediate problem without considering the effects of such a modification on other areas of the boiler. Many of these modifications have indeed met the original intent and created an erosion problem elsewhere in the boiler. This is compounded by the lack of documentation on the number and rational behind the modifications. A “Backpass” study scheduled for completion in Sept. 2000 is in the Lost Generation Action Plan. It is recommended that an E&CF chart be drawn/used to analyze all past modifications made to the unit. This would be a daunting task, but extremely useful in resolving and understanding all the erosion problems.

4.4.6 Unit 4 Air Pre-heater Failure 166,744 Mwh.

Ron Bertram, Jim Justice and Randy Wilde provided some detail about this incident, as there was no documentation regarding this failure.

Problem Description

Two modifications to the air pre-heater had been done to facilitate the early operation of the bag filter during unit start-up. These were to slow down the rotational speed of the pre-heater and delay its in-service operation during unit start-up. The net effect of these modifications on the air pre-heater was an increase in the number and extent of thermal cycles. This resulted in the thermal fatigue failure of a cold end basket retaining frame/clip. This allowed the basket to shift and bind the air pre-heater causing extensive damage.


Figure 4-8 diagrammatically shows the causes and correctable opportunities. The two operational changes significantly contributed the accelerated thermal fatigue failure of the cold end basket-retaining frame. Both of these conditions have been corrected and an extensive inspection and repair effort is included in the Lost Megawatt Action Plan.

4.4.7 Units 4 Flyash Erosion 39,288 MW/Hrs.

Mike Rosner provided the background to this problem.

Problem Description

Units 4 & 5 have a history of boiler tube failures due to flyash erosion in the back passes. This is because of a marginal boiler design and over firing conditions. When over firing for extra megawatts, operations increases excess air in order to prevent clinker formation. This results in


4-10

higher gas volumes and increased velocities. The short-term effect is less clinker formation. In the longer term this mode of operation increases flyash erosion resulting in unwanted tube leaks. This significantly impacts the reliability of the unit


Fig. 4.7 represents diagrammatically the information provided. The flyash erosion problem is the result of a marginal boiler design as in furnace size – BTU/square foot, choice of the most economical fuel and the operational philosophy. Erosion rate is dependent on a number of factors – velocity being a major factor. The relationship between erosion rate and velocity, is exponential - velocity to some power between 2 and 3. So, velocity hence gas flow should be kept at a minimum and under control. A flue gas velocity distribution test is scheduled for 2003. Extensive efforts are planned in the Lost Megawatt action Plan to repair and mitigate the effects of flyash erosion.

4.4.8 Unit 3 Turbine H.P. Nozzle Pluggage 100,628 Mwh.

The following provided some documentation and described the turbine pluggage problem - John Huff, Joe Baker and Dave Viser

Problem Description

Shortly after switching from hydrazine to Eliminox for dissolved oxygen control in 1993, the Unit 3 turbine/generator MW output started to decrease. In 1996 an investigation was launched - a “fault tree” analysis was drawn. This analysis identified that the reduced output was due to a restriction in the high-pressure (HP) turbine inlet nozzle. Results of a chemical cleaning the nozzle showed the deposits to be copper. Analysis of the economizer inlet water indicated that the copper was coming from the feedwater system. It was assumed the source of copper was the condenser. The HP turbine was cleaned several times, and each time nine to twelve pounds of copper were removed


The Behavioral Justification methodology was used to analyze the problem. Figure 4-10 shows four correctable opportunities.

• The replacement of hydrazine with Eliminox in 1993 followed an industry wide trend to remove hydrazine, a known carcinogen, from power plant sites. Substitutes were not as effective and many units observed an increase in copper corrosion and subsequent copper carryover in units operated in excess of 2300 PSI-drum pressure. Early in 2000 the unit has returned to hydrazine treatment. Hydrazine levels are adjusting to maintaining a negative oxidizing/reducing potential at the economizer inlet.

• The condenser is to be replaced with 29-4-C, a high performance stainless steel.

• It is also planned to replace copper metallurgy feedwater heaters with stainless steel as the need arises.



4-11

• These actions will prevent further migration of copper corrosion products into the boiler systems. However, there is copper in the water wall and superheater tubes which can be a major contributor to tube failures. A chemical cleaning of the waterwall tubes was done in Feb. of 2000.

4.4.9 Unit 4 LP Exciter Field Failure 45,229 Mwh.

Richard Bagienski was interviewed. After the interview details of the incident were provided.


Unit 4-exciter failed resulting in a unit trip. Inspection of the brushes and brush holders revealed that some misalignment between brush and brush holder tension arm. The brush arm arced and overheated the collector ring, which, in turn, caused other brushes to arc. The increase in resistance tripped the unit on low excitation.

Root Cause Analysis

Documentation provided was clear, concise, and complete, and is included in the appendix. The problem was quickly identified and the appropriate actions taken. Access to check brush alignment is difficult. The importance of ensuring correct alignment was not identified on the PM. This can be considered as the root cause. The PM has now been rewritten to include an inspection of the tension arms. The Lost Megawatt Action Plan has extensive Exciter work planned for both Units 4&5 in 2000.

4.4.10 Unit 4 HP Exciter Field Failure 33,500 Mwh

Richard Bagienski was interviewed.

Problem Description

Unit 4 was taken off line after analysis of an exciter field ground alarm indicated that there was a ground in the field about at 50% through the winding. After the unit was taken off line, the exciter field was isolated and the ground was verified. A copy of APS documentation covering this event is included in the appendix.

Root Cause Analysis

As in the LP exciter event (see 4.9.1), the problem analysis was well done. The exciter was shipped to General Electric for inspection and repair. The inspection revealed that a previous rewind was substandard. In 1992 an alternate vendor was used to do the rewind. This substandard rewind was considered to be the direct cause of this incident, Lack of proper QA/QC is the root cause of the problem.


4-12

4.5 Findings

• At each of the stations, the review team consisted of the two EPRI employees and a staff member from the other station involved in this exercise. This was a very positive aspect of the review as experiences and problem solutions at both stations were shared. The opinions/ideas of peers were welcomed.

• Root Cause Analysis is done by exception, i.e. as and when requested by management.

• Some staff members were trained in the Root Cause Analysis technique taught by Bobby Jones and Associates – Behavioural Justification, but they don’t regularly use any formal analysis technique.

• Root Cause analysis, commonly, consists of group brainstorming sessions, where solutions (often short-term) are formulated and actions allocated.

• These solutions are not documented and limited follow up is done on allocated actions.

• There was very little documentation available from which the review team could gather information. Therefore, in an attempt to reconstruct the incidents key information had to be obtained from staff members who had knowledge thereof. E&CF charts or Behavioral Justification diagrams were constructed from this information and causal factor determined.

• Of the ten lost generation incidents reviewed, only one had some usable documentation – Unit 3 Turbine Nozzle plugage. This investigation was started in 1996.

• In more than 70 % of the reviewed incidents, it was found that staff chose to rather replace/fix/modify equipment or plant, instead of assessing root causes.

• No documents were provided to support the1999 Lost Generation Action Plans and the Action Plans, therefore, could not be validated as required in the Scope of Work for this project. In almost all the incidents reviewed /reconstructed, the “fix” – repair/replace or modification did align with those recorded in the 1999 lost Generation Action Plan.

• Preventive/Predictive Maintenance schedules/activities are not implemented, as staff are focused on corrective maintenance, as in crisis management.

• There is a lack of problem ownership, as responsibility for resolution of problems is not specifically allocated, e.g. the turbine control valve chattering.

• The long-term health of the plant is neglected. The plant is run in excess of the design maximum continuous rating, e.g. boilers are over-fired to a point where clinkers start to form; continuous sootblowing is then used to remove this build up and this causes erosion which results in long-term damage and/or a tube leak.

• Project kick-off and exit meetings were held to confirm scope of work and give feedback respectively. Considerably less management staff attended the exit meeting.



4-13

4.6 Conclusions

The findings of the EPRI team led to the following conclusions being drawn:

4.6.1 Formally-documented Root Cause Analysis is not being done at the Four Corners. The most common reason for not doing a root cause was, ”we are too busy putting out fires”. This made verification of the actions identified in the 1999 Lost Generation Action Plan difficult, however through interviews actions could be verified.

4.6.2 Although no formal analysis is being done, the actions (solutions) identified in the 1999 Lost Generation Action Plans are considered short-term quick fix solutions.

4.6.3 Knowledge of a Root Cause Analysis process was limited even though staff had been trained in an analysis technique.

4.7 Recommendations

4.7.1 A formal structured Root Cause Analysis process needs to be implemented. This process should be developed based on a corporate directive requiring those major incidents (greater than a $X, 000 – value to be determined by Corporate), unit trips and a significant unexpected maintenance expenditure be investigated to ensure repeat failures/incidents/trips do not occur.

4.7.2 Staff should be trained/retrained in a Root Cause Analysis technique that includes Human Performance aspects.

4.7.3 The various recommendations identified in Section 4.4 – Root Cause Analysis should be evaluated and implemented.

4.8 Appendix 1: Root Cause Analysis Charts

(Excel file)

Note:

1. A number of the charts are incomplete as there was limited documentation and the allocated time per incident was constrained. However they do reflect the information given or presented by the various interviewees.

2. The charts need to be read in conjunction with the text in the main body of the report


4-14

6\1\99 6\1\99 6\11- 6\13 6\17- 6\19 6\20 6\21

7\1 7\??? 7\11- 11\26 11\26-12\2 12/2-12/25 12\25

*

S.S. Reports#4 control valvechattering

Turbine plannernotified

Unit curtailedto 710 MW 's

Valve closed tostop chatter

72 hour outage required forinspection

Load lim iter set to 93%

Unit shutdown for tube leak

No attempt made to repair or

inspect valve

Unit shut downfor tube leak

#4 control valvelinkage repaired

Bracket machined

Evidence of previous failure

Broken bolts

replaced

No failureanalys is of

bolts

S.S. reports that#4 valve is not

chattering

Unit operates with no loadcurtailment

Turbine planner &Operations Managerverify that #4 controlvalve is operational.

S.S. reports #4 valve ischattering

Operating with load

lim iter at 94%

Unit is run at over-pressure to meet

load requirements .

No investigation initiated

S.S. is notified that #4 valve iss tuck in closed

pos ition

Control operatorattempts to

openvalve.

Unit operateswith loadcurtailed

Load limiter set to 94%

Unit shut downfor tube leak

repairs

#4 control valvechecked and found

to beoperational

Governor pilot valve rebuilt withnon-OEM

reconditioned parts

Unit shutdowndue to switch-

yard failure

Curtailmentlifted after

startup - Unit at full load

197,471 Mwhlost due to #4 control valveproblems.

A

A

Lost Opportunity

No chatteringdetected

No writtenverification

Unit continues to operate with load

limiter at 94%

Operators unawarevalve checked and

found to be operational

Figure 4-1 Unit 5, #4 Control Valve 197,471 Mwh



4-15

I.D. C.O. Material

selection

C.O.

C.O.

C.O. C.O.

C.O.

1. Replace

skirt w/o

refactory ,

Screens,

White iron

pumps

N.C.

Lost Mwh

due to Scrubber

Outages

Recycle pump

failures

Foreign objectscause impeller

imbalance

Refactory fallingoff of sk irt

Pumpcavitation

Rubber-lined

piping failure

Cold wall effectpipe blis tering

High level of slurry

in scrubbers

Overflow plugged

withdebris

No levelindication

No levelindication

Blowdownnot

sufficent

No levelindication

Structuralfailure of

Dentis t Bowl

Telescope jammed

Tightc learances

Debrisbetween

pipe clearances

Figure 4-2 Unit 3 Scrubber Problems 33,446MW/Hrs.


4-16

* *

*

* *

Blistering inrubber-lined

piping.

Gum Rubberversus

Chloro Butyl

Gum rubber verysusceptible tocold wall effect

Rubber separatesfrom piping

Seperatedrubber plugs

normal overflow

Rubber piece20" diameter

Pipe diameteris 14"

Overflow pipe is long and almost

horizontal

Scrubber debrisplugs emergency

overflow

Pipe diameter is 14"

Overflow pipe is long and almost

horizontal

High waterlevel blocks

gas flow

No level ins trumentation

dP acrossscrubberincreasedsuddenly

Fans were fully -loaded - 100%

Unit 3 trip due to

high furnace pressure

A

A

Normal overflowis used for level

control

Figure 4-3 Unit 3, Scrubber Problems 33,446MW/Hrs.



4-17

*

*

Refactory applied tometal skirt

Purpose ofrefactory isnot known

Refactoryfalls off

metal skirt

No studs ormetal

hangersd

Base metal corrodes

Refactory lodgesin impeller

Impellerdesign

No pumpsuctionscreens

Free diameterpassage is 5"

Pump fails

Slinger ringswear out

Packingfails

Unit derated

Enviromental regsrequire pump to be

in service

Loss of MW due to recycle

pump failures

A

A

Rotor becomesimbalanced

Urethanecoated

Figure 4-4 Unit 3 Scrubber Recycle Pump Problems 33,446MW/Hrs.


4-18

* *

*

Battery PMperformed

Incorrect voltagemeter used totest batteries

Data not checked oranalyzed

DC inverteroutput voltage

drops.

AC power toinverter wasinterrupted

DC voltagedropped to less than 108 V

Batteries were bad

Replacementbatteries were

not ordered

BETA alarmpanel reset

BETA panelsensitive to

input voltage

Unit 4 tripsdue to loss ofbooster fans

New batteries ordered

A

A

Electriciansvisually inspect

batteries

Crystal growthin batteries

Indication ofend of life

Boosterfan trips

Figure 4-5 Unit 4, Loss of Booster Fans, 38, 413 MW/Hrs



4-19

. C.O. C.O.C.O. Assurefoundationintegrity ateach pinionreplacement.

C.O. Reviseprocedureto accommo-datedif ferencesbetw eenpulverizers

I.D..

Operarions w as not per-forming checks.Maint. began checkingfor surv ival.Now usingbettertechnology.Restoredlubemanposition.Improvedins trumen-tation.

UNIT 3, 3B PULVERIZEREXCESSIVE PINION

GEAR FAILURES

Foundationwas broken

Improper alignment be-tween gears

Pinion/Bullgears got too

hot

Improperlymanufactured

gears

Excessivevibrations &

stresses induced to foundations

Improperinstallation of

gears

Procedure was incorrect

Mechanicalfailure of

mounting bolts

Lube oilsystem

failed

Lube oilpump failed

Timelycheck of

pinion/bullgears notperformed

Improperlymanufactured

gears wereknowinglyaccepted

Figure 4-6 Unit 3 Pulverizer Pinion and Bull Gear Failures, 38,881 MW/Hrs.


4-20

C.O. C.O. C.O.

U3 REHEATER BACKPASS

FLYASH EROSION

Unquantifiedeffects of unitmodificationsin the boiler

High ash content in coal

High flue gasvelocities

Poor ashdistribution

Poor mill performance,fineness & distribution

Reheat & super-heat

outlet temperature

control

Furnace smallfor heat release

required

Close tubespacing inbackpass

Boiler damper operation

Ash momentumconcentrates

partic les in reheat backpass and S.H.

boundary wall.

Boiler damper operation

Reheat & super-heat

outlet temperature

control

Fuel level control

Poor millmaintenance

Ball chargeimproper

Boiler design

Figure 4-7 Unit 3 Backpass Flyash Erosion 39,378 MW/Hrs.



4-21

C.O.

C.O.

UNIT 4 AIRPREHEATER

FAILURE

Cold sidebaskets fell out

Basketfastners

failed

Accelerated thermal

fatigue of metal

APH was being removed from service

on cold s tartups toplace baghouse in

service earlier.

APH rotationspeed was set

too low.

Frame aroundAPH basket

failed

Thermal fatigueof metal.

Figure 4-8 Unit 4 Air Preheater Failure 166,744 MW/Hrs.


4-22

N.C. I.D.N.C.

Flyash Eros ionin Backpass of

Units 4&5

High flyash levels in

BHP coal

BHP coal islow costoption

Localized highvelocities in

flue gas

High flue gas volumes exis t

for boilerdesign

Flue gas haspoor

dis tribution

Perform fluegas veloc itydistribution

tests

Unit overfiredfor extra MW s

Excess air used to preventpluggage

Figure 4-9 Units 4 and 5 Backpass Flyash Erosion



4-23

C.O.

C.O.

UNIT 4 AIRPREHEATER

FAILURE

Cold s idebaskets fell out

Basketfastners

failed

Accelerated thermal

fatigue of metal

APH was being removed from service

on cold startups toplace baghouse in

service earlier.

APH rotationspeed was set

too low.

Frame aroundAPH basket

failed

Thermal fatigueof metal.

Figure 4-10 Unit 3 HP Turbine Nozzle Plugging, 100,628 MW/Hrs.


4-24

MW Hour Lost Area MW/Hrs. Lost

Unit 5 #4 Control Valve 197,471

Unit 4 Air Preheater Failure 166,744

Unit 3 Turbine Copper Fouling 100,628

Unit 4 LP Exciter Field Ground 45,229

Unit 3 Boiler Tube Leaks 39,378

Unit 4 Flyash Erosion 39,288

Unit 4 Battery Failure 38,413

Units3 Pulverizers 34,881

Unit 4 HP Exciter Field Ground 33,500

Unit 3 Scrubber, and Recycle Pumps 33,446

Total 728,978



4-25

UNIT # Type Furnace MFG. Start up DateFuel Description Oper Pressure MW Rating Pollution Control

1natural circ/front wall fired Riley Stoker 1962sub-bituminous 1800 170wet scrubbers 2natural circ/front wall fired Riley Stoker 196320-25 % ash 1800 170wet scrubbers 3natural circ/front wall fired Foster Wheeler 19649200 btu/ft3 2000 220wet scrubbers 4super critical/front & rear fired B&W 1969 " 3500 755baghouses/wet scrubbers5super critical/front & rear fired B&W 1970 " 3500 755baghouses/wet scrubbers

net


4-26

CHOLLA PLANT

1999 Lost Generation Incident Review


In an increasingly competitive environment, the ability to reliably produce the lowest cost power is essential for the long-term survival of a utility. With this vision in mind Arizona Public Services Company (APSC) requested EPRI to review the 1999 Lost Generation Action Plans of Cholla Plant. The intent of this review was to provide assurance or not that the “true” root causes of lost generation have been identified, and appropriate action plans are in place to address these causes.

The reviews took place between 19 June to 30 June 2000 at the Cholla Plant. Eleven incidents were reviewed, of which three had some documentation backing up the plant’s root cause analyses. The review team, two from EPRI and one from Cholla and one from Four Corners used a series of interviews with staff to re-construct the incidents. A number of staff involved with the incidents were interviewed. Two methods, Behavior Justification and Event and Causal Factor Charting were used to re-construct the incidents. Diagrams/charts were constructed from information that depended, almost entirely, on the interviewees’ long-term recollection of the incidents. Verification of information was difficult, as there was a lack of formal, incident-specific documentation and because of the time that had elapsed between the incidents and the analyses. In many incidences, perceptions and memory of what happened, and in what sequence, varied, emphasizing the importance of performing root cause analysis in a timely manner, and documenting the results.

Based on the information obtained during the two weeks at the plant, the EPRI team uncovered many opportunities for improvements in operations, equipment maintenance, inter-departmental communications at all levels of management, human performance, root cause documentation and engineering response. The Lost Generation Action Plans as developed by the plant need these improvements to ensure that they have a sound base. Documentation of incident investigations, no matter how minor, provide an audit trail to ensure a historical record of what has occurred at the plant, how it was analyzed, what the causes of the problem were and how they were solved. Lack of this audit trail often leads to repeat incidents because staff are not aware of what has happened previously/how problems were solved. This often leads to short-term, inadequate solutions, rather than structured, long-term and cost-effective solutions.

No formal structured root cause analysis process exists. Some plant employees have been trained in a root cause analysis technique, but its use is limited and the results are not documented. The staff members interviewed were all cooperative, forthcoming and interested in using a root cause analysis tool to solve long-term problems at the plant.

Each incident reviewed has a number of recommendations. These recommendations need to be evaluated and implemented. The main recommendation is that a formal structured approach to incident investigation needs be developed and implemented. A corporate directive should be developed to give guidance on the requirements. As a start all unit trips/forced outages, load loss



4-27

and failures to meet expected output should be subjected to thorough root cause analysis. Incidents where plant damage has occurred should also be investigated and any significant capital expenditure needs to be backed up with a justification that is based on a thorough root cause analysis. Training a Root Cause Analysis Technique needs to be implemented.

4.10 Introduction

In an ongoing effort to improve reliability and reduce maintenance costs in the APS system, EPRI was contracted to review the 1999 Loss Generation Action Plans at both the Cholla and Four Corners plants. The main objectives of this review were:

4.10.1 To provide assurances that the ‘true’ root causes of lost generation have been identified, and appropriate action plans are in place to address these causes:

4.10.1.1 determine and verify the true root causes of technical and/or production problems, and

4.10.1.2 identify preventive actions to effectively manage the identified problems and minimize the affect on availability.

The team consisted of, two from EPRI and two from APSC, one from Cholla – Jack Stant and one from Four Corners – Dan Wilson. Jack Stant Cholla participated in both reviews. The review took place between June 19 though June 30 at the Cholla Plant. Over fifteen staff members were interviewed regarding the ten incidents reviewed. These incidents represented a loss of 481,455 Mwh. The incidents selected represented the major Mwh losses and were prioritized by the respective plant management teams. The selection was based on their significance, available documentation, and availability of staff having knowledge of the incidents.

The Behavioral Justification method of analysis was used to analyze a number of the incidents - APS personnel have been trained in this method. Three of the ten incidents had adequate documentation.

Events and Casual Factors (E&CF) Charting was used as an alternative method. In this method the sequence of events is charted chronologically, from left to right, and information/conditions relating to events in the sequence are charted directly underneath the events. Events are shown as text within rectangles, joined by arrows, ending in a circled unwanted event (reason for the analysis). The information/conditions are shown in ovals, joined by lines that show their association. Dotted ovals or rectangles indicate information that still needs to be verified. Causal factors are identified by asking the question “ Which of these factors, if eliminated, would have either prevented the incident from occurring or at least minimizing its impact?” From these causal factors Root Cause are determined. Root Cause is the cause/s which, if eliminated, would prevent this event recurring or minimize its impact.


4-28

4.11 Plant Description

4.11.1 The Cholla power plant is situated in NE Arizona and consists of four corner fired units, manufactured by Combustion Engineering and went into service between 1962 and 1981. They are all drum units operating at a pressure of 2000-PSI. Unit 1, rated at 110 Mw, has SO2 scrubbers. Unit 2 rated at 245 Mw is equipped with SO2 scrubbers and an absorber. Unit 3, with a precipitators and is rated at 260 Mw. Unit 4, rated at 380 Mw, is fitted with a precipitator and an absorber. Units 1, 2, and 3 have Westinghouse turbine/ generator and Unit 4 has a GE turbine/generator train.

4.12 Incidents Investigated

The incidents selected and the Mwh lost are listed in the following tables:

Table 4-1 Cholla - Megawatt-hour loss summary report-fourth quarter 1999

Megawatt hour loss area Mwh lost

Unit 2 Waterwall Hydrogen Damage 188,944

Unit 4 Scrubber Outage 76,917

Unit 4 Air Pre-heater Blockage 48,740

Unit 4 High Opacity Shutdown 42,731

Unit 1 Mechanical Flyash Collector 41,146

Unit 2 A ID Booster Fan Motor Failure 30,464

Unit 2 Cable Tray Fire 28,899

Unit 1 Turbine Thrust Trips 9,971

Unit 1 ID Booster Fan Bearing Failure 7,544

Unit 4 Air Pre-heater Guide Bearing Fire 6,099

Total 481,455



4-29

4.12.1 Mechanical Dust Collector Retrofit 41,146 Mwh.

Those interviewed were: El Pahi, Dennis Stewart and Mike Machusak

Problem Description

The original Dust Collector on Unit 1 was becoming too expensive to maintain because of the increasing number of failures also because of its size, number, and location of tubes it was also difficult to maintain. The design as such made it difficult to address equipment wear problems. In 1997, an Engineering Service Request (ESR) was written in order to find a permanent solution – to replace the dust collectors. In order to replace the collector during the March 1998 the project was placed on a fast track. This was the first opportunity as the unit outage frequency had been changed from two to three years. This allowed three-months between Request for Proposals and installation of the equipment. Because of this time constraint, no modeling studies were required or performed prior to installation.

Performance tests showed that the new collector met opacity requirements, but failed to meet the pressure drop (dP) specification. The dP was 7 inches versus the 3.5 inches specified. In May of 1999, two modifications based on modeling studies made by the manufacturer were installed on the dust collector at the vendor’s expense. Subsequent tests showed slight improvement to the dP, but it still did not meet guarantee. The fans could not accommodate the additional pressure drop. As a consequence the unit was de-rated by 5 Mw.


The E&CF chart in Figure 4-11 identifies a number of causal factors. These are:

• Lack of long term planning. The poor performance of the dust collector was known some time before it was decided to proceed with a retrofit. The Engineering Services Request to resolve the collector problems was written in late 1997. Shortly there after, the scheduled outage interval was increased from two years to three years. This increase in frequency changed the priority to perform the modification during the 1998 Spring outage. Failure to do so would result in a three year delay of the project, and engineering studies predicted that the collector’s remaining life was less than three years

• A bid package was put together using inlet design and pressure drop specifications, and sent out in January of 1998 to three vendors, with a deadline of three weeks for responses. Due to time constraints, no modeling studies were required. ABB, the OEM for the collector, did not respond.

• The bids were not thoroughly analyzed and or evaluated. No design review was performed to validate the supplier’s technical guarantees.

• Contract management was questionable. After installation, the collector failed acceptance testing due to a high-pressure drop of seven inches. In May 1999 the contractor modified the plant based on a modeling study. Test showed a decrease of 1.6 inches in dP to approximately 5.4 inches against a guarantee of 3.5 inches. This additional dP was greater


4-30

than the capability of the induced fans at full load. This resulted in a 5 Mw de-rate on the unit. No additional requirements were put onto the contractor.

• A feasibility study is being undertaken to either replace the fans, or changing the blade design to overcome the collector pressure drop and return the unit to full load capability

The above represent the casual factors that if removed would have either prevented this incident from happening or at least would have minimized its impact. The root causes would need to be established from these causal factors by answering the question – What management processes should have been in place and enforced to ensure timely and cost effective project management?

4.12.2 Unit One ID Booster Fan Bearing Failures 7,544 Mwh.

No documentation was provided however El Pahi and Jim Fogel gave some background to this bearing failure.


Repeat booster fan bearing failures have occurred. The bearing are water-cooled sleeve bearings. The failure mode has generally been overheating caused by lack of adequate lubrication. The ash-laden gas that is conveyed by the fan erodes the fan blades causing an imbalance in the fan rotor. This imbalance results in increased vibration. The vibration is transmitted through the bearing housing to the cooling water supply housing, cracking the inlet nipple. This allows water to displace the oil in the bearing sleeve, causing the bearing over heat and fail.

Root cause Analysis Chart

Figure 4-12 shows the E&CF chart drawn from the information obtain from the interviews.

Plant personnel assumed that the increased fan vibration due to flyash erosion resulted in accelerated metal fatigue of the cooling water supply nipple. The following actions have been taken to resolve the problem:

• Cooling water has been replaced with an air- cooled system.

• In 1998 the worn/damaged bearing housing on “A” fan was replaced and the fan shaft repaired.

• A newly installed vibration monitoring system now trips the fan at seven mils.

• Installation of an external lube oil-cooling system.

The above actions have so far halted the failures. However, these actions only address the symptoms and/or give additional protection. None of the above address the causes, which are vibration, caused by fan imbalances, caused by erosion caused by excessive ash-laden gas.



4-31

4.12.3 Unit one Turbine Thrust Trips 9,971 Mwh.

Limited documentation was made available. George Ross and Tim Vachon provided some background to this incident.


One thrust trip setting was desired for all units at the plant. However, Unit 1 Turbine always ran with a higher thrust of plus or minus 10 mils compared to an expected value of plus or minus 6 mils. This was aggravated by the fact that the Bentley system and manual methods did not agree on thrust position. This resulted in numerous spurious trips of the Unit 1 turbine.

During the spring 1998 outage, the abnormally worn thrust bearing was replaced. The turbine was thoroughly inspected, and all possible contributors to the abnormal thrust were examined. New thrust probes were also installed and the thrust was statically calibrated. 2 mils of thrust were available for adjustment. On returned to service, the unit tripped on high thrust. For some unknown reason the thrust zero point had moved to -15 mils from the cold set point. While off load the probes were adjusted and the unit returned to service. The unit again tripped on high thrust. While off load the probes were again adjusted. Several trips occurred and adjustments made to the trip limit until a trip setting was obtained where the unit did not trip on high thrust. The probes could not be adjusted online. The above actions have eliminated the trips however the turbine now runs with a thrust value of between 12 to 15 mils which is a concern to plant staff.


The E&CF chart in Figure 4-13 identifies the causal factors for the repeated trips after the new probes were fitted. The reason for the high thrust in the first place still needs to be established. There are three causal factors identified.

• Incorrect calibration of the newly installed probes. Inexperience/lack of know how and insufficient data are the main reason for the incorrect calibration.

• The decision to limit the meter range in order to maintain plant standards. Since the cause of the excessive thrust had not been identified, a conservative approach was taken to protect the turbine from damage.

• The inability to adjust the probes while the turbine is online. Conservative adjustments were made to the thrust trip set points each time the turbine tripped since a large adjustment could have had serious consequences.

Although the trips have been eliminated, there is concern over the turbine’s excessive thrust readings. The investigation is ongoing and listed in the Lost Megawatt Action Plan. Those interviewed revealed an in-depth knowledge of the problem and are committed to its elimination.


4-32

4.12.4 Unit Two “A” Booster ID Fan Motor Failure 30,464 Mwh.

Those interviewed were: Andre Norwood, Lee Holly, Jim Wilber, Allen Bushman, Jack Stant and Joe Lacey


In June of 1999 two attempts were made to start Unit 2 A Booster ID fan following a planned unit outage. Both attempts resulted in an instantaneous ground fault. An investigation revealed that a power supply cable that had been disconnected for testing purposes during the outage had not been reconnected. The fan motor insulation was tested acceptable after reconnection of the cable and placed in service. While in service the vibration level increased. On July 11, the unit was returning to service after a tube leak repair and the fan was started six times for washing and balancing. On the sixth start the motor failed and burned-out. The badly damaged motor sent off site for repair.


The E&CF chart in Figure 4-14 shows the sequence of events. The analysis of this incident was difficult and required extensive interviews and review of available documentation. The following needs to be read in conjunction with the events mapped in Figure 4-14

• During the 1999 spring outage, APS Generation Engineering personnel performed extensive Doble testing on the Unit’s large electrical equipment. Test results on the “A” booster fan motor showed some deterioration of the motor cables. In order to quantify the problem, the testing procedure was altered. This consisted of disconnecting the cables at the breaker panel, which is not normally done. No one in the plant was informed of this change in the scope of work. Manpower scheduling problems necessitated using four different contract electricians on the job. The breaker cabinet was returned to service with the middle phase disconnected. The Doble test showed that the cable had deteriorated and needed to be replaced during the next planned outage.

• On June 18, operations attempted to start the fan for balancing. The fan tripped on instantaneous ground fault. No evidence of damage could be found. The protection flags were reset and a second attempt was made to start the fan. This time it tripped on overload. The production supervisor contacted electrical maintenance personnel to investigate the fan failure. A motor insulation test indicated that the motor had an open phase. At this point, the Outage contract electrical foreman, who was still on site, remembered that the cables had been disconnected during the outage. It was then discovered that the cable was still disconnected. After returning the cables to their proper configuration, the motor was tested and found acceptable, and the fan returned to service. No Doble testing was performed on the motor or the cable. Fan balance and cleanliness was verified.

• The fan operated for seventeen days with no problems. On July 6, the fan inboard bearing vibration increased to 3.5 mils horizontal and 2.5 mils vertical. The outboard bearing remained constant at 1.9. Plant personnel assumed that fan vibration was the result of deposition of material on the fan blades as a result of scrubber de-mister failure. However,



4-33

the de-mister problem had been addressed during the spring outage and appeared to be functioning properly. There was no detailed analysis done on the increasing vibration.

• On July 10, the unit was shut down for tube leak repairs, and the fans removed from service once the boiler was cooled. On July 11, based on an increase in vibration levels, the fan was washed. The fan was started six times between 19:20 and 00:06, a period of four hours and fourteen minutes. Analysis of the start and stop times – table in figure 4-14 shows that the plant was in compliance with the industry standard of not starting a motor more than three times in any one hour period. From previous washing experience a decrease in the vibration levels is noticed after the first washing cycle. This was not the case with this fan - washing of the blades had negligible effect on vibration levels. On the sixth start, the motor failed immediately.

The following represents the causal factors i.e. those causal factors that if were not present would have either prevented this incident from occurring or at least minimized its impact.

• The original Doble testing work scope was expanded without notifying the Plant Clearance Authority. Because of this, no one on the plant staff was aware that the cables had been disconnected for testing.

• The two ground faults may have significantly contributed to the failure. However, this cannot be accurately determined as the motor was repaired without any failure analysis being done on it.

• Doble testing was not performed on the fan motor following the two unsuccessful fan starts on June 18. Doble testing is a more rigorous test and may have revealed deeper problems.

• No analysis of the increased vibration levels. It was assumed that ash was lodged on the fan blades. This assumption was made in spite of the fact that the de-mister problem had been corrected, and the vibration increase was on the inboard bearing only.

• Excessive fan starts. A careful look at the fan start timetable shows that the production supervisor was following the guidelines regarding the time limit on fan starts. This guideline is part of the plant’s oral history and could not be found in any written procedures.

4.12.5 Unit Four Air Pre-heater Pluggage 48,740 Mwh.

The following were interviewed - Layne Miller, Jack Stant, Pete Beltran and Mark Soloman

Problem Description

In September of 1999, Unit 4 was removed from service due to a blocked air pre-heater. The first indications of blockage were observed in late 1998 Plant instrumentation and alarms indicating blockage were either ignored or misdiagnosed by both operations and maintenance staff. The blockage continued unabated until the unit was force to be taken out of service. During this period two opportunities to inspect the air pre-heater and or check instrumentation were missed. An inspection of the air pre-heater revealed that the sootblower nozzles had broken off, rendering the sootblower system ineffective.


4-34


The E&CF chart in Figure 4-15 identifies a number of causal factors resulting in this incident.

• In mid-1998 no investigation was undertaken or initiated when plant personnel noticed that no opacity spikes occurred during air pre-heater sootblowing. This is a normal occurrence when sootblowing the air pre-heater in operation. An investigation would have lead to the realization that the air pre-heater was blocking up.

• In October of 1998, operations wrote a Work Order on high dP on Unit 4 air pre-heater B wheel. Maintenance diagnosed it as a transmitter problem.

• In February of 1999, operations wrote another Work Order on high dP on both A” and “B” wheels of Unit Four air pre-heater, specifying potential blockage of the air pre-heater. Again, the diagnosis was faulty transmitters.

• Plant engineers do not trend or review data on the pre-heater. Trending the dP would have highlighted the blockage earlier.

The air pre-heater soot blowing nozzles have been added to the unit’s inspection routine. It is also planned to install pre heater airside soot blowers. This will solve the mechanical problems identified in this root cause analysis. However, the habit of ignoring or not repairing instrumentation with known problems (see Figures 4-17 and 4-19) is a serious trend that needs to be addressed.

4.12.6 Unit Four Scrubber Problems 76,917 Mwh.

Those interviewed were Jack Stant and Pat Reynolds

Problem Description

Unit 4 stack SO2 levels were increasing to unacceptable levels. Examination of operating parameters showed low flue gas flow through the scrubbers. The high dP across the de-misters confirmed that some blockage was taking place. During the spring 1999 planned outage, Unit 4 scrubber was inspected to gather data to be used for analysis. In May of 1999 the unit was removed from service and extensive repairs and modifications made to the scrubber. In the spring of 2000, proper nozzles were installed on Unit 4. Post outage SO2 and dP are within limits.


Figure 4-16 represents the analysis completed by plant staff using the Behavior Justification Root Cause Analysis methodology. All three identified correctable opportunities were addressed during the outages.

• The de-misters were replaced with a type less prone to plugging.

• Correctly sized nozzles were installed.



4-35

• The control scheme for spraying was improved in order to give Operations control over spraying frequency and duration.

This analysis was simple, but effective in identifying the problems and solutions before shutting down the unit for repairs. Advantage was taken of opportunities to inspect the equipment for further information. The approach was well planned and coordinated. This is an example of how plant problems/incidents should be managed.

4.12.7 Unit Four High Opacity Shutdown 42,731

Those interviewed were: Jack Stant, Elbert Watts, Junior Ward, Layne Miller, Mark McKenzie, Pete Beltran and Mark Soloman

Problem Description

Due to a strike at the McKinley mine, coal was purchased from an alternate supplier. On 5 June 2000, 5,500 tons of this coal was bunkered to Unit 4. Unknown to plant staff, the coal was a 30% ash coal instead of the expected 22%. The fly ash removal system was unable to cope with the increased ash loading, and Transformer Rectifier (RTs) sets in the precipitator began to short out because of high ash hopper levels. Unit load was decreased to control opacity, but was ineffective and opacity reading continued to increase. On June 11 the unit was taken off load to prevent further violations of opacity limits.


Although this event was not in the original scope of work, it was included at the request of Cholla plant management. An E&CF chart was drawn. Figure 4-17 shows the charted sequence of events. Four causal factors were identified

• The coal quality directive as written did not address the ash content of the coal received. APS purchased coal from an alternate mine that had previously supplied coal. Ash content is not addressed in the specifications, however the mine informed APS that coal in storage had an ash content no greater than 22% in fact the coal contained 30 % ash.

• The practice of verifying ash hopper empty was no longer done. It was customary for operators to check hopper level prior to starting the emptying sequence. This practice had not been passed onto the new operators and consequently did not get done on a routine basis.

• Hopper level indication was inoperative. The level indicator were considered unreliable and therefore were neglected.

• Preventive maintenance on the “Nuva” Feeders had been suspended and as a result numerous failure occurred simultaneously, which could not be kept up with.

4.12.8 Unit Two Waterwall Tubes Hydrogen Damage 188,944 Mwh.

Those interviewed were: Dan Nass and Kim Belknap


4-36

Problem Description

In the early nineties several poor quality water excursions, caused by condenser tube leaks, occurred on Unit 2. This, coupled with a lack of chemical cleaning, resulted in internal tube scaling - hydrogen damage. In 1999 fifteen tube failures occurred, of which, eleven were identified as ‘Hydrogen Damage’.


Figure 4-18 represents the root cause analysis completed using the Behavior Justification methodology.

• The unit was operated with high chlorides, which entered the boiler via condenser leaks.

• Poor control of the demineralizer regeneration process contaminated the makeup water with acid, reducing boiler pH. The boiler was operated for hours with a pH of seven.

• The practice of draining or chemically cleaning the boiler following severe pH excursions was not followed.

• The unit operated with high dissolved oxygen, producing high levels of corrosion products in the feedwater system, which eventually deposited on waterwall tubes.

• The chemical cleaning in 1997 was the first in eleven years.

The effects of the poor chemical control in the early nineties was felt in 1996 when the unit began experiencing tube failures due to hydrogen damage. Appropriate action has and is being taken by the plant staff. The condenser has been replaced. Makeup water supply has been contracted out and is of a good quality. EPRI guidelines are followed and the unit chemistry has been improved. Waterwall panels have been replaced. Management has placed a high priority on maintaining an effective boiler water quality control program.

4.12.9 Unit Two Cable Tray Fire 28,899 Mwh.

Those interviewed were: Jack Stant, Pete Beltran and Walt McMillan

Problem Description

In March of 1999, scaffolding boards lying on top of a cable tray located above a pulverizer hot air duct ignited. The fire melted the leads to the Unit 2 service water pumps, which supply cooling water to Units 2 and 3 soot blower air compressors. The air compressors had not tripped in spite of high temperatures and were manually removed from service by the Shift Supervisor to prevent further damage. The Shift Supervisor ordered Unit 2 to be shut down and load on Units 3 and 4 to be reduced. While lowering load on Unit 4, it tripped off line due high furnace pressure resulting from damper-control low air pressure. Extensive damage was done to Unit 2 soot blower air compressor because the wires on the high temperature trip were shorted out and not repaired since replacement was being considered.



4-37


The Behavior Justification analysis – figure 4-19 performed by plant staff in 1999 was reviewed. The E&CF charting method – figure 4-20 was also used as a comparison. Both analyses are lengthy and contain a number of causal factors. The following discussion will be based on the E&CF chart analysis.

• Scaffolding boards had been left on top of the pulverizer hot air duct for an unknown period of time. The duct conveys combustion air at a temperature of 500 degrees F to the pulverizers. The location is near the coal transfer point and coal dust settles onto the ductwork. The area is enclosed.

• The fire melted the leads to both service water pumps, which led to loss of cooling water to the soot blowing air compressors (SBAC). Unit 2 service water pumps provide cooling water to both Units 2 and 3, as the Unit 3 pumps had been abandoned for ten years.

• The Unit 4 Shift Supervisor discovered the loss of service water cooling pumps, as the supervisor for Units 1,2, &3 had assumed the role of Fire Commander. He immediately ordered the shut down of Unit 2.

• The Supervisor then manually took the Unit 2 and 3 SBACs off line. They had not tripped on high temperature since the high temperature trip wires had shorted and not been repaired.

• An alternate source of cooling water, using the fire water system, could not be located because the operators were not familiar with the plant layout.

• Unit 1’s soot blowing air system was isolated and load reduced on Units 3 and 4 to compensate for loss of soot blowing air. Unit 4 SBAC could not be used as it was down for repairs.

• The Assistant Operator (AO) attempted to isolate the Unit 4 air systems but was unable to do so, as there was no procedure for him to follow, and he was inexperienced. The AO was manually closing 4B ID Fan damper, but 4A damper had insufficient instrument air pressure to follow. The resulting high furnace pressure tripped the unit.

An analysis of this incident reveals that if the scaffold plank had not been left on the hot duct, this incident would not have occurred. Therefore the primary root cause of this incident is poor housekeeping and safety culture. Eighteen months prior to this incident a similar fire, involving scaffold boards, had occurred on Unit 4. The other seven causal factors identified contribute to the loss of Unit 4 and the extensive damage incurred by the air compressors. As a result of the BJ analysis performed in 1999, an action plan was put together and is reviewed below.

• The shorted wires on the air compressors were immediately repaired. The trip system was replaced.

• The Unit 3 service water pump was returned to service. However, the pump supply is from the Unit 3 cooling tower and is too high a temperature for use as a cooling water source for the air compressors. The project is now with engineering.

• Perform a monthly ‘ combustibles’ inspection of the units. This was done monthly in the second half of 1999, but has only been done once in 2000.


4-38

• Write a procedure for isolating Unit 4 from Units 1,2, & 3. A rough draft was written in March of 1999. As of June 2000, no further action has been taken.

• The Unit 4 ID fan damper drives have been replaced with Jordan drives, and no longer depends on instrument air.

• In order to ensure a 30-day turn-around on sootblower air compressor overhauls, rebuild kits have been purchased.

4.12.10 Unit Four Air Pre-heater Guide Bearing Fire 6,099 Mwh.

Those interviewed were: Jack Stant, Dan Hyde, Pete Beltran, Walt McMillan and Bryant Peterson

Problem Description

Unit 4 had been on line 27 hours following a unit shutdown to perform an air heater wash when a high temperature alarm on the 4A Air Pre-heater guide bearing in the control room was received. The AO who investigated the alarm reported a fire and the fire alarm sounded. The Shift supervisor and AO attempted, five times, to extinguish the flames. At this point, the SS ordered that the shut down of the unit. The Joseph City fire department arrived on site but could not make radio contact with the Supervisor. Members of the station fire team arrived and put the fire out. The bearing was visually inspected and found fit for purpose. The unit was returned to service.

Root cause Analysis Chart

Figure 4-21 shows the E&CF chart used to analyze this incident.

• The high bearing temperature alarms, which prompted the AO to inspect the guide bearing, was the result of fire in the vicinity to the thermocouple.

• The area around both A and B guide bearings is packed with about ten inches of flyash, which on its own, is not considered to be a fire hazard. Over a period of time, the flyash had become soaked with bearing oil from oil spillage and leaks.

Three conditions are necessary for a fire.

• Fuel -The oil, which had soaked the flyash, had a flash point of 440 degrees F.

• Temperature- the air passing through is 560-580 degrees F.

• Oxygen- The SS reported that the flames were coming from up under the guide bearing, which was covered with flyash. Only when the nozzle of the fire extinguisher was inserted all the way into the flyash was the fire put out.

A plant walk down by the review team revealed the exact same conditions, which had led to the fire, still existed. In an attempt to determine the temperature of the flyash, it was discovered that



4-39

the bearing seal had failed and hot gas was being blown out, and was providing the heat as well as the oxygen source.

Approximately fifteen feet from the bearing was a drum of oil, two buckets containing oil, oil soaked rag and gloves. As discussed in incident 4.9, poor housekeeping is the root cause of this incident. The fact that the exact same conditions still existed, a year later, shows a lack of follow through/follow-up by supervisory staff and the Fire Chief as they were given that responsibility.

4.13 Findings

• At each of the stations, the review team consisted of the two EPRI employees and a staff member from the other station involved in this exercise. This was a very positive aspect of the review as experiences and problem solutions at both stations were shared. The opinions/ideas of peers were welcomed.

• Root Cause Analysis is done by exception, i.e. as and when requested by management. Based on the information provided, Cholla management requested more analyses than Four Corners. These are undertaken by the Operation Technical Advisor – Jack Stant

• Many staff members were trained in the Root Cause Analyses technique taught by Bobby Jones and Associates – Behavioral Justification, but they don’t regularly use any formal analysis technique.

• Root Cause analysis, commonly, consists of group brainstorming sessions, where solutions (often short-term) are formulated and actions allocated.

• Problem solutions are not well documented and in some case not documented at all.

• Allocated actions/solutions to brain storming analysis are rarely followed up

• There was very little documentation available from which the review team could gather information. Therefore, in an attempt to reconstruct the incidents key information had to be obtained from staff members who had knowledge thereof. E&CF charts were constructed from this information and causal factor determined.

• Of the ten lost generation incidents reviewed, three were adequately documented using the Behavioral Justification technique.

• In more than 50 % of the reviewed incidents, it was found that staff chose to rather replace/fix/modify equipment or plant, instead of assessing root causes.

• In almost all the incidents reviewed /reconstructed, the “fix” – repair/replace or modification did align with those recorded in the 1999 Lost Generation Action Plan.

• Preventive/Predictive Maintenance schedules/activities are not implemented, as staff are focused on corrective maintenance, as in crisis management.

• There is a lack of problem ownership, as responsibility for resolution of problems is not specifically allocated, e.g. Cable tray/duct fires.

• Project kick-off and exit meetings were held at both plants. At Cholla the management team showed a keen interest and participated.


4-40

4.14 Conclusions

The following conclusions have been derived from the Findings above:

4.14.1 Formal documented Root Cause Analysis is not being done on all major incidents. The most common reason for not doing a root cause was, ”we are too busy putting out fires”. This made it difficult to verify the actions identified in the 1999 lost Generation Action Plan.

4.14.2 Although no formal analysis is being done, the actions (solutions) identified in the 1999 Lost Generation Action Plans are considered adequate short-term solutions. Long term solutions, come from an in-depth root cause analysis of the incident and/or problem.

4.14.3 Knowledge of a Root Cause Analysis process was limited even though staff had been trained in an analysis technique.

4.15 Recommendations

4.15.1 A formal structured Root Cause Analysis process needs to be implemented. This process should be developed based on a corporate directive requiring those major incidents (greater than a $X, 000 – value to be determined by Corporate), unit trips and a significant unexpected maintenance expenditure be investigated to ensure repeat failures/incidents/trips do not occur.

4.15.2 Staff should be trained/retrained in a Root Cause Analysis technique that includes Human Performance aspects.

4.15.3 The various recommendations identified in Section 4.4 – Root Cause Analysis should be reviewed, evaluated and implemented.

4.16 Appendix 2

Root Cause Analysis Charts (Excel file)

Note:

1. A number of the charts are incomplete as there was limited documentation and the allocated time per incident was constrained. However they do reflect the information given or presented by the various interviewees.

2. The charts need to be read in conjunction with the text in the main body of the report



4-41

C.O. N.C.

C.O. Condenser being replaced with S.S. 29-4-C

C.O. C.O.

Unit 3HP Turbine NozzleCopper Deposition

Copper inmain steam

Copper incondensate

Coppermetallurgy in

condenser

Coppermetallurgy inHP feedwater

heaters

Copper infeedwater

Coppermetallurgy inLP feedwater

heaters

Switched fromhydraz ine toEliminox in 93

Returned tohydraz ine in early 2000

Copper levelsin Econ. Inlet

reduced

Drum pressure2250-2300 PSI

Figure 4-11 Unit One, Mechanical Dust Collection Retrofit, 41,146 Mwhs


4-42

*

Repeated Booster ID

Fan Bearing Failures

Bearing overheats.

Cooling water displaces

lube oil

Cooling waterenters lube oil

reservoir

Cooling watersupply nipple

cracks

Fatigue No hi-hibearing

temperature trip

Rigiddesign

Housingworn

No analys isof failed nipple

High FanVibration

Fan inbalance due to flyash

erosion

Figure 4-12 Unit One, ID Booster Fan Bearing Failures, 7,544 Mwhs



4-43

1998

*

*

*

Unit 1 TurbineThrust Trips.

Thrust exceededtrip set point

Turbine thrustmeasured by

cold static methods

Trip setpointdetermined by

cold staticmeasurements

Unit came upon load

Turbine thrustgoes increasingly

negative with increased load.

Turbine expands with increased

temperature

New thrustprobes

installed

Probescalibrated

based on pastexperience

Lack ofexperience

Lack ofdata

Turbine overhaul component

replacement

Turbine has more thrust

range than otherCholla units

Different thrust

locatingmechanism

Meter rangelimited to maintain

plant standards

No capabilityto adjust probes

on line

Repeated tripsdue to high

thrust

W orn thrustbearing replaced

Probesreplaced

Figure 4-13 Unit One Turbine Thrust Trips, 9,971 Mwhs


4-44

6/18 6/18

*

6/21 7/6-7/8 7/10 7/11

* *

Unit 2 A Booster

ID Fan MotorShorted Out

Elec. Eng. Doble tested2A Boosterfan motor

Operator attempts to start fans for

balance check

Electric ianmeggered themotor & cable

Fan started forbalance checks

Motorchecked out OK

Center phasecable Dolby

tested marginal

Used fourdifferent elects.

for the tests

Fan tripped on instantaneous

ground

Operator againattempted tostart the fan

Fan trippedon overload

Motor checkedout OK

Megger testindicates open

phase

Balancingchecked out

OK

Fan placedin service, ran

for 17 days

Fan inboard bearing vibration

increases

No investigationto determine cause

ProductionSupv.

investigates

Electrical foremanremembers that

cable had been disconnected

Fan verified c leanafter June outage

Vibrationlevels

acceptable

Fan has repeatedstarts for washing

and balancing

InboardHorizontal 3.5

Vertical 2.5

Fan started6 times, no

procedures or guidelines DATE START STOP ELAPSED

TIME---------------------------------------------------------------------7/10 19:20 19:24 19:54 20:05 34 M in. 20:24 21:11 30 21:22 21:31 58 23:07 23:08 1057/11 00:06 00:06 59 Tripped

Outboard remained

constant at 1.9

Contract electrician

fails to reconnectcables

A

A

No detailreadings/analysis

done

First washing had negligible

effect onvibration levels

Unit down for scheduled

outage

Cable disconnected at

breaker cabinet forDolby testing

Planner notnotified of change

in work scope

Flags were downand reset

Unit 2 shut down

for boiler tuberepair

Decis ion madeto wash fans

Decis ion based on assumedfouling of fan

blades

Figure 4-14 Unit Two, A Booster ID Fan Motor Failure, 30,464 Mwhs



4-45

Oct-98 Feb-99 Mar-99 A pr-99

*

**

**

*

May-99

*

Operator ob-serves high dP on "B" wheel.

(Alarm?)

Operator ob-serves high dPon "A" & "B"

wheels.

Cable trayfire outage40 hours

Unit Outagefor scrubber

repairs 192 hrs.

Unit loadcurtailed- ran

out of fan

Opacity spikescaused by soot

blowing stop

Instrument shop verifiestransmitteraccuracy

Unit Four A irPreheaterplugged

9/99

Operator ob-serves high dP

on A & Bwheels.

W .O. W ritten

C429475

W .O. W ritten

C432684Fowarded toEngineering

W .O. W rittenC434285

"Not inst. problem,change alarm setpoints"

Discussed atmorning meeting

Operationstrouble shooting

problem

Missedopportunity

Missedopportunity

A

Unit Outageto wash wheels

Found APH Soot blowing nozzle

fallen off

W .O.specifiedpluggage

Maintenance diagnosedincorrectly

Maintenance diagnosedincorrectly

Observedby plant

personnel

No W orkOrder written

APH dP data notreviewed or trended

Maintenance diagnosed incorrectly

A

Figure 4-15 Unit Four, Air Preheater Pluggage, 48,740 Mwh


4-46

"A"

C.O. See "A"

C.O. I&SC.O. C.O. I&S

C.O. S

Unit 4 stackSO2 is too

high

Low flue gasflow through

scrubber

Plugged demis ters

reduc ing flow

Too muchcarryover

from lowerdemis ters

Incorrectnozzles

Spray frequency

is incorrect

Lower demis tersare plugged

Spray durationis too short

Spraying isineffec tive

Operator can adjust to as low as one minute

Upper demis tersare plugged

Spraying isineffec tive

Incorrectnozzles

Figure 4-16 Unit Four Scrubber Outage, 76,917 Mwh



4-47

5/31 6/3 6/5 6/6 6/7

*

* *

*

6/8 6/8 6/8 6/11 6/11

6/16 6/3 6/5 6/6 6/7

5,500 tons of30% ash coalbunkered to

Unit 4

Opac ity baseline rise

continues

High hopperindications on

"B" side

Operations uninformedof high ash

TR's trippingon A&B side

Elect.'s workingon "A" s ide high

hopperindications

A/O's beginto manuallypull hoppers

Unit hasopacity

exceedences

Coal was30% ash

"A" sidealarms not

working

Obsolete, partsnot available

Coal purchasedfrom alternate

supplier

Analys is notreceived untilafter loading.

5,000 tons/daytypical burn

Notspiking

P.M.'s notperformed

High levelin flyashhoppers

Decis ion made to shut unit

down

Unit outagedue to high

opacity

Nuvafeeders not adequately

maintained

Did not meetspecs

Practice ofhopper emptyverification no longer done.

A

A

Operators did notrealize the s ituation

or consequencesthey were fac ing

Dumped A18, 17,&14, and B18

hoppers to ground

Unit on line

after tubeleak

Opacitytrending

upward. 7%

Un-noticed by operators

Load isdropped

Operatorsverify ing A side

hoppers

27% opac ity

rule used

Operators unaware level alarm inoperative

Hopper level indication

is unreliable

Coal quality directive did not address

ash content

McKinley mine

on strike

Internal direc tivebeing rewritten

Training

Do PM's

Repair

No load reduction

cons idered

Hopperinspections

"A" s ide hadmore

damage

Prec ip not inspected

during outage

Figure 4-17 Unit Four, High Capacity Shutdown, 42,731 Mwh


4-48

M S S

Chloride contaminationin boiler water

Operating with

condenser leaks

Contaminates in

boiler makeup water

Heavy scale depositson boiler

waterwalls

Lack of properchemical c leaning

Lack of chemical

cleaning to removescale

Samples taken fromnon-

representative area

Did not boroscopewater wall

tubes

Non-complianceto procedure

No procedurein place

Impurities infeedwatersystem

Operating withcondenser tube

leaks

Corros ion infeedwatersystem

Impurities inmakeup

water

PoorpH

control

Operatingwith highdissolvedoxygen

Flowassistedcorrosion

Demineralizerregenerationmalfunction

Figure 4-18 Unit Two Waterwall Tubes Hydrogen Damage, 188,944 Mwh



4-49

*

*

*

*

* * *

**

*

Coal dustaccumulates

on hot airduct

Unknownmaintenanceactivityperformed

AO reportsfire

Fire brigaderesponds

S.S. ordersUnit 2

shutdown

S.S. shutsdown U-2

SBAC

S.S. orders U-1 SBA

isolated & 1ASBAC started

S.S. shutsdown U-3

SBAC

S.S. ordersU-3 to drop

load

S.S. ordersU-4 to drop

load

AO attemptsto isolate

U-4 airsystems

AO closes4B ID Fan

damper

Unit 4trips

Poor dustcontrol ontransfer points

500 degreegas temp

Planklef t onduct

Fire inU2 bottomash enc l.

Pow er f eeds to

SW pumpsgrounded

Hotenough to

smell

Cooled byserv ice w ater

High temptrip w iresshorted

High temptrip w ires

not repaired

Hotenough to

smell

Cooled byserv ice w ater

High temptrip w iresshorted

Nosoot blow ing

air

Nosoot blow ing

airNot

successful

AO newto unit

Noprocedures

3 to 5percent

High furnacepressure

Inst. A ir pressuretoo low to open4 A ID Fan damper

SW pumpdischargepressureat zero

Fire loopcooling H2Osupply notidentif ied

Similar incidenton U4 earlier(18 Months?)

Fire loop cooling H2O supply not identif ied

U-3 SW pumps

abandoned

U-4 SBA Cout of

serv ice

U-4 SBACout of

service

Loss of28,899

MW hours

Investigated?

A

A

Figure 4-19 Unit Two Cable Tray Fire, 28,899 Mwh


4-50

C.O. M1,2,3,4

Tw o units of f andone curtailed

U-3 curtailed

Not able to blow soot

Loss of adequatesootblow ing air to

plant

Units 2,3,&4SBA C's not running

U-4 SBAC notrunning

SBA C shut dow nfor maintenance

See prev iousBobby Jones

U-3 SBCA notrunning

Same as U-2 SBA C

U-2 SBA C not running

SS shut of f

High temperaturetrips did not w ork

Bearing temperatureover the high

temperature tr ip poimt

Los t cooling w aterto SBA C

U-2 serv ice w aterpumps not running

No pow er f eedsto pumps

Both motors pow erleads in tray melted

to ground.

Cable tray f ire

Wires shorted Did not repair U-3 serv ice w aterpumps not

running

Pumps shutdow n for 10

years

U-2 shut dow n due

to saf ety

Unknow ndamage due tocable tray f ire

Unit 4 tripped of f

High f urnacepressure

Operator low ered B

ID damper 3-5%i h d

Operator w aslow ering load

SS instructedoperator to low er

load

Not able toblow soot

Units 2,3, &4SBA C's not

running

A damper w ould not raise in A uto

A ir pressure w as at

or below 50 lbs.

Units 2,3,&4SBA C's not

running

Tie valvebetw een 2/3 and

4 w as open

Operator did not c lose tie valve

* Cont. in upper left

* Cont. in upper left

ID

ID CO M1,2,3,4

CO M1,2,3,4

ID

Figure 4-20 Unit Two Cable Tray Fire March 1999, 28,899 Mwh



4-51

0:18 9/24 14:46 27-Sep 17:48 28-Sep 17:56 28-Sep

*

*

*

1:18 9/25 15:46 28-Sep 18:48 29-Sep 18:56 29-Sep

*

*

*

U-4 shutdown forAH wash

U-4 on linefor 27 Hrs

Hi Tempalarm on4A guidebearing

AO confirmedreports fire atguide bearing

SS & AO could not put

out fire

SS takesunit offline

SS callsJoe c ityfire dept

6099 MW

Hours lost

SS & AOput out

fire

Maintenancechecks guide

bearing

Unit backon line

HighdP

No work done on guide bearing

Normalstart up Temp

spiked

Fire damagedthermocouple

Oil soakedflyash packedaround housingappox. 10" deep

Temp. meas.@ 420 plus F

above seals on A & B

Oil Flash point 440 F

Air seal leak ing

on A side

Air outtemp 560-

580 F

SS tries fivetimes to putout fire

Flamesemanating

from bearinghousing

Could notmake radio

content

Joe Cityfire brigade

too late

$700 labor charged to

W O

Firedamaged

thermocouple

Guide bearinginspected andfound to be OK

4A guidebearing temp

in alarm

Air sealleak ing on A guide bearing

A

Figure 4-21 Unit Four Air Preheater Guide Bearing Fire, 6,099 Mwh


4-52

Problem Description Root Cause MW/Hrs. Lost Unit 1 Mechanical Flyash Collector 4.1 Unit 1 ID Booster Fan Bearing Failure 4.2 Unit 1 Turbine Thrust Trips 4.3 9971 Unit 2 A ID Booster Fan Motor Failure 4.4 30,464 Unit 4 Air Preheater Pluggage 4.5 Unit 4 Scrubber Outage 4.6 76,917 Unit 4 High Opacity Shutdown 4.7 Unit 2 Waterwall Hydrogen Damage 4.8 188,944 Unit 2 Cable Tray Fire 4.9 28,899 Unit 4 Air Preheater Guide Bearing Fire 4.10 6,099


5-1

5 BOILER RELIABILITY OPTIMIZATION PROJECT PROGRAM AUDIT AT HAWAIIAN ELECTRIC COMPANY

5.1 EXECUTIVE SUMMARY

As a result of Hawaiian Electric Corporation embarking on a Remaining Useful Life and Generation Asset Management program in 1999, and including in the project the Boiler Reliability Optimization Program, EPRI was requested to provide an independent review of this area of the project.

During week 18 June 2001, a series of interviews and meetings were held with a number of HECO personnel and a number of documents were reviewed. The EPRI team found that progress had been made in successfully implementing a number of planned activities. The team also identified a number of opportunities where the current efforts could be enhanced and which, when implemented, would give additional assurance that the goals set for availability and reliability would be achieved. These opportunities are highlighted in the main portion of the report.

5.2 Introduction

In 1999 Hawaiian Electric Company Inc. (HECO) embarked on a Remaining Useful Life and Generation Asset Management program, to enable all the HECO generating plant to continue to operate well beyond the 20-year horizon. One of the projects in this program was the Boiler Reliability Optimization Program. The objectives of the Boiler Reliability Optimization Program are:

• To reduce short- and long-term boiler maintenance costs, while maintaining or improving overall boiler reliability.

• To develop and implement boiler maintenance strategies using EPRI’s Streamlined Reliability Centered Maintenance process.

The project was started in the last quarter of 1999 with a Boiler Tube Reduction and Cycle Chemistry Improvement training program, then continued in 2000 with a pilot Streamlined Reliability Centered Maintenance project, which is currently being implemented. Progress has been made in implementing many of the actions, along with many other initiatives as part of the overall RUL/GAM program.

The program had reached a point where it was expeditious to have an independent review of the current project status and plan to give assurance that the goals set would be reached. EPRI was

EPRI Licensed Material Boiler Reliability Optimization Project Program Audit at Hawaiian Electric Company

5-2

requested to undertake such a review. The review would give management the assurance that the overall RUL/GAM program and Boiler Reliability Optimization project goals will be met.

The specific objectives of this review are:

• To review the current Boiler Reliability project action plan

• To identify cost effective improvements; and

• To recommend changes in their action plan to address the identified gaps.

5.3 Findings

5.3.1 Boiler Reliability Optimization Project Plan

• There are basic plans, outlined in the PDM Team Strategic Plan 2001 and the Boiler Reliability Optimization Program – Envisioned Role of BRO Team Members, but they differ. The plans are fairly loosely structured and therefore lack focus, giving little direction for the development of specific action plans. There is no overall strategy with key deliverables and dates.

• Roles and responsibilities of the key players are not clearly defined. This results in misunderstandings and confusion about who should be doing what and when. This impacts the cost effectiveness of this program.

• An engineer was specifically employed to assist with the implementation of the Boiler Reliability Optimization Program. His energies have been redirected into solving a variety of other technical problems, and he has, therefore, been unable to devote the necessary time and attention to his primary role. This has occurred because of the above-mentioned lack of direction regarding roles and responsibilities.

• Boiler performance goals – tube leaks – are outlined in the Boiler Tube Failure Reduction/Cycle Chemistry Improvement Program Management Mandate. Short-term goals, as outlined, have only been partially achieved. Long term goals may be achieved but are aggressive.

• The Boiler Reliability Optimization Strategic Plan that is being followed (shown in Appendix 1), shows that a number of deadlines have not been met, or have been extended. The plan has been filled out with the relevant information regarding the current progress and updated deadlines.

5.3.2 Project Activities

5.3.2.1 BTFR/CCI Program

• A BTFR/CCI Improvement Program Management Mandate was issued in December 1999, after the BTFR/CCI training was completed. The mandate outlines the ground rules and


Boiler Reliability Optimization Project Program Audit at Hawaiian Electric Company

5-3

guidelines for the implementation of a Boiler Tube Failure Reduction Program. This gives direction and focus to the efforts of the core Boiler Tube Failure Reduction team.

• This Management Mandate has an annual review cycle. To date this document has not been reviewed and/or revised.

• Some of the short- and long-term goals outlined in the Mandate have been achieved.

• As a result of the Mandate being issued, a comprehensive, standardized Boiler Tube Failure report format was developed, and is being followed.

• The Boiler Maintenance Workstation (BMW) program, which is a Windows-based software program used to track and trend boiler tube failure data and information, was purchased some time ago. Software compatibility problems have now been solved and the database is being populated with drawings and tube leak data. Initial training on this program was given to a selected group some years ago, and no other employees have been trained since, so knowledge of its true benefits and uses is restricted.

• In excess of $3million has been budgeted for the procurement and installation of core water and steam chemistry instrumentation. Phase 1, consisting of 3 instruments per boiler is to be installed by the end of August 2001 for Kahe units. Installations of phase 1 instruments are scheduled for completion by 12/31/01 and 3/31/02 for Waiau and Honolulu respectively. This will enhance the control of water chemistry, which will minimize boiler tube leaks that result from poor water chemistry, improving reliability.

5.3.2.2 Streamlined Reliability Centered Maintenance

• A pilot project on three systems – Boiler Air and Gas, condensate and Feedwater – at two sites, has been successfully completed. The resultant maintenance tasks have been accepted and are being integrated into the work order management system for Kahe 3. This provides a basis for cost effective maintenance.

• There is no overall preventive/predictive maintenance basis for the boiler pressure parts, as in tubes. Repairs are carried out as and when failure occurs. This strategy has an impact on the overall reliability and maintenance costs.

5.3.2.3 Root Cause Analysis

• Training in basic Root Cause Analysis was done some time ago, but has not been formally used in practice besides investigation involving boiler tube failures. Problem solving is done on an ad hoc basis and repeat failures are occurring, causing problems with reliability and impacting costs.

• The importance of Root Cause Analysis being integrated in the work management process is not being recognized. It is the most effective means of ensuring improvements in reliability and saving costs, by preventing/minimizing failures from occurring or recurring.


5-4

5.3.3 Condition based data collection

• A hardware breakdown structure does not exist for the boiler i.e. uniquely identifying the system, sub-systems and components within the boiler. Therefore tracking the condition and or performance of each system, sub-system is difficult.

• Condition based data is collected on some sub-systems and components. However the data collection-analysis-action process is no consistent across all boiler components. This makes it difficult to determine the true condition of each sub-system or component and prioritize the inspection and repair work on the boiler.

5.4 Conclusions

In an effort to provide assurance that HECO’s current initiatives in the Boiler Reliability Optimization project are adequate and cost effective, reviewers from EPRI studied the given findings and concluded that:

5.4.1 The Boiler Reliability Optimization Team and others associated with reliability improvements have made a significant effort to implement the boiler tasks associated the RUL/GAM project. All involved need to be congratulated on their efforts and having the courage to request that a “third party” review their program.

5.4.2 Basic project management principals of time, cost and quality have not been followed in the development of an adequate program and schedule.

5.4.3 Some project planning has been done, but it is ineffective. The effectiveness of the team’s efforts is impaired because of the lack of clarity in the plan on roles and responsibilities, direction, objectives and deadlines.

5.4.4 There is a lot of emphasis and effort being focused on the Boiler Tube Failure Reduction program and the program is achieving its objectives.

5.4.5 The current efforts do not provide for an adequate maintenance strategy regarding boiler tubes.

5.4.6 As root cause analysis has not been rigorously practiced since the training sometime ago, it is doubtful that a significant proportion of that knowledge and skill gained is available.

5.4.7 Significant financial resources have been allocated to install state of the art water and steam chemistry instrumentation however little resources have been set aside for the on going maintenance of these newly installed instruments, some of which will be installed shortly

5.5 RECOMMENDATIONS

The following recommendations are given to enhance the current Boiler Reliability Optimization initiatives.



5-5

5.5.1 A project leader should be designated in writing, whose prime responsibility it would be to oversee the implementation of this project.

5.5.2 Develop a comprehensive plan/schedule for the implementation of all boiler-related activities. This plan should identify:

– The people responsible for the various activities identified in the program, and should clearly define their roles.

– What these activities are – as in scope

– The duration of each activity.

– The resource loading required to implement these activities

A typical Microsoft Project plan would provide this framework.

5.5.3 The Management Mandate and various other programs mandates/directives (i.e. Root cause Failure, Streamlined RCM etc) should be reviewed and updated, according to the given schedule, as they serve as the bases for Boiler Reliability Optimization. A two yearly review cycle is considered to be the norm.

5.5.4 In order to compliment the current BTFR initiatives, it is recommended that a Failure Modes and Effects Analysis (FMEA) be done on boiler pressure parts, as a means of identifying potential failure mechanisms, and the financial effects of their occurrence. This analysis will also focus the organization on what the most significant failures are, so that a quantum leap in reliability can be made with fewer organizational resources being utilized. FMEA is an engineering technique used to define, identify and eliminate known and/or potential failures, problems, etc. on a system and/or components within that system. Appendix 2 gives an example of an FMEA analysis.

5.5.5 Continue with the current SRCM initiatives on the remaining systems, i.e. to develop a cost effective maintenance program, based on system functionality that will enhance system reliability. This will make optimum use of available maintenance resources and provide a documented base for future additions/revisions to the maintenance program.

5.5.6 Develop a hardware breakdown structure for each boiler. The structure will allow for each system, sub-system and component to be uniquely identified. This numbering system can be used in MIMS, Plant View and BMW thus enabling the condition of each system, sub-system and component to be tracked and enabling cost effective run-repair-replace decisions to be made.

5.5.7 Clearly define and document the roles and responsibilities of the individuals whose responsibility is to continuously monitor the overall condition of each boiler system, sub-system and component and to develop a short and long term inspection plan for each boiler and that of managing the Boiler Reliability Optimization Project.

5.5.8 Training in a suitable Root Cause Analysis technique/process be given to a small number of individuals. It is suggested that a number of key individuals be trained to become “trainers” of a suitable process that meets the needs of HECO. These individuals can


5-6

then develop a Root Cause Analysis Awareness course that can then be given to engineers, technicians, operators and craftsperson.

5.5.9 An administrative mandate/policy/procedure to be drawn up to: standardize what gets investigated; outlines the investigation standards and processes to be followed; the roles and responsibilities of the various team members and the reporting requirements. See appendix 3 for a typical list of headings that appear in a procedure.

5.5.10 A maintenance strategy be developed and documented in the CMMS system for the new cycle chemistry instrumentation being installed. The information required to develop the maintenance requirements can be sort from in house expertise, the OEM and or EPRI.



5-7

Appendix 1: Boiler Reliability Optimization Strorage Plan

HECO BRO IMPLEMTATION PLAN AS PER STRATEGIC PLAN Page 1

No. Activity By Whom Goal By When Progress to date

Revised completion date

1 Formalize and Organize Unified BRO Team and Procedures

Bill and Bob Isler ♦ Issue BTFR Procedure Manual

♦ Issue CCI Procedures Manual

1Q01 3Q01

Draft 90% complete

Final Draft 08/31/01 Finalize and issue 12/31/01

2 Establish Formal Use of BTFR and RCFA Process for Boiler Tube Failures

Bill, supported by BRO Team

Fully Implement BTFR and RCFA Process

2Q01 Completed and established 08/16/01

3 Formalize Data Collection, Management and Distribution of Boiler Information

Bill, supported by PDM specialist, Ed Chang

Develop Process Map on Data Collection and Storage

3Q01 90% established Not yet formalized

Predict completion by 12/31/01

4 Assist with Development of Boiler Start Up Procedures and Combustion and Flame Management Guidelines

Jose, Lane, Ed, Bill, Barney and Russell

Completed Procedures in place

3Q01 No progress to date. Still waiting for schedule guidelines from Operations

5 Develop Air Pre-Heater Maintenance Program

Bill and Larry ♦ Develop Condition Baseline for all Air Pre-Heaters.

♦ Plan in place.

4Q01

Approximately 40% complete on baseline history.

Predict completion by 12/31/01.

6 Develop Feed-water Heat Exchanger Maintenance Program

Bill McCraw and PDM Specialist

♦ Develop Condition Baseline for All Heat Exchangers.

♦ Select Mapping software.

2Q01

Progress is subject to unit outages

Predict selection by 09/31/01.


5-8




7 Develop and Implement Lay-up Procedures for Feedwater Headers

Bill, Barney and Russell

Refine and Implement existing Procedures

2Q01 Approximately 60% complete.

Predict Waiau Honolulu by 12/31/01.

8 Develop and Implement Lay-up Procedures for Boilers

Bill, Ed, Barney and Russell

Issue Procedures for the Cycling of Units

4Q01 Approximately 40% complete.

Predict Waiau Honolulu by 12/31/01.

9 Install and Implement Nitrogen Blanketing Systems and all Feedwater Heaters

Bill and Barney Started 10/00 – Complete 4Q02 Waiau by 03/31/02.

.

10 Optimize Boiler Chemical Cleaning Frequency, using Condition Based Techniques. Oversee the Chemical Cleaning Process

Bill, Barney, Russell and Ed

Develop and have in place the programs and Procedures

4Q01 Practice in place.

Formalize program and procedures by 12/31/01.

11 Develop Strategy and Install all Core Cycle Chemistry Instrumentation

Bill, supported by Barney, Russell and P&E

♦ Install the 2000 increment

♦ Develop Strategy for implementation for remaining core instrumentation

2Q01 2Q01

Strategy is completed and included in 5year plan.

Phase I will be 85% complete by 12/31/01.

12 Develop Plan to Maintain Cycle Chemistry Instrumentation

Larry and Bill Plan to be in place 2Q01 Dormant – waiting for completion of Phase I CCI.



5-9




13 Evaluate Cycle Chemistry Procedures and Software

Bill, Bob Isler, Barney and Russell

Select system. ChemExpert will probably be chosen

3Q01 Dormant – waiting for EPRI Solutions to convert current software to Windows NT. (expected by 08/31/01).

Predict system selection by 12/31/01.

14 Train Operators in Cycle Chemistry following installation of instruments

Bill, Barney and Russell

♦ Develop Training Program

♦ Take the lead in Implementing Training

4Q01 1Q02

No progress to date. Doubtful will meet schedule completion.

15 Develop and Implement Air In-leakage Improvement Program

PDM Group, Barney and Russell

Implement Formal Program 2Q01 Practice in place. Formal program not yet written..

Predict 2Q02


5-10

Appendix 2

FMEA Worksheet

System: Heat recovery Sub-system: Economizer Component: Economizer inlet header

FMEA Process

Failure Modes/

Failure mechanism

Effects of failure

Sev

erit

y (1

– 1

0) Causes of Failure

Fre

qu

ency

(1

– 1

0) Detection

Method

Det

ectio

n

(1-

10)

RPN Recommended Action

Cracks on the internal diameter, parallel to the stub tube axis – Thermal fatigue

If detected during a periodic inspection, repairs would necessitate extending the outage. If, however the cracks went undetected failure would result in a forced outage with the consequent loss of revenue.

8 1. Start –up or shut down procedure not being followed

2 8 128 Conduct periodic reviews/audits of compliance to the start-up and or shut down procedures

8 2. No start-up or shut down procedure available or

4 8 256 Compile separately a start-up and shut procedure detailing the steps and precautions to be taken by the operator

8 3. Start-up and or shut down procedures is inadequate and highlights no precautions.

6 8 128 Conduct periodic reviews/audits of compliance to the start-up and or shut down procedures



5-11

Appendix 3

List of Typical headings that appear in a policy and or a procedure

1. Group name

2. Document type: policy, standard, guideline, procedure etc

3. Reference number

4. Revision number

5. Description of revisions made

6. Review date

7. Date authorized

8. Date when document came into effect

9. Compiled by

10. List of designations who have approved the document

11. Authorized by

12. Title

13. Background/preamble/introduction

14. Purpose

15. Scope of document

16. References

17. Definitions and or abbreviations

18. Responsibilities

19. Responsibility matrix

20. Requirements: step by step of what is required to be done

21. Work process flow and responsibility matrix

22. Distribution list

23. Appendices

© 2001 Electric Power Research Institute (EPRI), Inc.All rightsreserved. Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc.EPRI. ELECTRIFY THE WORLD is a service mark of the ElectricPower Research Institute, Inc.

Printed on recycled paper in the United States of America

1006537

Target:

Predictive Maintenance Program Development andDiagnostic Tools

EPRI • 3412 Hillview Avenue, Palo Alto, California 94304 • PO Box 10412, Palo Alto, California 94303 • USA800.313.3774 • 650.855.2121 • [email protected] • www.epri.com

About EPRI

EPRI creates science and technology solutions for

the global energy and energy services industry. U.S.

electric utilities established the Electric Power

Research Institute in 1973 as a nonprofit research

consortium for the benefit of utility members, their

customers, and society. Now known simply as EPRI,

the company provides a wide range of innovative

products and services to more than 1000 energy-

related organizations in 40 countries. EPRI’s

multidisciplinary team of scientists and engineers

draws on a worldwide network of technical and

business expertise to help solve today’s toughest

energy and environmental problems.

EPRI. Electrify the World

SINGLE USER LICENSE AGREEMENT

THIS IS A LEGALLY BINDING AGREEMENT BETWEEN YOU AND THE ELECTRIC POWER RESEARCHINSTITUTE, INC. (EPRI). PLEASE READ IT CAREFULLY BEFORE REMOVING THE WRAPPING MATERIAL.

BY OPENING THIS SEALED PACKAGE YOU ARE AGREEING TO THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TOTHE TERMS OF THIS AGREEMENT,PROMPTLY RETURN THE UNOPENED PACKAGE TO EPRI AND THE PURCHASE PRICE WILLBE REFUNDED.

1. GRANT OF LICENSEEPRI grants you the nonexclusive and nontransferable right during the term of this agreement to use this package only for your ownbenefit and the benefit of your organization.This means that the following may use this package: (I) your company (at any site ownedor operated by your company); (II) its subsidiaries or other related entities; and (III) a consultant to your company or related entities,if the consultant has entered into a contract agreeing not to disclose the package outside of its organization or to use the package forits own benefit or the benefit of any party other than your company.

This shrink-wrap license agreement is subordinate to the terms of the Master Utility License Agreement between most U.S. EPRImember utilities and EPRI.Any EPRI member utility that does not have a Master Utility License Agreement may get one on request.

2. COPYRIGHTThis package, including the information contained in it, is either licensed to EPRI or owned by EPRI and is protected by United Statesand international copyright laws.You may not, without the prior written permission of EPRI, reproduce, translate or modify thispackage, in any form, in whole or in part, or prepare any derivative work based on this package.

3. RESTRICTIONS You may not rent, lease, license, disclose or give this package to any person or organization, or use the information contained in thispackage, for the benefit of any third party or for any purpose other than as specified above unless such use is with the prior writtenpermission of EPRI.You agree to take all reasonable steps to prevent unauthorized disclosure or use of this package.Except as specifiedabove, this agreement does not grant you any right to patents, copyrights, trade secrets, trade names, trademarks or any otherintellectual property, rights or licenses in respect of this package.

4.TERM AND TERMINATION This license and this agreement are effective until terminated.You may terminate them at any time by destroying this package.EPRI hasthe right to terminate the license and this agreement immediately if you fail to comply with any term or condition of this agreement.Upon any termination you may destroy this package, but all obligations of nondisclosure will remain in effect.

5. DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIESNEITHER EPRI,ANY MEMBER OF EPRI,ANY COSPONSOR, NOR ANY PERSON OR ORGANIZATION ACTING ON BEHALFOF ANY OF THEM:

(A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USEOF ANY INFORMATION,APPARATUS, METHOD, PROCESS OR SIMILAR ITEM DISCLOSED IN THIS PACKAGE, INCLUDINGMERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON ORINTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY’S INTELLECTUAL PROPERTY, OR (III) THAT THISPACKAGE IS SUITABLE TO ANY PARTICULAR USER’S CIRCUMSTANCE; OR

(B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANYCONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OFSUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS PACKAGE OR ANY INFORMATION,APPARATUS,METHOD, PROCESS OR SIMILAR ITEM DISCLOSED IN THIS PACKAGE.

6. EXPORTThe laws and regulations of the United States restrict the export and re-export of any portion of this package, and you agree not toexport or re-export this package or any related technical data in any form without the appropriate United States and foreigngovernment approvals.

7. CHOICE OF LAW This agreement will be governed by the laws of the State of California as applied to transactions taking place entirely in Californiabetween California residents.

8. INTEGRATION You have read and understand this agreement, and acknowledge that it is the final, complete and exclusive agreement between youand EPRI concerning its subject matter, superseding any prior related understanding or agreement. No waiver, variation or differentterms of this agreement will be enforceable against EPRI unless EPRI gives its prior written consent, signed by an officer of EPRI.

boiler reliabiliti optimization 2001

Documents