Institutionalizing Alarm Management

How to improve safety, meet environmentaltargets and optimize operations while

increasing profitability

By: Jeff Gould, Matrikon Inc.

2

INTRODUCTION

Most plant personnel equate alarm management with reducing alarms, but this is just one of the techniques used to achieve effective alarm management. The true goal is more strategic: to give operators the right information to prevent abnormal situations and to prevent the escalation of those abnormal situations that cannot be avoided.

Poor alarm management practices result in billions of dollars lost every year to accidents, equipment damage, unplanned plant or unit outages, off-spec production, regulatory fines, and the huge, intangible costs related to environmental and safety infractions. As a result, it’s clear that implementing an alarm management solution is one of the top strategies for bringing hard benefits to any organization. This paper reviews the benefits of alarm management in general, the most common causes of alarm problems, and describes a proven approach that a variety of industry leaders have adopted in order to realize the benefits of proper alarm management practices.

The objectives of this document are to:

1. Review alarm management benefits

2. Provide a brief overview of the problems that alarm management solutions address

3. Outline the phases of a typical alarm management implementation

4. Examine common approaches used by real companies to achieve positive results

ALARM MANAGEMENT BENEFITS

Alarm management projects are often thought of as insurance policies against the likelihood of costly process disturbances rather than as a source of tangible financial benefits and direct ROI. But sound alarm management practices are now being recognized as the cornerstone of regulatory compliance, improved operational integrity and productivity, and enterprise-wide business improvements that repay investment quickly.

More and more, sound alarm management is becoming a precondition for extending a facility’s operating license. Moreover, with health and safety regulatory agencies closely monitoring the way that facilities operate, alarm management can also provide a concrete demonstration of a plant’s commitment to safety, the

environment, and the well being of the local community. Too often it takes significant downtime due to an unexpected and costly incident before management will endorse an alarm management project. Fortunately, more and more companies are proactively engaging in these types of initiatives.

So, is there a hard cost associated with alarm management? It’s hard to argue with the HSE, Responsible Care™, OSHA, OSH, ISO, global regulatory bodies and insurance companies. For example, the insurance industry spends $22 billion per year on equipment damage claims and, as a result, is forcing many insured facilities to implement alarm management solutions. The Health and Safety Executive is also cracking down on companies by enforcing alarm management deadlines for facilities if

3

they want to retain their operating licenses. Finally, many companies that are members of the Responsible Care™ program are likely already in the midst of improving their alarm management programs, since this has been strongly recommended for the past several years.

In addition to licensing, regulatory compliance and incident prevention, improved alarm management provides further business benefits:

• Proper staffing levels must be maintained to handle abnormal situations, but sound alarm management practices make it possible to assess and then streamline resource requirements. Loops per operator and consideration of other operator responsibilities provides an abstract picture of the true workload. By factoring alarm rates and operator actions into this equation, a more precise and concrete representation of workload is created. If the ultimate goal is control room consolidation, measuring alarm loads and pursuing alarm improvements can ensure that the reduced staff count can safely and efficiently control the process.

• Studies show that facilities lose, on average, 3% to 8% of their total production due to abnormal situations every year. Note that this figure only takes smaller cumulative losses into account, and not the larger, well-publicized catastrophes!

• Reduced insurance premiums are often available for insured facilities upon completion of an alarm management project, depending, of course, on the insurer.

• Some alarm management software, such as Matrikon’s ProcessGuard, include Alarm Performance Indicators™ that let users benchmark best-in-class performance throughout an organization. These alarm KPIs, when compared to a site’s unplanned outages, lost production, revenue, trips/incidents, and profit margins, can then be compared across all of the plants in an organization to better understand the relative performance of each and to identify opportunities for improvement.

• Finally, alarms identify problems that cost money. If you have nuisance alarms, either you need more operators or you run the risk of encountering significant upsets. If they are legitimate alarms, this is an indicator that the operation is not healthy and that it’s costing you money. A few common examples of the latter phenomenon are:

o Alarms that indicate variability or tuning problems o Alarms that indicate valve problems o Operator actions showing the need for specific training or advice

Fundamentally, alarms are indicators of lost revenue or profit, and alarm management solutions are implemented to reduce these losses and thus increase profitability by optimizing operational performance.

THE ALARM MANAGEMENT CHALLENGE

Twenty years ago, hard-wired alarms were the main mechanism for alerting control room staff to potential problems. Given the cost of running wire, new alarms were added sparingly, and operators were rarely flooded with alarms during abnormal situations. Instead, they relied heavily on strip charts, panel lights, and field operator support to diagnose problems.

The advent of the Distributed Control System (DCS) provided significant benefits for operations by improving control, as well as by alerting operators to potentially costly or dangerous situations. But,

4

since the perception was that adding a new alarm to the DCS cost nothing, new alarms were often configured without following a rigorous engineering process. As a result, most facilities alarmed virtually every reading, creating a much more expensive problem: alarm floods that overwhelmed operators and prevented them from effectively assessing the root cause of problems. Increased levels of automation and fewer operators taking responsibility for increasingly larger plant areas compounded the problem further. Even seasoned operators were significantly challenged by the task of understanding how to monitor and handle specific events, especially during stressful, safety-related situations where their colleagues’ health and safety were at risk.

Another problem, often overlooked, is the misdiagnosis of seemingly insignificant alarms. Understanding the meaning of an alarm and the consequence of not responding to it is critical for operator effectiveness. For example, if a redundancy status alarm rings in on your DCS, do your operators know if it will shut down the entire control system and thus the plant? Most engineers are unaware of the true implications of certain alarms, so it’s unrealistic to expect that operators will know either. Proper alarm engineering can help personnel address these issues by providing online documentation and alerting operators with consistent displays of alarm severity and how much time the operator has to respond before the situation escalates to a new, more urgent, and potentially more expensive level.

The final issue that many operators, engineers, and plant managers face is a lack of understanding how to establish and execute an effective alarm management project, leading them to avoid such projects in favor of “higher priority” issues. Yet hundreds of plants have gone through alarm management exercises to date. With the vast experience available in industry today, and with skilled facilitators and effective products from reliable vendors, plants can implement alarm management solutions quickly and effectively to achieve the benefits outlined above.

INSTITUTIONALIZING AN ALARM MANAGEMENT PROGRAM

Clearly alarm management is a valuable way to leverage and sustain plant assets (equipment, devices, DCSs, and people). So how exactly do you go about tackling the alarm management problem?

The first key to a successful project is getting site or corporate buy-in. In order to make the necessary cultural changes within the organization and realize the maximum benefits, the majority of stakeholders need to get behind the project. Chances are good that if your plant manager has read this far, he has recognized the need for an alarm management solution – what follows will outline the process and options (both business and technical) to effectively institutionalize alarm management.

Matrikon advocates a phased methodology that uses a proven process and infrastructure that will suit any site’s needs based on

financial and resource availability.

Phase 1: Installation of Enterprise Alarm and Event Historian

A history of alarm system performance is necessary to provide concrete evidence of your facility’s current performance. First, this makes it possible to define practical and concrete goals within your alarm management philosophy document that are realistic and attainable. But more importantly, this

5

information guides your entire alarm management process, enabling you to focus on problem areas and achieve measurable (Six Sigma) improvements. A successful alarm management solution must sustain improvements made to the alarm system and to your investment by monitoring plant performance over an extended period of time. Time, after all, is part of what has caused alarms to be added and their performance to deteriorate due to plant and equipment degradation, process and equipment changes, expansions, incidents, etc..

Enterprise alarm and event historians that collect alarms and events in real-time also provide an operational and engineering support tool. For example, the inside and outside operators can leverage an alarm historian for shift hand-off procedures or as a tool to review and monitor what happened on the previous shift. An alarm historian can also be used for incident reviews by correlating alarms with operator actions and process data, as well as to verify safe shutdowns and startups.

Matrikon has partnerships with a large number of the top oil, gas, and chemical producers in North America and Europe. From our most recent product steering meetings, our partners identified a deficiency in alarm management offerings currently on the market. One of the outcomes from our most recent industry board of direction is an effective and easy-to-use Matrikon product module: Alarm Performance Indicator™. ProcessGuard’s Alarm Performance Indicator™ enables companies to benchmark all plant areas throughout the entire enterprise to compare alarm performance across similar facilities. With this information available online, in real-time, the entire enterprise can continuously monitor plant alarm performance, which is a key indicator of production reliability and availability.

Figure 1 – Matrikon’s Alarm Performance Indicator™ developed from industry board direction.

The chart above doesn’t measure alarm frequency; rather, it measures the average alarm rate versus the maximum alarm rate. This can be done on an area-by-area basis across one or more facilities, or on a plant-by-plant basis. This view provides instant insight into how plants across your enterprise are performing, and which ones face the greatest risks.

Clearly, your alarm management resources should be directed both to the areas of greatest risk and to those places where there is the potential for the greatest improvement. Financial and safety gains can also be measured by correlating corporate financial and operating KPIs to alarm performance. By measuring corporate KPIs against the alarm performance indicators throughout the life of the alarm management project, a reasonable benefits analysis is achieved.

A simple, rolled up view of overall alarm performance is also available for senior technical management, or for plant management, so that they don’t need to worry about whether numerical targets are on track. Instead, a summary of the plant state is illustrated in Figure 2, below. If a plant exceeds EEMUA targets, this state is called “Predictive”. If a plant meets EEMUA benchmarks, then the state is

6

“Robust”. If a plant is slightly below EEMUA standards, it may still be “Stable”, but with more serious alarm problems a facility will be assigned a “Reactive” or “Overloaded” status.

Figure 2: Alarm Performance Per Plant

Figure 3, below, illustrates trends that show the details about how each of these performance categories is derived, which enables the appropriate engineer or technician to start investigating or understanding the alarm problem by focusing on the correct issues. For example, you may have constant alarm issues because your average alarm rate is too high, in which case you should begin by examining your nuisance alarms. Or you may have alarm flooding problems, in which case you should examine chattering and parent/child alarms and how they are impacted by your control strategy.

The bottom line is that measuring enterprise alarm performance and monitoring it over time can be easy with the right tools. In this case, personnel throughout a plant can measure and monitor performance by simply looking at a web page with no additional work required.

7

Figure 3 - Alarm performance details for categorizing plants as stable, unstable, reactive, etc.

Phase 2: Create an Alarm Philosophy

Once you have benchmarked current performance, it is possible to determine and document objective and measurable goals for your alarm management initiatives. This is a key part of the second phase: creating an alarm philosophy document. An alarm philosophy document contains the guiding principles by which you configure alarms and measure ongoing alarm performance levels. Most philosophy documents cover the following topics at a minimum:

1. What is an alarm?

2. How are priorities set, based on criticality and time available to respond?

8

3. General alarm considerations: e.g. How to deal with BADIO alarms?

4. Alarm performance criteria and resolution activities

These are just a few of the key issues to consider when developing your alarm philosophy. It’s strongly recommended that you consult the Engineering Equipment Materials Users Association (EEMUA) #191 publication available on the www.eemua.co.uk website for more detailed information. In addition, generic philosophies based on real manufacturing facilities across various industries are also available. Please contact Matrikon if you would like further details.

Phase 3: Top 20 Review

The next step is the Top 20 Review. This analysis is intended to identify particularly problematic alarms on a regular basis. To accomplish this, the Top 20 Review should be incorporated into a site’s existing operational policies. It may be part of a monthly Tail Gate Safety meeting or, more often, the review is done during a weekly operational or engineering meeting. This review is performed regularly because the process, equipment, and outside environment are constantly changing, and thus alarm parameters need to adapt to these changing conditions. Periodic monitoring keeps alarm counts down and helps identify other problems, such as tuning, valve sizing, transmitter issues, and many other performance-limiting or safety-related issues.

Figure 4 - Typical Alarm Resolution Breakdown

Once the Top 20 review has been incorporated into a regularly scheduled meeting, the process takes very little time, typically between 5 and 10 minutes, plus the time to fix problems that are costing money or that pose a clear safety or environmental hazard. The benefits far outweigh the amount of time invested.

The process you follow for alarm reviews will vary, and should take into account your site’s operating practices and available time. In general, however, most sites include an analysis of the most frequent

Alarm Resolution

53%

20%

7%

6%

5%

2%

1%

6%

Modify Trip Value or Remove Redundant Alarm Improve the Instrument or DeviceModify Instrument Range or CV Range Retune ControllerImprove Control System Logic Improve Process UnitImprove Operation Method Other

9

alarms and an analysis of any recent floods or trips for the previous week or month. The details of the review meeting can be as simple as bringing in a chart of the top 20 alarms along with flood analysis charts, then verbally reviewing what the major alarm problems are and what should be done about them. In most cases, change and work orders can be signed on the spot to make the process as efficient as possible.

A more detailed report may be desirable in some circumstances, and templates are available for weekly or monthly reports that can be used repeatedly to give consistent, detailed information for each alarm and its potential cause.

Note that many customers only implement phases 1 through 3, as they find that the overall reduction in alarms from these steps is around 50% to 75% of their current alarm levels and enables them to meet the targets they have outlined in their philosophy document. Further steps may still be taken, however. The following shows sample ProcessGuard reports extracted from a real site’s review report.

Figure 5 – Top 20 Alarm Distributions for October 31 to January 31

TAG Wk1 Wk2 Wk3 Wk4 Wk5Wk6Wk7 Wk8 Wk9Wk10Wk11Wk12Wk13Wk14 Wk15Wk16 Total4IT1581C 1014 0 0 0 0 0 331 0 0 8340 0 0 3265 28 0 0 129784ANN558 986 124 10851694 976 469 1400 166 377 209 338 191 361 552 977 528 104334TETG4HROFFTB 974 0 0 0 0 0 0 2239 0 0 0 0 0 0 0 2626 58394PNL0013IND4 0 0 0 0 0 0 119 78 0 0 0 0 0 0 0 5456 56534PT2427A1 802 10261599 0 61 430 220 0 0 91 1362 0 0 0 0 0 55914TEIPBFP4APSB 0 0 0 0 0 0 0 0 0 0 0 0 73 2239 2202 690 52043BG3530 1265 622 541 309 932 323 115 1025 0 0 0 0 0 0 0 0 51323ANN0151 596 473 505 479 631 527 539 621 380 0 0 0 0 0 0 0 47513ANN0322 620 841 780 1212 750 309 0 75 29 0 0 0 0 0 0 0 46164PT1570 0 0 0 0 0 0 0 2697 0 730 48 402 0 424 0 0 43013ANN0192 819 325 418 446 471 424 438 79 696 0 0 0 0 0 0 0 41164AT1102A 0 0 0 0 0 158 1450 401 2039 0 0 0 0 0 0 0 40483BE3510 3731 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 37314TE2301A1A 0 0 0 0 3597 0 0 0 0 0 0 0 0 0 0 0 35974ACOLEAKE 74 113 428 0 0 0 40 0 0 0 0 27 0 49 1762 1009 35024BFPMSLO 0 96 0 0 0 0 0 935 0 1540 186 208 346 63 0 0 3374ABASPLPR 0 0 0 0 0 0 40 0 0 0 0 0 887 1292 803 165 3187ABASPAPL 0 0 0 0 0 0 0 0 0 0 0 0 893 1294 798 163 31483ANN0158 467 256 230 411 285 233 320 614 329 0 0 0 0 0 0 0 3145WSADPFLR 0 0 198 41 122 825 0 0 514 0 1322 79 0 0 0 0 3101

Total Count & (Total count - Top 20) Distributed over time

05000

1000015000200002500030000

Wk1 Wk2 Wk3 Wk4 Wk5 Wk6 Wk7 Wk8 Wk9 Wk10 Wk11 Wk12 Wk13 Wk14 Wk15 Wk16

Total Alarm Count (Total Count - Top 20 Count)

Figure 6 – Total Alarm Count and Top 20 Alarms Distributed Over Time

10

Figure 7: Alarm Distribution, Alternate View

Figures 8 and 9: Mill 4A1 Under Bowl Pressure Figure 8 (left) shows values for 4PT2427A1 from October 2002 until January 2004. Figure 9 (right) shows the alarm count by week for this tag over the same period.

The following figure (below) is a more focused trend of the values that caused the alarms in figures 8 and 9 above. The low limit for this alarm seems to be set to approximately –1.25. On the days where there is a problem, the alarm skips between the alarm value of –1.25 and a return value slightly above this point. Results seen here are similar to those of other alarm count spikes for this tag.

Alarm Distribution

3B GP-125%

3B T-4016%

B T-8013%

3B D-59%

3B D-69%

3B CAUSTIC2%

3B A-HTR4%

3B HTRFEED2%

3B ALYZRCB2%

SYSTEM PPA2%

OTHER4%

3B FLARE1%

3B E-811%

3B H-5A1%

3B FLASH4%

3B D-35%

3B GP-13B T-40B T-803B D-53B D-63B D-33B FLASH3B A-HTR3B CAUSTICSYSTEM PPA3B ALYZRCB3B HTRFEED3B H-5A3B E-813B FLAREOTHER

0200

400600800

100012001400

16001800

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

11

Figure 10 – Alarm Value Trend Segment for 4PT2427A1

The chattering seen here is a product of process variability at the low limit. It may be possible to reduce the alarm count by adjusting this limit.

Phase 4: Documentation and Operator Assist

In some cases a more comprehensive alarm rationalization is required. The objective is to re-engineer the alarm priorities and trip limits consistently, and to provide operators with online documentation for:

the causes of specific alarms

how to verify that an alarm does in fact indicate a problem

what corrective actions should be taken

the consequences if the alarm is not handled properly

This online information helps operators to respond more effectively to abnormal situations.

The review process entails having the P&IDs in hand and reviewing and documenting every tag. In addition, the consequences of not responding to an alarm are documented, along with the maximum time that can elapse without taking action before this consequence occurs or the situation escalates. Typically most companies follow the guidelines taken from EEMUA #191 documentation to then assign a priority (see below):

Urgency of Response

Minor Consequence

Moderate Consequence

Major Consequence

More than 45 minutes

Low

Low

High

15 to 45 minutes

Low

High

Emergency

Less than 15 minutes

Low

Emergency

Emergency

12

Another key benefit of this review is that it is an effective way to capture the knowledge possessed by veteran operating staff and passing it on to a less experienced crew.

For documentation and rationalization, Matrikon’s Alarm MOCCA (Management of Change) product is instrumental in quickly rationalizing alarms. Conventional products can limit reviews to 100 tags per day; but MOCCA’s ease of use enables a faster review, and hence allows you to review more tags in a shorter period of time. MOCCA facilitates a faster review process by organizing and linking tags in a logical manner (tag name or P&ID) and through smart, auto fill-in technology. For example, the documentation fields provide intelligent auto-complete options.

Figure 11 – ProcessGuard MOCCA: Documentation, Master Alarm Database, Operator Assist

After going through all this effort, all alarm documentation can be viewed online, in real-time, via the web, either through Matrikon’s real-time viewer by selecting the alarm that just rang in and choosing to view the documentation.

Phase 5: Management of Change

The benefits of an alarm management project can be lost over time as incremental, untracked changes to alarms begin to have a greater and greater effect on the system’s overall performance. A management of change solution ensures that all changes to configured alarms are correct, consistent, and properly recorded.

13

Currently, management of change is a manual process for most alarm systems, resulting in increased workload for personnel, not to mention inconsistencies and errors when alarms that operators have adjusted during their shifts are not re-enabled or reconfigured to their proper, engineered settings. Ideally, a management of change solution should automate and enforce the change approval process while providing a complete and accurate audit trail of the change process – an essential step for proving regulatory compliance and directing internal alarm management efforts. By monitoring and verifying both authorized and unauthorized changes to the engineered alarm settings (including trip points, priority, controller mode, set points, alarm creation, deletion, disabling, and inhibiting on any tag), a management of change solution lets operators stay on top of the alarm management process.

Further, all changes to alarm settings should be stored in a master alarm database, along with the documentation for each alarm (including what changes were made, by whom, and the reason for the change). By automatically synchronizing with the DCS, the management of change system should notify appropriate personnel of any discrepancies.

Matrikon’s Alarm MOCCA (Management of Change Configuration Assistant), showing the standard interface along with specific audit trail information in a secure, web-based environment.

Phase 6: Dynamic Alarming Dynamic alarm management is often viewed as the ideal goal: in essence, telling the operator exactly what the problem is and how to resolve it, even while the plant state changes over time. First and foremost, the key to effective dynamic alarm management is to ensure that the alarms on the control system are accurate and reliable, and thus this should be one of the last phases in any alarm management process. If there are a lot of ineffective or nuisance alarms on the DCS, then the dynamic alarm management package must also deal with that excess information, substantially reducing its ability to provide accurate information to the operator.

14

Once the DCS alarm system housekeeping has been taken care of, a key question that usually arises is whether the dynamic alarm management package should reside on or off of the DCS. Dynamic alarm management solutions, in reality, are in their infancy, but both on and off-DCS solutions have their place. The following is a list of several dynamic alarm management options:

1. Simple on DCS logic using cut out or suppression logic 2. On or off DCS bulk changing of alarm settings for well known plant states 3. On DCS logic for dynamic limit changes 4. Off DCS operator advisory for root cause alarm problems (which alarm in the flood?) 5. Off DCS operator advisory for predictive process disturbances or equipment monitoring

From a safety perspective, fast response times are needed and for this reason DCS logic is often the best solution. However, there are certain states that are recognizable, and in some cases, can only be recognized by an intelligent external application. Regardless of the chosen solution, implementing dynamic alarm solutions on or off the DCS requires a complete operational model for the entire plant, or for all of the areas that will be monitored. All the different process conditions, operating states, and potential failures must be identified and handled appropriately. Given the far-reaching possibilities of dynamic alarm management, it’s important to outline and identify your goals clearly, what it will take to maintain the solution, and the expected benefits throughout your facility, both in the short term and in the long term.

The Matrikon Advantage

Our specialty is our ability to connect with any DCS in real-time, as well as our ability to deliver total alarm management solutions: our product offering covers the entire alarm management footprint, including:

• Real-time collection and analysis

• Documentation and operator assistance

• Master alarm database

• Dynamic alarming

Further, our comprehensive engineering services include installation, alarm analysis, alarm & control rationalization, and advanced process control optimization.

In line with our corporate philosophy, Matrikon is committed to delivering value to our clients and to lead our industry in technology and service. Our Alarm Management offering is tightly integrated with our Process Suite of products, including:

• ProcessNet for web-based collaboration, monitoring, custom reports, and correlation of alarms and events with process historians and maintenance management systems.

• ProcessDoctor for comprehensive control performance monitoring and improvement. Often alarms are generated due to simple tuning issues - ProcessDoctor leverages ProcessGuard to

15

better understand controller status and then provide a clear picture of how all control assets are performing in a plant area or even in an entire facility.

• ProcessMonitor for predictive alarming and condition-based maintenance applications.

• OPC Connectivity for data access to every major DCS on the market.

Conclusion

Providing operators with enough information to prevent abnormal situations and to diminish the impact of unpreventable abnormal situations is the key to an effective alarm management process. The phased alarm management approach offers a proven, measurable and easily implemented methodology that will guide a plant to a higher level of safety, environmental and production standards. The significant costs associated with accidents, equipment damage, unplanned plant or unit outages, off-spec production, regulatory fines, etc., will be drastically reduced as each phase is realized. Alarm Management is not just about reducing alarms; it is about responsible plant management and increased plant efficiency.

For more information about Matrikon’s alarm management solutions, or if you have any questions or comments about this paper, please contact the author, Jeff Gould, Matrikon’s Alarm Management Solutions manager, at 1.780.448.1010, or email solutions@matrikon.com, or visit our website: http://www.matrikon.com.

About the Author

Jeff Gould is the ProcessGuard Product Manager responsible for overseeing the development, marketing, and sales of Matrikon’s Alarm Management product line. Jeff joined Matrikon in 1996 and helped to develop Matrikon’s driver business, and for the last few years has been responsible for growing and shaping Matrikon’s alarm management solutions. Jeff graduated as a computer engineering technologist and his background includes several years of software development management, and sales and marketing of software solutions for the process control industry.