Enterprise PI - How do I manage all of this?

Robert Raesemann

JJacksonville, FL

Introduction

8th Largest Municipal Utility in the Country

– Electrical System

– Water and Wastewater

– Competitive Pressures

• Regulated by Local, State, and Federal Authorities

• De-regulation

My Place in All of This

• Background– Electrical Engineering– DCS, PLC, and Data Acquisition

• Current job in IT group, Enabling Technologies Section– Develop and deploy tools that improve business

processes– Ex. PI System, PC Based HMI systems, etc...

PI Used to be simple

About 1 1/2 years ago...

• one PI server connected to DCS

• maybe 20 users connected to it on LAN

• 2 client applications– Processbook – Datalink

• Users could tolerate a little downtime

PI System used by plant personnel for all of the standard

uses

• We have already received tremendous benefits– Troubleshooting– Operations– Efficiency monitoring

It quickly became more complex

• 5 PI interface nodes with many more planned• 1 Master PI server• Several custom developed applications• WAN connections (ATM,Fast Ethernet,…)• Users all over the company who can’t live

without it• PI being used for regulatory reporting

The Future Holds More Value and Capability at the Expense of

Complexity• Multi-tier architectures

• Multiple PI servers

• More (distributed) applications

• More interconnection between PI and other Enterprise systems (ERP, Work order systems, …)

• More Users (remember, Everyone is a knowledge worker now)

In the Future PI Reliability Depends on Many Things

• PI code base is very mature and stable

• New systems depend on reliable hardware, network connections, and software configurations– Networks fail– Servers fail (hard disks and power supplies)– Synchronization of databases

IT Groups Have Already Faced Many of these Problems

• Move to client-server architecture increased complexity

• Software applications live on multiple systems which must all work together

• Users spread out over WAN

Simple Network Management Protocol (SNMP)

• Internet standard for communicating system status to a central network management console.

• Applications maintain information in a managed information base (MIB)

• Agents on computer monitor hardware and software to insure proper operation. Send SNMP Traps to alert manager of problems based on policy configuration.

Monitoring our PI system with SNMP

• PI itself has no SNMP capabilities.

• Third party software enables instrumentation of PI processes on NT machines.

• Custom developed agent monitors PI data stream.

• PI-SDK will be awesome - Server Agent

Basic Policy to Monitor a PI server

• All PI services are running.

• Dr. Watson not present.

• Disk utilization below 80% (tunable threshold with warning and critical levels)

• CPU utilization below 90% (tunable threshold with warning and critical levels)

• All archive files present

Basic Policy to Monitor a PI Interface server

• API Buffer is running

• apibuf.dat file not getting larger

• Interface service is running

• PI Interface Agent– Custom agent to monitor data stream

The PI Interface Agent

• Monitors connection to PI server

• Counts Bad Inputs, I/O Timeouts, and Shutdowns for a particular interface– Alerts when number of errors exceeds

thresholds

• Alerts if points not updating– Checks a sample point on interface to make

sure that it updates regularly

The PI Interface Agent

Monitoring custom applications

• Custom applications write errors to the NT event log.

• NT Log Agent monitors the log for errors in the PI applications and reports back to management console.

• Benefits– Better support to PI data customers.– Better data availability

Event Management

• Management server generates messages based on data collected by polling agents and from traps sent by agents

• Automated actions (scripts) can be build to respond to the situation

Monitoring the Server Hardware

• HP Netservers have SNMP agents called Netserver Assistants

• Monitor over 100 physical parameters on server

• Voltages, temperatures, fans running

• Hard Drive Arrays, ECC RAM

• Power supply failures

Recap of Current Monitoring

• PI server hardware

• PI NT services

• NT server health

• PI data stream

• Web pages that access PI are monitored by an IIS agent

Can I Do More With This?

• Code developed for PI agent can be used to model physical systems in Unicenter

• Stable integration of PI with other enterprise wide data systems

• Provide monitoring and management capabilities beyond what is currently possible

Wish to use PI data in new ways to obtain the competitive advantage• Better Monitor Critical Systems

• Decision centered business model– Provide better decision making support to

managers and engineers.

• Build real-time enterprise resource planning capability.

Better Monitoring of Critical Systems

• Continuous Emissions Monitoring System

• Monitors plant air emissions

• Strict EPA requirements for monitoring, reporting, and maintaining.

• System checked daily by technicians

• EVP (designated representative) signs reports– Criminal penalties associated with

misinformation

What is Planned

• Agent monitors health of CEMS system and pages technicians and environmental personnel immediately if there is a problem

• Agent not only monitoring real-time data from PI but also checking Work Order System, Emissions Database

• Monitor current health, maintenance practices, and emissions targets

Smart Plants

• The design of the next generation “Smart” plant is influenced by the capability to manage the information and control systems.

W ide AreaN etw ork(W AN )

Program m able Log ic C ontro ller(PLC )

C EM SInstrum enta tion

P I In terface C om puterrunning U nicenter TN G N T Agent and

PI In terface Agent

E lsag Ba ileyD istribu ted C ontro l System

(D C S)

M ultip le P I In terfaceM achines d istribu tedacross severa l JEA

facilities

M ain P I Serverrunning U nicenter TN G

N T System Agent

Process D ataArch ive

Program m able Log ic C ontro ller(PLC )

E lsag Ba ileyD istribu ted C ontro l System

(D C S)

P I In terface C om puterrunning U nicenter TN G N T Agent and

PI In terface Agent

T he P I In te rface A gen t runs on in te rfacem ach ines d is tribu ted th roughou t the W ideA rea N e tw ork . A ny e rro rs de tec ted by thein te rface m ach ine show up on U n icen te rT N G W orldV iew sc reens .

N ote: A P I Server Agent isp lanned la ter th is year as O SIexpands the P I AP I. The TN GAgents out o f the box havealready greatly reduced theam ount o f tim e spent tom ainta in the server.