eareckson air force station, alaska, power plant generator protection project narrated by ted...

Eareckson Air Force Station, Alaska, Power Plant

Generator Protection ProjectNarrated by Ted Creedon, P.E

Chief Engineer

Creedon Engineering

04/22/23 © Creedon Engineering Rev 1.3 2

You will learn:

• Why a fiber LAN based protective relay architecture was selected

• The (new) skills required to design and commission LAN based generator relays

• The Cyber Security strategy used• What equipment is needed to program and test the

relays• Why Zero Defect Contracting was used• Why training was done prior to design• Project Deliverables• Surprising positive impact on other engine/generator

control systems


• “Cobra Dane collects exoatmospheric, multiple-object intelligence data on Russian reentry vehicles and can extract and reduce the data on site. The system also provides early-warning and attack assessment for missiles that would impact the continental United States and detects and catalogs satellites for the U.S. Air Force’s Spacetrack systems. “

Cobra Dane Radar Mission

From USAF website


Bus Power Plant Existing Condition

• Unscheduled outages damage Cobra Dane components ($100K-$1.5M/outage)

• Plant can’t hold frequency

• 4 hours to sync generators

• Uncalibrated metering and relays

• Disconnected bus differential wiring


Power Plant Existing Condition

• Governor actuators hunt causing excessive EGB-50 wear

• 2301 speed control’s frequency response detuned (see Running Hz on following slide)


Site Environment

• Winter wind 180 knots

• Summer ground fog

• Bad weather delays travel and spare part shipment for weeks at a time

• Temp +19 to +56F

• Seismic 1 G


Other Factors

• Plant personnel want new equipment

• “We have to move ahead with new technology”

• Plant on PACAF and MDA commanding general’s daily reports

• 45 year old equipment maintained by very creative personnel


Project Scope

• Hard dollar USAF Contract

• Design/build

• Contractor decides what to do w/ USAF approval

• Protective relay replacement is in budget and doable

• New engine controls are not in budget


Relay Replacement Options

• Opt #1: Recalibrate and repair existing induction disk relays

• $2k / relay just for service estimate

• Flash hazard

• Unacceptable to all



• Opt # 2: replace with electronic relays 1:1

• Reuse existing wiring and doors

• Recommended by protection engineer

• Risk of incorrect existing wiring causing startup problems

• Risk of excessive on site labor

• Flash hazard



• Opt # 3: Rack and stack relays• New dutch doors required• No capability to add general purpose 4-20

ma I/O, rtd’s etc• RS 232 communication is unacceptable• Copper 300v network wiring doesn’t meet

Code• Architecture does not support future

expansion



• Opt #4: Modular relays• New dutch doors to cover 15KV breakers while

troubleshooting• Removes maximum existing wiring• Function/card vs function/box supports future

expansion• Dual fiber LAN – will survive switchboard fire• Oscillography will be useful when

troubleshooting multiple generator problems



• Opt #4: • Hard wire problems now “soft wire”• Can be prototyped and precommissioned in

Anchorage• Minimum parts count• Can remotely troubleshoot over modem/LAN• Weren’t sure it would be acceptable to the site.

This is why O&M training was held first.



• Opt #4 con’t:

• Cyber security is a problem

• Keep 86 lockout relays hardwired

• Passive backplane with plug in CPU and I/O option cards is a well proven architecture

• 32 bit DSP on VT/CT cards offloads CPU

• The relay is a small Cray



• Option 4 is the only option that enables Zero Defect Contracting:– Zero injuries– Zero defects– Zero disputes– On time– On budget

• In that order


Anticipated Problems

• Option # 4 gave the Contractor the best chance to avoid:

• Safety issues (medivac could take days)

• Can reduce flash hazard risk with software

• The power plant having start up problems at the point of no return (Island goes dark)

• Warranty issues ($10-$50K per trip)


Feature Copper LAN Copper RS 232/485 Fiber LAN

terminations/link 16RS232 2 - 7 (Depends on

TD,RD,CTS,RTS,DSR, GND,DTR)

2

Redundant links no no yes

Incorrect terminations w orks at reduced data rate w orks at reduced data rate doesn't w ork at all

Meets code in 120 VDC compartments No

No - if 600 volt cable is used need termination barriers and

line protectorsYes

Snif fable/tappable Yes Yes With great dif f iculty

Works during faultsNo - according to mfgr's tests,

TCP/IP packets dropped and need to be retransmitted

CMMR problem causd by touch potential shif ting ground levels Immune to faults

Ground Loops Slightly Susceptable very susceptable Immune to ground loops

Distance 100 meters >100 metersSingle mode multiple KM or multi

mode 1,000M

Works if broken Degraded performance Degraded performance No

Muxable Yes- LAN sw itch With dif f iculty - adds ports Yes - Fiber sw itch

Addressable Yes No Yes

Speed 1GB 345K max 1GB

Future speed enhancements (required for 20+ year life)

Replace cable and sw itches Replace cable and muxes Replace sw itches, reuse cable


TYPICAL MODULAR RELAY ARCHITECTURE, OPTIONS AS

INSTALLED

Note: Governor speed control card was actually prototyped by vendor but never manufactured.

CPU + DUAL FIBER LAN CARD

Standard 4CT/4VT + DSP

4 Form-A (voltage with optional current) outputs, 8 digital inputs

Standard 4CT/4VT +DSP

2 Form-A (voltage with optional current) and 4 Form-C outputs, 4 digital inputs

FUTURE 4-20MA VOLTAGE REGULATOR PID LOOP OR SETPOINT CONTROL

FUTURE 4-400MA ENGINE GOVENOR ACTUATOR PID LOOP OR SETPOINT CONTROL


Gen 110.1.1.1

Gen 210.1.1.2

Gen 310.1.1.3

Gen 410.1.1.4

Gen 510.1.1.5

Gen 610.1.1.6

Fiber Switch#1

10.1.1.16116 PORT

Fiber Switch#2

10.1.1.5116 PORT

Workstation10.1.1.51

Dual Fiber LAN

Laptop10.1.1.152

Copper LAN

Laptop10.1.1.153

Copper LAN

FUTURE FOUNDATION

FIELDBUS SENSORS & ACTUATORS

FUTURE PID LOOP KW SHARING

FUTURE PID LOOP VAR SHARING

FUTURE ENGINE PID LOOP SPEED

CONTROL


• The next 3 slides show all the other engine and generator controls that could be replaced with 6 additional modular relays (if VAR and KW load sharing were available)

• However, 4-20ma I/O could be used for set point engine/generator control

• Feeder relays can also be added


Unified Plant Architecture

• It is now possible to manually control an entire power plant with set point control without having to interface different manufacturer’s equipment

• Complete automatic control would be feasible if PID was available on the DSP I/O cards

• Use LAN for KW and VAR sharing signals


Project Team

• Chief Engineer & Electrical Contractor– Runs project, provides network & data engineering– Builds prototype system in Anchorage– Builds test bench on site– Writes O&M manuals

• Chief Protection Engineer– Short circuit & coordination study– Drawings – Programs relay and tester (w/ chief engineer)– Commissions relays on site

• Lifts each wire and verifies connectivity• Functional checkout &Start Up


Project Team

• Switchboard wireman– Reviews design and consults– Builds & pre wires new doors– On site demolition and installation

• Fiber person– Installs and tests fiber on site

• CAD drafter

• Vendor trainer


Project Team

• The minimum experience level required was 25 years per person


Deliverables

• Short circuit and coordination study• Relay settings & relay program• Automated 6 voltage, 6 current relay tester

program• On site test reports• Hyperlinked O&M manuals (800 pages)

– 300 typeset custom pages + 500 pages relay manual (vendor manual had to be republished)

– LaTex was used to typeset the custom pages– MS word broke at 30 pages due to color photos

• Project data required 3 CD’s (~1.5 GB)


Deliverables

• Drawings – @ 30 drawings per– Existing as builts– Demolition– New work– New as builts

• Specs– On drawings– Relays and fiber switches specified prior to

bid


Submittals

• Vendor does no shop drawings

• Vendor builds exactly what is shown

• Engineering firm required to produce “zero defect, shop drawing quality work” by contract

• O&M manuals– In color– Hyperlinked, with index and table of contents


Project Schedule

• Sign contract

• 1 week Training

• 1 week site visit

• Prepare drawings

• Fabricate and pre wire doors

• Short circuit and coordination study

• Write relay and tester programs


Project Schedule

• Ship equipment

• Demolition, installation and commissioning on site

• Write O&M manuals

• As builts

• This all had to be done under a hard dollar, hard completion date, US Government Contract


Training

• 1 week custom training with vendor, gear fabricator, owner, engineers

• 12 people

• 6 laptops and 6 relays

• 2 fiber switches

• Owner buys into architecture and orders SCADA packages


Training

• At the end of a week:

• We’re exhausted

• Owner is happy

• Vendors are happy

• The engineers knew they had their work cut out for them

• (During negotiations the remark “After all its “just another relay” ” was made….)


Training

• If we hadn’t done the training first, the project may have failed

• Training cost $16.5K in Anchorage


Test Bench

• 3 computers– Relay software– Tester software– Linux box for network packet sniffing

• 2 modular relays• 2 fiber switches• 1 6 Voltage, 6 Current programmable tester -

GFE• Everything on fiber/copper LAN, (RS-232 was

too slow to use)


Test Bench

• The test bench allowed complete simulation of all faults and trip points

• Pick up, drop out and trip were observed and compared with calculated values.

• 9 different protection groups (subsets of protection elements) were needed to isolate interaction between settings.

• Splitting the protection elements into test groups was unexpected and not covered by the training course.


Test Bench

• We had a great deal of confidence in the design because of the testing in Anchorage

• Plant personnel comfortable with SCADA GUI, not with the relay or tester programming and GUI’s

• Additional plant personnel were trained in Anchorage using the test bench.


Programming

• Training Gave us a heads up on the relay programming

• Had we known, we would have had relay tester training. The generator test program didn’t exist

• The test program required as much time and thought as the relay program

• About 6 weeks of 7/10’s for 2 engineers• 2 engineers, a protection engineer and a data

engineer/protection engineer are required.


A Neat Tool

• We used a spreadsheet to organize the protection elements, their settings and test block connections


On Site Surprises

• There was one unexplained zit in the protection circles..– It was resolved on island– We had excellent 24/7 relay vendor support

during start up – yes, we woke engineers up

• The tester died during test out of generator #1, on Friday at 9 AM– The tester vendor shipped a replacement that

was ON SITE at 8 PM the next day – from 11,000 miles away


On Site Surprises

• Oh, by the way, the relay program didn’t work on island– The protection engineer accidentally hit the

automatic firmware update feature in Anchorage

– It was incompatible with the download software

– We had a new CD in 24 hours

• Which brings up another concern:


Cyber Security• It not a protective relay, it’s a computer on a LAN• TFTP is a problem with the fiber switch – it automatically tries to

load new code on startup. Disable tftp.• Web server can’t be disabled• Passwords use reversible encryption!• Fiber LANs are difficult to compromise• Use unroutable IP addresses (10.x.x.x)• Physical security – 5 rings

– Shemya is surrounded by 28 F water– Air Police with M-16’s patrol the power plant– Trusted employees– Isolation and disconnection from any network– Fiber

• Yearly cyber security audit required• Bit by bit firmware verification (Linux diff) do not rely on hash

schemes (MD5, SHA1, etc)


Cyber Security

• If you’re not convinced that Cyber Security is a problem, read page 268 of ISBN 0-89141-821-0, “At The Abyss” by Thomas C Reed, Former Secretary of the Air Force. He details the sabotage of a Soviet gas line with firmware trojans planted in the Urgenoi gas field pump stations in 1983.


Example of what happens if on a intranet or internet

• Hack the http server• [Sat Aug 19 21:33:36 2006] [error] [client 24.20.123.11] request failed: URI too long (longer than 8190)• [Sat Aug 19 21:34:08 2006] [error] [client 24.20.123.11] File does not exist: /srv/www/htdocs/_vti_bin• [Tue Aug 22 18:08:23 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/scripts• [Tue Aug 22 18:08:23 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/MSADC• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/c• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/d• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/scripts• [Tue Aug 22 18:08:24 2006] [error] [client 24.236.230.63] File does not exist: /srv/www/htdocs/_vti_bin• Hack the ftp server• Aug 31 07:39:47 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:39:47 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:39:47 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:39:51 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:39:52 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:39:52 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:40:00 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:40:00 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:40:00 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:40:11 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:40:11 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]• Aug 31 07:40:11 redcloud pure-ftpd: ([email protected]) [WARNING] Authentication failed for user [Administrator]• Aug 31 07:40:24 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [user] [Administrator]• Aug 31 07:40:25 redcloud pure-ftpd: ([email protected]) [DEBUG] Command [pass] [<*>]

• > 202.40.190.42• Server: 10.1.1.xxx• Address: 10.1.1.xxx#53

• Non-authoritative answer:• 42.190.40.202.in-addr.arpa name = ritt-190-42.ranksitt.net.• www.ranksitt.net is in Bangladesh• www.sovam.com is a Russian Army site in Krasnoyarsk• Protective relays and fiber switches should have internal firewalls as a Cyber Security option

http://www.ranksitt.net/

http://www.sovam.com/


Data from the Relay

• Data from the relay displayed on a laptop in the control room:– Enabled accurate setting of generator voltage

and frequency• Synchronization problem completely eliminated

– Enabled tuning of the 2301’s providing good frequency response

• Transient load sharing improved

– Eliminated frequency oscillations


Data from the Relay

• Enabled rebuilding the engine fuel rack linkages, eliminating dead band.

• Allows 2 generators to accept 3500 KW each, when 1 of 3 on line goes down the other 2 now accept 100% load

• Eliminated complaints from Cobra Dane

• Electrical faults in Cobra Dane no longer trip generator breakers


One problem

• While re-roofing the power plant one relay power supply was shorted out due to rain infiltration

• The relay power supply repair cost $2K• The relay program was loaded from its

own CD (6 CD’s each differing only by IP address) and installed

• Yes, the relay should have been recommissioned ($50K). But it wasn’t.


One problem

• And it worked just fine!


Follow Up

• As of 8/25/2006 (2 years later)

• Plant personnel receive periodic factory training

• Are planning relay improvements and additions designed and installed themselves

• They are totally self reliant


The Future• SCADA/PMCS software can be user written in .NET 2 or .NET 3

– Event driven– Free AES encryption– The relay as a data source– Better graphical user interfaces

• More add in cards for– Speed + KW share– Voltage + VAR share– DSP for engine/generator controller apps

• Interface cards to other plant protocols – – Fieldbus – use redundant sensors that are NIST traceable– I/O interfaces to other protocols: CAN, etc.

• User programmable PID, etc..• Cyber Secure Certs by mfgrs


Address

tcreedon /at/ easystreet /dot/ com

Ted Creedon, P.E.Chief EngineerCreedon Engineering5740 SW Childs RdLake Oswego Or 97035503 620 0492

eareckson air force station, alaska, power plant generator protection project narrated by ted...

Documents

budget creedon engineering

ted creedon

relay replacement optionsopt

reuse existing wiring

protective relay architecture

network wiring doesnt

new technologyplant

new equipmentwe