Making the Case for Redundant SCADA Communication Systems How to prevent a SCADA communications failure from being a major catastrophe Andrew C. Degner City of Gresham Water Resource Operations and Regulatory Compliance Manager

Water/Wastewater Lineage: Herbie & George

City of Tualatin – Home of the Crawfish Festival

Content Slide

SCADA System Architecture

System Telemetry

Schematic of SCADA Nodes

RTU – SCADA Pack 100 – Schneider Electric

Analog Radio Modem Dataradio Integra TR INB48-510-T

Antenna Types

MTU PLC-Allen-Bradley 1756-L61

Map of 21 RTUs w/ Radio Modems

Weekly 1:11 Manager’s Meeting Tuesday, May 17, 2016 at 1:45 pm

System Wide Comm Failure

Redundant Cellular System

Additional Cellular Components

SCADA Cellular Comm Status Display

Schematic of Redundant Cellular Sites

Map of 10 RTUs w/ Cellular Redundancy

2014 SCADA/Telemetry Maintenance Evaluation PURPOSE: In the ongoing effort to improve maintenance, operation, and security of the SCADA Telemetry system: * Identify and stock critical spare parts * Computer hardware life cycle replacement planning *Control and communication hardware life cycle and replacement planning *SCADA software backups and failure recovery planning *Process control enhancements to reduce Operator adjustments and possible error

*SCADA SYSTEM SECURITY AND RISK

Disrupting Radio Telemetry COUNTER MEAURES: ADD REDUNDANT COMMUNICATIONS TO CRITICAL SITES (FLOW MONITORING, FLOW CONTROL, TANK LEVELS)

Summary of Comm Failure Events

• CAUSE: A contractor for a not to be named global leader in athletic shoe manufacturing, located 13 miles NW of Tualatin started up a telemetry system using a unlicensed frequency that blocked the City of Tualatin’s licensed frequencies (451.4250 – 457.6120 MHz)

• Three separate contractors worked to troubleshoot and resolve issue at

cost of $3,318 • City of Tualatin Water Division staff assigned 72 hours overtime to

resolve issue • Radio communications off-line from May 17 through May 23 (6 days)

• Able to resolve issue w/o filing formal complaint to FCC

Lessons Learned • CONDUCT A COMPLETE SCADA/TELEMETRY MAINT/RISK EVALUATION • BUDGET $$$ FOR HARDWARE/SOFTWARE RECOMMENDATIONS

OUTLINED IN THE EVALUATION • TRAIN OPERATIONS STAFF IN TROUBLE SHOOTING SCADA/TELEMETRY

SYSTEMS • ON-CALL AGREEMENTS IN PLACE WITH “COMPETENT” CONTRATORS

FOR SCADA/TELEMETRY ANNUAL MAINTENANCE, REPAIR, AND UPGRADES

• NEVER TAKE A ROUTINE STAFF MEETING FOR GRANTED

Thanks to the following:

• MICK WILSON – CITY OF TUALATIN (RETIRED)

• JERRY POSTEMA – CITY OF TUALATIN PWD

• CARL SERPA – PORTLAND ENGINEERING

• HAPPY HERBIE THE GERMAN PLUMBER

Thank You!