Tools & Methods for Hardening Communication Security of Energy Delivery Systems

Dr. Yow-Jian Lin Applied Communication Sciences

Dr. Sami Ayyorgun Applied Communication Sciences

Dr. Robin Berthier University of Illinois at Urbana-Champaign

Cybersecurity for Energy Delivery Systems Peer Review July 24-26, 2012

Team Members

• ACS – Sami Ayyorgun – Shrirang Gadgil – Jaewon Kang – Vikram Kaul – Yow-Jian Lin – Sunil Samtani

• DTE Energy

– Andre Guibert De Bruet – Lanse LaVoy – Catherine Martinez – Gerald Vermeulen

• EPRI – Larry Burnette – Glen Chason – Galen Rasche – Scott Sternfeld

• UIUC

– Robin Berthier – Rakesh Bobba – Roy Campbell – Salman Malik – Xueman Mou – Edmond Rogers – Bill Sanders

2

• Objectives – Identify Vulnerabilities and Deficiencies of

communication protocols commonly used in EDS (e.g., ICCP, C37.118, C12.22)

– Design, develop, and demonstrate a modular and extensible ADEC-G (Agent-based, Distributed, Extensible Cybersecurity for the Grid) system for

• Monitoring/detecting abnormal EDS protocol usage

• Ensuring security coverage

• Technical Approach – An online system with stateful model

based checkers (SMBCs) that helps utilities monitor EDS protocol communication contexts and flag abnormal session behaviors

– An offline framework that security tool developers, operators, and auditors can use to verify security properties (leverages formal methods)

• Schedule √ 03/31/2011: Threat Matrix and Cyber

Security Scenario Report

√ 11/04/2011: System Design Document – 09/28/2012: Initial Software Release – 06/24/2013: Final System Demo – 09/20/2013: Final Software Release

• Performers: Applied Communication Sciences (formerly Telcordia Technologies)

• Partners: DTE Energy, EPRI, UIUC

Summary: Tools & Methods for Hardening Communication Security of Energy Delivery Systems

3

• Motivation – EDS communicating pairs are restricted subject to

• Bilateral agreement • Limit on number of accepted connections • Firewall rules • VPN

– “Insider” threats remains an issue • Compromised hosts or disgruntled employees can exploit EDS protocol

deficiencies/vulnerabilities to steal protected data or to cause service disruption or even system damage

• ADEC-G focuses on patrolling protocol communication behaviors, as part of a defense-in-depth strategy to security – Complements other cybersecurity approaches such as encryption, access control, filtering,

key management, or malware detection – Online system monitors and detects abnormal protocol behaviors using the following

techniques: • Deep-packet inspection • Event sequence, volume, and temporal order tracking • Cross-session comparison

– Offline formal verification techniques ensure security property coverage • Protocol communication behaviors that violate a security property can/will be detected by

available checkers in ADEC-G online system

ADEC-G Focus: Abnormal EDS Protocol Usage

4

Agent GUI

ADEC-G System Overview

ADEC-G Agent

5

Offline Verification Framework

Verification Framework GUI

Modular architecture – Facilitates extensibility, including support for new protocols – Enables customized deployment for specific protocol(s) – Allows integration with other tools

• Packet sniffing, interception, injection for packet extraction and insertion

and replay via raw socket, libpcap, netfilter/iptable • Protocol Dissectors for protocol specific packet parsing

– E.g., Ethernet, IP, TCP, UDP, ICCP, C37.118, C12.22 • Context Establishment for tracking session state of interest

– E.g., ICCP bilateral table ID, invocation ID, relevant TCP context for an ICCP session

• Stateful Model Based Checkers (SMBCs) for matching session context against rules governing protocol behaviors

– E.g., C37.118 data frame length should match with the expected data length derived from the most recent CFG2 frame

• Rules for configuring monitoring/detection policies and actions – Rule example: c2; LengthMismatch; 0; ALARM; ENABLED

• Recommended Actions for implementing appropriate responses – E.g., WARNING, ALARM, DROP, or DISCONNECT

ADEC-G Agent

6

Stateful Model Based Checkers (SMBCs) – State machines model behaviors and states of

interest, based on protocol specifications, vulnerability analysis as well as device and network configurations

– Exposed self-contained conditions signify the reach of certain states

– Context updates trigger state transitions and the evaluation of relevant rules

– Detection is based on observed behaviors, not signature of code segment

Rules and Model-Based Checkers

7

SMBC for checking suspicious SBO-device operation in ICCP

Rules – Extensible Rule language with a set of exposed state variables and self-contained conditions

for each supported protocol – General form: ID; Condition; TimeWindow, Action; Status

• ID: a unique rule identifier • Condition: singleton or compound of self-contained and/or user defined (based on exposed state

variables) conditions • TimeWindow: a window of state context subject to evaluation • Action: WARNING, ALARM, DROP, or DISCONNECT • Status: Enabled or Disabled

Fall-back IDLE Attack

IDLE ARMED

[MMS Event Notification Req.](with Timeout, Local Reset, Failure)

SELECTED

[MMS Event Notification Req. Rate > Threshold_2](with Timeout, Local Reset, Failure)

Non-Select IDLE Attack

[MMS Read Rsp. w/ Failure Rate > Threshold_1]

(for SBO device)

• Rule editing – Adding/updating rules for changing

monitoring/detection needs • Monitoring

– Inspect/track communication sessions and context variables

– Alarm/indication/notification of rule triggering

– Graphical display of instantaneous cumulative values for selective variables

• Visualization – Dynamic update of links/sessions – Overlay of triggered sessions on topology – Independent control of displayed “topology”

at physical and individual protocol level • Link expansion and collapse • Node clustering • Topology modification

• Access control – Authentication, authorization

ADEC-G Agent Technical Features

8

Exploitation and Detection of Vulnerability: An example using C37.118

ADEC-G Agent GUI showing active C37.118

session between ACS.PMU and ACS.PDC

When an ADEC-G rule is triggered, the display shows the active session

that triggered the rule in RED

– On the C37.118 tab, the LengthMismatch detector is triggered

– The logical topology shows the network elements and the session that cause the trigger

9

• Offline Framework – Validate formally the security monitoring architecture of an ICCP deployment

• Objective – Support security operators, engineers at utilities, and developers to leverage

formal model checking to build resilient monitoring solutions

ADEC-G Offline Analysis: Overview

Input from developers ICCP Specifications

(state machine)

ICCP Security Property

ICCP Checkers

Topology and config. templates

Input from end users ICCP deployment

topology

ICCP configuration parameters

Security Property parameters (incl. checker selection) Backend

(Formal verifier)

Frontend (GUI)

Formal checkers libraries

Output

Validated checkers

Failing checkers + Guide on

fixing issues

Coverage model

10

11

Formal Verification of Checker Design

(TSCount/EvalTime>Thresh) --> Checker.AttackDetected

1 Design of formal models based on protocol and device specifications

2 Simulation and state-space exploration

3 Verification of security properties

Formal Verification Framework (under development)

12

Define network topology

Define security property Define monitoring agent location and configuration

Run formal analysis to verify soundness of monitoring architecture

• Challenges

1. Near-real-time operation of ADEC-G for fast alarm-generation subject to resource limitations.

2. What additional checkers and rules should be implemented? Which ones are of more “value” to utilities?

3. Integration with and leveraging of the existing systems.

• Feasibility

– Multi-threading, parallelization, code optimization for ‘1’.

– Work closely with DTE Energy, other utilities, and vendor collaborators on trials for ‘1, 2, and 3’.

Technical Challenges and Feasibility

13

DTE DR-SOC Existing Application

14

ispatchable ustomer eneration – Overall Architecture example

DR-SOC

Customer Network

Firewall

4 MW

EMS / SCADA

ICCP

DRSOC Box

40 kV Tie 5823

Internet

Firewall

DRSOC Properties: • SCADA-like system that monitors

and controls over 200 nodes with a total of more than 240MW of power (to be doubled in 12mo).

• Interfaces with the utility SCADA/EMS over ICCP: e.g. –Tie 5823 load data passed to DRSOC via

ICCP. –DRSOC Box retrieves Tie 5823 load

data. –DRSOC Box controls generator output

to remain within 1/3 ratio requirement. –Power not needed by Customer’s plant

is exported to grid.

ADEC-G Agent ADEC-G Utility and Impact: • Provides provable security

against vulnerabilities in integrating 3rd-party control units into the utility system.

• Provides automatic monitoring and inspection of all the data links in a scalable manner.

• Provides configurable alarms and detectors.

• Provides flexible and extensible (e.g. for adding other protocols) environment.

DTE DR-SOC: Existing Application ADEC-G Usage example 1

15

DTE DR-SOC: Existing Application ADEC-G Usage example 2

16

• Major Accomplishments – ADEC-G online system is operational for tracking ICCP, C37.118, and C12.22 protocol

sessions • Demonstrated in lab settings against application session hijacking, denial of service,

and man-in-the-middle attacks • Installed at DTE Energy DRSOC to monitor ICCP traffic in operational setting • Sub-msec packet processing → currently at 5-10K packets/sec on average

– Formally verified soundness of ICCP checkers – Designed and implementing ICCP validation framework

• Actual vs. Planned – Tools development is on track at feature enhancement stage; actively engaging potential

users in the loop. – Slightly under budget due to delay in having the whole team contracts done; progressing

smoothly since then.

Progress to Date

17

• Discussed and shared findings with many hardware/software vendors (e.g., SEL, GE, Landis+Gyr, GPA, N-Dimension, etc.) – Continued interaction with vendors is critical in improving

technology/product maturity for the industry.

• Trials with Utilities and Vendors – Conducting ADEC-G trial in DTE Energy. – Held ADEC-G workshop jointly with EPRI during OpenSG in Feb.’12. – In addition to DTE, demonstrated ADEC-G to SEL, Landis+Gyr,

Florida Power & Light, and TVA.

• Challenges – Identifying the right POCs in utilities and vendors, who are

proactive in dealing with potential security vulnerabilities.

Collaboration/Technology Transfer

18

• ACS, UIUC, and EPRI are developing a coordinated schedule of reach-out to utilities and vendors for potential ADEC-G trials. In addition to demonstrating ADEC-G capabilities, in these trials we will also explore various use options of ADEC-G to enhance its benefits to the industry. Some of these use options include the following: – By leveraging ADEC-G’s modular design, we will explore the integration of various

system components (through APIs) with other vendors’ solutions: e.g. • SMBCs can be used for stronger IDS capability alongside of access control solutions, • ADEC-G GUI can interact with other deep-packet inspection modules, • Context establishment module can supply rich context data to aid other detection logic.

– By leveraging ADEC-G’s dual-mode of operation (in sniff or inline modes), we will

explore providing monitoring, detection, and/or mitigation services as needed by potential customers: e.g.

• Sniff-mode operation (which is non-intrusive) would facilitate user confidence by providing monitoring and detection services.

• Inline-mode operation could be used for human-in-the-loop mitigation services.

– By leveraging the Formal Verification Framework, we can provide validation of protocol specification completeness and deployment coverage of selected security policies being used by other tools.

Technology Transfer Plan

19

• Planned work items for the remaining period of performance – Event logging and replay capabilities for post-event/forensic analysis – Added monitoring and detection support (e.g., additional exposed

state variables, SMBCs, etc.) – Possibly additional protocols (e.g., DNP-3) – Soundness verification of all checkers and their security property

coverage – Development of user-friendly offline verification framework for ICCP – (Trials, Improvements), (Trials, Improvements), …

• Potential follow-on work – ADEC-G agent correlation for cross-referencing/checking across a

wider monitoring and control area – Incorporating physical checkers into ADEC-G

Next Steps

20