download the presentation slides here

November 2015

Prof. O. A. Mohammed, IEEE SmartGridComm_Keynote, Miami, FL USA 1

Energy Systems Research Laboratory, FIU

Design and Simulation Issues for Secure Power Networks as Resilient

Smart Grid Infrastructures

Prof. O. A. [email protected]

Tel: 305-321-5622

Energy Systems Research LaboratoryDepartment of Electrical & Computer Engineering

Florida International UniversityMiami, Florida

Keynote Presentation atIEEE SmartGridComm 2015

November 5, 2015Miami, Florida

Prof. O. A. Mohammed, IEEE SmartGridComm, Miami, FL USA

Energy Systems Research Laboratory, FIU Prof. O. A. Mohammed, IEEE SmartGridComm, Miami, FL USA

November 2015



• Management of increased levels of distributed and renewable energy sources. (control challenge)

• Integrating a wide variety of systems governed by different regulations and owned by different entities.(interoperability challenge)

• The variable nature of renewable energy sources. (Generation uncertainty )

• Real time energy forecasting and energy management system for generation and demand balancing. (Demand uncertainty)

• New distributed architectures with many microgrids. (Resiliency)

Challenges of integrating distributed resources



• Interoperability between different protocols and applications (in software layer).

• Identify the communication network and bandwidth required to collect measurement and control remote sites (Distributed control).

• Data availability (Delay, corrupted data, denial of service,…etc.)

• Data security and privacy


November 2015



0 20 40 60 80 100

Power Grid

Water

Nuclear

Government Facilities

Healthcare

Percentage (%)

Sec

tors

Power Grid Cyber Attack RiskTrustworthy Critical Infrastructures

Critical infrastructures increasingly targeted by cybercriminalsSome Governmental initiatives– Identified by NSF as a key research area

– Critical infrastructure protection (CIP) set by the US Presidential Directives

– North American electric reliability corporation (NERC) CIP requirements, 2013

0

50

100

150

200

250

300

2009 2010 2011 2012 2013

Nu

mb

er o

f A

ttac

ks

Year

28X



Smart Grid Cyber Infrastructure


November 2015



Smart Meter Security Threats

SMART Meter Vulnerability:• The AMI network is open to external unsecured

environments such as cellular channels, power line carriers and radio signals.

Cellular,Power line modem

Radio Signal (900MHZ)

Adversary

• The AMI can provide a communication path to customer systems such as building management systems (BMS) through the customer gateway.

• If the adversary succeeds in penetrating into the AMI network and pretending to be a valid smart meter management system, he can easily send a disconnect signal to millions of customers.

AMI: Advanced Metering Infrastructure

Dis

conn

ectio

n si

gnal

.In

corr

ect p

rice

Inco

rrec

t Loa

d D

ata


ZigBeeWiFi, etc.


Smart Meter Security Threats• In January 16, 2014 Proofpoint, Inc. uncovered

Cyber attack involving conventional household "smart" appliances. The global attack campaign involved more than 750,000 malicious communications coming from more than 100,000 everyday consumer gadgets such as home-networking routers, connected multi-media centers, televisions and at least one refrigerator that had been compromised and used as a platform to launch attacks.

Secure Measures

• The network topology should prevent interaction between customers in the NAN.

• Price signal should be authenticated

• Smart meters use X.509 authentication certificate.

• Most of the smart meters doesn't update the certificate for life time (that is a problem). Example latest discovered bug “Heartbleed” in OpensSSL used to compromise the certificate

Zigbee, Wifi

X.509 Certificate is an Authentication ProtocolBetween Smart Meter and Utility. Uses SSL Certificate

Example: an attack on a customer appliance


November 2015



Smart Grid Cyber Infrastructure (FAN Threats)



Field Area Network (FAN)• FAN shared multi service IP

network cover Distribution automation, Integrated Distributed resources, Demand Response and field devices

• Based on Broad Band wireless resources. FAN routers has WIFI interface for field technician.

• Data integrity and confidentiality should be ensured for smart meter data and field devices.

• If adversary succeed to compromise FAN router the intruder could easily send wrong signal to switches or field devices NIST reference Model

NIST Publication 1108 Page 35


FAN routers located on the pole

November 2015



Smart Grid Cyber Infrastructure (WAM Threats)



Security challenges:• Most of the protocols were developed for efficient data

transmission in isolated control network without considering the security required for wide spread and open system.

• Phasor Measurement units PMU depend on external clock source which can be spoofed or jammed.

• PMU protocols ( C37.118 and IEEE 1334 ) doesn't support authentication.

Prof. O. A, Mohammed, IEEE SmartGridComm, Miami, FL USA

Spoof

Network Attack

November 2015



• State estimator can detect bad data form faulty meters or communication errors

• Stealth attack can be designed to be hidden from state estimator.

• Several types of stealth attack can be performed against state estimator such as (state, framing and topology attack)


Bad data from faulty meter

Bad data Identified

State estimator stealth attack

Bad data not identified


To design security aware WAM, different factors should be considered in the communication and system design such as:

Data authentication (insure the source of the Data)

Data integrity (detect corrupted or changed data)

Proper location of highly secured and encrypt meters to prevent state estimator attack.

Data mining techniques could be used to detect altered data.


November 2015



• Cyber Physical security should not only be considered in the cyber component but also the power system network topology should be designed to be resilient in cases of attack.

• The control system should be designed to withstand cyber attack and cyber component failures.

• Centralized control suffer from single point of failure problems.

• Successful attack against centralized control system could lead to serious damage and loss of service



• Decentralized control reduce the risk of single point of failures and loss of service.

• Risk of attacking area and loss of service still high


November 2015



• Distributed control minimize the risk of cyber attack.

• Each node exchange information and cooperate with neighbor node to improve the system stability.

• Attack detection can be improved by data mining from different sources

completely distributed multi-agent control



• The types and levels of data protection used to encrypt or authenticate signals should be coordinated with signal sensitivity and impact on the system stability.

• The attack detection should rely on physical system characteristics as well as the cyber security rules

• Cyber attack countermeasures should consider the dynamics and the special nature of the power system.


November 2015



• We need new modeling and simulation tools to capture the dynamic nature of both cyber and physical components.

• We need test bed facility that will provide the ability to perform experiments involving integration of different technologies in one system (communication, software and physical components)

• Simulation tools and test bed implementations should provide the ability to test different types of vulnerability and Launch attack scenarios



• Complex cyber physical infrastructure environment is required for identifying possible vulnerabilities and testing solutions

• The infrastructure should provide modular and flexible structure to implement different physical topologies and operational scenarios

• The test environment should seamlessly integrate with different protocols and support interoperability (standards???).


November 2015



Our verification Platform–The Test Bed has the following capabilities and components:

Phasor measurement Units monitoring, protection, and control

Distributed and renewable energy sources wind, solar, Fuel cells, etc.

New operational schemes protective digital relaying Wide Area Protection

Intelligent protection schemes and their application for Prevent cascading outages Islanding situations Grid blackout

Emulation of Plug-In-Hybrid and Electric Vehicles (PHEVs) and (PEVs) Energy Storage systems, SOC and SOH for batteries

Integration of Hybrid AC-DC systems micro grid solutions for residential and industrial applications. Enhancement of Energy Efficiency and EMS

Cyber infrastructure Communication network, servers, communication protocols, HIL



So, Various Composable Modules areIntegrated in the Test Bed


November 2015



IEC61850 Relays

Transmission lines

Buses

Synchronizers

SCADA

PMUs

Battery ChargingAnd

PEV system

Pulse load/super cap. micro grid

DC architecture

Software platform

Battery management

system

Wind, PV, Storage microgrid

Prof. O. A. Mohammed.


Configurable transmission Network

Remote controlCircuit Breaker

Remote controlload emulator

Synchronous generatorand control agent

Hybrid MicrogridDC Microgrid

Communication network

LinuxMulti Agent

Communication Protocols

November 2015



• Communication middleware is required to enable information exchange between different controllers.

• should provide portability and interoperability between different system component.

• should provide time predictable performance, low latency and overhead to meet the real time application requirements.

• The communication middleware must support large system expansion and adding new types of data.

• The Communication Middleware could be message centric or data centric



• Message centric middleware enables different nodes to send messages to each other without regard to physical location or message contents.

• Messages must be predefined based on required data exchange and operation case which limit the system expansion.

• The application is responsible of insuring correct data format and parsing received information which add extra overhead in development control application.

• Data filtering is done at the application level which lead to poor utilization of network bandwidth.


November 2015



• Data centric middleware provide data aware communication approach which overcome the disadvantages of message centric. The message is built by the middleware to exchange and update the system state.

• Messages are derived from the system data model. (no predefined message structure)

• Data format validation and parsing information are done on the middleware level. (reduce the complexity of control application)

• Data filtering is done at the middleware level which improve the utilization of the network bandwidth.



Efforts saved By

Data centric

Middleware

Dat

a ce

ntr

ic is

ch

ose

n t

o b

e U

sed

at

th

e sm

art

gri

d T

est

bed


Emphasized by Industry (see Smart grid

Interoperability Panel)

November 2015



• Data Distribution service is a communication standard based on data centric and publisher subscriber approach created by Object Management Group (OMG).

• Supported By Standard application Programming Interface API which simplify integration with different applications.

• Utilize real time publisher subscriber protocol (RTPS)

• IEC 61850 implements RTPS communication.

(provide interoperability between different vendors)Prof. O. A, Mohammed, IEEE SmartGridComm, Miami, FL USA


• No message broker or server which avoid single point of failure.

Communication failure

DDS DATA BUS

Client server communication schema

• Single point of failure• Low update rate• High latency

DDS publisher/subscriber communication schema

• Reliable communication (no single point of failure)• High update rate• Low latency (no intermediate message broker)

Tra

nsm

ittin

g no

des

Transmitting nodes Receiving nodes

Rec

eivi

ng n

odes


November 2015



Unicast communication• Multiple stream for multiple

destination• Consume high bandwidth• Not suitable for remote or

distributed control where multiple agents need to access the same data

• Multiple copy of the same data sent from the source to Each distention.

• Consume high bandwidth

• Not suitable for Wide area measurement since allocated bandwidth for remote site usually low.

• Transmitting multiple stream add extra processing overhead on the transmitting nodes and reduce the update rate

Three copy of the same data



• Single stream for all destination

• Reduce the network bandwidth

• Reduce the processor overhead

• Suitable for low bandwidth link and remote sites

• Reach set of Quality of Services profiles QoS (Predictable delivery)

• QoS defines the data transmission priority, life time, ordering based on time stamp and allowed latency

Multi cast communication• Single stream for multiple

destination• Optimize network bandwidth• suitable for remote or distributed

control where multiple agents need to access the same data

Single copy of the data


November 2015



DDS Gatewy

DAQ/RTU

DATABASE

Distributed Data service DDS

DDS Gatewy

PROTECTION RELAYS

DDS Gatewy

LOAD EMULATORES

DDS Gatewy

GENERATION CONTROL

DDS Gatewy

RENEWABLE ENERGY EMULATORS

DDS Gatewy

Power system analysis SW packages

DDS Gatewy

Smart Meters

DDS Gatewy

ENERGY STORAGE

MANAGMEN

DDS Gatewy

DEMAND SIDE MANAGMENT

The gateway is the bridge between different types of protocols and DDS



• Performance tests were performed to measure the latency and maximum possible update rate for the implemented DDS.

• 10,000 messages were sent with different message rate (from 50 to 1000 Message/sec) and the delay time is observed.

• The test was repeated with different message size (from 32 B-63 kB)


November 2015



Performance Test for DDS Unicast and Best effort QoS.

Performance Test for DDS Multicastand Best effort QoS

Max 1.2 ms delay Max 1.4 ms delay

Max 0.3 ms delay

Higher message size Higher message size


Max 0.35 ms delay


Micro Grid Component Infrastructure

• DC Microgrids Components– Modular MIBBC– Battery Bank– Programmable Load emulator– DC bus module– High Freq. Inverter– more

• Hybrid PEVs Charging system Components

– LC Filter– Bidirectional ac/dc converter– Bidirectional dc/dc converter– Hardware Overview– Lithium-ion Battery– Management,

Diagnostics Software

• AC/DC interconnected network components

– Uncontrolled rectifier

– Dynamic load emulation

– AC Filter

– DC Filter

– DC/Dc Boost Converter

– AC/DC Measurements and Protection

– DC Power Module

– Medium voltage DC Transmission line model

• Components of Microgrids for pulse load studies

– Super capacitor bank– Bidirectional converter– Multi port Boost Converter– Lead Acid Battery Bank– Programmable DC Load


November 2015



Smart Grid Test-bed Components (The CPS infrastructure)• Distributed control components

– Embedded agent platform– HIL simulators– DDS infrastructure– Smart Meters– Developed Interface library

• Phasor measurement Components– PMUs– PDCs– RTACs– Interface for control and protection modules– Real time phasor data server



Hybrid HW/SW CPS Simulation Environment


High Level ArchitectureIEEE Standard for

Federated Simulation

November 2015



Test Bed Interface Library• Interface Library developed to

provide control function and data collection capability from local and remote network

• Real time publisher subscriber protocol (RTPS) insure interoperability

• Provide Flexible environment for Distributed control test and validation

• Currently available interface for generators, Micro Grids, CB, measurement nodes, PMU, Load emulators and renewable energy emulators



Generator block (slack mode) provide

interface to• Start and shutdown the

generator• Control generator speed• Read terminal voltage, line

current and frequency in real time

Generator block (Power control mode) provide

interface to• Start, shutdown and

synchronize the generator• Control generator torque• Read terminal voltage, line

current and frequency in real time

Microgrid blockprovide interface to

• Control active and reactive power flow from the Micro grid.


November 2015



Wind turbine emulatorprovide interface to

• Control wind speed• active and reactive

power reference• Voltage and current

feedback in real time

PV emulatorprovide interface to

• Control the PV panel temperature and irradiance

• Voltage and current feedback in real time

Load emulatorprovide interface to

• Load active and reactive power

Network configuration block

Bus measurement block Circuit Breaker control block



• To ensure proper operation and initialization an automated startup algorithm where developed using finite state machine.

• The controller is built using state flow toolbox

• The automated startup algorithm is divide into two layer.

• The top layer (supervisory layer) which monitor the state of the CB and generators and ensure correct startup sequence

• The bottom layer which handle the generators startup and synchronization process


November 2015



Generator in Pcontrol mode



Control algorithm modeled by Simulink

Developed interface library

Control signal

Con

trol

sig

nal

Control signal

Feedback Fee

dbac

k

Feedback

SIMULATION

Real System


Different types of attacks can be performed on actual comm.

network

Attack models can be simulated in real time(e.g. false data injection, delayed attack, …)

November 2015



Generator Interface Block

Load emulatorsInterface Block

Bus measurements Interface Block

Control Model

Load pattern



0 20 40 60 80 100 120 140 160 1800

10

20

30

40

50

60

70

80

90

100Frequency

Time(sec)

Fre

quen

cy(H

z)

0 20 40 60 80 100 120 140 160 1800

20

40

60

80

100

120Load Bus Voltage

Time (sec)

Vol

tage

(V

)

0 20 40 60 80 100 120 140 160 1800

100

200

300

400

500

600

700

800

900

1000Generator Output Power

Time (sec)

Pow

er (

W)

0 20 40 60 80 100 120 140 160 1800

0.5

1

1.5

2

2.5

3Generator Current

Time (sec)

Cur

rent

(A

mp)

Bus voltageGenerator Frequency

LoadGenerator Current


November 2015



Multi Agent Data Information Model for Power Systems

Decentralized control is established using multi agent frameworks

Agents interact and cooperate to achieve a global or local objective (through an optimization function)

In future active distribution networks, simultaneous power system operations will be controlled by the system operator and private microgrid operator entities:

• Frequency and voltage support (Ancillary Service) • Online DER Scheduling (Optimal Dispatch) • Market models (Auctions, Dynamic Pricing)

There is a need to perform concurrent control. Multi agent control applications are required.

Agent Entity

Peer-to-peer Communication



• An agent needs to interact with its environment through sensors and actuators.

• A sensor acquires the data from the outside world and the actuator responds according to the agent’s decision.

Agent Platform

Sensors

Decision Making

Actuators

Environment

Perception

Action

How Can We Link Power System Physical Objects to Agent Platforms?

• For Actual Multi Agent Field Implementation :Need to link agent objects to distributed industrial control systems.


November 2015



Can IEC 61850 Meet Decentralized Control of Active Distribution Network Demands?

The smart grid concept covers an extensive control, automation and protection applications.

IEC 61850 does not meet all the required forms of monitoring and information exchange demands.

Active distribution networks require dynamic adjustment of primary, secondary and tertiarycontrol levels.

• Frequency and voltage support (Ancillary Service) • Online DER Scheduling(Optimal Dispatch) • Market models (Auctions, Dynamic Pricing)

Advanced intelligent multi agent frameworks are necessary with a flexible ability to create tailor‐made decentralized control schemes while allowing the legacy protocols.

AESPO Agent

Microgrid Agent

DER Agent

Microgrid Agent

DER Agent

Microgrid Agent

DER Agent

DER Agent

DER Agent

DER Agent

Tertiary Level Control

Agent Communication

Secondary Level Control

Agent Communication



Agent codes are developed and performed in real‐time power system applications

OPC UAMIDDLEWARE

Cloud Communication

Interface

Client / Server

Java ClientFIPA Messages

IEC 61850Manufacturing Message Specification (MMS)


MULTI AGENT FRAMEWORK

The foundation for intelligent

physical agents (FIPA)

November 2015



Proposed Multi Agent Framework The foundation for intelligent

physical agents (FIPA) is an organization which intends to evolve inter-operable agent communications with semantically meaningful messages, such as how messages are transferred and presented as objects.

Taking the specific benefits of two major frameworks, We want to provide a flexible framework for active distribution network application layers

Merging IEC 61850, FIPA and the open connectivity unified architecture (OPC UA) standards

OPC UA a middleware for abstracting various IED protocols into an interoperable interface for secure and reliable data exchange.

OPC UAMIDDLEWARE

Cloud Communication

Interface

Client / Server

Java ClientFIPA Messages

IEC 61850Manufacturing Message Specification (MMS)



Conclusions• The smart grid requires new set of modeling, simulation and

experimental tools to implement and validate new designs.• The smart grid test bed provide unique integrated environment for

testing developed techniques involving cyber and physical component

• The DDS provide real time performance and distributed architecture which simplify the data exchange and control implementation.

• The RTPS ensure the interoperability between different nodes• The interface library for the smart grid test-bed provide flexible

and powerful tools to integrate with simulation packages.• The federation simulation provide flexible tool for co-simulation of

CPS system to identify full system behavior. • Agent based control schemes provide decentralized operation

capability for power systems with data information modeling and protocols.


download the presentation slides here

Documents