Connecting the RTDS to a Multi-Agent System Testbed Utilizing the GTFPGA

Mark Stanovich, Raveendra Meka,Mike Sloderbeck

Florida State University

Introduction

Power systems are becoming much more cyber-physical

Computational resourcesData communication facilities

Desire to explore distributed control of electrical systemsExisting RTDS infrastructure to simulate electrical system simulationNeed to add computational and data communication facilities

Distributed Controls Testbed

Support a variety of software

Operating systems• E.g., Linux, Windows, Vx Works

Applications and programming languages• E.g, Matlab, C++, Java, JADE

Data communicationsE.g., TCP/IP

Cost effectivePortable

Versalogic “Mamba” SBCs (x86 Core 2 Duo processor)

*Designed by Troy Bevis

Connecting RTDS to the Distributed Controls Testbed

Need to exchange signals between computational units and RTDS

Receive sensor readingsSend commands

Digital and analog I/O wires

Tedious for large number of wiresSignal mapping changes frequently

GTFPGA

Xilinx ML507 boardFiber protocol capability (2 Gbps) to/from RTDS GPC/PB5 cardsSupported in RSCAD libraries for small and large time steps64 bidirectional 32-bit signals in large time step, available for Ethernet-based communication

Fiber optic

Ethernet

Embedded PowerPC processor

Fiber protocol decoding / encoding

GTFPGA Flexibility

GTFPGA provides a flexible mechanism to exchange dataReroute signals in software

Support multiple experimental setupsAutomatable• Faster• Less error prone

Computational units may not have native I/O capabilities

CommunicationsMamba #1 Mamba #6

CommunicationsMamba #1 Mamba #6

Ethernet

Fiber optic

GTFPGA

Data is exchanged between FPGA and RTDS every timestepTCP/IP server

Exchanges data between computational platforms and FPGACode runs on PowerPC processorMultiple computational units can connectPort number to identify desired signals mappingLow performance

Fiber optic

RTDS

Computing Board #1

Computing Board #6

Ethernet

PowerPC

Fiber optic encoder/decoderG

TFP

GA

Shipboard Distributed Control

*Work by Qunying Shen

FREEDM (NSF Center)

Proposed a smart-grid paradigm shift to take advantage of advanced in renewable energy

Plug and play energy resources and storage devicesManage resources and storage through distributed intelligenceScalable and secure communication backbone

Distributed Grid Intelligence (DGI)Control software for the FREEDM microgridManage distributed energy resources and storage devices

Solid State Transformer (SST)Power electronics based transformerActively change power characteristics such as voltage and frequency levelsInput or output AC or DC powerImprove power quality (reactive power compensation and harmonic filtering)

Distributed Grid Intelligence (DGI)

Data Communications

DGI issues power commandsConvergence

DGIs collaborate to set equal loading on all SSTs

DGI proceeds through a series of phases

Group ManagementState CollectionLoad Balancing

Power Convergence

0 10 20 300

5

10

15

20

25

Real Time (s)

Pow

er (k

W)

SST1 (measured)SST2 (measured)SST3 (measured)SST4 (measured)SST5 (measured)

Need for Flexible Communications

Each DGI requires two signals to

RTDS60 total signals

Round Trip Latency

InterferenceNumber of competing connections

Send value to RTDS and wait for return to be incremented

MambaGTFPGARTDS

Round Trip Latency

Round Trip Latency

Time (ms)

coun

t

0 50 100 150 200 250 3000

500

1000

1500

2000

012345

Number of competing

connections

Various Alternatives

Bulk transfer to separate distribution board

TCP/IP implementation degrades with contention

Use PCIe to exchange data with host PC

Host PC handles TCP/IP connectionsGTFPGA handles communication with RTDS

Fiber optic

PC

Ie

Ethernet

EmbeddedPowerPC processor

RTDS Interface Module

GTFPGA PCIe Communications

RTDS provides FPGA logic to decode/encode signalsXilinx provide logic to communicate over PCIeWrite “glue” to put the two togetherTCP/IP server

Port to a Linux implementationDriver

Exchange data over PCIe

Xilinx PCIe Communications

RTDS Optical Fiber Interface Module

Host PC (Linux)

Driver

TCP/IP Server

GTFPGA PCIe

Xilinx CoregenImplementation creates an FPGA project that communicates using PCIe protocolReads and writes are directed to FPGA RAM

Add RTDS Interface Module to Coregen’d projectRedirect signals

Write and read data made available by RTDS interface module

RAM

RTDS Interface

RTDS

Host PC (Linux)

RAM

Xili

nx B

oard

PCIe Host PC Software

User-space driverMemory mapped I/O

TCP/IP serverEach control process utilizes a different portConfiguration file used to setup RTDS to computational unit mapping

Host PC (Linux)

Driver

TCP/IP Server

Round Trip Latency

10,000 round trip timings

300 microsecond

latency

Round Trip Latency(Initial Implementation)

Time (ms)

coun

t

0 50 100 150 200 250 3000

500

1000

1500

2000

012345

Number of competing

connections

Round Trip Latency(Vary Competing Connections)

Time (ms)

coun

t

0 0.5 1 1.50

200

400

600

800

1000

1200

1400

16000510152023

Number of competing

connections(Each connection

exchanges 4-bytes)

Round Trip Latency(Vary Transfer Size)

Time (ms)

coun

t

0 0.5 1 1.5 2 2.50

500

1000

1500

200011530456064

4-byte Signals

Exchanged

Future/Continuing Work

Diversify and expand the number of computational unitsDifferent architectureReduced computational power

DMA rather than memory-mapped I/OEach signal potentially results in one PCIe transactionReduce variability due to changes in number of signals exchanged

Co-simulationUtilizing GPU facilitiesPseudo real-time Simulink

• RTDS signaling to “clock” Simulink

Conclusion

GTFPGA offers a very flexible and scalable solutionExtend communicate with external computational units

Utilizing Ethernet interface directly on the GTFPGA results in large latenciesPCIe interface of GTFPGA can be used to reduce latenciesUtilizing the PCIe interface

Latencies are significantly reducedLarger number of connections are supported

Opportunity to view PCIe implementation on tour

Acknowledgement

This work was partially supported by the National Science Foundation (NSF) under Award Number EEC-0812121 and the Office of Naval Research Contract #N00014-09-C-0144.

Contact Information

Mark Stanovich – stanovich@caps.fsu.eduMike Sloderbeck – sloderbeck@caps.fsu.eduRaveendra Meka – meka@caps.fsu.edu

Future/Continuing Work

Diversify and expand the number of computational units

Different architectureReduced computational power

Data communications emulationTopologiesWirelessCharacteristics• Dropped packets• Latencies