data visualisation and exploration - smart grid forums · pdf file 2016-07-04  ·...

Data Visualisation and Exploration Turning high volumes of complex, varied, real-time data into user friendly visuals for advanced decision support and rapid action

6.04.2016 Grid Analytics Europe 2016 (Amsterdam)

Matthias Stifter (AIT) Ingo Nader (Teradata) Konrad Diwold (Siemens)

Content

 Methodologies to support grid planning and operation  Power SnapShot Analysis  Express Grid Data Access

 Interactive data discovery technologies  Parallel processing, MapReduce, Performance

 Use Cases  Identification of Unsymmetry  Relations of network state events  Validation of inverter voltage control characteristics  Assignment of meters based on communication activities and

voltage measurements

207/04/2016

Motivation for application of data analytic methods in power system analysis

307.04.2016

Motivation for data analytic methods

 Volume of Data not process-able in table calculation tools  e.g., MS Excel 1 million rows are way too much

 Processing time longer than analysis time  e.g., Development of algorithm takes days and computation

takes several weeks

 Data resolution for the problem is higher than ususal  e.g., Load forecast from monthly data to hourly or minutes.

 Interactivity for data exploration  immediate feedback and direct response to selections

407/04/2016

Intelligent Data Analytics Research Fields

 Search and Analysis  Information Retrieval, Computer Vision, Process Analysis,

Statistics, Data Mining, Algorithmic Efficiency …

 Semantic Processing  Information Extraction, Knowledge Engineering, Semantic Web,

…  Cognitive Systems and Prediction

 Machine Learning, Reasoning, Decision Support, …

 Visualisation and Interaction  Visualisation, Visual Analytics, Rendering …

507/04/2016

Data Scientists?

607.04.2016

http://www.forbes.com/sites/danwoods/2012/03/08/hilary-mason-what-is-a-data-scientist/

Data Scientists?

707.04.2016

Data Scientists!

807.04.2016

Analytics and visualisation methodologies to support grid planning and operation

907.04.2016

Power Snap Shot Analysis Method (PSSA)

Full determined network state with synchronized measurements  Synchronized measurements per meter: 1 sec-RMS of V, I, P, Q per phase  10 most interesting snapshots out of 900 seconds are selected  > 100 Million snapshots for 40 networks

1007.04.2016

Voltage drop diagram of one feeder for one snapshot Voltages displayed on the geographical map (GIS)

Optimising visualisation interactivity to ensure data can be fully explored and anomalies identified effectively

1107.04.2016

Interactive analysis and visualisation framework

1207.04.2016

Network

Snapshot

Meter

Voltage Drop

Histogram Inverter Q(U)

Unsymmetry

Low Voltage Network Model Validation

 Extract, Transform, Load (ETL) to PostgreSQL / Aster DB  Data discovery / exploration (z.B.: voltages, model validation, Trends)  Simulation, Analysis and Visualisation  Connection to ASTER/Teradata from PSSHost (MapReduce, parallelisation)

1307.04.2016

Low Voltage Network Model Validation

1407.04.2016

Low Voltage Network Model Validation

1507.04.2016

Low Voltage Network Model Validation

1607.04.2016

Low Voltage Network Model Validation

1707.04.2016

Low Voltage Network Model Validation

1807.04.2016

Low Voltage Network Model Validation

1907.04.2016

Low Voltage Network Model Validation

2007.04.2016

Deviations

Deviations

Low Voltage Network Model Validation

2107.04.2016

Deviations

Deviations

Dependency on Daytime /

Load situation

Data Discovery Plattform for parallel processing

 Combining open source and commercial solutions  Data analysis methods for massive parallel processing  MapReduce functions based on R, Java, Python

2207.04.2016

MapReduce Example: Maximum and minimum voltages

 From all meters get maximum and minimum voltage per snapshot

2307.04.2016

Maximum and minimum voltage of phase 1 per weekday and the corresponding averaging.

Performance Tests: Maximum and minimum voltages

 Issues of MapReduce function  Import of 900 million measurements not possible  import only voltages  Access via JDBC DB connection to PostgreSQL/Aster not possible

 Run on cluster

 Run locally

2407.04.2016

Benchmark Fetch time In-DB/ Java total

Aster MapReduce (beehive) 2*3GB + 2GB 14 min 2 sec 14 min

Aster MapReduce (beehive) 2*6GB + 4GB 9 min 2 sec 9 min

Aster Java JDBC (local) 6GB ~ 179 sec ~

PostgreSQL JDBC (worker) 3GB 50 min 127 sec 52 min

PostgreSQL JDBC (local) 6GB 36 min 63 sec 40 min

Rows type U_eff Date

100 million 14.1 millions June 2014

900 million 97 million August 2015

Effective interworking of data scientists and domain experts to maximise exploration effectiveness

2507.04.2016

2607.04.2016

LET‘S CRUNCH SOME DATA!YEP!

… OR AS RANDALL MUNROE* WOULD HAVE ILLUSTRATED IT …

*xkcd.com

2707.04.2016

CAN WE MAKE A HISTOGRAM OF

THESE VOLTAGES

YEP

I‘M USING R …

Example for data discovery process: Unsymmetry

 Voltage histogram per phase for one network

 all snapshots

2807/04/2016

Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this

snapshot (determined by the lowest voltage of all of the three phases).

2907.04.2016

WAIT A MINUTE … WHAT ARE THOSE

OUTLIERS ON PHASE 1? THERE *ARE*

SOME OUTLIERS

LET‘S LOOK AT ONE SNAPSHOT I`M USING R …

Example for data discovery process: Unsymmetry

 Voltage histogram per phase for one network

 one snapshot

3007/04/2016

Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this

snapshot (determined by the lowest voltage of all of the three phases).

3107.04.2016

WHAT IS THE REASON … ? WANT ME TO APPLY COLLABORATIVE FILTERING AND PLOT AN AFFINITY

GRAPH TO SHOW RELATIONS?

YEP! I`M USING ASTER‘S CFILTER …

WHAT?

VISUALISE THESE UNSYMMETRY EVENTS!

Analysis of unsymmetric events

 Interactive visualisation of unsymmetry events

 Connection relates events at same time but not same event

 Visualisation using Collaborative Filtering und Affinity Graph

3207/04/2016

Visualization for of relations of number of voltage unbalance events (color and width of edges) which are occuring at the same time at meters (color and size of nodes).

Detection of isolated unsymmetry events

 Same event connecting meters for all snapshots  Event = strong asymmetry between voltages  Visualisation with Collaborative Filtering und Affinity Graph

3307/04/2016 Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other

meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.

3407.04.2016

WOW! YEP!

CAN YOU SHOW ME THE METER ID AND FEEDER ID?

I`M JOINING SOME TABLES …

WHAT‘ IS THAT BUBBLE?

ISOLATED EVENTS, UNRELATED TO THE REST

Detection of isolated unsymmetry events

 Same event connecting meters for all snapshots  Event = strong asymmetry between voltages  Visualisation with Collaborative Filtering und Affinity Graph

3507/04/2016 Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other

meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.

3607.04.2016

I THINK THIS IDs BELONG TO RIPPLE CONTROLLED SINGLE

PHASE SWITCH DEVICES WHAT?

I‘M USING POWER FACTORY …

REMOTELY CONTROLLED WARM WATER BOILERS!

AHA ….

Detection of isolated unsymmetry events

 Network Model and load information

3707/04/2016

Example Network Model in Power System Analysis Application “PowerFactory”

MU381802326

60241152

60241150

KK4109128 60241151

60241149

60241148

KK66771

STRANG4_0857204

60241145

DA36917

DA36916

DA36913MA36915

KÜMA3691260241146

60241142

KÜMA36911

MA309894

MA309895

DA36910_60241143