data visualisation and exploration - smart grid forums · pdf file 2016-07-04  ·...

Click here to load reader

Post on 17-Jun-2020

0 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • Data Visualisation and Exploration Turning high volumes of complex, varied, real-time data into user friendly visuals for advanced decision support and rapid action

    6.04.2016 Grid Analytics Europe 2016 (Amsterdam)

    Matthias Stifter (AIT) Ingo Nader (Teradata) Konrad Diwold (Siemens)

  • Content

     Methodologies to support grid planning and operation  Power SnapShot Analysis  Express Grid Data Access

     Interactive data discovery technologies  Parallel processing, MapReduce, Performance

     Use Cases  Identification of Unsymmetry  Relations of network state events  Validation of inverter voltage control characteristics  Assignment of meters based on communication activities and

    voltage measurements

    207/04/2016

  • Motivation for application of data analytic methods in power system analysis

    307.04.2016

  • Motivation for data analytic methods

     Volume of Data not process-able in table calculation tools  e.g., MS Excel 1 million rows are way too much

     Processing time longer than analysis time  e.g., Development of algorithm takes days and computation

    takes several weeks

     Data resolution for the problem is higher than ususal  e.g., Load forecast from monthly data to hourly or minutes.

     Interactivity for data exploration  immediate feedback and direct response to selections

    407/04/2016

  • Intelligent Data Analytics Research Fields

     Search and Analysis  Information Retrieval, Computer Vision, Process Analysis,

    Statistics, Data Mining, Algorithmic Efficiency …

     Semantic Processing  Information Extraction, Knowledge Engineering, Semantic Web,

    …  Cognitive Systems and Prediction

     Machine Learning, Reasoning, Decision Support, …

     Visualisation and Interaction  Visualisation, Visual Analytics, Rendering …

    507/04/2016

  • Data Scientists?

    607.04.2016

    http://www.forbes.com/sites/danwoods/2012/03/08/hilary-mason-what-is-a-data-scientist/

  • Data Scientists?

    707.04.2016

  • Data Scientists!

    807.04.2016

  • Analytics and visualisation methodologies to support grid planning and operation

    907.04.2016

  • Power Snap Shot Analysis Method (PSSA)

    Full determined network state with synchronized measurements  Synchronized measurements per meter: 1 sec-RMS of V, I, P, Q per phase  10 most interesting snapshots out of 900 seconds are selected  > 100 Million snapshots for 40 networks

    1007.04.2016

    Voltage drop diagram of one feeder for one snapshot Voltages displayed on the geographical map (GIS)

  • Optimising visualisation interactivity to ensure data can be fully explored and anomalies identified effectively

    1107.04.2016

  • Interactive analysis and visualisation framework

    1207.04.2016

    Network

    Snapshot

    Meter

    Voltage Drop

    Histogram Inverter Q(U)

    Unsymmetry

  • Low Voltage Network Model Validation

     Extract, Transform, Load (ETL) to PostgreSQL / Aster DB  Data discovery / exploration (z.B.: voltages, model validation, Trends)  Simulation, Analysis and Visualisation  Connection to ASTER/Teradata from PSSHost (MapReduce, parallelisation)

    1307.04.2016

  • Low Voltage Network Model Validation

    1407.04.2016

  • Low Voltage Network Model Validation

    1507.04.2016

  • Low Voltage Network Model Validation

    1607.04.2016

  • Low Voltage Network Model Validation

    1707.04.2016

  • Low Voltage Network Model Validation

    1807.04.2016

  • Low Voltage Network Model Validation

    1907.04.2016

  • Low Voltage Network Model Validation

    2007.04.2016

    Deviations

    Deviations

  • Low Voltage Network Model Validation

    2107.04.2016

    Deviations

    Deviations

    Dependency on Daytime /

    Load situation

  • Data Discovery Plattform for parallel processing

     Combining open source and commercial solutions  Data analysis methods for massive parallel processing  MapReduce functions based on R, Java, Python

    2207.04.2016

  • MapReduce Example: Maximum and minimum voltages

     From all meters get maximum and minimum voltage per snapshot

    2307.04.2016

    Maximum and minimum voltage of phase 1 per weekday and the corresponding averaging.

  • Performance Tests: Maximum and minimum voltages

     Issues of MapReduce function  Import of 900 million measurements not possible  import only voltages  Access via JDBC DB connection to PostgreSQL/Aster not possible

     Run on cluster

     Run locally

    2407.04.2016

    Benchmark Fetch time In-DB/ Java total

    Aster MapReduce (beehive) 2*3GB + 2GB 14 min 2 sec 14 min

    Aster MapReduce (beehive) 2*6GB + 4GB 9 min 2 sec 9 min

    Aster Java JDBC (local) 6GB ~ 179 sec ~

    PostgreSQL JDBC (worker) 3GB 50 min 127 sec 52 min

    PostgreSQL JDBC (local) 6GB 36 min 63 sec 40 min

    Rows type U_eff Date

    100 million 14.1 millions June 2014

    900 million 97 million August 2015

  • Effective interworking of data scientists and domain experts to maximise exploration effectiveness

    2507.04.2016

  • 2607.04.2016

    LET‘S CRUNCH SOME DATA!YEP!

    … OR AS RANDALL MUNROE* WOULD HAVE ILLUSTRATED IT …

    *xkcd.com

  • 2707.04.2016

    CAN WE MAKE A HISTOGRAM OF

    THESE VOLTAGES

    YEP

    I‘M USING R …

  • Example for data discovery process: Unsymmetry

     Voltage histogram per phase for one network

     all snapshots

    2807/04/2016

    Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this

    snapshot (determined by the lowest voltage of all of the three phases).

  • 2907.04.2016

    WAIT A MINUTE … WHAT ARE THOSE

    OUTLIERS ON PHASE 1? THERE *ARE*

    SOME OUTLIERS

    LET‘S LOOK AT ONE SNAPSHOT I`M USING R …

  • Example for data discovery process: Unsymmetry

     Voltage histogram per phase for one network

     one snapshot

    3007/04/2016

    Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this

    snapshot (determined by the lowest voltage of all of the three phases).

  • 3107.04.2016

    WHAT IS THE REASON … ? WANT ME TO APPLY COLLABORATIVE FILTERING AND PLOT AN AFFINITY

    GRAPH TO SHOW RELATIONS?

    YEP! I`M USING ASTER‘S CFILTER …

    WHAT?

    VISUALISE THESE UNSYMMETRY EVENTS!

  • Analysis of unsymmetric events

     Interactive visualisation of unsymmetry events

     Connection relates events at same time but not same event

     Visualisation using Collaborative Filtering und Affinity Graph

    3207/04/2016

    Visualization for of relations of number of voltage unbalance events (color and width of edges) which are occuring at the same time at meters (color and size of nodes).

  • Detection of isolated unsymmetry events

     Same event connecting meters for all snapshots  Event = strong asymmetry between voltages  Visualisation with Collaborative Filtering und Affinity Graph

    3307/04/2016 Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other

    meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.

  • 3407.04.2016

    WOW! YEP!

    CAN YOU SHOW ME THE METER ID AND FEEDER ID?

    I`M JOINING SOME TABLES …

    WHAT‘ IS THAT BUBBLE?

    ISOLATED EVENTS, UNRELATED TO THE REST

  • Detection of isolated unsymmetry events

     Same event connecting meters for all snapshots  Event = strong asymmetry between voltages  Visualisation with Collaborative Filtering und Affinity Graph

    3507/04/2016 Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other

    meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.

  • 3607.04.2016

    I THINK THIS IDs BELONG TO RIPPLE CONTROLLED SINGLE

    PHASE SWITCH DEVICES WHAT?

    I‘M USING POWER FACTORY …

    REMOTELY CONTROLLED WARM WATER BOILERS!

    AHA ….

  • Detection of isolated unsymmetry events

     Network Model and load information

    3707/04/2016

    Example Network Model in Power System Analysis Application “PowerFactory”

    MU381802326

    60241152

    60241150

    KK4109128 60241151

    60241149

    60241148

    KK66771

    STRANG4_0857204

    60241145

    DA36917

    DA36916

    DA36913MA36915

    KÜMA3691260241146

    60241142

    KÜMA36911

    MA309894

    MA309895

    DA36910_60241143