Download - Data Visualisation and Exploration - Smart Grid Forums · 2016-07-04 · Data Visualisation and Exploration Turning high volumes of complex, varied, real-time data into user friendly

Data Visualisation and ExplorationTurning high volumes of complex, varied, real-time data into user friendly visuals for advanced decision support and rapid action

6.04.2016 Grid Analytics Europe 2016 (Amsterdam)

Matthias Stifter (AIT)Ingo Nader (Teradata)Konrad Diwold (Siemens)

Content

Methodologies to support grid planning and operation Power SnapShot Analysis Express Grid Data Access

Interactive data discovery technologies Parallel processing, MapReduce, Performance

Use Cases Identification of Unsymmetry Relations of network state events Validation of inverter voltage control characteristics Assignment of meters based on communication activities and

voltage measurements

207/04/2016

Motivation for application of data analytic methods in power system analysis

307.04.2016

Motivation for data analytic methods

Volume of Data not process-able in table calculation tools e.g., MS Excel 1 million rows are way too much

Processing time longer than analysis time e.g., Development of algorithm takes days and computation

takes several weeks

Data resolution for the problem is higher than ususal e.g., Load forecast from monthly data to hourly or minutes.

Interactivity for data exploration immediate feedback and direct response to selections

407/04/2016

Intelligent Data Analytics Research Fields

Search and Analysis Information Retrieval, Computer Vision, Process Analysis,

Statistics, Data Mining, Algorithmic Efficiency …

Semantic Processing Information Extraction, Knowledge Engineering, Semantic Web,

… Cognitive Systems and Prediction

Machine Learning, Reasoning, Decision Support, …

Visualisation and Interaction Visualisation, Visual Analytics, Rendering …

507/04/2016

Data Scientists?

607.04.2016

http://www.forbes.com/sites/danwoods/2012/03/08/hilary-mason-what-is-a-data-scientist/

Data Scientists?

707.04.2016

Data Scientists!

807.04.2016

Analytics and visualisation methodologies to support grid planning and operation

907.04.2016

Power Snap Shot Analysis Method (PSSA)

Full determined network state with synchronized measurements Synchronized measurements per meter: 1 sec-RMS of V, I, P, Q per phase 10 most interesting snapshots out of 900 seconds are selected > 100 Million snapshots for 40 networks

1007.04.2016

Voltage drop diagram of one feeder for one snapshot Voltages displayed on the geographical map (GIS)

Optimising visualisation interactivity to ensure data can be fully explored and anomalies identified effectively

1107.04.2016

Interactive analysis and visualisation framework

1207.04.2016

Network

Snapshot

Meter

Voltage Drop

HistogramInverter Q(U)

Unsymmetry

Low Voltage Network Model Validation

Extract, Transform, Load (ETL) to PostgreSQL / Aster DB Data discovery / exploration (z.B.: voltages, model validation, Trends) Simulation, Analysis and Visualisation Connection to ASTER/Teradata from PSSHost (MapReduce, parallelisation)

1307.04.2016


1407.04.2016


1507.04.2016


1607.04.2016


1707.04.2016


1807.04.2016


1907.04.2016


2007.04.2016

Deviations

Deviations


2107.04.2016

Deviations

Deviations

Dependencyon Daytime /

Load situation

Data Discovery Plattform for parallel processing

Combining open source and commercial solutions Data analysis methods for massive parallel processing MapReduce functions based on R, Java, Python

2207.04.2016

MapReduce Example: Maximum and minimum voltages

From all meters get maximum and minimum voltage per snapshot

2307.04.2016

Maximum and minimum voltage of phase 1 per weekday and the corresponding averaging.

Performance Tests: Maximum and minimum voltages

Issues of MapReduce function Import of 900 million measurements not possible import only voltages Access via JDBC DB connection to PostgreSQL/Aster not possible

Run on cluster

Run locally

2407.04.2016

Benchmark Fetch time

In-DB/Java total

Aster MapReduce (beehive)2*3GB + 2GB 14 min 2 sec 14 min

Aster MapReduce (beehive)2*6GB + 4GB 9 min 2 sec 9 min

Aster Java JDBC (local)6GB ~ 179 sec ~

PostgreSQL JDBC (worker)3GB 50 min 127 sec 52 min

PostgreSQL JDBC (local)6GB 36 min 63 sec 40 min

Rows type U_eff Date

100 million 14.1 millions June 2014

900 million 97 million August 2015

Effective interworking of data scientists and domain experts to maximise exploration effectiveness

2507.04.2016

2607.04.2016

LET‘S CRUNCHSOME DATA!YEP!

… OR AS RANDALL MUNROE* WOULD HAVE ILLUSTRATED IT …

*xkcd.com

2707.04.2016

CAN WE MAKE A HISTOGRAM OF

THESE VOLTAGES

YEP

I‘M USING R …

Example for data discovery process: Unsymmetry

Voltage histogram per phase for one network

all snapshots

2807/04/2016

Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this

snapshot (determined by the lowest voltage of all of the three phases).

2907.04.2016

WAIT A MINUTE …WHAT ARE THOSE

OUTLIERS ON PHASE 1?THERE *ARE*

SOME OUTLIERS

LET‘S LOOK AT ONE SNAPSHOTI`M USING R …

Example for data discovery process: Unsymmetry

Voltage histogram per phase for one network

one snapshot

3007/04/2016

Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this

snapshot (determined by the lowest voltage of all of the three phases).

3107.04.2016

WHAT IS THE REASON … ?WANT ME TO APPLY COLLABORATIVE FILTERING AND PLOT AN AFFINITY

GRAPH TO SHOW RELATIONS?

YEP!I`M USING ASTER‘S CFILTER …

WHAT?

VISUALISE THESE UNSYMMETRY EVENTS!

Analysis of unsymmetric events

Interactive visualisation of unsymmetry events

Connection relates events at same time but not same event

Visualisation using Collaborative Filtering und Affinity Graph

3207/04/2016

Visualization for of relations of number of voltage unbalance events (color and width of edges) which are occuring at the same time at meters (color and size of nodes).

Detection of isolated unsymmetry events

Same event connecting meters for all snapshots Event = strong asymmetry between voltages Visualisation with Collaborative Filtering und Affinity Graph

3307/04/2016Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other

meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.

3407.04.2016

WOW!YEP!

CAN YOU SHOW ME THE METER ID AND FEEDER ID?

I`M JOINING SOME TABLES …

WHAT‘ IS THAT BUBBLE?

ISOLATED EVENTS, UNRELATED TO THE REST


Same event connecting meters for all snapshots Event = strong asymmetry between voltages Visualisation with Collaborative Filtering und Affinity Graph

3507/04/2016Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other

meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.

3607.04.2016

I THINK THIS IDs BELONG TO RIPPLE CONTROLLED SINGLE

PHASE SWITCH DEVICESWHAT?

I‘M USING POWER FACTORY …

REMOTELY CONTROLLED WARM WATER BOILERS!

AHA ….


Network Model and load information

3707/04/2016

Example Network Model in Power System Analysis Application “PowerFactory”

MU381802326

60241152

60241150

KK4109128 60241151

60241149

60241148

KK66771

STRANG4_0857204

60241145

DA36917

DA36916

DA36913MA36915

KÜMA3691260241146

60241142

KÜMA36911

MA309894

MA309895

DA36910_60241143

DA36908

DA36909

KÜMA36907_60241144

60817295

60241147

KK104661

KK49883

KK49874

STRANG3_0857203

DA36934_60241138

MA36933

MA36932

MA36931

DA36930_60241139

MA36929

MA36928

MA36926

MA36927

MA36925

MA36923

DA36924_60241140

60241133602411316024113460241135

MU56021705

MU56019271

60241132

60241128

60241130

6024112960241127

KK66770

60241126

60241137

60241136

MA36922

MA36921

DA36918

DA36920_60241141

KK66769

KK66768

KK66767

STRANG2_0857202

Trafostation_08572_LITTRING_HS

ON_08572_LITTRING_NS-Verteiler

KK66766

STRANG1_0857201

V~

Slack emulator

V~

UL3

N a

us P

SS

V~

UL2

N a

us P

SS

V~

UL1

N a

us P

SS

PV _ PB NR_ 9 .6 kW p[ 3] (1 ..

3035185

543307_XAY2Y 4x150

543308_XAY2Y_4x50

PV_PBNR_9.6kWp[3]

3035194

543305_XAY2Y_4x50

510500/543304_XAY2Y_4x150

PV_ PB NR _9 .6 6 kW p [3 ]( 3 ..

510501_XAY2Y_4x50

30351834292717

PV_PBNR_4.7kWp[1]

510502_HA_frd_Cu_4x16

3034906

3035249

315777_YY_4x16

543302/543303_XAY2Y_4x150

LS/TR Schalter_STRANG4

3035254

3 1 5 8 0 1 e _A l_ 4 x5 0

3 1 5 8 0 1 d _A l_ 4 x5 0

3 1 5 8 0 1 c _ A l_ 4 x5 0

3 1 5 8 0 1 b _ A l_ 4 x5 0

3 1 5 8 0 1 a _ A l_ 4 x5 0 ÜA56057846

510503_XAY2Y_4x50510504_HA_frd_Cu_4x16

3035176

keine AM-ID/315800_XAY2Y_4x150

3035192

PV_PBNR_9.89kWp[3]

ÜA56050556

315797_YY_4x16

3 15 7 9 6 c _ A l_ 4 x5 0

3 15 7 9 6 b _ A l_ 4 x5 0

3 15 7 9 6 a _ A l_ 4 x5 0

3 1 57 9 5 a _ A2 Y _ 4 x5 0

3035181

PV _ PB NR_ 9 .6 6 kW p[ 3 ](. .

3 1 5 7 9 5 b _ A 2 Y _ 4x 5 0

3 1 5 7 9 4 _ A l_ 4 x5 03035180

PV _ PB NR_ 9 .6 6 kW p[ 3 ](. .

315793_XAY2Y_4x50

keine AM-ID_XY2Y_4x16

4414524

PV_ P BN R_4 . 6kW p[ 1] (1 ..

315791_HA_frd_Cu_4x16

3035182

315792_XAY2Y_4x150

31790_AYY_4x150


3035174

PV_PBNR_4.8kWp[1]

ÜA56043883 1 5 8 0 5 p_ A 2 Y _ 4 x9 5

3 1 5 8 0 5 o_ A 2 Y _ 4 x9 5

3 1 5 8 0 5 n_ A 2 Y _ 4 x9 5

315805m_A2Y_4x953035178

315805l_A2Y_4x95

PV_PBNR_11.5kWp[3]

3 1 5 8 0 5 k_ A 2 Y _ 4 x9 5

315805j_A2Y_4x95

315805i_A2Y_4x95

3 1 5 8 0 5 h_ A 2 Y _ 4 x9 5

3 1 5 8 0 5 g_ A 2 Y _ 4 x9 53035191

3 1 5 8 0 5 c_ A 2 Y _ 4 x9 5

3 1 5 8 0 5 b_ A 2 Y _ 4 x9 5

3 1 5 8 0 5 d_ A 2 Y _ 4 x9 5

315805f_A2Y_4x95

ÜA56043169

3035175

PV_PBNR_4.6kWp[1]

3 1 5 8 0 5 e_ A 2 Y _ 4 x9 5

ÜA56046050

3 1 5 8 0 5 a_ A 2 Y _ 4 x9 5

315817_HA_frd_Cu_4x16

3035187

315818_HA_frd_Cu_4x16

3035172

315819_HA_frd_Cu_4x16

30351964416159

315816_HA_frd_Cu_4x16

3035163

315814_XAY2Y 4x150

315815_AYY_4x150

315812_AYY 4x95

315813_XAY2Y_4x150

315820_XAY2Y_4x50

30351993035198

3035170

315821_AYY_4x50

3035190

315824_XAY2Y_4x50

PV _ PB NR_ 9 .8 7 kW p [3 ]( ..

3035164

315823_XAY2Y_4x50

PV_PBNR_9.66kWp[3]

3035162

3158

25_X

AY

2Y_4

x50

315822_AYY_4x95

3035248

315826_XAY2Y_4x50

PV_PBNR_9.87kWp[3]

315827_AYY_4x95

30351933035184

PV_PBNR_3.22kWp[1]

315828_HA_frd_Cu_4x16

44127033035197

3035186

P V_ PB NR _1 4 . 26 k Wp [3 ]

315811_AYY_4x150



ON_Trafo_19654_160kVA_30.1/0.42_kV

315810_AYY_4x150

30_kV_Netz

3035179

3807.04.2016

ALL DEVICES ARE INSTALLED IN A MULTISTORY FLAT

SO WHAT?

SOMEONE CONNECTED ALL ON THE SAME PHASE!

USUALLY CONNECTED TO ONE OF THE THREE PHASES RANDOMLY … …STATISTICALLY EVENLY DISTRIBUTED

AMONG THE THREE PHASES!

!?!

Defining more events to explore relations among them

3907.04.2016

Interactive Analysis of Events and Relations

Definition of events, e.g. total power, single infeed of active or

reactive power, unbalance

Interpretation Frequent event is a slight

unbalance (asym_3v_voltage) Singe phase infeed causes no

unbalance High voltage independent from

other events

4007/04/2016Relation of events which happen at the same moment in time, for individual meters.

Events of high_single_voltage_253v are independent from feed-in and asymmetry events.

Single phase feed-in

Interactive Analysis of Events and Relations

Definition of events, e.g. total power, single infeed of active or

reactive power, unbalance

Interpretation Frequent event is a slight

unbalance (asym_3v_voltage) Singe phase infeed causes no

unbalance High voltage independent from

other events

4107/04/2016Relation of events which happen at the same moment in time, for individual meters.

Events of high_single_voltage_253v are independent from feed-in and asymmetry events.

Single phase feed-in

Asymmetric 600W reactive power



Peak 4kW reactive power

Validation of inverter voltage control characteristics to ensure proper operation after commissioning

4207.04.2016

Validation of inverter voltage control characteristics

Voltage dependent control of reactive power Q(U) Stochastic 1 sec samples (preserve privacy), supported by many meters

4307.04.2016

Q(U) characteristic with deadband and overvoltage injection limitation Q(U) control characteristic for a single phase inverterreconstructed with voltage and reactive power measurements with1 second values (PSSA). Note: Deviations because of differingparameters of the control characteristic.


Only sum measurement available at household level

Filter of points on curve and linear/polynomial regression

4407.04.2016

-500

0

500

1000

1500

2000

230 232 234 236 238 240 242 244

Reac

tive

Pow

er [V

Ar]

Voltage [V]

Meter_Q_eff_1 Meter_Q_eff_2 Meter_Q_eff_3Submeter_Q_eff_1 Submeter_Q_eff_2 Submeter_Q_eff_3

y = 167,11x - 38297y = 243,89x - 56357y = 296,29x - 69098

y = 235,56x - 56216y = 264,89x - 63102

y = 259,37x - 61820

-500

0

500

1000

1500

2000

230 232 234 236 238 240 242 244

Reac

tive

Pow

er [V

Ar]

Voltage [V]

Meter_Q_eff_1 Meter_Q_eff_2 Meter_Q_eff_3Submeter_Q_eff_1 Submeter_Q_eff_2 Submeter_Q_eff_3

Q(U) control characteristic for a three phase inverter reconstructed with linear regression. The measurements for the validationhave to be filtered according to specific criteria out of the point cloud to identify the gradient of the characteristic.


Visualisation for one network

4507.04.2016

Projects ISOLVES and iNIS (Source: Andreas Abart, Netz

Oberösterreich and AIT)

Assignment based on Communication Activity on Network Level

4607.04.2016

Real-time Voltage measurements from Smart Meters

3/5 min mean actual values Update every 2 minutes

4707.04.2016

Express Grid Data Access (EGDA)

LV Dashboard showing Transformer voltages and tap position, as well as voltage measurements from the grid

Communication activity within a network

Number of Activities over time between successive events

4807.04.2016

Activity pattern of three sensor of the same network over a day (19.08.2014)

Communication activity of different networks

Number of Activities over time between successive events

4907.04.2016

Activity pattern of three sensor of different networks over a day (19.08.2014)

Topology Assignment on network level

Based on communication we can assign meters to networks

5007.04.2016

Filter weakcoorelations

Activity triggeredby tap change

Assignment based on Voltage Measurement on Feeder Level

5107.04.2016

Voltage Correlations between Meters (1)

Correlation coefficients of voltages per phase of all meters

5207.04.2016

Correlations of voltages per single phase of all meters for ca. 400 measurements (SnapShots)

Phase 1 Phase 2 Phase 3


Correlation coefficients of voltage differences of two phases of all meters

5307.04.2016

ΔU Phase 1-2 ΔU Phase 2-3 ΔU Phase 3-1

Correlations of voltage differences between two phases of all meters for ca. 400 measurements (SnapShots)


Unsymmetry is distinct on a feeder different between measurements (SnapShots)

5407.04.2016

Voltage drop diagram of a network with for feeders für a single SnapShop


Correlation coefficients of modified unsymmetry factor of all meters

5507.04.2016

Modified Unsymmetry Factor k

푘 =1 − 3− 6훽1 + 3− 6훽

훽 =푈 + 푈 + 푈

(푈 + 푈 + 푈 )

Correlations of modified unsymmetry factor k of all meters for ca. 400 measurements (SnapShots)

Conclusion

Discovery and exploration Open source ecosystem (R, python/anaconda, Java, PostGRES, …) Commercial solutions operationalize methods Visualisation and Interactivity

Data Analysis Process Data scientist and domain experts Data Analytic Sprints 2-3 days for producing and discussing results

Data and Sources Smart Meters can do much more than daily energy consumption (or 15

min aggregated profile) A lot of data is produces can be discarded or aggregated Quality is hard to achieve Courage to use incomplete data

5607.04.2016

AIT Austrian Institute of Technologyyour ingenious partner

Matthias StifterEnergy DepartmentElectric Energy Systems

AIT Austrian Institute of TechnologyGiefinggasse 2 | 1210 Vienna | AustriaT +43(0) 50550-6673 | M +43(0) 664 81 57 944 | F +43(0) [email protected] | http://www.ait.ac.at

Download - Data Visualisation and Exploration - Smart Grid Forums · 2016-07-04 · Data Visualisation and Exploration Turning high volumes of complex, varied, real-time data into user friendly

Top Related