Data Visualisation and ExplorationTurning high volumes of complex, varied, real-time data into user friendly visuals for advanced decision support and rapid action
6.04.2016 Grid Analytics Europe 2016 (Amsterdam)
Matthias Stifter (AIT)Ingo Nader (Teradata)Konrad Diwold (Siemens)
Content
Methodologies to support grid planning and operation Power SnapShot Analysis Express Grid Data Access
Interactive data discovery technologies Parallel processing, MapReduce, Performance
Use Cases Identification of Unsymmetry Relations of network state events Validation of inverter voltage control characteristics Assignment of meters based on communication activities and
voltage measurements
207/04/2016
Motivation for application of data analytic methods in power system analysis
307.04.2016
Motivation for data analytic methods
Volume of Data not process-able in table calculation tools e.g., MS Excel 1 million rows are way too much
Processing time longer than analysis time e.g., Development of algorithm takes days and computation
takes several weeks
Data resolution for the problem is higher than ususal e.g., Load forecast from monthly data to hourly or minutes.
Interactivity for data exploration immediate feedback and direct response to selections
407/04/2016
Intelligent Data Analytics Research Fields
Search and Analysis Information Retrieval, Computer Vision, Process Analysis,
Statistics, Data Mining, Algorithmic Efficiency …
Semantic Processing Information Extraction, Knowledge Engineering, Semantic Web,
… Cognitive Systems and Prediction
Machine Learning, Reasoning, Decision Support, …
Visualisation and Interaction Visualisation, Visual Analytics, Rendering …
507/04/2016
Data Scientists?
607.04.2016
http://www.forbes.com/sites/danwoods/2012/03/08/hilary-mason-what-is-a-data-scientist/
Data Scientists?
707.04.2016
Data Scientists!
807.04.2016
Analytics and visualisation methodologies to support grid planning and operation
907.04.2016
Power Snap Shot Analysis Method (PSSA)
Full determined network state with synchronized measurements Synchronized measurements per meter: 1 sec-RMS of V, I, P, Q per phase 10 most interesting snapshots out of 900 seconds are selected > 100 Million snapshots for 40 networks
1007.04.2016
Voltage drop diagram of one feeder for one snapshot Voltages displayed on the geographical map (GIS)
Optimising visualisation interactivity to ensure data can be fully explored and anomalies identified effectively
1107.04.2016
Interactive analysis and visualisation framework
1207.04.2016
Network
Snapshot
Meter
Voltage Drop
HistogramInverter Q(U)
Unsymmetry
Low Voltage Network Model Validation
Extract, Transform, Load (ETL) to PostgreSQL / Aster DB Data discovery / exploration (z.B.: voltages, model validation, Trends) Simulation, Analysis and Visualisation Connection to ASTER/Teradata from PSSHost (MapReduce, parallelisation)
1307.04.2016
Low Voltage Network Model Validation
1407.04.2016
Low Voltage Network Model Validation
1507.04.2016
Low Voltage Network Model Validation
1607.04.2016
Low Voltage Network Model Validation
1707.04.2016
Low Voltage Network Model Validation
1807.04.2016
Low Voltage Network Model Validation
1907.04.2016
Low Voltage Network Model Validation
2007.04.2016
Deviations
Deviations
Low Voltage Network Model Validation
2107.04.2016
Deviations
Deviations
Dependencyon Daytime /
Load situation
Data Discovery Plattform for parallel processing
Combining open source and commercial solutions Data analysis methods for massive parallel processing MapReduce functions based on R, Java, Python
2207.04.2016
MapReduce Example: Maximum and minimum voltages
From all meters get maximum and minimum voltage per snapshot
2307.04.2016
Maximum and minimum voltage of phase 1 per weekday and the corresponding averaging.
Performance Tests: Maximum and minimum voltages
Issues of MapReduce function Import of 900 million measurements not possible import only voltages Access via JDBC DB connection to PostgreSQL/Aster not possible
Run on cluster
Run locally
2407.04.2016
Benchmark Fetch time
In-DB/Java total
Aster MapReduce (beehive)2*3GB + 2GB 14 min 2 sec 14 min
Aster MapReduce (beehive)2*6GB + 4GB 9 min 2 sec 9 min
Aster Java JDBC (local)6GB ~ 179 sec ~
PostgreSQL JDBC (worker)3GB 50 min 127 sec 52 min
PostgreSQL JDBC (local)6GB 36 min 63 sec 40 min
Rows type U_eff Date
100 million 14.1 millions June 2014
900 million 97 million August 2015
Effective interworking of data scientists and domain experts to maximise exploration effectiveness
2507.04.2016
2607.04.2016
LET‘S CRUNCHSOME DATA!YEP!
… OR AS RANDALL MUNROE* WOULD HAVE ILLUSTRATED IT …
*xkcd.com
2707.04.2016
CAN WE MAKE A HISTOGRAM OF
THESE VOLTAGES
YEP
I‘M USING R …
Example for data discovery process: Unsymmetry
Voltage histogram per phase for one network
all snapshots
2807/04/2016
Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this
snapshot (determined by the lowest voltage of all of the three phases).
2907.04.2016
WAIT A MINUTE …WHAT ARE THOSE
OUTLIERS ON PHASE 1?THERE *ARE*
SOME OUTLIERS
LET‘S LOOK AT ONE SNAPSHOTI`M USING R …
Example for data discovery process: Unsymmetry
Voltage histogram per phase for one network
one snapshot
3007/04/2016
Histogram of the voltages per phase on one snapshot showing strong asymmetric voltages. Note: the vertical dashed line marks the trigger of this
snapshot (determined by the lowest voltage of all of the three phases).
3107.04.2016
WHAT IS THE REASON … ?WANT ME TO APPLY COLLABORATIVE FILTERING AND PLOT AN AFFINITY
GRAPH TO SHOW RELATIONS?
YEP!I`M USING ASTER‘S CFILTER …
WHAT?
VISUALISE THESE UNSYMMETRY EVENTS!
Analysis of unsymmetric events
Interactive visualisation of unsymmetry events
Connection relates events at same time but not same event
Visualisation using Collaborative Filtering und Affinity Graph
3207/04/2016
Visualization for of relations of number of voltage unbalance events (color and width of edges) which are occuring at the same time at meters (color and size of nodes).
Detection of isolated unsymmetry events
Same event connecting meters for all snapshots Event = strong asymmetry between voltages Visualisation with Collaborative Filtering und Affinity Graph
3307/04/2016Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other
meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.
3407.04.2016
WOW!YEP!
CAN YOU SHOW ME THE METER ID AND FEEDER ID?
I`M JOINING SOME TABLES …
WHAT‘ IS THAT BUBBLE?
ISOLATED EVENTS, UNRELATED TO THE REST
Detection of isolated unsymmetry events
Same event connecting meters for all snapshots Event = strong asymmetry between voltages Visualisation with Collaborative Filtering und Affinity Graph
3507/04/2016Visualizations of the number of voltage unbalance events at the same time (color of edges) related to other
meters. Note that the isolated events on the bottom are happening unrelated to all other events in the network.
3607.04.2016
I THINK THIS IDs BELONG TO RIPPLE CONTROLLED SINGLE
PHASE SWITCH DEVICESWHAT?
I‘M USING POWER FACTORY …
REMOTELY CONTROLLED WARM WATER BOILERS!
AHA ….
Detection of isolated unsymmetry events
Network Model and load information
3707/04/2016
Example Network Model in Power System Analysis Application “PowerFactory”
MU381802326
60241152
60241150
KK4109128 60241151
60241149
60241148
KK66771
STRANG4_0857204
60241145
DA36917
DA36916
DA36913MA36915
KÜMA3691260241146
60241142
KÜMA36911
MA309894
MA309895
DA36910_60241143
DA36908
DA36909
KÜMA36907_60241144
60817295
60241147
KK104661
KK49883
KK49874
STRANG3_0857203
DA36934_60241138
MA36933
MA36932
MA36931
DA36930_60241139
MA36929
MA36928
MA36926
MA36927
MA36925
MA36923
DA36924_60241140
60241133602411316024113460241135
MU56021705
MU56019271
60241132
60241128
60241130
6024112960241127
KK66770
60241126
60241137
60241136
MA36922
MA36921
DA36918
DA36920_60241141
KK66769
KK66768
KK66767
STRANG2_0857202
Trafostation_08572_LITTRING_HS
ON_08572_LITTRING_NS-Verteiler
KK66766
STRANG1_0857201
V~
Slack emulator
V~
UL3
N a
us P
SS
V~
UL2
N a
us P
SS
V~
UL1
N a
us P
SS
PV _ PB NR_ 9 .6 kW p[ 3] (1 ..
3035185
543307_XAY2Y 4x150
543308_XAY2Y_4x50
PV_PBNR_9.6kWp[3]
3035194
543305_XAY2Y_4x50
510500/543304_XAY2Y_4x150
PV_ PB NR _9 .6 6 kW p [3 ]( 3 ..
510501_XAY2Y_4x50
30351834292717
PV_PBNR_4.7kWp[1]
510502_HA_frd_Cu_4x16
3034906
3035249
315777_YY_4x16
543302/543303_XAY2Y_4x150
LS/TR Schalter_STRANG4
3035254
3 1 5 8 0 1 e _A l_ 4 x5 0
3 1 5 8 0 1 d _A l_ 4 x5 0
3 1 5 8 0 1 c _ A l_ 4 x5 0
3 1 5 8 0 1 b _ A l_ 4 x5 0
3 1 5 8 0 1 a _ A l_ 4 x5 0 ÜA56057846
510503_XAY2Y_4x50510504_HA_frd_Cu_4x16
3035176
keine AM-ID/315800_XAY2Y_4x150
3035192
PV_PBNR_9.89kWp[3]
ÜA56050556
315797_YY_4x16
3 15 7 9 6 c _ A l_ 4 x5 0
3 15 7 9 6 b _ A l_ 4 x5 0
3 15 7 9 6 a _ A l_ 4 x5 0
3 1 57 9 5 a _ A2 Y _ 4 x5 0
3035181
PV _ PB NR_ 9 .6 6 kW p[ 3 ](. .
3 1 5 7 9 5 b _ A 2 Y _ 4x 5 0
3 1 5 7 9 4 _ A l_ 4 x5 03035180
PV _ PB NR_ 9 .6 6 kW p[ 3 ](. .
315793_XAY2Y_4x50
keine AM-ID_XY2Y_4x16
4414524
PV_ P BN R_4 . 6kW p[ 1] (1 ..
315791_HA_frd_Cu_4x16
3035182
315792_XAY2Y_4x150
31790_AYY_4x150
LS/TR Schalter_STRANG3
3035174
PV_PBNR_4.8kWp[1]
ÜA56043883 1 5 8 0 5 p_ A 2 Y _ 4 x9 5
3 1 5 8 0 5 o_ A 2 Y _ 4 x9 5
3 1 5 8 0 5 n_ A 2 Y _ 4 x9 5
315805m_A2Y_4x953035178
315805l_A2Y_4x95
PV_PBNR_11.5kWp[3]
3 1 5 8 0 5 k_ A 2 Y _ 4 x9 5
315805j_A2Y_4x95
315805i_A2Y_4x95
3 1 5 8 0 5 h_ A 2 Y _ 4 x9 5
3 1 5 8 0 5 g_ A 2 Y _ 4 x9 53035191
3 1 5 8 0 5 c_ A 2 Y _ 4 x9 5
3 1 5 8 0 5 b_ A 2 Y _ 4 x9 5
3 1 5 8 0 5 d_ A 2 Y _ 4 x9 5
315805f_A2Y_4x95
ÜA56043169
3035175
PV_PBNR_4.6kWp[1]
3 1 5 8 0 5 e_ A 2 Y _ 4 x9 5
ÜA56046050
3 1 5 8 0 5 a_ A 2 Y _ 4 x9 5
315817_HA_frd_Cu_4x16
3035187
315818_HA_frd_Cu_4x16
3035172
315819_HA_frd_Cu_4x16
30351964416159
315816_HA_frd_Cu_4x16
3035163
315814_XAY2Y 4x150
315815_AYY_4x150
315812_AYY 4x95
315813_XAY2Y_4x150
315820_XAY2Y_4x50
30351993035198
3035170
315821_AYY_4x50
3035190
315824_XAY2Y_4x50
PV _ PB NR_ 9 .8 7 kW p [3 ]( ..
3035164
315823_XAY2Y_4x50
PV_PBNR_9.66kWp[3]
3035162
3158
25_X
AY
2Y_4
x50
315822_AYY_4x95
3035248
315826_XAY2Y_4x50
PV_PBNR_9.87kWp[3]
315827_AYY_4x95
30351933035184
PV_PBNR_3.22kWp[1]
315828_HA_frd_Cu_4x16
44127033035197
3035186
P V_ PB NR _1 4 . 26 k Wp [3 ]
315811_AYY_4x150
LS/TR Schalter_STRANG1
LS/TR Schalter_STRANG2
ON_Trafo_19654_160kVA_30.1/0.42_kV
315810_AYY_4x150
30_kV_Netz
3035179
3807.04.2016
ALL DEVICES ARE INSTALLED IN A MULTISTORY FLAT
SO WHAT?
SOMEONE CONNECTED ALL ON THE SAME PHASE!
USUALLY CONNECTED TO ONE OF THE THREE PHASES RANDOMLY … …STATISTICALLY EVENLY DISTRIBUTED
AMONG THE THREE PHASES!
!?!
Defining more events to explore relations among them
3907.04.2016
Interactive Analysis of Events and Relations
Definition of events, e.g. total power, single infeed of active or
reactive power, unbalance
Interpretation Frequent event is a slight
unbalance (asym_3v_voltage) Singe phase infeed causes no
unbalance High voltage independent from
other events
4007/04/2016Relation of events which happen at the same moment in time, for individual meters.
Events of high_single_voltage_253v are independent from feed-in and asymmetry events.
Single phase feed-in
Interactive Analysis of Events and Relations
Definition of events, e.g. total power, single infeed of active or
reactive power, unbalance
Interpretation Frequent event is a slight
unbalance (asym_3v_voltage) Singe phase infeed causes no
unbalance High voltage independent from
other events
4107/04/2016Relation of events which happen at the same moment in time, for individual meters.
Events of high_single_voltage_253v are independent from feed-in and asymmetry events.
Single phase feed-in
Asymmetric 600W reactive power
Asymmetric 1000W reactive power
Asymmetric 200W reactive power
Peak 4kW reactive power
Validation of inverter voltage control characteristics to ensure proper operation after commissioning
4207.04.2016
Validation of inverter voltage control characteristics
Voltage dependent control of reactive power Q(U) Stochastic 1 sec samples (preserve privacy), supported by many meters
4307.04.2016
Q(U) characteristic with deadband and overvoltage injection limitation Q(U) control characteristic for a single phase inverterreconstructed with voltage and reactive power measurements with1 second values (PSSA). Note: Deviations because of differingparameters of the control characteristic.
Validation of inverter voltage control characteristics
Only sum measurement available at household level
Filter of points on curve and linear/polynomial regression
4407.04.2016
-500
0
500
1000
1500
2000
230 232 234 236 238 240 242 244
Reac
tive
Pow
er [V
Ar]
Voltage [V]
Meter_Q_eff_1 Meter_Q_eff_2 Meter_Q_eff_3Submeter_Q_eff_1 Submeter_Q_eff_2 Submeter_Q_eff_3
y = 167,11x - 38297y = 243,89x - 56357y = 296,29x - 69098
y = 235,56x - 56216y = 264,89x - 63102
y = 259,37x - 61820
-500
0
500
1000
1500
2000
230 232 234 236 238 240 242 244
Reac
tive
Pow
er [V
Ar]
Voltage [V]
Meter_Q_eff_1 Meter_Q_eff_2 Meter_Q_eff_3Submeter_Q_eff_1 Submeter_Q_eff_2 Submeter_Q_eff_3
Q(U) control characteristic for a three phase inverter reconstructed with linear regression. The measurements for the validationhave to be filtered according to specific criteria out of the point cloud to identify the gradient of the characteristic.
Validation of inverter voltage control characteristics
Visualisation for one network
4507.04.2016
Projects ISOLVES and iNIS (Source: Andreas Abart, Netz
Oberösterreich and AIT)
Assignment based on Communication Activity on Network Level
4607.04.2016
Real-time Voltage measurements from Smart Meters
3/5 min mean actual values Update every 2 minutes
4707.04.2016
Express Grid Data Access (EGDA)
LV Dashboard showing Transformer voltages and tap position, as well as voltage measurements from the grid
Communication activity within a network
Number of Activities over time between successive events
4807.04.2016
Activity pattern of three sensor of the same network over a day (19.08.2014)
Communication activity of different networks
Number of Activities over time between successive events
4907.04.2016
Activity pattern of three sensor of different networks over a day (19.08.2014)
Topology Assignment on network level
Based on communication we can assign meters to networks
5007.04.2016
Filter weakcoorelations
Activity triggeredby tap change
Assignment based on Voltage Measurement on Feeder Level
5107.04.2016
Voltage Correlations between Meters (1)
Correlation coefficients of voltages per phase of all meters
5207.04.2016
Correlations of voltages per single phase of all meters for ca. 400 measurements (SnapShots)
Phase 1 Phase 2 Phase 3
Voltage Correlations between Meters (2)
Correlation coefficients of voltage differences of two phases of all meters
5307.04.2016
ΔU Phase 1-2 ΔU Phase 2-3 ΔU Phase 3-1
Correlations of voltage differences between two phases of all meters for ca. 400 measurements (SnapShots)
Voltage Correlations between Meters (3)
Unsymmetry is distinct on a feeder different between measurements (SnapShots)
5407.04.2016
Voltage drop diagram of a network with for feeders für a single SnapShop
Voltage Correlations between Meters (4)
Correlation coefficients of modified unsymmetry factor of all meters
5507.04.2016
Modified Unsymmetry Factor k
푘 =1 − 3− 6훽1 + 3− 6훽
훽 =푈 + 푈 + 푈
(푈 + 푈 + 푈 )
Correlations of modified unsymmetry factor k of all meters for ca. 400 measurements (SnapShots)
Conclusion
Discovery and exploration Open source ecosystem (R, python/anaconda, Java, PostGRES, …) Commercial solutions operationalize methods Visualisation and Interactivity
Data Analysis Process Data scientist and domain experts Data Analytic Sprints 2-3 days for producing and discussing results
Data and Sources Smart Meters can do much more than daily energy consumption (or 15
min aggregated profile) A lot of data is produces can be discarded or aggregated Quality is hard to achieve Courage to use incomplete data
5607.04.2016
AIT Austrian Institute of Technologyyour ingenious partner
Matthias StifterEnergy DepartmentElectric Energy Systems
AIT Austrian Institute of TechnologyGiefinggasse 2 | 1210 Vienna | AustriaT +43(0) 50550-6673 | M +43(0) 664 81 57 944 | F +43(0) [email protected] | http://www.ait.ac.at