realization of big data platform for pmu · realization of big data platform for pmu - real-time...
TRANSCRIPT
© Hitachi, Ltd. 2014. All rights reserved.
Realization of Big Data Platform for PMU - Real-Time Data Processing & Historical Data Analysis -
March 27, 2014
Yutaka Kokai Shinji Fujiwara
Hitachi America, Ltd. Hitachi, Ltd.
© Hitachi, Ltd. 2014. All rights reserved.
1.Technical requirements for Control Center
2.High-Speed Data Access Platform for Control Center
3.Live Demo ~ PMU Application for Control Center ~
Contents
1
© Hitachi, Ltd. 2014. All rights reserved.
1. Technical requirements for Control Center
2
© Hitachi, Ltd. 2014. All rights reserved.
1-1 PMU Installation in North America
3
PMU installation in North America
The number of installed PMUs are growing in North America
Source: NASPI Oct 22, 2013
0
200
400
600
800
1000
1200
March 2012 March 2013
Num
be
r
About 500
1,126
Source: DOE Synchrophasor Technologies and their Deployment
in the Recovery Act Smart Grid Programs, August 2013
In the case of WECC,
481 PMUs are installed.
Source: DOE Synchrophasor Technologies and their Deployment
in the Recovery Act Smart Grid Programs, August 2013
The number of installed
PMUs are increased.
Now the usage of PMU data
needs to be considered.
© Hitachi, Ltd. 2014. All rights reserved.
Platform for Big Data
Analysis is required
Information Storage
& Retrieval System
for Big Data Analysis
(PMU data)
1-2 Database Requirement for PMU Data
4
In the case of 500 PMUs are installed ……
PMU data will be accumulated about 40 TB / year! And the number of PMUs are expected to increase more and more in the future...
・ ・
・ ・
・
・
・ ・
PDC
PDC
PMU
PMU
PMU
PMU
Operator
94[bytes] 500 31536000 [seconds/year]
≒ 40 TB (8 phasors, 4 analogs, 2 digitals)
30 [messages/sec]
PMU data size × Number of PMUs × Message rate × Time
© Hitachi, Ltd. 2014. All rights reserved.
1-3 PMU Applications for Control Center
5
Historical Data Analysis Real-Time Data Processing
Wide-area monitoring
Voltage stability analysis
Oscillation analysis
Islanding
Mode meter
State estimation
Power plant model validation
Frequency response analysis
Post event analysis
Baselining
Fault location
・
・
・
・
・
・
To realize applications using PMU data, both “Real-Time Data
Processing” and “Historical Data Analysis” are required.
© Hitachi, Ltd. 2014. All rights reserved.
2. PMU Data Analysis Platform for Control Center
6
© Hitachi, Ltd. 2014. All rights reserved.
2-1 PMU Data Analysis Platform for Control Center
7
PDC PMU
PMU
PDC PMU
PMU
Super
PDC
Stream Data
Processing
In-memory
Data Cache
Analysis
Applications
Send
Sensor Data (every a few sec)
Online Power System Analysis
Load
Sensor Data
(every 5 to10 min)
Model updates
Model
DBMS
Archive
Files Access
Historical Data
(in 2 to 3 sec)
Real-Time Data Processing
Historical Data Analysis
Off-line Analysis
Historical Data Analysis: Quick Data Loading and High-Speed data access to historical data, comfortable data viewing and quick power system analysis.
Real-Time Data Processing: Online power system analysis using the PMU data in real time.
© Hitachi, Ltd. 2014. All rights reserved.
2-2 History of Hitachi’s Database Management System
8
© Hitachi, Ltd. 2014. All rights reserved.
Hitachi Advanced Data Binder Platform *1 (HADB) - x100 fast Database Engine *2
- Configured with Hitachi server and storage system □ High-speed Data Loading for Huge Data □ Multi-Table JOIN over Multiple Data Sources
Ad-hoc Data Analysis
High-speed Database Engine
Hitachi Server
Hitachi Storage
2-3 High-speed Data Access Platform for Control Center
9
What’s the New Technology ?
BI Customer Value
Business Improvement
MicrosoftⓇ
Excel
High-speed Data access Platform
Hitachi Advanced Data Binder Platform *1 The outcome of “Development of the fastest database engine for the era of very large database, and Experiment and evaluation of strategic social services enabled by the database engine” project (Principle Investigator: Prof. Masaru Kitsuregawa, University of Tokyo and also Director-General, National Institute of Informatics), supported by the Japanese Cabinet Office’s FIRST Program (Funding Program for World-Leading Innovative R&D on Science and Technology).
*2 Compared with Hitachi conventional technology. The performance was measured based on standard analytical benchmarks. The speeding-up effect on various queries are different. We compared the performance of searching a fixed amount of data which met a certain condition in the database.
© Hitachi, Ltd. 2014. All rights reserved.
2-4 Innovative Technology of Database Engine
10
“Out-of-Order Database Engine” technology *3
- SQL is automatically divided into thousands of task and each task issues asynchronous I/O to get data from disk - These tasks are executed by Out-of-Order control method Thousands of Asynchronous I/O improves the SQL processing
従来型データベースエンジン
時間
決定的な処理順序
同期入出力発行
超高速データベースエンジン
時間
超大量
非同期入出力発行
非決定的な処理順序
Conv. DB Engine (In-Order) Out-of-Order DB Engine
Most of CPU time is I/O wait SQL is executed by thousands of task
without waiting I/O completion
Sparse I/O Dense I/O
*3 A principle proposed by Prof. Kitsuregawa (Univ. of Tokyo and also Director-General, National Institute of Informatics) and Project Associate Prof. Goda (Univ. of Tokyo)
Small number of
Sync. I/O
Conv.
DB Engine In-order
processing
Out-of-Order
DB Engine Out-of-Order
processing
Large number of
Async. I/O
time time
© Hitachi, Ltd. 2014. All rights reserved.
Search processing (μs)
Synchronous I/O processing (ms)
Server
Storage
Server
Storage
[Conventional Technology] In-order exec.
[New Technology] Out-of-Order exec.*4
Task assignment
Search processing
Wait for I/O completion
Disk I/O
Key point
DB engine automatically divides search process into thousands of tasks and executed them asynchronously.
I/O wait time causes long proc. time
•Search process is divided into thousands of tasks, and they are carried out in parallel.
•Using asynchronous I/O, DB server issues as many I/O as possible.
•DB server continues execution of each task in I/O completion sequence.
A B C D
E
B
C
D
E
A
B
C
D
E
A
11
*4 A principle proposed by Prof. Kitsuregawa (Univ. of Tokyo and also Director-General, National Institute of Informatics) and Project Associate Prof. Goda (Univ. of Tokyo)
2-5 Technology in Detail
© Hitachi, Ltd. 2014. All rights reserved.
2-6 HADB broke TPC-H Benchmark Record
© Hitachi, Ltd. 2014. All rights reserved.
2-7 Demonstration
13
Please watch a Live Demonstration
© Hitachi, Ltd. 2014. All rights reserved.
2-7 Demonstration (Cont’d)
14
© Hitachi, Ltd. 2014. All rights reserved.
2-7 Demonstration (Cont’d)
15
© Hitachi, Ltd. 2014. All rights reserved.
3.Live Demo ~ PMU Application for Control Center ~
16
© Hitachi, Ltd. 2014. All rights reserved.
3-1 PMU Application for Control Center
17
Imports measurement data generated at a speed of tens of thousands per second without delay. Retrieves requested data within a few seconds from 100 TB scale historical data.
Technological Merits
HADB: Hitachi Advanced Data Binder
PDC: Phasor Data Concentrator
PMU: Phasor Measurement Unit
STATCOM: Static Synchronous Compensator
IS & R: Information Storage and Retrieval
Hydro power plant
Thermal power plant
STATCOM
Wind farm
PDC
Historical
Data
PMU
PMU
PMU
Information Storage
& Retrieval System
Transmission Network Operator
Operator Real-time
Check
1. Fault
2. Fault event
Sensor data are sent every a few seconds
Real-time problem analysis
HADB
Power Grid
3. Analysis
PMU
Visualize historical data
High-speed data import
(45 MBytes/sec; every 5 minutes)
Retrieves data in a few seconds
from a 100TB-scale historical data
© Hitachi, Ltd. 2014. All rights reserved.
3-2 Demonstration System Configuration
18
Client PC PMU Application for Control Center
Data source : Time series data from Phasor Measurement Units (dummy) - # of PMUs: 500 - Measurement interval: 33.3 milliseconds Data import operation: Imports data in every five minutes (400 MBytes) Size of historical Data: 1.2 Tbytes (14.5 Billion records, 11 days) DB server : HA8000/RS220 (12 cores/24 threads) + AMS2500 (64 HDDs) Client PC : Notebook PC DBMS : Hitachi Advanced Data Binder Platform
: Open Source DBMS
Voltage Trend of PMUs
PMU : Phasor Measurement Unit
HADB
DB Server
HA8000/ RS220
AMS2500
Historical Data
Application Server Sensor Data
Sensor data are imported periodically
Open Source DBMS
Historical Data
Traditional DBMS based IS & R
HADB based IS & R
1st Demo
2nd Demo
NW
© Hitachi, Ltd. 2014. All rights reserved.
3-3 Demonstration
19
1st Data Import Demonstration
2nd Data Retrieve Demonstration
© Hitachi, Ltd. 2014. All rights reserved.
3-4 Summary - Data Import Performance
20
4.5 MBytes/sec
45.0 MBytes/sec
x 10.0 Faster
Traditional DBMS based IS & R
HADB based IS & R
© Hitachi, Ltd. 2014. All rights reserved.
3-5 Summary - Data Retrieve Performance
21
14.5 sec 1.1 sec
x 13.2 Faster
Traditional DBMS based IS & R
HADB based IS & R
© Hitachi, Ltd. 2014. All rights reserved.
March 27, 2014
Hitachi America, Ltd.
END
22
•Infrastructure Systems Company, Power Information & Control Systems Division
•Information & Telecommunication systems Company, IT Platform Division Group, IT Platform R & D Management Division
Hitachi, Ltd.
Realization of Big Data Platform for PMU - Real-Time Data Processing & Historical Data Analysis -
© Hitachi, Ltd. 2014. All rights reserved.
Trademarks
23
• Microsoft and Excel are either registered trademarks or trademarks of Microsoft Corporation in the United
States and/or other countries.
• TPC Benchmark and TPC-H are trademarks of the Transaction Processing Performance Council.
• Other company and product names mentioned in this document may be the trademarks of their respective
owners. Throughout this document Hitachi has attempted to distinguish trademarks from descriptive terms
by writing the name with the capitalization style used by the manufacturer, or by writing the name with
initial capital letters. Hitachi cannot attest to the accuracy of this information. Use of a trademark in this
document should not be regarded as affecting the validity of the trademark.