time series forecasting in smart grid data managementdbst/material/20111025_165_boehm.pdf · data...

© Prof. Dr.-Ing. Wolfgang Lehner |

Time Series Forecasting in Smart Grid Data Management

Matthias Böhm

TU Dresden Database Technology Group

October 26, 2011

124, 165

© Prof. Dr.-Ing. Wolfgang Lehner | | 2

> Disclaimers

I’m not an expert from the energy domain but I will do my best to explain the context of smart grids.

Smart grids are a vision – be aware that we‘re talking about the future!

Data management in smart grids is a huge topic. This talk covers only a small, DB-relevant subset: time series forecasting.


1

2

3


> Transformation of the Energy Sector

Increasing Energy Demand Increasing world population Increasing industrialization Strong increase in MENA

(middle east, north africa) Moderate increase in EU (Europe)

Problems with Traditional Energy Resources Exhausted fossil resources (ratio 1:106) Risk/share of nuclear power

General Political Goal: Increased Integration of RES (Renewable Energy Sources) Different technologies Centralized and Decentralized


[Hans Müller-Steinhagen: „DESERTEC: Strom aus der Wüste für eine Klima und Ressourcen schonende Energieversorgung Europas”, TU Dresden, 2010]

(TUD @ DUN since 07/2011)


> Example Decentralized Generation

Photovoltaic Highest production

per required space EEG Sachsen: ca 911 kWh p.a. / kWp

Parents‘ House (9.8 kWp)

Sister‘s House (8.0 kWp)



> Types of Renewable Energy Sources

Storable Energy Hydro power Biomass Solarthermal power Geothermal power

Fluctuating Energy Wind power Photo Voltaic Waves / Tides

Fluctuations require balancing energy demand/supply

[Hans Müller-Steinhagen: „DESERTEC: Strom aus der Wüste für eine Klima und Ressourcen schonende Energieversorgung Europas”, TU Dresden, 2010]

CRES PV

22 kWp (Greece)

CRES Windpark 2410 kWp

(Greece)

1 Week (20.10-26.10)



> Outline

Motivation and Introduction Background Smart Grids MIRABEL Project

Time Series Forecasting in DBMS Background Advanced Analytics in DBMS Background Model-Based Forecasting Forecasting in Relational DBMS

Forecast Query Optimization Techniques Hierarchical Forecasting Context-Aware Model Maintenance Publish Subscribe Forecast Queries

Conclusion



>

Background Smart Grids



> Future Vision: Smart Grids

Smart Grids Increased flexibility of energy networks via ICT (monitor, control) Goals: more RES, active customer involvement, balancing demand/supply Example: EU European SmartGrids Technology Platform (set up in 2005)

http://www.smartgrids.eu/documents/vision.pdf

Peer-to-Peer

Micro-Grids

Large Power Plants

Virtual Power Plants

Data Management Challenges


Smart Meter: foundation for

smart grids (bi-directional

communication)

http://www.smartgrids.eu/documents/vision.pdf


> Overview Data Management Challenges

Large-Scale Distributed Systems Number of stakeholders, number of nodes, amount of data

High Availability / Fault Tolerance Basically available, soft state, eventual consistent

Near-Realtime Data Synchronization and Integration High update rates, low latency, protocol/schema/format heterogeneity

Advanced Analytics Time series forecasting, scheduling/ balancing, classification, clustering, association

rule mining, complex event detection



> People

Prof. Dr.-Ing. Wolfgang Lehner

[email protected]

Dr.-Ing. Matthias Böhm [email protected]

Dipl.-Inf. Ulrike Fischer [email protected]

Dipl.-Medien-Inf. Lars Dannecker [email protected]

http://www.mirabel-project.eu/


EU FP7 project

mailto:[email protected]






http://www.mirabel-project.eu/


> Some Smart Grid Research Projects

Project Description

ADDRESS Real-time communication architecture enabling active demand and real-time request responses

AEOLUS Real-time prediction and distributed control of large-scale off-shore wind farms

EDISON* Electric vehicles as storage in distributed and integrated markets and open networks

EU DEEP Business solutions for enhancing distributed energy resources via a demand-pull approach

FENIX Aggregation of distributed energy resources into large scale virtual power plants

MeRegio Regions with power supply systems that are optimized with respect to their greenhouse gas emissions

MIRABEL* Balancing of energy demand and supply based on specified consumption and production flexibilities

MORE MICROGRIDS

Multi-micro grids management operation systems and centralized/ decentralized control strategies

Smart House- /Smart Grid

Inter-influences of smart houses and smart grids and hierarchical control concept



> Balancing Potential

MeRegio Data (Karlsruhe/Stuttgart) Elasticity to dynamic pricing

of real customers Three levels (SNT, NT, HT),

varying during the day, announced day-ahead

Significant differences to control group of up to 10%

Additional Conclusions (SAP Research User Study, E-Energy) Acceptance depends on device (e.g., dish washer, tumble dryer) Acceptance if comfort not lost and user keeps full control

Realistic time shifts: 0.5 to 3 hours. Main motivator: financial benefit, ecological benefit nice-to-have Fear of inapplicability and loss of flexibility



>

MIRABEL Project



> MIRABEL Consortium

Project start: 01/2010 Project end: 12/2012



> MIRABEL Approach – Flex Offers

Consumer and producers (households, SMEs) Have schedulable (flexible) demand and supply flex-offers

Flexibilities Time (flexibility interval), Amount of electricity (profile), and/or Price


kW

t

8pm earliest starting time

6 am latest starting time

8 am

2h

Profile

Flex-offer Earliest starting time: 8pm

Latest Starting time: 6am

Profile: Washing machine, 30°, normal

Price: 15ct/kWh or less

Flexibility interval


> MIRABEL Approach – Big Picture

Supply Demand

Scheduling

FlexOffers with flexibilities

(time, amount)

Forecasting Aggregation BG 11 BG 12

BG 1

MBA

Balance Group 2

Use Cases (demand and supply):

- Production schedules - Electric heat pumps, - Electric vehicles, - Washing machines, - Dryer - Dishwashers, - Photo voltaics, - Urban wind, and - Micro combined heat and power



> Example Use Case: RES Integration


Demand

Supply

Flex-offers

Non-schedulable demand

Non-schedulable RES


> Motivation Forecasting

Objectives Accurate forecasting of energy demand and supply System architecture integration Efficiency, robustness, scalability

Flex Offers

Flex Offers

Supply Demand Scheduling

RES Supply Forecasts

Demand forecasts

FlexOffer Forecasts

Accurate/efficient forecasting as precondition for scheduling



> Outline




Conclusion



>

Advanced Analytics in DBMS



> MAD Skills

Magnetic „Attract data and

practitioners“ Use all available data

sources independent of their quality

Agile „Rapid iteration: ingest, analyze, productionalize“ Continuous and rapid evolution of physical and

logical structures ELT (Extraction, Loading, Transformation)

Deep „Sophisticated analytics in Big Data“ Extended algorithmic runtime environment Ad-hoc advanced analytics and statistics


[Jeffrey Cohen, Brian Dolan, Mark Dunlap, Joseph M. Hellerstein, Caleb Welton: MAD Skills: New Analysis Practices for Big Data. PVLDB 2(2):1481-1492 (2009)]


> Data Mining Techniques

Predictive Techniques Descriptive Techniques

Nominal Data

Numerical Data

Classification

Clustering

(Outlier Detection)

Time Series Forecasting

(Recommendations)

T1 T2

T3 T4

Regel:

Association Rule Mining

Sampling



> Time Series Forecasting in DBMS


Advanced Analytics / Forecasting in DBMS

Analysis (e.g., R, SPSS)

DBMS

Traditional (on top)

Full Integration

Predictive DBMS

DBMS

DBMS UDF UDF

Partial Integration (extension

functionalities) (bi-directional)

Analysis (e.g., R, SPSS)

[SIGMOD’10] [VLDB‘11]

[VLDB’07] [VLDB‘08] [CIDR‘11] [ICDE‘12]

[Microsoft’11] [Oracle‘11]

other commercial DBMS

(general/special- purpose)

[SIGMOD’06] [SIGMOD‘08]


>

Model-Based Forecasting



> Overview Model-Based Forecasting

Forecast Model Statistical time series description (model)

(Recursive) Forecasting Process Model Identification Model Estimation Forecasting and Model Update Model Evaluation Model Adaptation


Model Identification

Model Estimation

Forecasting

Model Evaluation

Model Adaptation

Model Maintenance

Time Series Data

Model Type AR(2)

Model Parameters φ1=0.55, φ2=0.45

New Time Series

Values Ui

Forecasting Values Fi+h

Model Creation Model Usage


> Model Identification

Forecast Model Types / Classes


Base Forecast Models

Exponential Smoothing

Machine Learning

(Auto)Regression

Domain-Specific Extensions

HWT (Single-Equation)

EGRV (Multi-Equation)

BN (Bayesian Networks)

SVM (Support Vector Machines)

SVR (Support Vector Regression)

ANN (Artificial Neural Networks)

Black-Box Gray-Box

White-Box

AR MA

ARMA ARIMA

SARIMA ARMAX

MLR (Multiple Linear Regression)

SESM (Single Exponential Smoothing)

DESM (Double Exponential Smoothing)

TESM / HoltWinters

(Triple Exponential Smoothing)


> Forecast Model Types

EGRV-Model (Engle, Granger, Ramanathan, and Vahid-Arraghi)

Multi-equation autoregressive model (ensemble) White-box model tailor-made for energy demand

Core Idea: Time Series Decomposition

Simple models with many specific variables (30-50)



> Model Estimation

Problem Instantiate a forecast model w.r.t.

meta model and training data set

Example Forecast Model Type AR(2):

Error Metric: MSE

Horizon h=1 Meregio Customer 40

Energy Demand

Parameter Estimator L-BFGS-B


2211ˆ −− ⋅+⋅= ttt yyy φφ

( )∑=

−n

iii yy

n 1

2ˆ1

eMSE=827,354.4

0.23

0.56

eMSE= 211,204.7


> Model Usage

Forecasting Use the estimated forecast model Create h forecast values (forecast horizon) Update model state for new measurements (e.g., exponential smoothing)

Example Forecast (EGRV) SMAPEvshort

=0.0021 SMAPElong

=0.0755


21 23.056.0ˆ −− ⋅+⋅= ttt yyy

30min 1year


> Model Maintenance

Model Evaluation Goal: Trigger model adaptation only if necessary

Fixed Interval Techniques (# updates, time interval)

Continuous Evaluation Techniques (threshold, on-demand)

Model Adaptation Goal: Adapt the forecast model to the changed time series (if necessary)

Model Re-Identification Model Re-Estimation (old model as start point)


h=2


>

Forecasting in DBMS



> Example Forecast Query


SQL Forecast Query

Logical Query Plan Forecast operator Ψ

(create/reuse + forecast)

SELECT S_Date, S_Qty FROM Article, Sales WHERE A_Anr = S_Anr AND A_Name = ’Article A’ FORECAST 2

… S_Anr S_Date S_Qty

… 1 2011-10-24 6

… 1 2011-10-25 5

… 2 2011-10-26 1

… 1 2011-10-26 7

A_Anr A_Name …

1 Article A …

2 Acticle B …

S_Date SUM

2011-10-24 6

2011-10-25 5

2011-10-26 7

2011-10-27 6

2011-10-28 6.5

Article

Result

⋈S_Anr=A_Anr

Sales

Ψk=2

Q:

σ A_Name= ‚Article A'

πS_Date, S_QTY


> Forecast Query Compilation

Logical Plan Rewriting Cost model: accuracy and efficiency Example

Physical Plan Rewriting Operator alternatives (create model, scan model, etc) Operator parameterization (model type, estimator, etc)


Increased Model Creation Efficiency Possibly Decreased Model Accuracy (Increased Plan Costs (join, project),

Model reuse possibilities)

Sales2

⋈Sales1.Date=Sales2.Date

πSales1.Date, Sales1.Amount – Sales2.Amount

Sales1

Ψk=2 Ψk=2

Sales2

⋈Sales1.Date=Sales2.Date

πSales1.Date, Sales1.Amount – Sales2.Amount

Sales1

Ψk=2


> Schema Architecture

From 3-Layer to 4-Layer Schema Architecture


App

T

Storage

External Schema

Internal Schema

Conceptual Schema

Logical data independence

Phyiscal data independence

T

Storage

Conceptual (Statistical)

Schema

Internal Schema

Conceptual (Data)

Schema

Logical data independence

Phyiscal data independence

App External Schema

T

Logical model independence

Transparency allows for optimizations (accuracy/efficiency)

3-Layer Schema Architecture (ANSI/SPARC)

4-Layer Schema Architecture


>

Physical Models ( )

Schema Architecture in Depth


Index Structures

Logical Access Paths

Physical Access Paths

B+-Tree BitMap Compression

Partitioning

Materializations

Base Relations R S

…

…

Conceptual (Statistical)

Schema

Internal Schema

Conceptual (Data)

Schema

Model/Time Series Index Structures

Configurations

Logical Models

T

M1 M2 M3

M1 M11 M12 := +

M11 AR(2), MSE, (R⋈ (σ Name=AS)),

L-BFGS-B, h=6, τSMAPE=0.1

Model Index

Skip-Lists

Similarity Indexes

Logical Computation

Schemes

Physical Computation

Schemes


> Outline




Conclusion



> Overview MIRABEL Forecasting Approach

Forecast Queries

Forecasting and Maintenance

Streams of New Measurements

Forecasting Component

Qi

Scheduling, Aggregation, Monitoring

Ui

Energy Producer (Supply)

Energy Consumer (Demand)

Forecast Queries

Forecast Models

Model Evaluation Model Adaptation

Domain-Specific Forecast Models:

EGRV, HWT

Publish/Subscribe Forecast Queries

Partitioning and Parallelization Physical

Design (Hierarchies,

Horizons, Sampling)

Weather Integration



>

Hierarchical Forecasting

[Ulrike Fischer, Matthias Böhm, Wolfgang Lehner: Offline Design Tuning for Hierarchies of Forecast Models.

BTW 2011:167-186]



> Problem and Solution Overview


1. Deploy once and use many times 2. Keep and maintain a subset of models

(Physical Design of FM Hierarchies) DWH Model Pool

Query Interface

Updates Forecast Queries

HTC

HD2

Mobiles

Smart

Nokia

SELECT date, SUM(sales) FROM facts WHERE pgroup = „HTC“ GROUP BY date FORECAST 1 month

Scan

Aggregate

BuildModel

Forecast

facts

Forecast

MHTC Forecast

MHD2

Forecast

MSmart

Aggregation

Forecast

MMobiles

DisAgg

Key

1. Aggregation

2. Disaggregation Model Advisor

Workload Preference

Create Configuration

Analyze

Error + Cost

Configuration


> Optimization Problem

Configuration CW

Find best set of forecast models for a multi-dimensional aggregation hierarchy and given workload

Cost Model Efficiency: Maintenance Cost BW (# FM in CW) Accuracy: Configuration Error EW (sum of errors over W using best)

Optimization Objective Linearized costs

Optimization Algorithms Problem: Exponential search space Greedy Algorithm (start bottom-up, monotonic maintenance costs) Heuristics (recursive, decomposition, correlation, disagg error)


]1,0[ with )1( minmax

∈

−+ ααα

BB

EE W

T

W

CW

Weighted Accuracy

Weighted Efficiency


> Experimental Evaluation

Complete (C) All models, only direct forecasts

Bottom-Up (B) Only models at level one, others use aggregation

Top-Down (T) Only one model for top element, others use disaggregation

Greedy (G)


Maintenance Costs Accuracy


>

Context-Aware Model Maintenance

[Lars Dannecker, Robert Schulze, Matthias Böhm, Wolfgang Lehner, Gregor Hackenbroich: Context-Aware Parameter Estimation for Forecast Models in the Energy Domain.

SSDBM 2011:491-508]




Problem of Evolving Energy Time Series Optimal parameters change over time (with seasonal behavior) Rough search space (many local minima)

Basic Idea Model adaptation by reusing forecast models

w.r.t. current context

Case-Based Reasoning (Learning how to solve new problems from past experience)

Retain: Save FM with their context

Retrieve: Search FM for current context

Revise: FM refined by local/global optimization Time Series Forecasting in Smart Grid Data Management

Problem-Solution Case Base

Revise Retain

Retrieve

}{ ip}{ ip}{ ip

}{ ip

Local/global optimization

}{ ip′


> Model History Tree

Decision Tree Decision node:

splitting attr, value Leaf node:

forecast models Splitting attribute (highest PIQR), splitting value (partitioning median)

Experimental Evaluation

Time vs. Accuracy; TSESM, UK Demand Time vs. Accuracy; EGRV, UK Demand

Most beneficial for complex forecast models

Robust w.r.t. different models, error metrics,

data sets …



>

Publish-Subscribe Forecast Queries

[Ulrike Fischer, Matthias Böhm, Wolfgang Lehner, Torben Bach Pedersen: Publish-Subscribe Forecast Queries.

submitted for publication]




Problem Applications often continuously require forecast

values and do complex processing Forecast queries inefficient if complex algorithm

or only small changes

Publish-Subscribe Forecast Queries (PSFQ) Notify application for significant new forecast values


Example MIRABEL Scheduling (Goal: Supply – Demand = 0)

Forecasting

Scheduling

Supply Demand

Subscribe Publish

SELECT datetime, energydemand FROM customers WHERE customer_id = 30 FORECAST 2 THRESHOLD 0.1

Different Possibilities 1) Send requested values h (many horizon violations) 2) Send max values h+max (many theshold violations) 3) Send h+k values


> Optimization Problem

Horizon Extension k

Find best horizon extension for given time series and subscriber

(Subscriber) Cost Model Horizon violation incremental costs FI

Threshold violation complete costs FC

Optimization Objective Determine k(s) that minimize overall costs

Optimization Algorithms


horizon extension

+⋅

−

+∆

++

∆≥+=

=

∆

=∆∑

otherwise )1(11

)(

)(

with

,

1,

kFk

DkhF

DkkhFC

CC

IC

C

Dk

n

iDktotal

ii

ii

Offline-Static Offline-Time Slices Online


> Experimental Evaluation

Offline Algorithms

Real-World Experiment


Offline-Static Offline-TimeSlice


> Outline




Conclusion



> Conclusions

Smart Grids Problems of traditional energy sources Flexible energy networks via ICT (RES, customers, balancing) Data management challenges (e.g., time series forecasting)

Time Series Forecasting Time series forecasting required for balancing Advanced analytics / time series forecasting in DBMS (functionality, transparency) Transparency allows for optimization (efficiency, accuracy) Forecast query processing and optimization techniques Domain-specific forecasting models and

optimization techniques

Lots of optimization potential and directions for future research


© Prof. Dr.-Ing. Wolfgang Lehner |


Matthias Böhm

TU Dresden Database Technology Group

October 26, 2011

124, 165, 206?


>

Adaptive Re-Optimization Project: GCIP Dissertation “Cost-Based Optimization of Integration Flows” (03/2011) First adaptive, cost-based re-optimizer for EAI, ETL, MOM systems

In-Memory Indexing / Query Processing Project: DEXTER Generalized prefix trees with transaction management Query processing on generalized prefix trees

Time Series Forecasting in DMS Project: MIRABEL, FFQ Efficient time series forecasting for evolving time series Forecast query processing and physical design tuning

Other Research Projects Resiliency-aware data management Architecture-aware adaptive query processing (Database programming languages)

My Background


time series forecasting in smart grid data managementdbst/material/20111025_165_boehm.pdf · data...

Documents