energaware d3.4 energy data management …energaware.eu/downloads/energaware_d3.4_energy data...

D3.4 – Energy data management,

communication and processing

Document number D3.4

Document title Energy data management, communication and processing

Version 1.0

Status Final

Work package WP 4

Deliverable type Report

Contractual date of delivery 31/07/2016

Actual date of delivery 27/01/2017

Author Luís Miguel Pinho (ISEP), António Barros (ISEP)

Contributors UOP, ISEP, UPC, ADV

Keyword list Data middleware platform

Dissemination level PU

D3.4 – Energy data management, communication and processing

Dissemination level: PU

EnerGAware - 649673 Version 1.0 of 26

Amendment history

Version Date Author (unit) Description

0.1 2016.09.08 ISEP ToC definition

0.2 2016.11.20 ISEP First draft for review

0.3 2016.12.07 UOP Review

0.4 2016.12.09 ADV Review

1.0 2016.12.30 ISEP Final version

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 649673. Disclaimer: The sole responsibility for the content of this material lies with the authors. It does not necessarily represent the views of the European Union, and neither EASME nor the European Commission are responsible for any use of this material.




Executive summary

This report presents the work of tasks 3.5 and 3.6 of the project, detailing the energy data collection and communication middleware platform implemented in the project, as well as the processing rules implemented to deliver the energy data and rewards to the game. For interoperability and reuse, the middleware platform is based on a private FIWARE instantiation.




Table of contents 1. ENERGAWARE ICT INFRASTRUCTURE OVERVIEW ................................................................................. 8

1.1 INTRODUCTION ............................................................................................................................................ 8 1.2 SYSTEM OVERVIEW ....................................................................................................................................... 8

2. ENERGY DATA MANAGEMENT PLATFORM......................................................................................... 11 2.1 INTRODUCTION .......................................................................................................................................... 11 2.2 ARCHITECTURE OF THE SOLUTION ................................................................................................................. 11

3. ENERGAWARE DATA MANAGEMENT AND COMMUNICATIONS ARCHITECTURE ............................ 14 3.1 ENERGAWARE DATA REPOSITORY ............................................................................................................... 14

3.1.1 Home energy consumption collection ................................................................................. 14 3.1.2 Weather data collection ........................................................................................................ 16 3.1.3 Home gameplay collection ................................................................................................... 17

3.2 DATA IMPORT SERVICES .............................................................................................................................. 18 3.2.1 Replicate the Concordia server database .......................................................................... 18 3.2.2 Updating the home energy consumption history ............................................................... 19 3.2.3 Automatic weather data retrieval ........................................................................................ 19 3.2.4 Household gameplay data retrieval ..................................................................................... 19

4. DATA PROCESSING AND EXPORT ...................................................................................................... 21 4.1 INTRODUCTION .......................................................................................................................................... 21 4.2 HOME ENERGY CONSUMPTION FILE EXPORT .................................................................................................. 21 4.3 WEATHER DATA FILE EXPORT ........................................................................................................................ 23 4.4 HOME GAMEPLAY FILE EXPORT .................................................................................................................... 23 4.5 COMPUTE ENERGY CONSUMPTION REDUCTION FOR GAME REWARDS .............................................................. 24




Figures Figure 1. System overview .................................................................................................................................. 9

Figure 2. Architecture of the solution ............................................................................................................. 12

Figure 3. Energy readings and weather data export widget .................................................................... 21




Tables Table 1. orion_history collection schema. ..................................................................................................... 15

Table 2. weather_data collection schema. ................................................................................................. 16

Table 3. game_data collection schema. ...................................................................................................... 17

Table 4. Line format of CSV energy data file export. .................................................................................. 22

Table 5. Line format of CSV weather data file export. ................................................................................ 23

Table 6. Line format of CSV game play data file export. ........................................................................... 24




Glossary and abbreviations

API Application Programming Interface

CSV Comma Separated Values

DBMS Database Management System

ECR Energy Consumption Reduction

FI-PPP Future Internet Public-Private Partnership

HTTP Hypertext Transfer Protocol

IoT Internet of Things

MADIS Meteorological Assimilation Data Ingest System

NOAA US National Oceanic and Atmospheric Administration

PWS Personal Weather Stations

UK United Kingdom

URL Uniform Resource Locator




1. EnerGAware ICT Infrastructure Overview

1.1 Introduction

The EnerGAware project aims to assess the contribution of a serious game to change energy consumption behaviour and habits towards a more efficient energy use. An analysis of such a contribution requires the collection of data to determine how energy use has evolved over time, and how this possible evolution is related with the experience of the game. Therefore, two main lines of data are collected:

Energy consumption over time

Game experience

The energy consumption (electricity and gas) is automatically monitored with a 15 minutes period, by reading the actual electricity and gas meters of the pilot homes. The series of collected values along time provide the evolution of energy consumption for each home. However, the energy consumption is related to the weather conditions, namely the air temperature. As such, an analysis must compensate for temperature variations along the year, which requires that daily weather parameters, especially air temperature, be recorded together with the energy consumption data.

The game experience for each home is given by the overall time that the home residents have played the game and the progress that they have achieved throughout the game, expressed by the sum of points earned.

The EnerGAware Middleware aggregates all this data (energy consumption, weather and game experience) that will be used in the post analysis to assess the contribution of the game in changing energy consumption behaviour.

1.2 System overview

The EnerGAware Middleware is responsible to maintain a repository of three different types of data, retrieved from different sources (as represented in Figure 1):




Energy consumption readings, available from the Advanticsys Concordia Server (Deliverable 4.1). This server is responsible to collect data received from the monitoring infrastructure and provides a set of services that export the energy consumption data.

Local (Plymouth) weather data, available from weather services (currently Weather Underground). The weather data can be provided by automatic online sources or by periodic files. Pilot households’ game experience are available from the Fremen Corp. Game Server. This server harvests the gaming experience data, and provides a set of services that aggregates and exports the relevant game experience data.

Figure 1. System overview

Besides collecting and storing the mentioned sets of data, the EnerGAware Middleware must export the same data in a suitable format for a post analysis conducted by the specialist partners of the EnerGAware consortium. These data are anonymised, such that it allows partners to discriminate between pilot homes, but it is impossible to identify the physical dwellings to which the data refers to.

The game incorporates a system of player rewards determined by energy savings in the real, physical world, such that these rewards can be used by the player to improve his progress in the




game. The EnerGAware Middleware must compute these real-world energy savings and provide this data to the Game Server.

Finally, data such as overall energy consumption and achieved reduction may be displayed to the general public, after being anonymised and aggregated, such that individual homes cannot be discerned nor identified.




2. Energy data management platform

2.1 Introduction

The EnerGAware Middleware is responsible for maintaining a repository of data from multiple sources:

Energy meter readings, whose infrastructure and operation is automatically managed by the Advanticsys Concordia Server. These data allow project partners to discriminate between homes, but it is anonymised such that it is impossible to determine the actual dwellings the data refers to.

Game experience, whose tracking is performed by Fremen Corp. Game Server. These data allow project partners to discriminate between homes, but it is anonymised such that it is impossible to determine the actual dwellings the data refers to.

Weather data, which can be sourced from either an automatic web weather service (Weather Underground), or weather services which provide periodic weather log files manually.

Furthermore, the EnerGAware Middleware must provide web services to conveniently export relevant data:

To the EnerGAware consortium partners: files containing the energy meter readings, weather data and game play experience.

To the Game Server: computed values of energy savings for each individual home.

2.2 Architecture of the solution

The EnerGAware Middleware is built upon the FIWARE platform1 which establishes a set of Application Programming Interfaces (API) which facilitates the development of applications for the Internet of Things (IoT). The FIWARE platform is supported by the Future Internet Public-Private Partnership (FI-PPP) project of the European Union2, and provides in public access an open-source

1 https://www.fiware.org 2 https://www.fi-ppp.eu/




reference implementation of each of its components. Typical IoT applications acquire data from multiple and diverse sources types but all related to a specific context, which is then processed and registered, such that applications can answer requests on the working context. The mission of the EnerGAware Middleware aligns with the IoT paradigm, therefore FIWARE is a viable platform to build upon the required functionality.

Within the FIWARE platform, the EnerGAware Middleware functionality is achieved by several components, represented in Figure 2:

the Orion context broker that manages the information including registrations and subscriptions of publishers/subscribers of data;

the mongoDB persistent database;

the Wirecloud web user interface;

the Context Manager that provides the modules handling all data acquisition and data export.

Figure 2. Architecture of the solution




The Context Manager module is responsible for the acquisition and processing of data from the energy meter readings, the game experience and the weather conditions. In addition, the Context Manager is also responsible to handle the requests of specific data (either bulk data for energy consumption behaviour analysis or game rewards based on energy savings). The Context Manager executes on top of a Node.js3 runtime environment and its functionality is, therefore, implemented in the Javascript programming language. The Node.js is an open-source event-driven Javascript runtime that executes on the server side. Node.js executes on a single thread, and I/O calls are treated asynchronously (non-blocking), such that other concurrent operations can be executed; when a I/O call finishes, a callback function is called to process the results. This characteristic allows Node.js to support multiple concurrent requests without the execution cost of thread context switching and the possibility of deadlock, as no locks are used.

The EnerGAware Middleware stores the acquired data persistently in a MongoDB4 database. MongoDB is an open-source document-oriented Database Management System (DBMS). Unlike traditional relational databases that organise data into tables and relations, MongoDB organises data in JSON5 documents (i.e. records) that are gathered in collections, instead. Furthermore, MongoDB allows dynamic schemas, such that documents in the same collection do not need to present the same format.

The web user interface is built upon the Wirecloud6, a FIWARE application mashup component. Wirecloud is a web mashup platform that allows an end user to create a personal dashboard from independent widgets. Wirecloud widgets are developed in HTML and Javascript, and must comply with a set of rules, such that they can be registered into a Wirecloud widget repository where users can select and load preferred widgets into his/her dashboard.

3 https://nodejs.org/ 4 https://www.mongodb.com 5 http://www.json.org 6 https://catalogue.fiware.org/enablers/application-mashup-wirecloud

3. EnerGAware data management and communications architecture

3.1 EnerGAware data repository

The EnerGAware data repository is supported by the MongoDB DBMS. MongoDB stores data in documents (records) that are grouped in collections. The EnerGAware Middleware keeps three collections:

orion_history – a collection of all energy readings collected from the participant pilot homes;

weather_data – a collection of daily weather parameters;

game_data – a collection of game experience data.

3.1.1 Home energy consumption collection

The orion_history collection is the repository where energy readings are stored in the EnerGAware Middleware. The Advantic Concordia system monitors the participant pilot homes and samples the respective energy meters (electricity and gas) with a period of 15 minutes. Each document (record) in the orion_history contains the valid energy readings from all homes sampled at a determined time instant. Consequently, each document contains the time stamp of the instant at which the Concordia server started polling the monitoring devices located at the participant dwellings, and an array of home records. Each home record contains the home identifier, a code that identifies the home (for internal differentiation purposes) but does not allow determining and locating the actual dwelling nor the tenants that live in that dwelling. In addition, each home record contains the sampled energy readings.

Each document in this collection complies with the format defined in Table 1.




Field Type Description

date Date time

Timestamp of the instant the Concordia server issued a polling request.

homes Array Array of individual home records. Each home has a private record containing the sampled values from the respective energy meters.

homes[]. id Text

Code that univocally identifies a home. It is the monitoring aggregator device identifier, in the format “EA #nnn”, where nnn is an 3-digit integer.

This code does not allow the identification of the actual dwelling or household tenants.

homes[]. type Text Type of record. The record represents a “Home”.

homes[]. signals Array Array of records of energy reading samples from one home.

homes[]. signals[]. value Float Value read from the energy meter.

homes[]. signals[]. type Text Type of record. The record represents a “number”.

homes[]. signals[]. modDate Date time

Timestamp at which the remote monitoring device transmitted the energy reading.

homes[]. signals[]. signalid Integer

Code that identifies univocally the sensor that read the energy meter. This code does not allow the identification of the actual dwelling or household tenants.

homes[]. signals[]. magnitude Text Type of physical measure. Possible values are “Active energy” (for electricity) and “Volume” (for gas).

homes[]. signals[]. unit Text Unit of the measurement. Possible values are “kWh” (for electricity) and “m3” (for gas).

Table 1. orion_history collection schema.




3.1.2 Weather data collection

The weather_data collection registers the daily weather statistics of Plymouth, UK, the city where the pilot dwellings are located (from Weather Underground). These data, namely the daily average temperatures are mandatory for the analysis of energy consumption behaviour, such that the effect of outside temperature can be compensated and energy consumptions can be compared despite different weather conditions between consecutive years.

Each document (record) is characterised by the date the weather data refers to; in addition each document contains data relative to temperature, pressure, humidity and wind. Each document follows the format defined in Table 2.


date_utc Date Date of the day that the weather data refers to.

mean_temperature Float Mean temperature of the whole day (in Celsius).

max_temperature Float Maximum temperature recorded during the day (in Celsius).

min_temperature Float Minimum temperature recorded during the day (in Celsius).

mean_pressure Float Mean atmospheric pressure recorded during the whole day (in Pascal).

mean_wind_speed Integer Mean wind speed recorded during the whole day (in Km/h).

Humidity Integer Mean relative atmospheric humidity recorded during the whole day (in percentage).

windDirection Text Predominant wind direction throughout the day (in cardinal/intermediate directions).

Table 2. weather_data collection schema.




3.1.3 Home gameplay collection

The game_data collection registers the overall progression in the serious game in each of the pilot homes. Each household may have more than one instance of the game being played by multiple people living in the same dwelling. The data in this collection aggregates the possible multiple game progressions per home.

This collection maintains a daily record of the game progression for every household. The schema of this collection is defined in Table 3.


date Date time Day date (YYYY/MM/DD) that the weather data refers to.

home_id Text Code that univocally identifies a home. This is the household identifier, in the format “DW nnn”, where nnn is a 3-digit integer.


time_playing Time Time (HH:MM:SS) that was spent playing the game. This value sums the playing times by all people living in the same dwelling identified by home_id.

missions_accomplished Integer Number of game missions accomplished while playing the game. This value sums the missions accomplished by all people living in the same dwelling identified by home_id.

score Integer Number of points earned while playing the game. This value sums the points earned by all people living in the same dwelling identified by home_id.

Table 3. game_data collection schema.




3.2 Data import services

The EnerGAware Middleware stores data that is collected from external sources (i.e. Advantic Concordia server, Fremen Corp game server, weather web service). The EnerGAware Middleware is designed to automatically retrieve and store the data from the multiple sources. The functionalities to retrieve data are implemented as routines that periodically issue requests to web services that provide specific and relevant data. The data import functionalities are the following:

import the energy readings and add to the energy consumption history;

import the game data and update the current game progression status;

import the weather data for a determined day and add to the weather history.

3.2.1 Replicate the Concordia server database

The Advantic Concordia server manages all operations of remote monitoring and maintains a database with the history of all values sampled from the energy meters at the pilot homes. The EnerGAware Middleware maintains an updated replica of this collection of sampled values.

The EnerGAware orion_history collection was initially loaded with all values that were already collected by the monitoring system, by a service that requests the whole data history until the time instant the service was called. This service was developed in Javascript and executes on the Node.js runtime.

This service first enquires the Advantic Concordia server for the list of all sensors that are deployed at the pilot homes, using the SOAP request getEnerGAwareSignals(). Once the service knows the list of code identifiers of all sensors, it requests all the values sampled by each sensor, using the SOAP request getEnerGAwareTimeSeriesData(). The returned energy meter values are registered into the orion_history collection.

Although the service that replicates the Advantic Concordia server database was called once to generate the first state of the orion_history collection, it can be used if whenever a new replica of the current state of the Advantic Concordia server data history is required.




3.2.2 Updating the home energy consumption history

The energy meters of the pilot homes are automatically monitored, and their current values are sampled every 15 minutes, once the monitoring sensors and aggregator are installed in a dwelling. These samples are sent to the Advantic Concordia server, which is responsible to manage the whole monitoring infrastructure.

The EnerGAware Middleware has a service that periodically requests the last valid (sensor errors are removed by pre-processing in the Concordia server) values sampled from the sensors, and stores them in the orion_history collection. This service was developed in Javascript and executes on the Node.js runtime, being activated every 15 minutes, to match the sampling period of the monitoring infrastructure. This service enquires the Advantic Concordia server for the last valid sampled values using the SOAP request getEnerGAwareSignalLastValue(); the returned energy meter values are registered into the orion_history collection.

3.2.3 Automatic weather data retrieval

The weather data from Plymouth is retrieved automatically from the web service Weather Underground (wunderground.com). Weather Underground is a weather service that provides worldwide historical weather data either from weather stations operating at airports, from the US National Oceanic and Atmospheric Administration (NOAA) Meteorological Assimilation Data Ingest System (MADIS), or from Personal Weather Stations (PWS) that voluntarily contribute to the Weather Underground service.

Weather Underground provides a public API that allows to request the weather data from one determined city or from a specific weather station, for a given past date.

The EnerGAware Middleware has a service that requests the weather data of the previous day and registers this data in the weather_data collection. This service is activated daily.

3.2.4 Household gameplay data retrieval

The progress experience by players of the serious game is tracked and registered in the Fremen Corp game server. This server provides a web service that returns the current accumulated game progress per household in terms of total hours played, missions accomplished and points scored, for all pilot homes selected to play the game; the returned data follows the JSON format.




The EnerGAware Middleware has a service developed in Javascript that executes on the Node.js runtime that requests the accumulated game progress for all households, and registers this data into the game_data collection. This service is activated daily.




4. Data processing and export

4.1 Introduction

The EnerGAware Middleware provides services that process and export selected data that is relevant to different types of users and systems, upon request. These web services provide energy consumption and weather data files (for analysis) and energy consumption reductions for game rewards.

4.2 Home energy consumption file export

The EnerGAware Middleware provides a web service to request a downloadable file containing a series of timestamped energy readings of the pilot homes. This service is available to EnerGAware consortium registered users with granted access to the Wirecloud FIWARE component (authentication by login). The web service is provided by a Wirecloud widget that is loaded into the EnerGAware consortium user dashboard (as depicted in Figure 3), where the user can select from a list of possible options:

Figure 3. Energy readings and weather data export widget




Export All Data from All Houses. This option exports all registered energy readings, from all homes.

Export All Houses Energy Data From Month / Year. This option exports all energy readings registered during the selected month.

Export All Houses Data Between Dates. This option exports all energy readings registered between two selected dates.

Export All Energy Data From One House. This option exports all registered energy readings from one selected home.

The resulting output becomes available as a downloadable CSV text file in which each line is a record of energy readings (electricity and/or gas) for a given home at a given time instant. The record fields are delimited by semi-colons (‘;’), and each line follows the format:

<Timestamp>;<Home ID>;<Electricity meter reading>;<Gas meter reading>

The field values are interpreted as defined in Table 4.

Field Description

Timestamp Timestamp of the energy readings, UTC time.

Home ID Code that univocally identifies a home. It is the monitoring aggregator device identifier, in the format “EA #nnn”, where nnn is an 3-digit integer.


Electricity meter reading Value (in kWh) that was acquired from the electricity meter sensor.

Gas meter reading Value (in m3) that was acquired from the gas meter sensor.

Table 4. Line format of CSV energy data file export.




4.3 Weather data file export

The EnerGAware Middleware provides a web service that generates a downloadable file containing the log of the daily weather data summary for a selected month. This service is available to EnerGAware consortium registered users with granted access (authentication by user login at the Wirecloud FIWARE component), where the weather data export widget allows the user to select the month and year, and the resulting output is a downloadable CSV text file in which each line is a record of weather parameters for a specific date. The record fields are delimited by semi-colons (‘;’), and each line follows the format:

<Date>;<Mean Temp.>;<Max. Temp.>;<Min. Temp.>;<Mean pressure>;<Humidity>

The fields are interpreted as defined in Table 5.

Field Description

Date Date of the day the weather data refers to, UTC time.

Mean Temp. Mean temperature of the day, in Celsius (C).

Max. Temp. Maximum temperature of the day, in Celsius (C).

Min. Temp. Minimum temperature of the day, in Celsius (C).

Mean pressure Mean atmospheric pressure of the day, in hectopascal (hPa).

Humidity Mean relative humidity of the day, in percentage (%).

Table 5. Line format of CSV weather data file export.

4.4 Home gameplay file export

The EnerGAware Middleware provides a web service that generates a downloadable file containing the log of the daily cumulative game play data for all homes. This service is available to EnerGAware consortium registered users with granted access (authentication by user login at the Wirecloud FIWARE component), where the game play export widget allows the user to obtain a downloadable CSV text file in which each line is a record of game play cumulative data for a home at a specific date. The record fields are delimited by semi-colons (‘;’), and each line follows the format:

<Date>;<Home ID>;<Time playing>;<Missions accomplished>;<Score>




The fields are interpreted as defined in Table 6.

Field Description

Date Date of the day the cumulative game play data refers to, UTC time.

Home ID Code that univocally identifies a home. This is the household identifier, in the format “DW nnn”, where nnn is a 3-digit integer.


Time playing The cumulative time playing the game until the date. This value sums the playing times by all people living in the same dwelling identified by home_id.

Missions accomplished The cumulative number of game missions accomplished while playing the game, until the date. This value sums the missions accomplished by all people living in the same dwelling identified by home_id.

Score Cumulative number of points earned while playing the game, until the date. This value sums the points earned by all people living in the same dwelling identified by home_id.

Table 6. Line format of CSV game play data file export.

4.5 Compute energy consumption reduction for game rewards

The EnerGAware Middleware provides a web service that computes the energy consumption reduction of a selected home, comparing the energy utilisation between two different time periods. The energy savings can then be converted into game rewards.

This web service implements processing rules reduced and adapted from task 5.1 (Deliverable 5.1). The analysis presented in Deliverable 5.1 depends on a weather normalization coefficient which will be determined offline, based on the full data. This is thus not implemented in the real-time consumption reduction for game rewards. Also, due to the deployment of the monitoring




infrastructure, houses have different baseline periods which may prevent yearly comparisons. Therefore weekly and monthly comparisons were also implemented.

This web service is available as a HTTP POST request available at the URL

https://<server address>/getSelectedHouseECR

The POST request requires two parameters in the request message:

selectedHouse – the DW code of the selected home, in the form “DW #nnn”, and

timeService – one of four possible terms: “week”, “month”, “year_week”, “year_month”.

The timeService parameter defines the two different period that must be compared:

week – the energy consumptions of the most recent two consecutive weeks is compared;

month – the energy consumptions of the most recent two consecutive months is compared;

year_week – the energy consumption of the most recent week is compared with the consumption of the equivalent week in the past year;

year_month - the energy consumption of the most recent month is compared with the consumption of the equivalent month in the past year.

The Energy Consumption Reduction (ECR) is the ratio of the difference between the energy consumption in the past period (Epast period) and the energy consumption in the current period (Ecurrent period, i.e. last week or last month), and the energy consumption in the past period is given in percentage, as defined in equation:

ECR = (Epast period – Ecurrent period) / Epast period x 100

The computation of the ECR for the purpose of game rewards does not compensate different weather conditions between both considered periods, because there is no statistical data to determine the coefficient by which the outside mean temperature affects the energy consumption. Such coefficient is planned to be computed during the offline analysis in WP5.

The returned value of the Energy Consumption Reduction is in percentage (%). The returned value is “N/A” (not available) if there is no sufficient data to compute the ECR or the house code is not known.


References [1] EnerGAware D4.1 – Pilot implementation methodology, March 2016.

[2] EnerGAware D5.1 – Monitoring and evaluation methodology, October 2015.

energaware d3.4 energy data management …energaware.eu/downloads/energaware_d3.4_energy data...

Documents