3/14/2013

1

Time Series Data Management in

Air Quality Monitoring

AQE 2013, Telford/UK

Edgar Wetzel, 14 March 2013

Brief company presentation

� KISTERS AG

� software company with core competence: RDBMS-based time-series management

� +1.000 B2B and G2B customers world-wide

� +500 employees in offices all around the globe (4 continents)

� 50 years of experience since the company started back in 1963

© KISTERS AG 14/03/2013 - 2 -

3/14/2013

2

Measurement data: short term use

monitoring station

• permanent = continuous in time

• fixed = stable in space

monitoring station

• temporary = discontinuous in time

• mobile = moving in space

measurement data

• <site><date+time><value><flags>

data user

validation +

analysis

agency

regulatory

reporting archiving

multiple measurement data

� a time series!

© KISTERS AG 14/03/2013 - 3 -

multiple software packages

from distinct datalogger

and instrument manufacturers

Measurement data: quantity and variety

� data are measured all over the place

meteorology

ambient air quality

CEMS

concentration

quantity

deposition

NO2

NOX

O3

PM10

PM2,5

CO2

wind speed + direction

°C

© KISTERS AG 14/03/2013 - 4 -

3/14/2013

3

Measurement data: quantity

� monitoring networks consist of numerous stations

back-

ground

traffic industrystation type:

� each stations monitors several meteorological and air quality parameters

� large-scale trends can be determined using trend analysis on long-term data sets

© KISTERS AG 14/03/2013 - 5 -

What happens to the measured data?

� it is processed and ends up in regulatory reports

© KISTERS AG 14/03/2013 - 6 -

3/14/2013

4

What happens to the measured data?

� next it is archived and more often than not ...

... forgotten

© KISTERS AG 14/03/2013 - 7 -

� further use is limited due to ...

� limited availability

� difficult accessibility

� incompatible file formats

� inconsistencies

� unstructured archiving

� unrelated data sets

Each data points has an economic value! -1-

� measuring data costs money

� equipment

� human resources

� maintenance

� archiving data costs money

� albeit prices for data storage are steadily declining

� thus price is no limitation to long-term archiving of mass data!

© KISTERS AG 14/03/2013 - 8 -

3/14/2013

5

Each data points has an economic value! -2-

� there is a scientific and economic value in the information

potentially derived from the data

� information must be derived from the data

� data from different disciplines must be correlated

� communication and warning:

� defects – preventive maintenance – disaster

� early detection of critical situations – transparent information policy

� information quality increases by means of ...

� tried & tested validation procedures

� optimized analysis/evaluation tools

� data correlation algorithms

© KISTERS AG 14/03/2013 - 9 -

Long-term data analysis

� each new measurement point increases simultaneously both

� the economic value of the time series

� the statistical significance of the data pool

� BUT ...

� using many software makes evaluation difficult

� data piles don’t have any statistical or economic value if they are not

� properly organized/structured/managed/documented(meta data)

� accessible

� available

� quality assured

© KISTERS AG 14/03/2013 - 10 -

Data piles need to be managed by a single enterprise solution

software to maximize return on investment!

3/14/2013

6

How to improve?

� Time-Series Data Management Systems

� Enterprise Solution ... RDBMS-based advanced ...

� time-series data management

� time-series data analysis

� multi-tier architecture

� manage in one DB both

� current data (measured & calculated)

� legacy data (measured & calculated)

� assure all of the following functionality:

– centralization,

– validation,

– aggregation,

– processing,

– alerting,

– reporting,

– dissemination, and

– public information.

GUI applications

application server

RDBMS

data base

© KISTERS AG 14/03/2013 - 11 -

AquisNet – in brief

� AquisNet == Air quality information system – NETworks

� software family providing modules for the 2 main blocks of an AQM network:

� Data Center:

– centralization, validation, aggregation, processing, alerting, reporting &

dissemination

– RDBMS-based advanced time-series data management and analysis

� DAS: optional data acquisition system, i.e. the monitoring station software

– sample data from analogue, digital and smart sensors/analyzers

– run function tests and calibration cycles

– plausibility testing and local data storage

© KISTERS AG 14/03/2013 - 12 -

Field-proven software built on 15 years of experience

in managing huge amounts of current and legacy AQM

data!

3/14/2013

7

Air Quality Information System: core functions

Configuration &

Definitions

system widecatalogues/ key list

predefined and userdefinabledata groups

regulatory compliance rules

(EC, US EPA, UBA ...)

data access permissions

& data views

Operational Meta Data Management

sitestation

(measurement location)

parameter(channel)

time series definition

Time Series Data Management

data feeds(import/export)

data validation andediting

calculation of derived time series

data publishing

Time Series Data Analysis

statistics and

air quality indicators

manual or

automated reporting

graphics and

numeric tables

Performance Reporting (KPI)

© KISTERS AG 14/03/2013 - 13 -

What’s a time series for KISTERS?

� any data that is linked to a location and has a date/time stamp

� KISTERS will increase the value of the TS data by means of ...

� data centralization and subsequent validation

� appending quality and other information to the measured value

� creating an auditable track of data versions

� providing an organized data source for other uses

� providing an organized data sink for other data providers

© KISTERS AG 14/03/2013 - 14 -

transaction historycurrent state of

production time series

data quality state ,

changes Y/N +comments

3/14/2013

8

© KISTERS AG 14/03/2013 - 15 -

Typical time series data management workflow

reporting

produce report chosen report format report

processing

aggregate, analyze|station meta data| -> sequence

(<date+time><calculated value><quality flags>)derived data: temporary or permanent

validation

increase quality and set additional QA flags accordingly

|station meta data| -> sequence (<date+time><value><quality flags>)

2nd ... xth level QA checked production data

centralization and storage

receive from station and record in RDBMS|station meta data| -> sequence

(<date+time><value><quality flags>) original measurement data

data transmission

plausibility check <date+time><value><quality flags> 1st level QA checked data

measurement

measure <date+time><value> raw data

Data centralization and storage

� AquisNet SODA - KISTERS’ telemetry unit

� Data collection and communication on all channels

� Internet, phone, GSM & GRPS, radio communication, satellite, web services

� supports both push and pull modes!

� Central soft- and hardware for all data

� Consistent planning and monitoring

� Data security & user management

� Powerful and scalable

3/14/2013

9

Data validation

� AquisNet supports both ...

� fully automated algorithm-based validation

� visual inspection and validation

� All validation steps are ...

� documented in data quality flags

� in a versioning system allowing traceability & rollback

� commented

AquisNet DMO Graph/Table Editor

time series

explorer

interlinked

table + graph

views

multiple

time series +

scales/legends

+ graph types

© KISTERS AG 14/03/2013 - 18 -

3/14/2013

10

Data processing

� data security by means of controlled redundancy

� all incoming data are write-protected and will remain untouched

� a copy will be produced for validation and further processing

� automated data aggregation

� derived time series (mean, max, min, percentiles, ...) are calculated

automatically as new data comes in

� on the fly calculation of

� air quality indicators

� statistical indicators

� temporary time-series for ad-hoc analysis

AquisNet – time series analysis

programmable equations:

transform input time

series into derived time

series

regression calculation:

graph view + regression

equation

© KISTERS AG 14/03/2013 - 20 -

3/14/2013

11

Data reporting

� reports can be ...

� pre-defined by the user

� executed automatically following a pre-determined schedule

� executed manually on demand

� in the ownership of the system (all users), a single user, a user group

� numeric tables or graphs or a combination of both

� data formats for reports ...

� national or international standard formats (EC, US EPA, UBA, etc.)

� standard file formats (csv, pdf, jpg, ...)

� user-defined formats (optional)

If it’s in the DB (monitoring & station data, other data) it can be in a report!

All tables & graphs on the screen can be part of a report!

Reporting can be manual or automated!

AquisNet REP – time series graphs

time series line graphs:

multiple series, w/o

numeric table

3/14/2013

12

AquisNet REP – sample line/bar graphs

XY graphs with

labelsbox-whisker plotsline + step chart

tiled bar graphs

with labels

tiled line graphs

with sum bars

scatter plot +

linear regression

pollution rose

AquisNet – polar plots (w/o table)

wind rose

3/14/2013

13

AquisNet WEB–(option) web site content generation

Air Quality Index

AquisNet Web generates dynamic content that you can insert into

the existing framework of your own web site design!

© KISTERS AG 14/03/2013 - 25 -

air pollution network

synoptic map

AquisNet Dashboard –(option) desktop + mobile app

3/14/2013

14

... mobile apps ...

WISKIOther

�data hub: search and retrieve the data you need...

Why KISTERS is involved in interoperability ...

where do I find the

data I need for my

climate change

research?

KISTERS

DATA HUB

INTERNET

Web

Services

Other

WISKI

Other

WISKI

Other

WISKI

Web

Services

send request

receive data catalogue

Web

Services

Web

Services

send specific request

receive TS data

3/14/2013

15

AquisNet – hosting options

customer premisesAquisNet can be hosted at the

customer’s or remotely in the cloud!

AquisNet Data Center - future proof investment

� modern technology: flexible, scalable and customizable

� multi-tier software architecture:

� RDBMS – Business Layer – Rich Client

� RDBMS – Business Layer – Web Server – Browser Client

� single or multi-user with full-featured user-administration

� easy to integrate in existing IT environments:

� choice of RDBMS’s (Oracle, MS SQL) & operating systems (MS Windows, Linux)

© KISTERS AG 14/03/2013 - 30 -

� independent of any instrument or data logger manufacturer

3/14/2013

16

KISTERS + AquisNet – advantages in a nutshell

� KISTERS: 50 years of existence in the world-wide market

� solid business, markets and technological experience

� presence in all 4 corners of the world:

� close to the customers

� multi-lingual staff & intercultural experience

� dynamic + enthusiastic staff: solid mix of experienced and young employees

� AquisNet: vertical solution for the air quality community

� especially developed for AQM and related applications

� built on KISTERS time-series technology: solid, proven, future-proof

� +15 years of AquisNet

© KISTERS AG 14/03/2013 - 31 -

� +25 years of experience in time series data management

� close supervision, testing and implementation of new technology

� secured RDBMS-based technology with built-in redundancy (backup)

� transaction log and full rollback

� truly field-proven in various environments: +1000 B2B/B2G customers

KISTERS AG

Charlottenburger Allee 5 Tel. 02 41 96 71-0 info@kisters.de

D-52068 Aachen Fax 02 41 96 71-555 www.kisters.de

Vielen Dank für Ihre Aufmerksamkeit.

Thank you for your attention.

Nous vous remercions de votre attention.