qualimaster project presentation - europa€¦ · qualimaster project presentation qualimaster...

QualiMaster Information Package (D7.2)

March 2014

QualiMaster Project Presentation QualiMaster Consortium

Motivation

Promise of high volume data stream analysis: New and faster

insights from in-depth analysis of detailed data (high volume data

streams) from multiple sources (multiple perspectives/factors)

Some road blocks:

Processing pipelines over very high volume data are already a

challenge (volume, changing frequencies/bursts, noisiness)

Processing infrastructures have to be configured for worst case

(wasting of capacities)

Systematic approaches for exploiting the potential of hardware-

based solutions are missing

Value adding but resource demanding algorithms can often only be

run offline (delay of information for decisions)

QualiMaster Project, GA 619525 Project Presentation, March 2014 2

Project Vision

Vision of QualiMaster :

make high volume real-time data processing

a highly opportunistic process

that flexibly exploits

– data sources

– reconfigurable hardware

– families of approximate algorithms

in a configurable, demand-driven and adaptive way.

Project Presentation, March 2014 3 QualiMaster Project, GA 619525

The Big Picture


Financial Applications

Big

Data

Algorithms & Quality

Configurable Hardware

Pipieline Adaptation

A data processing

infrastructure

mastering reactive,

proactive, and

reflective adaptation

based on available

computation sources,

load, and needs

Systematic exploitation of

reconfigurable hardware

for stream data

processing

Market-based real-time systemic risk monitoring based

on in-depth co-dependence modeling and monitoring

combined with use of social Web data for supporting

and stabilizing the prediction of systemic risk

Analysis of financial

data streams (hundreds

of millions messages

per second), social web

sources (e.g. Twitter

with about 5000

Tweets per second)

and other sources

relevant for financial

risk analysis

Quality modeling for data

stream processing

pipelines and exploitation

of algorithmic families

with different quality/

performance trade-offs

QualiMaster Project Goal

Create a configurable data processing infrastructure

for analytics over real-time, high volume, and bursty data streams

which autonomously masters reactive, proactive, and reflective

adaptations

based on available computation sources, load, and needs

Via runtime adaptation of data processing pipelines by

– Algorithmic families with different quality/performance trade-offs

– Exploitation of reconfigurable hardware for stream processing


The QualiMaster Consortium

Focussed: 5 Partners from 3 countries (DE, UK, EL)

Balanced: 2 SMEs, three academic partners

Complementing competences

L3S Research Center, Leibniz Universität Hannover

Maxeler Technologies Limited

Stiftung Universität Hildesheim

Telecommunication Systems Institute

Spring Techno GmbH & Co KG

6 Project Presentation, March 2014 QualiMaster Project, GA 619525

Expected Outcomes

Project Presentation, March 2014 7

Families of scalable data processing

algorithms

- Different quality/efficiency trade-offs

- Well-defined quality parameters

- Focus: algorithmic classes

Techniques and tools for translating

and optimizing data stream

processing algorithms into

reconfigurable hardware

Models and tool support for pipeline

configuration

- Based on algorithm families

- Considering constraints

- Defining adaptation space

Novel methods for dynamic quality

adaptation of data processing

pipelines

- Reactive, proactive and reflective

adaptation

- Also covering cross-pipeline settings

QualiMaster Project, GA 619525

Expected Outcomes


Configurable data processing

software infrastructure

• Provides platform for adaptive

pipeline execution

• Exploits general-purpose as well

as reconfigurable hardware

+ Tools + algorithm management

• Instantiation for financial domain

Financial Use cases


Systemic Risk Analysis (Core Use Case)

Motivation:

– Systemic risk is key factor for stability of financial markets

– Strong correlation of financial markets increases risk

Aim: Real-time systemic risk monitoring

Markets co-dependency modeling and monitoring

Use of social Web data for supporting and stabilizing the prediction

of systemic risk

Risk Assessment for institutional financial clients

Also uses multi-variant and multi-market approach (different target)

Pre-trading risk analysis

Real-time, real money trading risk assessment

Main Challenges in the Project

Research challenges in individual research areas

Bridging Disciplinary Gaps

– Bringing stream processing (steps) to hardware

– Coupling adaptation & algorithmic families (algorithm

development and evaluation vs. system building)

– ...

Really understanding the Application Requirements

– Capturing the complexity and requirements of Systemic Risk

Analysis

– Going beyond existing solutions

Integration into running infrastructure + evaluation


Research Challenges - Data:

Big Data challenges:

– Volume: terabytes of aggregated data

– Velocity: data volume is growing at a high rate

– Variety: heterogeneous sources (structures/unstructured)

Real-time processing challenges:

– Latency: analysis performed on live streams with minimum latency

– Uncertainty: need to cope with the incompletence of the data

(Social) Web data challenges:

– Noise: finding and filtering relevant data on the web

– Quality: data on the web is generated without quality control

– Privacy: exploiting user generated data without breaking privacy


Quality-aware adaptation

– Identifying, modeling and measuring quality parameters of algorithms

– Identifying, modeling and measuring trade-offs (cost vs. quality)

– Quality propagation, end-to-end quality requirements and cost constraints

– Reactive, proactive and reflective adaptations

Combination of software and hardware

– Unified quality-aware model for software and hardware alternatives

– Identifying of what should/could be translated to hardware, and when?


Research Challenges - Algorithms:

The Cloud Computing Approach:


Web data

crawling

Filtering/

Sampling Sentiment

analysis Trend predictions

more load more computational resources

less load less computational resources

The QualiMaster Approach:

Configurable and adaptable data processing pipelines

– alternative execution paths with different quality and cost constraints


Web data

crawling

Filtering/

Sampling Sentiment






Web data

crawling

Filtering/

Sampling Sentiment


low load execution path x

high quality output





Web data

crawling

Filtering/

Sampling Sentiment


High load execution path y

lower quality (bounded error)





Web data

crawling

Filtering/

Sampling Sentiment


Can (part of) this

be translated to

hardware?

Development Process:


Iterative and incremental development process (RUP)

Initial phase: requirements and design

Two major iterations

Final phase

12-2014 12-2015 12-2016

Development

Integration

Evaluation

Initialization Phase

1st Iteration

2nd Iteration

Finalization

Phase

7-2015 9-2016 6-2014

Miletones

Project structured by 6 Milestones


QualiMaster Milestones

MS1 Foundations 6

MS2 QualiMaster Approaches and Basic Infrastructure 12

MS3 Core Building Blocks for the QualiMaster Infrastructure 19

MS4 Interim QualiMaster Technology & Infrastructure 24

MS5 Advanced Building Blocks for the QualiMaster Infrastructure 31

MS6 Final QualiMaster Technology & Infrastructure 36

MS6

MS5

MS4

MS3 MS2

Milestones:


12-2014 12-2015 12-2016

Initialization Phase

1st Iteration

2nd Iteration

Finalization

Phase

6-2014 7-2015 9-2016

Foundations

Approach

and Basic

Infrastructure

MS1

Core

components

Interim

Infrastructure

Advanced

Components

Final

Infrastructure

www.qualimaster-project .eu


Stay Informed!

qualimaster project presentation - europa€¦ · qualimaster project presentation qualimaster...

Documents