sc2 workshop 1: big data europe (bde) - project overview & food workshop

BIG DATA EUROPEIntegrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges

negative conotationBig Data has often a

2 mai 2023www.big-data-europe.eu

Big Data in Marketing


Big Data in Intelligence


BigDataEurope aims to help maximizing the societal value of Big Data

Health, demographic change and wellbeing; Food security, sustainable agriculture and forestry,

marine and maritime and inland water research, and the Bioeconomy;

Secure, clean and efficient energy; Smart, green and integrated transport; Climate action, environment, resource efficiency and

raw materials; Europe in a changing world - inclusive, innovative and

reflective societies; Secure societies - protecting freedom and security of

Europe and its citizens.2 mai 2023www.big-data-europe.eu

The three Big Data „V“ – Variety is often neglected

Quelle: Gesellschaft für Informatik

© Fraunhofer-Allianz Big Data 7

Proactive Maintenance at Rolls Royce New Business Model integrating Sensor Data & Big Data Analytics

Dr. Dirk Hecker

Condition Monitoring, Proactive maintenance, „Power-by-the-

hour“,

as-a-service Business Model – payment modell by flight hours

Quelle: www.springboeck.ch/SR_Technics.htm

© Mark Hillary | Flickr


The rolling SmartphoneNew Business Models for the Automotive Industry with Data Value Chains

Dr. Dirk Hecker

Windshield wiper as rain sensors for micro wether prognosis

Automotive industry can become data provider for other industries

Quel

le: G

TÜQuelle: www.farm

ing-simulator.com


Predictive Analytics

Dr. Dirk Hecker

From Business Intelligence to Big Data Analytics

Business Intelligence Monitoring Predictive Analytics

What happened before?

What happens now?

What will happen soon?

What should happen?

Prescriptive Analytics„the last Mile“

“prescriptive analytics suggests decision options on how to take advantage of a future opportunity”

Quelle: BMW Quelle: www.7-forum.com Quelle: BMW Quelle: Volvo

BigDataEurope Rationale Show societal value of Big Data Lower barrrier for using big data

technologieso Required effort and resourceso Limited data science skillso Lack of Generic Architectures, components

Help establishing cross-lingual/organizational/domain Data Value Chainso Multiple Data Sourceso Required: Integration, Harmonisation2 mai 2023www.big-data-europe.eu

BigDataEurope: Objectives


COORDINATIONStakeholder

Engagement (Requirements

Elicitation)

SUPPORTDesign, Realise,

EvaluateBig Data Aggregator

PlatformCreate and

Manage Societal Big Data Interest

Groups

Cloud-deployment ready

Big Data Aggregator

Platform

CSA Measures

Results

Orthogonal Dimensions of Big Data Ecosystems

Generic Big Data Enabling Technologies

Data Value Chain

Data Generation & Acquisition

Data Analysis & Processing

Data Storage & Curation

Data Visualization &

Usage

Data-driven Services

Socie

tal C

halle

nges

Dom

ain

Spec

ific D

ata

Asse

ts

& T

echn

olog

y

Healthcare

Food Security

Energy

Intelligent Transport

Climate & Environment

Inclusive & Reflective Societies

Secure Societies

Stakeholder Engagement Cycle

© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS

Open-Source Technologies for Big Data Apps (small selection :-)

???

14


Big Data - Technologies

Volume

VelocityVarietyStorm

15


Groups of Technologies

Big Data Technolo

gies

Data Storage Technologies

Data Processi

ng

Workflow Coordina

tion

Querying/

Processing

Search

Data Export/ Import

Data AnalysisStatistics

Text Mining

16


Big Data Requirements

Analysis of historical dta Millions of entries Varying analysis quesitions Years of input data => Big Data Batch Processing

Interactive analysis by online queries Thousands of users online Extremely fast response

time Super high availability => Big Data Databases

Analysis of actual data with low latency in "real-time" React to newest trends Low-Latency change

detection Real-time online monitoring => Big Data Stream

ProcessingBut how to put it together ?17


The „traditional“ Hadoop Ecosystem + NoSQL componentsa Big Data Management System

ZooKeeper

askaban

Kafka

cassandravoldemort

MongoDBCouchDB

elastic searchsolrlucene


Batch Function

Speed Function

Data Storage

pages withpostings

Batch View

Realtime Viewme

ssag

e pass

ing

message passing

Application

Horizontal Scalability in the Lambda Architecture

19

> volume

> users

> users, volume

> velocity> volume, velocity

Blueprint of the Data Aggregator Platform

Follows typical Lambda Architecture

Integrated on top of existing Big Data distribution

+ Semantic Layer (Retaining Semantics using LD approach )

Batch Layer

Speed Layer

Data Storage

Real-time data &

Transactions …

Batch View

Real-time Viewm

essa

ge p

assin

g

message passing

Applications & ShowcasesReal-time dashboardsDomain-specific BDE apps

Big Data AnalyticsIn-stream Mining

BDE Platform &

Intelligence

Input dataStreamSpatialSocialStatistical TemporalTransactionalImagery

BDE Platform based on BigTop

Packaging Smoke testing VirtualizationPackage RPMs and DEBs, so that you can manage and maintain your own cluster.

Integrated smoke testing framework

Vagrant recipes, raw images, and docker recipes for deploying BigData infrastructures from zero.2 mai 2023www.big-data-europe.eu

+ Semantic Layer - Retaining Semantics using Linked Data

Data Aggregator Platform Challenges

Ingest semantic (RDF) and non-semantic (CSV, JSON, XML, …) datao Integrate various mapping techniques

(R2RML, CSV on the Web, JSON-LD) preserve semantics, provenance and

metadata in Big Data processing chainso Preserve URI/IRIso Preserve triples

Exploit semantics for aggregations2 mai 2023www.big-data-europe.eu

Current Activities – Year#1 2015 BDE Societal Workshops (7)

Plannedo Schedule on Website

7 W3C Interest Groups set up: Please Join!o SC1: HEALTH https://www.w3.org/community/bde-health/joino SC2: FOOD & AGRICULTURE https://www.w3.org/community/bde-food/o SC3: ENERGY https://www.w3.org/community/bde-energy/ o SC4: TRANSPORT https://www.w3.org/community/bde-transport/o SC5: CLIMATE & ENVIRONMENT

https://www.w3.org/community/bde-climate/o SC6: SOCIETIES https://www.w3.org/community/bde-societies/ o SC7: SECURITY https://www.w3.org/community/bde-secure-societies/www.big-data-europe.eu

http://www.big-data-europe.eu/first-round-of-bigdataeurope-workshops-announced/

https://www.w3.org/community/bde-health/join

https://www.w3.org/community/bde-food/

https://www.w3.org/community/bde-energy/

https://www.w3.org/community/bde-transport/

https://www.w3.org/community/bde-climate/

https://www.w3.org/community/bde-societies/

https://www.w3.org/community/bde-secure-societies/

BDE Partners

Sören AuerBig Data Europe CoordinatorFraunhofer IAIS & University of [email protected]

Thanks


mailto:[email protected]

Energy/Climate Example: Greenshifting


sc2 workshop 1: big data europe (bde) - project overview & food workshop

Data & Analytics