sc4 pilot 1 · 2020. 4. 7. · stream and batch processor l. selmi - bde - societal challenge on...

SC4 Pilot 1 L. Selmi - BDE - Societal Challenge on Transport Brussels 22.09.2016

Upload: others

Post on 21-Sep-2020

1 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

SC4 Pilot 1

L. Selmi - BDE - Societal Challenge on Transport

Brussels 22.09.2016

The Societal Challenge

L. Selmi - BDE - Societal Challenge on Transport

From a data perspective we see endless streams of data collected by sensors and devices immersed in a graph of infrastructures

Objective of the SC4 Pilots

L. Selmi - BDE - Societal Challenge on Transport

A scalable, fault-tolerant and flexible

platform based on open source frameworks

that can process unbounded data sets and

graphs.

Microservice Architecture

L. Selmi - BDE - Societal Challenge on Transport

Message Broker

L. Selmi - BDE - Societal Challenge on Transport

Apache Kafka is a high-throughput distributed durable messaging system

Apache Kafka

Kafka Cluster

L. Selmi - BDE - Societal Challenge on Transport

Apache Kafka

Stream and Batch

Processor

L. Selmi - BDE - Societal Challenge on Transport

Apache Flink is an open source platform for distributed stream and batch data processing.

Apache Flink

Flink Cluster

L. Selmi - BDE - Societal Challenge on Transport

Apache Flink

Storage and Indexing

L. Selmi - BDE - Societal Challenge on Transport

Elasticsearch is a distributed open source document database built on top of Apache Lucene

Elasticsearch Cluster

L. Selmi - BDE - Societal Challenge on Transport

Rserve

L. Selmi - BDE - Societal Challenge on Transport

R is a free software environment for statistical computing. It is used in the pilot to run the map-matching and the prediction algorithms.

The R Project

Pilot 1 Architecture

L. Selmi - BDE - Societal Challenge on Transport

BDE Components

L. Selmi - BDE - Societal Challenge on Transport

A Pipeline

L. Selmi - BDE - Societal Challenge on Transport

SC4 - Pilot 1 Visualization

L. Selmi - BDE - Societal Challenge on Transport

SC4 Pilot 1 can process

real-time FCD data for

map-matching and

aggregations (e.g. average

speed)

Future Work

L. Selmi - BDE - Societal Challenge on Transport

◎ Pilot 2: integration of the FCD historical data and prediction algorithm

◎ Pilot 3: more data sets (BT sensors, NYC taxi data, OpenStreetMap)

Database Systems 13 Stream Processing - GitHub Pages · Distributed stream processing engines, and “unified” batch/stream processing Proprietary systems: Google Cloud Dataflow,

Stream Processing with Apache Apex · Apache Apex - Overview • Apex - Enterprise grade, unified batch and stream processing engine • Highly Performant - Can reach single digit

Summer Internship PGDABM BATCH 2015-17 - …ccsniam.gov.in/images/pdfs/Batch_Profile_15_17_new.pdf · Summer Internship PGDABM BATCH 2015-17 . ... Bhaskar Yadav Stream : B.Sc

Pilot SC4 - Big Data Europe · Stream and Batch Processor BDE Workshop Brussels ... Apache Flink is an open source platform for distributed stream and batch data processing. Apache

Apache Beam: A unified model for batch and stream processing data

Goat Mountain Habib Selmi - Banipal

Data Management 13 Stream Processing - mboehm7.github.io · Distributed stream processing engines, and “unified” batch/stream processing Proprietary systems: Google Cloud Dataflow,

Contracts I - Selmi - Fall 2004_4

Adaptive Block and Batch Sizing for Batched Stream Processing …weisong.eng.wayne.edu/_resources/pdfs/zhang16-DyBBS.pdf · Spark Streaming is a batched stream processing frame-work,

Stream is the new Batch

Apache Flink · Stream and Batch in one System •Most systems are either stream or batch systems •In the past, Flink focused on batch processing –Flink‘s runtime has always

Big Data Europe Transport Pilot case, Luigi Selmi

Big Data: Challenges and Some Solutions207e4193-139b-40ec-9040... · 2017-11-29 · Stream and batch processing • Stream processing: continuous processing that continuously produces

Azure Stream analytics - GitHub Pages€¦ · Analytics on IoT Edge Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production

Digital Transformation · 2019-12-11 · Stream Sender . Batch Sender . Stream Receiver . SignalR . WebAPI . Security . UOM . Vertical Horizontal Splicing Coercion . Data Access

Ufuc Celebi – Stream & Batch Processing in one System

Micro-architectural Characterization of Apache Spark on Batch and Stream Processing Workloads

Apache Flink Meetup Berlin #6: Unified Batch & Stream Processing in Apache Flink

Unified Batch and Real-Time Stream Processing Using Apache Flink

Apache Flink- A System for Batch and Realtime Stream Processing

Neptune: Scheduling Suspendable Tasks for Unified Stream/Batch … · data across them, unified distributed dataflow frameworks evolved to support both batch and stream processing

Stream-wise breakup of 2015-17 Batch - ERNET year.pdf · Stream-wise breakup of 2015-17 Batch ... Biotechnology, Civil, Aerospace 22% Freshers ... Amrita School of Engineering, Bangalore

Unified Stream & Batch Processing with Apache Flink (Hadoop Summit Dublin 2016)

Dr Amina Aitsi Selmi

Batch and Stream Graph Processing with Apache Flink

Contracts I - Selmi - Fall 2003_4

About Intosaint, Nesrine Selmi, Algiers 8-9 April 2015

Daniel P. Selmi* - UC Davis School of Law · SELMI-63 STAN.L. REV. 591 (DO NOT DELETE) 3/6/2011 5:47 PM 591 THE CONTRACT TRANSFORMATION IN LAND USE REGULATION Daniel P. Selmi* Throughout

In-Memory Computing Summit Talk... · Pipeline API (Stream and Batch Processing) Streaming Readers and Writers (Kafka, FileWatcher, Socket, IMap & ICache streamer) Stream Processing

Adaptive Block and Batch Sizing for Batched Stream ...weisong/papers/zhang16-DyBBS.pdf · Adaptive Block and Batch Sizing for Batched Stream Processing System ... a batch size in

SL Name Stream & Batch

BREC – Batch Recorder Application for Dual Stream Recording … · 2011-05-29 · Merlin05 Batch Recorder (BREC) and Dual Stream Decoder Page 5 1. Overview 1.0 Introduction Skymicro’s

Torts Outline Selmi 2006 AWESOME With Rules & Cases & Hypos & Holdings

Simple)stream)processing - Computer Science …€¦ · Spark’“batch”’framework ... Discretized) Stream)Processing Spark Spark Streaming batches)of)X) seconds live data) stream

TOWARDS PORTABILITY AND BEYOND€¦ · THE BEAM MODEL • Unified batch and stream programming model • Stream: Batch is just a bounded stream • Massively parallelizable Transformations