Upload File

the stream processor as a database - francisco...§streaming applications are often not bound by the...

  • Home
  • Documents
  • The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck
30
Ufuk Celebi @iamuce The Stream Processor as a Database

Upload: others

Post on 12-Jul-2020

0 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

UfukCelebi@iamuce

The Stream Processoras a Database

Page 2: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

The(Classic)UseCaseRealtimeCountsandAggregates

2

Page 3: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

(Real-)TimeSeriesStatistics

3

StreamofEvents Real-timeStatistics

Page 4: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheArchitecture

4

collect messagequeue

analyze serve&store

Page 5: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheFlinkJob

5

case class Impressions(id: String, impressions: Long)

val events: DataStream[Event] = env.addSource(new FlinkKafkaConsumer09(…))

val impressions: DataStream[Impressions] = events.filter(evt => evt.isImpression).map(evt => Impressions(evt.id, evt.numImpressions)

val counts: DataStream[Impressions]= stream.keyBy("id").timeWindow(Time.hours(1)).sum("impressions")

Page 6: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheFlinkJob

6

case class Impressions(id: String, impressions: Long)

val events: DataStream[Event] = env.addSource(new FlinkKafkaConsumer09(…))

val impressions: DataStream[Impressions] = events.filter(evt => evt.isImpression).map(evt => Impressions(evt.id, evt.numImpressions)

val counts: DataStream[Impressions]= stream.keyBy("id").timeWindow(Time.hours(1)).sum("impressions")

Page 7: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheFlinkJob

7

case class Impressions(id: String, impressions: Long)

val events: DataStream[Event] = env.addSource(new FlinkKafkaConsumer09(…))

val impressions: DataStream[Impressions] = events.filter(evt => evt.isImpression).map(evt => Impressions(evt.id, evt.numImpressions)

val counts: DataStream[Impressions]= stream.keyBy("id").timeWindow(Time.hours(1)).sum("impressions")

Page 8: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheFlinkJob

8

case class Impressions(id: String, impressions: Long)

val events: DataStream[Event] = env.addSource(new FlinkKafkaConsumer09(…))

val impressions: DataStream[Impressions] = events.filter(evt => evt.isImpression).map(evt => Impressions(evt.id, evt.numImpressions)

val counts: DataStream[Impressions]= stream.keyBy("id").timeWindow(Time.hours(1)).sum("impressions")

Page 9: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheFlinkJob

9

case class Impressions(id: String, impressions: Long)

val events: DataStream[Event] = env.addSource(new FlinkKafkaConsumer09(…))

val impressions: DataStream[Impressions] = events.filter(evt => evt.isImpression).map(evt => Impressions(evt.id, evt.numImpressions)

val counts: DataStream[Impressions]= stream.keyBy("id").timeWindow(Time.hours(1)).sum("impressions")

Page 10: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheFlinkJob

10

KafkaSource map() window()/

sum() Sink

KafkaSource map() window()/

sum() Sink

filter()

filter()

keyBy()

keyBy()

State

State

Page 11: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

Puttingitalltogether

11

Periodically(everysecond)flushnewaggregates

toRedis

Page 12: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TheBottleneck

12

Writestothekey/valuestoretaketoolong

Page 13: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

Queryable State

13

Page 14: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

QueryableState

14

Page 15: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

QueryableState

15

Optional,andonlyattheendof

windows

Page 16: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

QueryableState:ApplicationView

16

Database

realtimeresults olderresults

Application QueryService

currenttimewindows

pasttimewindows

Page 17: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

QueryableStateEnablers§ Flinkhasstateasafirstclasscitizen

§ Stateisfaulttolerant (exactlyoncesemantics)

§ Stateispartitioned (sharded)togetherwiththeoperatorsthatcreate/updateit

§ Stateiscontinuous (notminibatched)

§ Stateisscalable

17

Page 18: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

StateinFlink

18

window()/sum()

Source/filter()/map()

Stateindex(e.g.,RocksDB)

Eventsarepersistentandordered (perpartition/key)

inthemessagequeue(e.g.,ApacheKafka)

Eventsflowwithoutreplicationor synchronouswrites

Page 19: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

StateinFlink

19

window()/sum()

Source/filter()/map()

Triggercheckpoint Injectcheckpointbarrier

Page 20: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

StateinFlink

20

window()/sum()

Source/filter()/map()

Takestatesnapshot Triggerstatecopy-on-write

Page 21: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

StateinFlink

21

window()/sum()

Source/filter()/map()

Persiststatesnapshots Durablypersistsnapshots

asynchronously

Processingpipelinecontinues

Page 22: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

QueryableState:Implementation

22

QueryClient

StateRegistry

window()/sum()

JobManager TaskManager

ExecutionGraph

StateLocationServer

deploy

status

Query:/job/state-name/key

StateRegistry

window()/sum()

TaskManager

(1)Getlocationof"key-partition"of"job"

(2)Lookuplocation

(3)Respondlocation

(4)Querystate-nameandkey

localstate

register

Page 23: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

QueryableStatePerformance

23

Page 24: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

Conclusion

24

Page 25: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

Takeaways§ Streamingapplicationsareoftennotboundbythestream

processoritself.Crosssysteminteraction isfrequentlybiggestbottleneck

§ Queryablestatemitigatesabigbottleneck:Communicationwithexternalkey/valuestorestopublishrealtimeresults

§ ApacheFlink'ssophisticatedsupportforstatemakesthispossible

25

Page 26: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

TakeawaysPerformanceofQueryableState

§ Datapersistenceisfastwithlogs• Appendonly,andstreamingreplication

§ Computedstateisfastwithlocaldatastructuresandnosynchronousreplication

§ Flink'scheckpointmethodmakescomputedstatepersistentwithlowoverhead

26

Page 27: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

Questions?§ eMail:[email protected]§ Twitter:@iamuce§ Code/Demo:https://github.com/dataArtisans/flink-

queryable_state_demo

27

Page 28: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

Appendix

28

Page 29: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

Flink Runtime+APIs

29

DataStreamAPI

RuntimeDistributedStreamingDataFlow

TableAPI&StreamSQL

ProcessFunction API

Building Blocks: Streams, Time, State

Page 30: The Stream Processor as a Database - Francisco...§Streaming applications are often not bound by the stream processor itself. Cross system interactionis frequently biggest bottleneck

ApacheFlinkArchitectureReview

30


Bottleneck Identification and Scheduling in Multithreaded ... · a bottleneck waiters and the threads executing a bottleneck exe-cuters. A single instance of a bottleneck can be

SAP Event Stream Processor: Troubleshooting Guide · SAP Event Stream Processor: Troubleshooting Guide ... Project log files are generated regardless of ... SAP Event Stream Processor:

SuperCISC Compiler C to VHDL compiler Processor actors are bottleneck Hardware acceleration

Caches I CSE 351 Spring 2017 - University of Washington · L16: Caches I CSE351, Spring 2017 Problem: Processor-Memory Bottleneck 8 Main Memory CPU Reg Processor performance doubled

Stream Processor Architecture

RAY TRACING ON A STREAM PROCESSOR - Computer · PDF file · 2004-03-05RAY TRACING ON A STREAM PROCESSOR ... In this dissertation we show how a ray tracer can be written as a stream

SAP Sybase Event Stream Processor- Design Patterns: CCL and … · 2019-11-12 · SAP Sybase Event Stream Processor- Design Patterns: CCL and SPLASH (1st edition) SAP Sybase Event

Memory & Caches I...L16: Caches I CSE351, Spring 2020 Problem: Processor-Memory Bottleneck 10 Main Memory CPU Reg Processor performance doubled about every 18 months Bus latency

A SEMINAR ON 1 . CONTENT 2 The Stream Programming Model The Stream Programming Model-II Advantage of Stream Processor Imagine’s

SPR1100 Stream Processor

The Imagine Stream Processor

YABench: A Comprehensive Framework for RDF Stream Processor Correctness and Performance Assessment

EE482S Lecture 1 Stream Processor Architecturecva.stanford.edu/classes/ee482c/slides/lect01_slides.pdf– EE482C – stream processor architecture • Course format – Readings –

Lean IT in a nutshell · Kaizen Toyota Way TWI Muri Muda Mura Muri 4m Standardized work Work standards Andon Bottleneck Lead time Bottleneck Cycle time Value stream mapping Gemba

The Stream Processor as a Database Apache Flink

Parallel Computing using Memristive Crossbar …eecs.ucf.edu/~velasquez/IDT.pdfParallel Computing using Memristive Crossbar Networks: Nullifying the Processor-Memory Bottleneck Alvaro

Stream Architecture: Rethinking Media Processor Design

SAP Event Stream Processor: Installation Guide for … Event Stream Processor: Installation Guide for Solaris Content Starting the Cluster Database and an ESP Node.....52 Granting

Chasing Bugs with the BeepBeep Event Stream Processor

SAP Event Stream Processor: Security Guide

EE482S Lecture 1 Stream Processor Architecturecva.stanford.edu/classes/ee482s/slides/lect01_slides.pdf– DLP across stream elements – TLP across sub-streams and across kernels –

SAP Event Stream Processor: Troubleshooting Guide. 22 SAP Event Stream Processor: Troubleshooting Guide Monitoring and Analysis Tools

QPSK Transport Stream Processor - Pixelmetrixpixelmetrix.com/datasheets/QPSK.pdf · signal integrity service integrity remote monitoring QPSK Transport Stream Processor The QPSK TSP

SAP Event Stream Processor: Installation Guide for Windows · PDF filePUBLIC SAP Event Stream Processor 5.1 SP 10 Document Version: 1.0 – 2015-06-24 SAP Event Stream Processor: Installation

The Stream Processor as the Database - Apache Flink @ Berlin buzzwords

Topic 5: Memory System Basics ECE 4750 Computer …...Memory Technology • Motivation for Caches •Classifying CachesCache Performance CPU-Memory Bottleneck Processor Main Memory

RAY TRACING ON A STREAM PROCESSOR

QPSK Transport Stream Processor - Pixelmetrix

How did I get here? Building confidence in a distributed stream processor

Bottleneck Elimination from Stream Graphs S. M. Farhad The University of Sydney Joint work with Yousun Ko Bernd Burgstaller Bernhard Scholz

SAP Event Stream Processor: Cockpit Guide · PDF file5.4 Project Administration.....124 Adding a Project ... SAP Event Stream Processor: Cockpit Guide. Stream Processor

Apache Flink Big Data Stream Processing · PDF fileApache Flink Big Data Stream Processing Tilmann Rabl ... Apache Flink! The case for Flink as a stream processor • Ideal basis for

SAP Event Stream Processor: Getting Started Guide Event Stream Processor: Getting Started Guide ... 5.3 Editing in the CCL Editor ... SAP Event Stream Processor: Getting Started Guide

Reintroducing the Stream Processor: A universal tool for continuous data analysis

Stream Processing with Apache Flink · Log event analysis using a stream processor 11 Web server Web server Web server High throughput publish/subscribe bus Serving layer Stream processors