flink. pure streaming

Flink Pure Streaming

Paco GuerreroBig Data & Solutions Architect 9/21/16

Not for Geeks

Life as Time

4

Anything as Time

Flink as Time

Streaming vs Batch

7

“Abstraction of reality used to facilitate information processing”

MicroBatch Batch

Batch

Batch

Batch

All Input

Batch

Batch Job

All Input

Batch

Batch Job

All Input

All Output

Nothing about timeTimestamps used as trick to keep real time fingerprint

Streaming

“Continuous processing of data that is continuously produced”

Streaming

“Streaming is the next programming paradigm for data applications, and you need to start thinking in terms of streams”

“Continuous processing of data that is continuously produced”

Streaming

“Streaming is the next programming paradigm for data applications, and you need to start thinking in terms of streams”

“Continuous processing of data that is continuously produced”

Data Stream: Infinite sequence of data arriving in a continuous fashion.

Streaming

“Streaming is the next programming paradigm for data applications, and you need to start thinking in terms of streams”

“Continuous processing of data that is continuously produced”

Data Stream: Infinite sequence of data arriving in a continuous fashion.

Stream processing is the backbone of the new data infrastructure.

Streaming

“Streaming is the next programming paradigm for data applications, and you need to start thinking in terms of streams”

“Continuous processing of data that is continuously produced”

Data Stream: Infinite sequence of data arriving in a continuous fashion.

Stream processing is the backbone of the new data infrastructure.

“The world beyond batch” A high-level tour of modern data-processing concepts. By Tyler Akidau

August 5, 2015 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

Streaming

Streaming

Streaming Job

Streaming

Streaming Job

Streaming

Streaming Job

Real Life Time !!

Streaming is the biggest change in

data infraestructure since Hadoop

Streaming

The biggest change is moving from

batch to streaming is handling time explicitly

Streaming

Micro Batch

Micro Batch

Micro Batch

Batch Job 1

Batch JobBatch Job 2

All Output

Batch Job 1

Micro Batch

Batch JobBatch Job 2

All Input

All OutputBatch Job 3All Output

All Output

Batch Job 1

Batch Frequency ?Timestamps keeps real time fingerprint

Micro Batch

Streaming Technologies

Batch StreamingMicro Batch

StateLess – Record acknowledgementsCPU bounded performanceNot expressive declarative functional API – Low Level APINot auto scalingLow level programmatic topology Poor Streaming Windows funcionalitiesNot compatible with Hadoop APIs

Streams

Streaming Technologies

Batch StreamingMicro Batch

StateLess – Record acknowledgementsCPU bounded performanceNot expressive declarative functional API – Low Level APINot auto scalingLow level programmatic topology Poor Streaming Windows funcionalitiesNot compatible with Hadoop APIs

Streams

Streaming Technologies

Batch StreamingMicro Batch

StateLess – Record acknowledgementsCPU bounded performanceNot expressive declarative functional API – Low Level APINot auto scalingLow level programmatic topology Poor Streaming Windows funcionalitiesNot compatible with Hadoop APIs

Streams

Streaming Technologies

Batch StreamingMicro Batch

StateLess – Record acknowledgementsCPU bounded performanceNot expressive declarative functional API – Low Level APINot auto scalingLow level programmatic topology Poor Streaming Windows funcionalitiesNot compatible with Hadoop APIs

Streams

Streaming Technologies

Streaming Technologies

Batch StreamingMicro Batch

Streaming Technologies

Batch StreamingMicro Batch

Streaming Technologies

Batch StreamingMicro Batch

Apache Flink

38

FlinkOpen Source Stream Processing Framework. Last available Release 1.1.1

Top Level Apache Project since Dec '14

FlinkOpen Source Stream Processing Framework. Last available Release 1.1.1

Top Level Apache Project since Dec '14

Main FeaturesNative Stream Low LatencyHigh throughputStatefulExactly-one guaranteesDistributedExpressive APIsAnd more ….

FlinkOpen Source Stream Processing Framework. Last available Release 1.1.1

Top Level Apache Project since Dec '14

Main FeaturesNative Stream Low LatencyHigh throughputStatefulExactly-one guaranteesDistributedExpressive APIsAnd more ….

Flink Flink

Flink

Flink Integration

YARN upcoming...

Flink Integration

Flink Integration

YARN upcoming...

Flink Integration

YARN upcoming...

upcoming...

Flink Integration

YARN upcoming...

Flink Stack

Flink Runtime Engine

Flink Runtime Engine

Distributed pipelined processing

Execute everything as Stream

Iterative ( cyclic ) dataflows

Mutable state in operations

Operate on managed memory (*)

Also works on batch !!

Job Manager

Client

Optimizer

Dataflow Graph

Flink Runtime Engine

Distributed pipelined processing

Execute everything as Stream

Iterative ( cyclic ) dataflows

Mutable state in operations

Operate on managed memory (*)

Also works on batch !!Workers ( Task Managers )

Job Manager

Client

Optimizer

Dataflow Graph

Execution Graph

Flink Runtime Engine

Stream Job

Batch Job

ML Job

Flink Runtime Engine

Graph Job

optimizer

optimizer

optimizer

optimizer

Stream Job

Batch Job

ML Job

Flink Runtime Engine

Graph Job

optimizer

optimizer

optimizer

optimizer

Tasks scheduled and executed in workers ( slots )

Tasks as chain of operators

Run operator logic in a pipelined fashion

State is kept in operators

Stream Job

Batch Job

ML Job

Flink Runtime Engine

Graph Job

optimizer

optimizer

optimizer

optimizer

Tasks scheduled and executed in workers ( slots )

Tasks as chain of operators

Run operator logic in a pipelined fashion

State is kept in operators

Stream Job

Batch Job

ML Job

Flink Runtime Engine

Graph Job

optimizer

optimizer

optimizer

optimizer

Tasks scheduled and executed in workers ( slots )

Tasks as chain of operators

Run operator logic in a pipelined fashion

State is kept in operators

Stream Job

Batch Job

ML Job

Flink Runtime Engine

Graph Job

optimizer

optimizer

optimizer

optimizer

If you want to know one thing about Flink is that you don't need to know

the internals of Flink

Events Time &

Windows

Fault Tolerance &

CorrectnessState Handling

Low Latency &

High ThroughputAPI Libraries SQL

Building Blocks

Events Time &

Windows

Fault Tolerance &

CorrectnessState Handling

Low Latency &

High ThroughputAPI Libraries SQL

Building Blocks

lTime references

lOut of order events

lPowerful Windowing

Event Times & Windowing

Event Times & Windowing

EventTime

EventTime

Event Times & Windowing

Flink Data Source

EventTime

EventTime

Ingestion Time

Event Times & Windowing

Flink Data Source

Flink Window Operator

EventTime

EventTime

Ingestion Time

Processing Time

Event Time: when data is generated

Ingestion time: when data is loaded from source

Processing time: when data is processed

Event time help to process out- of-order events and replay elements as the ocurred ( deterministic results )

Explicit handling of time. 3 choices:

Event Times & Windowing

Event Times & Windowing

Event time. Out or Order

1 2 3 5 7

4 6 8 9 10

Event time. Out or Order

1 2 3 5 7

4 6 8 9 10

Event time. Out or Order

Out or Order

1 2 3 5 74 6 8 9 10

1 2 3 5 7

4 6 8 9 10 1 2 3 5 74 6 8 9 104

Event time. Out or Order

Ingestion Time Windows

Out or Order

1 2 3 5 74 6 8 9 10

1 2 3 5 7

4 6 8 9 10 1 2 3 5 74 6 8 9 10

1 2 3

4

4 5

Event time. Out or Order

6 7 8 9 10

Event Time Windows

Ingestion Time Windows

Out or Order

1 2 3 5 74 6 8 9 10

Event time. Watermarks

1 2 3 5 7

4 6 8 9 10

Event time. Watermarks

1 2 3 5 7

4 6 8 9 10

1 2 3 54 6 8

1 2 3 54 6 8

1 2 3

4

4 5

Event time. Watermarks

6 8

Event Time Windows

Ingestion Time Windows

Out or Order

1 2 3 5 7

4 6 8 9 10

1 2 3 54 6 8 910

1 2 3 54 6 8 910

1 2 3

4

4 5

Event time. Watermarks

6 8 9 10

Event Time Windows

Ingestion Time Windows

Out or Order

Not event time before 5 will come

Late Time of 2

5

1 2 3 5 7

4 6 8 9 10

1 2 3 5 74 6 8 910

1 2 3 5 74 6 8 910

1 2 3

4

4 5

Event time. Watermarks

6 7 8 9 10

Event Time Windows

Ingestion Time Windows

Out or Order

Not event time before 10 will come

Late Time of 2

10

Windowing

Windows: grouping of events according to time, session*, count

Windowing

Windows: grouping of events according to time, session*, count

Powerful built-in windows:

Windowing

Windows: grouping of events according to time, session*, count

Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events.

Windowing

Windows: grouping of events according to time, session*, count

Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events.

Time: l Tumbling: trigger every X time with received events

l Sliding: trigger every X time with received events in last Y time

Windowing

Windows: grouping of events according to time, session*, count

Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events.

Time: l Tumbling: trigger every X time with received events

l Sliding: trigger every X time with received events in last Y time

Session: all events from session/user X until session time expired ( Gap )

Windowing

Windows: grouping of events according to time, session*, count

Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events.

Time: l Tumbling: trigger every X time with received events

l Sliding: trigger every X time with received events in last Y time

Session: all events from session/user X until session time expired ( Gap )

High level API for user windows: Window Assigner, Trigger, Evictor

Events Time &

Windows

Fault Tolerance &

CorrectnessState Handling

Low Latency &

High ThroughputAPI Libraries SQL

Building Blocks

lManaged operator state for backup/recovery

lSavepoints

Stateful Streaming

Op

Stateless StreamProcessing

Stateful Streaming

Op Op

State

Stateless StreamProcessing

Stateful StreamProcessing

lBuilt-in internal state in each operator for exactly-once semantics

lUser state can be declared in each operator to be saved locally in memory ( API, key/value pars )

lSnapshots: periodically local states in memory are persisted in lightweight distributed snapshots. No global pause !!

lCheckpoint as global consistent point-in-time snapshot build by set of distributed snapshots.

lPluggable state backend for snapshots:JobManager, HDFS, RocksDB

lSavepoints: user-triggered retained checkpoint

Events Time &

Windows

Fault Tolerance &

CorrectnessState Handling

Low Latency &

High ThroughputAPI Libraries SQL

Building Blocks

lExactly-once semantics with managed operator state

lDistributed Snapshotting Algorithm

Periodically

Chandy-Lamport Snapshots

“The global-state-detection algorithm is to be superimposed on the underlying computation: It must run concurrently with, but no alter, this underlying computation”

. Triggers snapshots asynchronously

. Embedded snapshots algorithm in stream of data ( barriers )

. No global pause, lightweight impact in performance

Handling Checkpoints

Periodically

Chandy-Lamport Snapshots

“The global-state-detection algorithm is to be superimposed on the underlying computation: It must run concurrently with, but no alter, this underlying computation”

. Triggers snapshots asynchronously

. Embedded snapshots algorithm in stream of data ( barriers )

. No global pause, lightweight impact in performance

Handling Checkpoints

Periodically

Chandy-Lamport Snapshots

“The global-state-detection algorithm is to be superimposed on the underlying computation: It must run concurrently with, but no alter, this underlying computation”

. Triggers snapshots asynchronously

. Embedded snapshots algorithm in stream of data ( barriers )

. No global pause, lightweight impact in performance

Handling Checkpoints

snapshot

Job Manager

Periodically pushes barriers for new state

New state X+1

Ack for Snapshot state X from Task N

Handling Checkpoints

snapshot

Job Manager

Handling Checkpoints

snapshot

Job Manager

Handling Checkpoints

snapshot

Job Manager

Handling Checkpoints

All Acks received

Register Checkpoint for restore in case of fail

Streaming Fault Tolerance

In case of fail, last global checkpoint is recovered ( recovery from partial checkpoint / individual snapshots is coming )

Need of stateful source like kafka to ensure end-to-end exactly-once semantic in case of fail.

Kafka sink doesn't guarantee end-to-end exactly-once ( multiple writes in topic ) ( at least-once )

Semantics in Flink:

At Least Once: never loses events, events might be reprocessed

Exactly once: neither reprocessed nor lost events.

Exactly once by default, with low impact in performance and development effort, unlike another tools like Storm or Spark.

If you want to know one thing about Flink is that you don't need to know

the internals of Flink

Events Time &

Windows

Fault Tolerance &

CorrectnessState Handling

Low Latency &

High ThroughputAPI Libraries SQL

Building Blocks

lPipelined runtime

lLatency vs throughput tunning

Exactly-once semantic with low impact in performance

Controllable checkpointing overhead

Higher throughput using processing time

Performance improvements thanks to:

. operator chaining during optimization phase

. own optimized serialization stack with code generation

Performance Tunning

Benchmark for “Streaming Computation” published by Yahoo. Dec 18, 2015https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at

Production use-case

l counting ad impressions group by campaign

l aggregations over a 10 second window

l save current aggregate value to Redis every second

Streaming Benchmark

Throughput vs Latency Graph

Throughput ( 1000 events / sec )

99 PercentileLatency ( ms )

Not Operator combinig in Storm, more complicate topology, more steps for events and more overhead

Apache Storm Without Tridentl At least once / Double counting after fail / Lost state after Failuresl CPU bounded

Apache Sparkl Latency increase with throughput

Apache Flinkl Exactly once / No double counting / No state lossl Limited by bandwidth between Kafka and Flink cluster l (1 GigE).

l kafka brokers within Kafka Cluster ( 10 GigE ) l Achieved 15 million messages /sec l ( before 3 million m/sec) with exactly once semantic

10,000,000 20,000,000

1 GigE

10 GigE

Performance Tunning

Events Time &

Windows

Fault Tolerance &

CorrectnessState Handling

Low Latency &

High ThroughputAPI Libraries SQL

Building Blocks

lHigh Level API

lWide range of basic and advanced operators

lJava , Scala. Python soon !!

API

API

Working on data streams ( bounded ? )

API

Working on data streams ( bounded ? )

Stream Processing: Explicit Handling of Time

API

Working on data streams ( bounded ? )

Stream Processing: Explicit Handling of Time

Java & Scala. Python coming. Java: Bean type classes vs Tuples with position addresses. Scala: case classes.

API

Working on data streams ( bounded ? )

Stream Processing: Explicit Handling of Time

Java & Scala. Python coming. Java: Bean type classes vs Tuples with position addresses. Scala: case classes.

Operators:

Sources: kafka, FileSystem, Cassandra …

Sinks: Kafka, HDFS, Cassandra ….

Transformations: Basic: map, flatmap, filter, grouping, iterate, project, join, cross, … Streaming: Windowing + Aggregations, Temporal Binary Iterative Stream operators

API

Working on data streams ( bounded ? )

Stream Processing: Explicit Handling of Time

Java & Scala. Python coming. Java: Bean type classes vs Tuples with position addresses. Scala: case classes.

Operators:

Sources: kafka, FileSystem, Cassandra …

Sinks: Kafka, HDFS, Cassandra ….

Transformations: Basic: map, flatmap, filter, grouping, iterate, project, join, cross, … Streaming: Windowing + Aggregations, Temporal Binary Iterative Stream operators

DataStream<?> DataSet<?>

Core API

1 implementation*, 2 interfaces

Source Map Reduce

Fliter

Join Sum Sink

Map

Source

Operators

Source Map Reduce

Fliter

Join Sum Sink

Map

Source

Operators

Source Map Reduce

Fliter

Join Sum Sink

Source

Filter

Operators

Source Map Reduce

Fliter

Join Sum Sink

Source

Filter

Operators

Source Map Reduce

Fliter

Join Sum Sink

Source

Reduce

Operators

Source Map Reduce

Fliter

Join Sum Sink

Source

Reduce

Operators

Source Map Reduce

Fliter

Join Sum Sink

Source

Join

Operators

Source Map Reduce

Fliter

Join Sum Sink

Source

Join

Operators

Source Map Reduce

Fliter

Join Sum Sink

Source

Operators

Events Time &

Windows

Fault Tolerance &

CorrectnessState Handling

Low Latency &

High ThroughputAPI Libraries SQL

Building Blocks

lEasy to use. SQL !!

lBased on Apache Calcite

API extension for DataSets y DataStreams

Based on relational Table abstraction

Table <=> Source / DataSet / DataStream

Operators like: where, select, as, groupBy, join, union, minus, distinct, orderBy, ...

Table API

Execute SQL-Like sentences on DataSets and Datastreams

Resuts returned as Table ( Table API ), convertible to DataStream or DataSets

SQL and Table API can be seamlessly mixed over DataStream/DataSets

Flink’s SQL support is not feature complete, yet.

Queries that include unsupported SQL will fail !!

SQL Support

SQL

Parsing and Logical plan for Table operators and SQL are optimized using Apache Calcite

Only supported a Subset of the comprehensive SQL standard

Apache Calcite provides with:

SQL Parsing

API for building expressions in relational algebra

Query planning engine

Provides SQL for Streaming Queries with windows aggregations

SELECT STREAM TUMBLE_END(rowtime, INTERVAL '1' HOUR) AS rowtime, productId, COUNT(*) AS c, SUM(units) AS units FROM Orders GROUP BY TUMBLE(rowtime, INTERVAL '1' HOUR), productId;

Apache Calcite

SQL Sentence

Apache Calcite: SQL to Logical

Plan as Relational Algebra

Flink Optimizer: Logical Plan to Execution Plan

If you want to know

one thing about Flink

is that you don't need

to know the internals of Flink

So … Batch

Batch on Stream

Stream: Unbounded Data Stream

Unbounded Data Stream

Batch on Stream

Stream: Unbounded Data Stream

Batch: Bounded stream ( dataset ) on a stream processor

Global window over the entire dataset

Optimization in operators for joins and grouping, with blocking data exchange if needed

Unbounded Data Stream

Bounded Data Set

Batch on Stream

Stream: Unbounded Data Stream

Batch: Bounded stream ( dataset ) on a stream processor

Global window over the entire dataset

Optimization in operators for joins and grouping, with blocking data exchange if needed

Unbounded Data Stream

Bounded Data Set

Batch on Stream

Stream: Unbounded Data Stream

Batch: Bounded stream ( dataset ) on a stream processor

Global window over the entire dataset

Optimization in operators for joins and grouping, with blocking data exchange if needed

Batch specific optimizations:

Cost-based optimizer: dataset size known before hand

Manage memory on / off-heap for join, sort, …

Optimization serialization stack for user-types

Bounded Data Set

Batch on Stream

Unbounded Data Stream

Conclusions

Conclusions

Conclusions

Flink Pure streaming engine matches real life. No Abstraction

Conclusions

Flink Pure streaming engine matches real life. No Abstraction

Batch on streaming

Conclusions

Flink Pure streaming engine matches real life. No Abstraction

Batch on streaming

Flexible Windowing Semantics with Explicit Time handling

Conclusions

Flink Pure streaming engine matches real life. No Abstraction

Batch on streaming

Flexible Windowing Semantics with Explicit Time handling

Competitive Performance, low latency and hight throughput

Conclusions

Flink Pure streaming engine matches real life. No Abstraction

Batch on streaming

Flexible Windowing Semantics with Explicit Time handling

Competitive Performance, low latency and hight throughput

Apache Beam, open sourced by Google, uses Flink as its first order runner forBatch and Streaming processing in partnership with Data Artisans.

100% Compliance of data processing model “what, where, when, how “

Indizen Technologies, S.L

Paseo de la Castellana, 130 - 4ª planta

28046 Madrid, Spain

Tel. 91 535 85 68

www.indizen.com

@indizen_corp¡gracia

s!

136

Francisco José Guerrero

Big Data & Solutions Architect

Tel. 91 535 85 68

Mov. XXX YYY ZZZ

p.guerrero@indizen.com