streaming analytics better than batch when and why - (big data tech 2017)

© Copyright. All rights reserved. Not to be reproduced without prior written consent.

Streaming analytics better than batch - when and why ?

_A. Kawa - D. Wysakowicz - K. Zarzycki_


Have you ever built cool Big Data

pipelines?


Example Use-Case

■ Can be done in batch and real-time■ User session analytics at Spotify

● Simple stats■ Duration, number of songs, skips,

searches etc.

● Advanced analytics■ Mood, physical activity, real-time content,

ads


Example Output

How long do users listen to a new edition of Discover Weekly?

_1. Dashboards_


Example Output


Australian users are listening to Discover Weekly too short !!!

_1. Dashboards_ _2. Alerts_


Example Output


Australian users are listening to Discover Weekly too short !!!

Recommend songs and ads based on

current activity.

_1. Dashboards_ _2. Alerts_ _3. Content_


1st - Batch Architecture

1h1h

1h

1h - 1d

1hUser

EventsUser

Sessions


1st - Batch Architecture

1h1h

1h

1d

1hUser

EventsUser

Sessions


The More Moving Parts …

⬇ The higher learning curve⬇ The more gluing code⬇ The larger administrative effort⬇ The more error-prone solution


Long Waiting Time

Image source: “Continuous Analytics: Stream Query Processing in Practice”, Michael J Franklin, Professor, UC Berkley, Dec 2009 and http://www.slideshare.net/JoshBaer/shortening-the-feedback-loop-big-data-spain-external

http://www.slideshare.net/JoshBaer/shortening-the-feedback-loop-big-data-spain-external

http://www.slideshare.net/JoshBaer/shortening-the-feedback-loop-big-data-spain-external


2nd - Micro-Batch Architecture

1m - 1h


♪ ♪

No Built-In Session Windows

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

[10:00 - 11:00) [11:00 - 12:00)


Late Data …

♪ ♪ ♪ ♪ ♪ ♪ Event Time

14:55 - 16:35

Processing Time


... Included in Current Batch

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪

♪

14:55 - 16:35 16:50 - …

Event Time

Processing Time


Out-Of-Order Data …

♪ ♫ ♪ Event Time

Processing Time


Out-Of-Order Data …

♪ ♫ ♪ ♪ ♪ ♫

♪ ♪ ♫

Event Time

Processing Time


... Breaks Correctness

♪ ♫ ♪ ♪ ♪ ♫ ♪ ♫ ♪ ♬ ♫

♪ ♪ ♫ ♪ ♫ ♪ ♬ ♫♪

Event Time

Processing Time


Problems

FILES, BATCHES, DATA LAKES


Solving Streaming Problem With Batch?


3rd - Streaming-First Architecture


User Session Windows

♪User 1 ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

User 3 ♪ ♪ ♪ ♪ ♪ ♪

Session gap eg. 15 minutes

♪

♪♪

5


User Session Windows

♪User 1 ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

User 3 ♪ ♪ ♪ ♪ ♪ ♪

Session gap eg. 15 minutes

♪

♪♪

5

[3,2]


Reading From Kafkaval sessionStream : DataStream[SessionStats] = sEnv

.addSource(new KafkaConsumer(...))

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪♪ ♪ ♪ ♪ ♪ ♪ ♪


Session Windows With Gapval sessionStream : DataStream[SessionStats] = sEnv


.keyBy(_.userId)

♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪ ♪

♪ ♪ ♪ ♪ ♪ ♪ ♪

User 1

User 2


Session Windows With Gapval sessionStream : DataStream[SessionStats] = sEnv


.keyBy(_.userId)

.window(EventTimeSessionWindows.withGap(Time.minutes(15)))

User 1 ♪ ♪ ♪ ♪ ♪ ♪

Session gap - 15 minutes

♪♪


Analyzing User Sessionval sessionStream : DataStream[SessionStats] = sEnv


.keyBy(_.userId)


.apply(new CountSessionStats())

User 1 ♪ ♪ ♪ ♪ ♪ ♪ ♪♪


Handling Late Eventsval sessionStream : DataStream[SessionStats] = sEnv


.keyBy(_.userId)


.allowedLateness(Time.minutes(60))


User 1 ♪ ♪ ♪ ♪ ♪ ♪ ♪♪ ♪


Triggering Early Resultsval sessionStream : DataStream[SessionStats] = sEnv


.keyBy(_.userId)


.trigger(EarlyTriggeringTrigger.every(Time.minutes(10)))



User 1 ♪ ♪ ♪ ♪ ♪ ♪ ♪♪


Sessionization Exampleval sessionStream : DataStream[SessionStats] = sEnv


.keyBy(_.userId)


.trigger(EarlyTriggeringTrigger.every(Time.minutes(10)))



Working example:https://github.com/getindata/flink-use-case

https://github.com/getindata/flink-use-case

https://github.com/getindata/flink-use-case


Modern Stream Processing Engines

■ Rich stream processing semantic● Built-in support for event-time windows● Accurate results for late / out-of-order events and replays● Early triggers

■ Low latency and high-throughput■ Exactly-once stateful processing


Modern Stream Processing Engines

■ Rich stream processing semantic● Built-in support for event-time windows● Accurate results for late / out-of-order events and replays● Early triggers

■ Low latency and high-throughput■ Exactly-once stateful processing

User survey:http://data-artisans.com/flink-user-survey-2016-part-1http://data-artisans.com/flink-user-survey-2016-part-2

http://data-artisans.com/flink-user-survey-2016-part-1/





How can I reprocess data?


Reprocessing Events In Flink

1. Take periodic snapshots of a job● It stores Kafka offsets, on-flight sessions, application state

2. Restart a job from a savepoint rather than from a beginning


What if data is no longer in Kafka?


Consuming Data From HDFS

■ Run your streaming code on HDFS (bounded data)● You need to read data in event-time based order


How to join with other data sets/streams?


Join With Other Datasets / Streams■ Flink can join windowed streams easily■ Join of data stream with data set is WIP

● Even with slowly changing data set!● Even keyed data

Stream 2

Stream 1

Joined Stream Input Stream Joined Stream

+

Id Name

1 John Doe

2 Jane Doe

Dataset

+

https://issues.apache.org/jira/browse/FLINK-2320


When is batch processing good?


Batch Processing Use-Cases

■ Ad-hoc analytics and data exploration● Notebooks, Spark/Flink/Hive, Parquet, complete data sets

■ Technical advantages● A large swaths of historical data in HDFS● High-level libraries in mature batch technologies


Batch Processing Use-Cases

■ Ad-hoc analytics and data exploration● Notebooks, Spark/Flink/Hive, Parquet, complete data sets

■ Implementation advantages● Offline experiments over large historical data

■ Historical events are usually stored in HDFS, not Kafka

● High-level libraries in batch processing technologies■ Spark MLlib, H2O

(when data arrives continuously)

don’t solve streaming problem

with batch jobs


I like this streaming API.Can I use it for batch?


Unified batch and streaming API

■ Not with raw Flink API■ But with Flink Table API■ Apache Beam


Who Are You, actually?

■ At GetInData, we build custom Big Data solutions● Hadoop, Flink, Spark, Kafka and more

■ Our team is today represented by

Krzysztof Zarzycki

Dawid Wysakowicz

Adam Kawa


■ Stream often the natural representation of your data

■ Stream processing is not only about low latency

Summary


Q&A


Thanks ! Big Data Tech Warsaw !


Log Abstraction

11:00 - 12:00

12:00 - 13:00

… …

10:00 - …

10:00 - …

10:00 - 11:00


Spark Structured Streaming

⬇ It’s still ALPHA and the APIs are still experimental⬇ Operates on top of micro-batches (Spark SQL engine)

⬆ Easy-to-learn API (Dataset/DataFrame)⬆ Rich ecosystem of tools and libraries e.g. MLlib⬆ Supports event-time

⬇ Sessionization not yet supported - SPARK-10816⬇ Queryable state not yet supported - SPARK-16738

https://issues.apache.org/jira/browse/SPARK-10816

https://issues.apache.org/jira/browse/SPARK-16738


Kafka Streams

⬇ No exactly-once (just at-least-once)⬇ Kafka as the only data source⬇ No bounded streams (batch) optimizations

⬆ Simplicity⬆ Embedded into application⬆ Supports event-time

⬇ Lack of session windows


Apache Beam

⬆ Unified API for batch and streaming⬆ Rich streaming processing semantics

⬆ Complex TriggerDSL⬆ Multiple runtime environments

⬆ Spark, Flink, Apex, Dataflow⬆ Side inputs and outputs

⬇ Verbose Java API⬇ New project - Top level since 01/2017


Google Dataflow

■ Runtime environment for Apache Beam in Google Cloud

⬇ No support for Iterative Computations⬆ Supports Side Outputs⬆ Works with every Google Cloud Service (Pub/Sub, BigTable

etc.)

streaming analytics better than batch when and why - (big data tech 2017)

Technology