reactconf 2014 - event stream processing

Dr Andy Piper

Push Technology

Reappt, a Push Technology product offers the enterprise grade Diffusion technology as a

service.

{MAKING SENSE OF THE FIRE -HOSE IN REAL-TIME}

{EVENT PROCESSING}

Copyright Push Technology 2014

Time @cobbscomedyclub

The past, the present and the future walked

into a bar

It was tense


Will the real Andy Piper please stand up?

– CTO at Push Technology

– Ex-BEA/Oracle

– Spring contributor (Spring DM) and Author

– Standards contributor – OMG, JCP etc

– PhD, Cambridge, Distributed Systems

✗ ✗


Agenda in 140 characters

What is it - What not? Why? History. Measure

infinity. Windows. Queries. Going fast –

reliably, distributed, distributed and fast and big


What is Event Stream Processing?



• It’s not stream processing

– Typically focused on local parallelism

– I have opinions but they get me in trouble



• Not event passing

– Event exchange not processing, e.g. JMS

– Stateless



• Not event mediation (brokering)

– Filtering, routing, and enrichment, e.g. ESB

– Stateless



“Event Stream Processing deals with the

task of processing streams of event data

with the goal of identifying the meaningful

pattern within those streams” – Wikipedia



• ESP is about querying data

streams

– Looking for something

– Haystack won’t stay still!

– Answers depend on multiple events

– Extremely stateful

Where the interesting questions

are!


Meta-analogy

“Producing thrust with a scramjet has been

compared to lighting a match in a hurricane

and keeping it burning” - NASA

“Event stream processing is like looking for

a needle in a haystack in a hurricane”


It’s like an inverted database

Query Event

Data Query

RDBMS CEP

• Data is ‘dynamic’

• Queries are ‘static’

• Data is ‘static’

• Queries are ‘dynamic’


Why bother?

• Too much data

• Time is integral to the questions

• Data is moving too fast

• Databases assume static datasets

?


History – Two schools of thought

• Database and make it time driven

• Logic approach with time constraints


Stream Processing History

• Tapestry – ’92– Early inverted database (not Apache!)

• Materialized views – ‘95– [A. Gupta and I. S. Mumick. “Maintenance of materialized views: Problems,

techniques, and applications.” 1995]

• David Luckham coined term CEP – “The Power Events”. 2001– Logic-based CEP

– Company acquired by Avaya

• Michael Franklin– Dataflow processing in PostgreSQL

– [“TelegraphCQ: continuous dataflow processing.” 2003]

• Aurora – ‘03– [Cherniack et al – “Scalable distributed stream processing.” 2003]

• STREAM – ‘03– [Arasu et al – “STREAM: The Stanford Stream Data Manager.” 2003]

• Borealis – ‘05 – [Abadi et al – “The design of the Borealis stream processing engine.” 2005]


Some definitions

• Tuple – a multi-set of elements ( e1, e2, … en )– A single tuple is a monad!

• Event or Data Stream 𝑺𝒏 - any ordered pair 𝒔, ∆ 𝒏

– 𝑠 is an unbounded sequence of tuples and

– ∆ is an unbounded sequence of positive real time intervals

– 𝑠 and ∆ are of equal length

• Event stream processing transforms event streams into new event streams through queries

• Outputs and inputs continuous– Operators are continuous queries


How do you measure an event

stream if it’s unbounded?

How do you measure infinity?


Measuring infinity

• Don’t do it

– But just event passing – where is the fun in

that?!

• Synopses – store summary information

– Continuous average = running total + items

• Windows – define working set

– Continuous average over last N items


Measuring infinity


Types of window

• Sliding

• Jumping (batching)

• Partitioned

• Time-based

• Others


What to do with a working set?

• Windows define the scope of interest

• Run queries against working set as it

changes

– Continuous Queries


When should you run queries?

• Run queries when output is not

idempotent

• When is that?

– Contents of the window changes – maybe?

– Time advances – possibly?

– Depends on window and query

Linking cause and effect in an efficient manner

lies at the heart of CEP and is why the answer

is not simply programming


How can we define queries on windows?

• Describe queries on windows using a

SQL-like syntax

• [Arasu et al. – “The CQL Continuous Query

Language: Semantic Foundations and Query

Execution” 2003]

SELECT AVG(price) FROM stream [ROWS N]


Querying windows

• SlidingSELECT * FROM s [ROWS 4 SLIDE 4]

• PartitionedSELECT a, b FROM s

[PARTITION BY b ROWS 3]

• Time-basedSELECT * FROM s [RANGE 30 SECONDS]


How do you make it fast?

• Generally in-memory the only way

• Operate as a gigantic state machine and

optimize like crazy

– Go reactive!

– Talk to Martin


Why must it be fast?

• Not reactive streams!

• Flow control causes causal paradox

• Stream processing must keep up


How do you make it resilient?

• Making stateful systems resilient has

challenges

• State generally changing extremely quickly


Resiliency approaches

• Save all the things and replay

– But infinite data?!

– Sometimes possible because append-only

• Save all the state

– Assumes there is less of it

– State is changing rapidly

– Too rapid to be effective


Resiliency approaches

• Elsewhere checkpoint and record changes

– Maybe we can record state and things

– Many commercial systems do

• No recording - identical parallel systems

– Synchronization an issue

– Catch-up an issue


How do you scale stream processing?

• Follow the crowd

• Distribute processing

• Multiple input sources

– If independent

– Flume

– Kafka


How do you distribute stream processing?

• DAG of event streams

– Inputs and outputs are event streams

– Nodes are operators or groups of operators

– Nodes can be distributed


Apache Storm

• Toolkit for creating distributed event flows

• Bolts (operators) and spouts (sources)

• Composed using a Clojure DSL

• Storm runs topologies

– Map-Reduce jobs finish – batch

– Topologies process forever – continuous


Apache Storm – a toolkit for distribution

(topology

{"1" (spout-spec twitter-feed-spout)}

{"2" (bolt-spec {"1"} filter :p "status" )}

{"3" (spout-spec database :p "retail" )}

{"4" (bolt-spec {"2"} top-n)}

{"5" (bolt-spec {"3" "4"} join :p "item" )}

...

)


How do you reliably distribute?

• State is now distributed

– Synchronization all but impossible

– Deterministic if relative order is preserved• Depends on operators and their effect

• [L. Lamport - “Time, Clocks, and the Ordering of Events in a Distributed System.” 1978]

– In theory a replay of things through the network will recover the state

– Alternative of storing the state for all the operators is harder


How do you reliably distribute?

• Different classes of recovery

– [Hwang et al. – “High-Availability Algorithms

for Distributed Stream Processing”. 2005]

• Precise recovery – failure effects hidden perfectly

• Rollback recovery – no data loss, but outputs

may be duplicated

• Gap recovery – data lost during recovery

• Reliable distribution overlaps distribution

– Upstream backup, reactive streams?


Reactive stream processing

• Message/event driven

• Discussed resiliency

• Continuous queries == responsive

– Push towards on-line queries

• Elasticity – harder


Stream Processing with Data

• Time dimension to data problems

• Data dimension to stream problems

• JOIN streams to tables

• Easy when small

• Large datasets harder

– Cache join data in memory?

– Push query into datastore?


Stream Processing with Big Data

• Time dimension to Big Data problems

– Velocity (vvv) implies stream processing

• Large dataset problem domain

• But now the data is distributed!


Shortcomings of Big Data


Fast Data Architecture


Fast Data Architecture

• Similar to recoverable architectures:

– Snapshot (queries) + incremental updates

– Current state = known state + changes

– Requires static queries - cached results

• Spark does this quite well


Fast data technology

• Storm – topology deployment

• Spark – logic queries on RDDs

• Spark streaming– repeating snapshots / micro-batch

– Fast data-ish

• Flume – fast ingest of log data

• Kafka – pub-sub messaging as distributed commit log

• Hadoop streaming– create M-R jobs using executable scripts

• Hive

• Cloudera Impala – MPP SQL query engine on top of Hadoop


Summary


Future Stream Processing

• Ease-of-use– CQL or Graphical - both have drawbacks

– Queries get really complicated really quickly

• Ease-of-use + distribution– Real systems challenge

• Fast data architectures

• Real-time machine learning– Spark ML Library

– Hadoop Mahout

• Interactive streaming queries – declarative and caching– Hive and Spark

reactconf 2014 - event stream processing

Software