![Page 2: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/2.jpg)
Who am I?• Director of Applications Engineering at data
Artisans• Previously working on streaming
computation at Twitter, Gnip and Boulder Imaging
• Involved in various kinds of stream processing for about a decade
• High-speed video, social media streaming, general frameworks for stream processing
![Page 3: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/3.jpg)
Overview• In stateful stream processing the bottleneck has
often been the key-value store• Accuracy has been sacrificed for speed• Lambda Architecture was developed to address
shortcomings of stream processors• Can we remove the key-value store bottleneck
and enable processing at in-memory speeds?• Can we do this accurately without Lamba
Architecture?
![Page 4: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/4.jpg)
Problem statement• Incoming message rate: 1.5 million/sec• Group by several dimensions and
aggregate over 1 hour event-time windows• Write hourly time series data to database• Respond to queries both over historical
data and the live in-flight aggregates
![Page 5: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/5.jpg)
Input and QueriesStreamtweet-id: 1,event: url-click,time: 01:01:01tweet-id: 2,event: url-click,time: 01:01:02tweet-id: 1,event: impression,time: 01:01:03tweet-id: 2,event: url-click,time: 02:01:01tweet-id: 1,event: impression,time: 02:02:02
Query Resulttweet-id: 1,event: url-click,time: 01:00:00 1tweet-id: 1,event: *,time: 01:00:00 2tweet-id: *,event: *,time: 01:00:00 3tweet-id: *,event: impression,time: 02:00:00 1tweet-id: 2,event: *,time: 02:00:00 1
![Page 6: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/6.jpg)
Input and QueriesStreamtweet-id: 1,event: url-click,time: 01:01:03tweet-id: 2,event: url-click,time: 01:01:02tweet-id: 1,event: impression,time: 01:01:01tweet-id: 2,event: url-click,time: 02:02:01tweet-id: 1,event: impression,time: 02:01:02
Query Resulttweet-id: 1,event: url-click,time: 01:00:00 1tweet-id: 1,event: *,time: 01:00:00 2tweet-id: *,event: *,time: 01:00:00 3tweet-id: *,event: impression,time: 02:00:00 1tweet-id: 2,event: *,time: 02:00:00 1
![Page 7: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/7.jpg)
Input and QueriesQuery Resulttweet-id: 1,event: url-click,time: 01:00:00 1tweet-id: 1,event: *,time: 01:00:00 2tweet-id: *,event: *,time: 01:00:00 3tweet-id: *,event: impression,time: 02:00:00 1tweet-id: 2,event: *,time: 02:00:00 1
Streamtweet-id: 1,event: url-click,time: 01:01:03tweet-id: 2,event: url-click,time: 01:01:02tweet-id: 1,event: impression,time: 01:01:01tweet-id: 2,event: url-click,time: 02:02:01tweet-id: 1,event: impression,time: 02:01:02
![Page 8: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/8.jpg)
Input and QueriesStreamtweet-id: 1,event: url-click,time: 01:01:03tweet-id: 2,event: url-click,time: 01:01:02tweet-id: 1,event: impression,time: 01:01:01tweet-id: 2,event: url-click,time: 02:02:01tweet-id: 1,event: impression,time: 02:01:02
Query Resulttweet-id: 1,event: url-click,time: 01:00:00 1tweet-id: 1,event: *,time: 01:00:00 2tweet-id: *,event: *,time: 01:00:00 3tweet-id: *,event: impression,time: 02:00:00 1tweet-id: 2,event: *,time: 02:00:00 1
![Page 9: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/9.jpg)
Query Resulttweet-id: 1,event: url-click,time: 01:00:00 1tweet-id: 1,event: *,time: 01:00:00 2tweet-id: *,event: *,time: 01:00:00 3tweet-id: *,event: impression,time: 02:00:00 1tweet-id: 2,event: *,time: 02:00:00 1
Input and QueriesStreamtweet-id: 1,event: url-click,time: 01:01:03tweet-id: 2,event: url-click,time: 01:01:02tweet-id: 1,event: impression,time: 01:01:01tweet-id: 2,event: url-click,time: 02:02:01tweet-id: 1,event: impression,time: 02:01:02
![Page 10: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/10.jpg)
Streamtweet-id: 1,event: url-click,time: 01:01:03tweet-id: 2,event: url-click,time: 01:01:02tweet-id: 1,event: impression,time: 01:01:01tweet-id: 2,event: url-click,time: 02:02:01tweet-id: 1,event: impression,time: 02:01:02
Query Resulttweet-id: 1,event: url-click,time: 01:00:00 1tweet-id: 1,event: *,time: 01:00:00 2tweet-id: *,event: *,time: 01:00:00 3tweet-id: *,event: impression,time: 02:00:00 1tweet-id: 2,event: *,time: 02:00:00 1
Input and Queries
![Page 11: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/11.jpg)
Time Series Data
01:00:00 02:00:00 03:00:00 04:00:000
25
50
75
100
125Tweet Impressions
Tweet 1 Tweet 2
![Page 12: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/12.jpg)
Any questions so far?
![Page 13: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/13.jpg)
Legacy SystemStream Processor
Hadoop
Lambda Architecture
Streaming
Batch
![Page 14: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/14.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
Stream Processor
![Page 15: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/15.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
Stream Processor
![Page 16: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/16.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
Stream Processor
![Page 17: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/17.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
Stream Processor
![Page 18: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/18.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
Stream Processor
![Page 19: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/19.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
Stream Processor
![Page 20: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/20.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
Stream Processor
![Page 21: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/21.jpg)
Legacy System
Lambda ArchitectureHadoop
Streaming
Batch
![Page 22: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/22.jpg)
• Aggregates built directly in key/value store
• Read/modify/write for every message
• Inaccurate: double-counting, lost pre-aggregated data
• Hadoop job improves results after 24 hours
Legacy System(Lambda Architecture)
![Page 23: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/23.jpg)
Any questions so far?
![Page 24: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/24.jpg)
Goals for PrototypeSystem
• Feature parity with existing system• Attempt to reduce hardware footprint by 100x• Exactly once semantics: compute correct results in
real-time with or without failures. Failures should not lead to missing data or double counting
• Satisfy realtime queries with low latency• One system: No Lambda Architecture!• Eliminate the key/value store bottleneck (big win)
![Page 25: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/25.jpg)
My road toApache Flink
• Interested in Google Cloud Dataflow• Google nailed the semantics for stream processing• Unified batch and stream processing with one
model• Dataflow didn’t exist in open source at the time (or
so I thought) and I wanted to build it.• My wife wouldn’t let me quit my job!• Dataflow SDK is now open source as Apache Beam
and Flink is the most complete runner.
![Page 26: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/26.jpg)
Why Apache Flink?• Basically identical semantics to Google Cloud Dataflow• Flink is a true fault-tolerant stateful stream processor• Exactly once guarantees for state updates• The state management features might allow us to eliminate the
key-value store• Windowing is built-in which makes time series easy• Native event time support / correct time based aggregations• Very fast data shuffling in benchmarks: 83 million msgs/sec on 30
machines• Flink “just works” with no tuning - even at scale!
![Page 27: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/27.jpg)
Prototype SystemApache Flink
Streaming
![Page 28: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/28.jpg)
Prototype SystemApache Flink
Streaming
![Page 29: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/29.jpg)
Prototype SystemApache Flink
Streaming
![Page 30: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/30.jpg)
Prototype SystemApache Flink
Streaming
![Page 31: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/31.jpg)
Prototype SystemApache Flink
Streaming
![Page 32: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/32.jpg)
Prototype SystemApache Flink
Streaming
![Page 33: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/33.jpg)
Prototype SystemApache Flink
Streaming
![Page 34: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/34.jpg)
Prototype SystemApache Flink
Streaming
![Page 35: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/35.jpg)
Prototype SystemApache Flink
Streaming
![Page 36: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/36.jpg)
Prototype SystemApache Flink
We now have a sharded key/value storeinside the stream processor
Streaming
![Page 37: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/37.jpg)
Prototype SystemApache Flink
Why not just query that!
We now have a sharded key/value storeinside the stream processor
Streaming
![Page 38: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/38.jpg)
Prototype SystemApache Flink
QueryServic
eWhy not just query that!
We now have a sharded key/value storeinside the stream processor
![Page 39: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/39.jpg)
Prototype System• Eliminates the key-value
store bottleneck
• Eliminates the batch layer
• No more Lambda Architecture!
• Realtime queries over in-flight aggregates
• Hourly aggregates written to database
![Page 40: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/40.jpg)
The Results• Uses 0.5% of the resources of the legacy
system: An improvement of 200x with zero tuning!
• Exactly once analytics in realtime• Complete elimination of batch layer and
Lambda Architecture• Successfully eliminated the key-value store
bottleneck
![Page 41: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/41.jpg)
How is 200x improvement possible?
• The key is making use of fault-tolerant state inside the stream processor
• Computation proceeds at in-memory speeds• No need to make requests over the network to
update values in external store• Dramatically less load on the database because only
the completed window aggregates are written there.• Flink is extremely efficient at network I/O and data
shuffling, and has highly optimized serialization architecture
![Page 42: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/42.jpg)
Does this matterat smaller scale?
• YES it does!• Much larger problems on the same
hardware investment• Exactly-once semantics and state
management is important at any scale!• Engineering time invested can be
expensive at any scale if things don’t “just work”.
![Page 43: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/43.jpg)
Summary• Used stateful operator features in Flink to
remove the key/value store bottleneck• Dramatic reduction in hardware costs (200x)• Maintained feature parity by providing low-
latency queries for in flight aggregates as well as long-term storage of hourly time series data
• Actually improved accuracy of aggregations: Exactly-once vs. at least once semantics
![Page 44: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/44.jpg)
Questions?
![Page 45: Stateful Stream Processing at In-Memory Speed](https://reader035.vdocument.in/reader035/viewer/2022070520/58f9a905760da3da068b6a1d/html5/thumbnails/45.jpg)
Thanks!