secret: a model for analyzing the execution semantics of ... · secret: a model for analyzing the...
Post on 03-May-2018
223 Views
Preview:
TRANSCRIPT
SECRET: A Model for Analyzing the Execution Semantics of
Stream Processing Engines
Nesime Tatbul ETH Zurich
Stream Processing: A Decade Ago [Aurora, VLDB’02]
• Monitoring applications require collecting, processing, disseminating, and reacting to real-time events from push-based data sources.
• “Store and Pull” model of traditional databases does not work well.
2
Data Base
DBMS Query Answer
Traditional Database Systems
Query Base
SPE Data Answer
Stream Processing Engines
Stream Processing: Today
3
• Tens of commercial products
• Countless academic prototypes
• Open-source distributed platforms and Hadoop extensions (e.g., Yahoo! S4, Twitter Storm, HStreaming)
• Streams Big Data [ The TIBCO Blog, http://www.thetibcoblog.com/ ]
⊂
Integrated Stream Processing (ISP)
• Integrating information from multiple, heterogeneous data sources has both been a key enabler and a major challenge in the DB community over the last 20 years.
• Today, similar integration support for SPEs is needed in three main forms: 1. across multiple streaming data sources 2. between SPEs and traditional DBMSs 3. over multiple SPEs
4
#1: Streaming Data Source Integration
• Goal: Integrated querying over multiple, potentially heterogeneous streaming data sources
• Example: Route planning based on information from news feeds, weather sensors, traffic cameras, etc.
• Main challenges: – Schemas of different sources can differ from one another and from the
input schemas of the already running continuous queries (CQs). – Input sources/the network can introduce imperfections into the stream. – Adapters may become a bottleneck.
• Current state of the art: – Commercial SPEs: adapters + SDKs – Research: Mapping Data to Queries [Hentschel et al.], ASPEN [Ives et al.]
5
#2: SPE-DBMS Integration • Goal: Integrated querying over SPEs and traditional
database and data warehousing systems • Example: Operational business intelligence, Continuous
analytics, Stream warehousing • Main challenges:
– Bridge the “data vs. operation” gap between the two worlds. – Find the right language and architecture primitives for the
required level of querying, persistence, and performance. • Current state of the art:
– Languages [STREAM CQL, StreamSQL, MATCH-RECOGNIZE] – Architectures [SPE-based (e.g., StreamBase); DBMS-based (e.g., Truviso, DejaVu, MaxStream)]
6
#3: SPE-SPE Integration • Goal: Integrated querying over multiple, potentially
heterogeneous SPEs – to exploit the advantages of distributed operation – to exploit specialized capabilities and strengths of SPEs – to provide higher-level monitoring over large-scale enterprises
with loosely-coupled operational units • Example: Supply-chain management • Main challenges:
– the need for functional integration – the need to deal with heterogeneity at different levels (e.g., query
models, capabilities, performance) • Current state of the art:
– MaxStream, ExoEngine, DSAM [Meyer-Wegener et al.]
7
Overview of our ISP Research
8
• SPE-DBMS integration – MaxStream: a federation middleware for ISP [BIRTE’09, ICDE’10, NTII’10] – DejaVu: declarative pattern matching in ISP systems [SIGMOD’09, Pervasive’09, DEBS’11] – SMS: storage and transaction management techniques for ISP [EDBT’09, EDBT’12]
• SPE-SPE integration – MaxStream – SECRET: a model to describe SPE query execution semantics [VLDB’10, VLDBJ’12] – ExoEngine: an architecture for virtualizing SPEs [Middleware’11]
Problem: Heterogeneity of SPEs • SPEs have differences in their query models:
– Syntax heterogeneity: • Language clauses/keywords for common query constructs
syntactically differ. – Capability heterogeneity:
• Support for certain query types differs. – Execution model heterogeneity:
• Underlying query execution models differ. • Not exposed to the application developer via language syntax.
• So, what?
– Difficult to build, debug, port, integrate applications.
9
Syntax for time-based sliding windows: - StreamBase: [SIZE x ADVANCE y TIME] - Coral8 : KEEP x SECONDS
Support for flexible slide: - StreamBase allows an arbitrary value for y. - Coral8 uses y=1msec by deafult.
You may get different results for: - StreamBase: [SIZE x ADVANCE 1 TIME] - Coral8 : KEEP x SECONDS
Example #1: StreamBase vs. Coral8
• Input Stream: InStream(Time, Val) {(30, 10), (31, 20), (36, 30), …} • Continuous Query: “Compute average value of the tuples for the last 5 seconds.”
• Result Stream: OutStream(Val)
– StreamBase: {(10), (15), (15), (15), (15), (20), (30), ...} – Coral8 : {(10), (15), (20), (30), ...}
10
WHY?
Our Solution • Goal:
– To describe, predict, and compare the basic query execution behaviors of diverse SPEs (i.e., focus on execution model heterogeneity)
• Approach: – A formal model based on experimenting with real systems – Design principles: expressive, simple, orthogonal, extensible
• Scope:
11
Version 1 [VLDB’10] Version 2 [VLDBJ, to appear] Data in-order streams (gaps, simultaneity) Queries time-based windows + tuple-based windows Systems Coral8, STREAM, StreamBase + Oracle CEP
SECRET Model Overview
• What affects query results produced by an SPE?
• Tick -> Report -> Content -> Scope
Query Result = F(system, query, input)
ScopE Content
12
window
Tick
Content
REport
ScopE
Tick REport
t0
Time & Order in SECRET
• Two notions of time: – tapp: application time of a tuple (t in SECRET) – tsys : system time (arrival time) of a tuple (τ in SECRET)
• Order assumptions:
– Tuples are partially ordered by their tapp values. – Tuples are totally ordered by their tsys values. – Batch-id (bid) defines a further ordering among simultaneous
tuples (i.e., tuples having the same tapp) [Jain et al.]. Tuples are partially ordered by their bid values.
13
ScopE in SECRET • Scope is based on
– window specification (size (ω) and slide (β)) – start of the first window (t0)
• Scope(t) defines the interval of “active window” at application time t. – Active window at time t is the open window with the earliest start time.
14
W1
t0+ω
W2
t0+β t0+β+ω W1’
W2’
t0 tapp
W3
t0+2β t0+2β+ω
Content in SECRET
• Content(t, τ) specifies the set of input tuples that are in Scope(t) as of system time τ.
• It can return different values at t, depending on the
arrival of tuples (due to τ).
15
REport in SECRET
• Report(t, τ) defines the conditions under which the window contents become visible for further query evaluation and result reporting.
• It can take a logical combination of the following: – content change – window close – non-empty content – periodic
16
Example #1 with SECRET
• Input: InStream(Time, Val) = {(30, 10), (31, 20), (36, 30), …} • Query: “Compute average value of the tuples for the last 5 sec.” • Result: OutStream(Val)
– StreamBase (window close&non-empty) = {(10), (15), (15), (15), (15), (20), (30), ...} – Coral8 (content change&non-empty) = {(10), (15), (20) ,(30), ...}
17
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 tapp
10 20 30
- - 10
10 15 15 15
- 15 - 15
- 20 20 30
30 … …
Tick in SECRET
• Tick(τ) defines the condition which drives an SPE to take action on its input.
• It can be based on one of the following: – tuple-driven: react to individual tuples – time-driven: react to all tuples with the same tapp value – batch-driven: react to subsets of tuples with the same tapp
value [Jain et al.]
• Note: tuple-driven and time-driven are in fact special cases of batch-driven.
18
Example #2: Coral8 - Difference in Tick • Input: InStream(Time, Val) = {(3, 10), (5, 20), (5, 30), (5, 40), (5, 50), ...} • Query: “Compute sum of the values for the last 4 seconds.” • Result: OutStream(Val)
– Coral8 (tuple-driven) = {(10), (30), (60), (100), (150), ...} – Coral8 (time-driven) = {(10), (150), ...}
19
30
10
20
40 50
0 1 2 3 4 5 6 7 8 tapp
10
30 60 100 150 10
150
Extending SECRET From Time-based to Tuple-based
• Windowing domain is different. – tid: tuple-id of a tuple (i in SECRET) – Tuples are totally ordered by their tid values. – Windows (size and slide) are defined in terms of tid’s instead of tapp’s.
• Scope and Content: – t -> i and t0 -> i0 – t is not unique -> i is unique
• Tick: – maps from tsys to tapp -> maps from tsys to tid – no gaps nor a notion of simultaneity in the tid domain => simpler to formulate
• Report: – Content-change and non-empty-content always true => simpler to formulate – Tick may invoke Report with multiple tid values (when: time- or batch-driven
tick + simultaneous input) => One of those that satisfy the Report condition is chosen (“evaporating tuples” [Jain et al.]) 20
Example #3: Tuple-based Windows • Input: InStream(Time, Val) = {(1, 10), (2, 20), (2, 30), (2, 40), (2, 50), (4, 60), ...} • Query: “Compute sum of the values for a tuple-based tumbling window of size 2.” • Result for an SPE S(Tick=time-driven, Report=window-close):
– OutStream(Val) = {(70), (110), …}
21
0 1 2 3 4 5 6
tid
(1,10) (2,20) (2,30) (2,40) (2,50) (4,60)
tsys
- 70 110
SPEs in SECRET SPE t0, i0 Report Tick
Coral8 content change window close non-empty content periodic
tuple-driven time-driven batch-driven
Oracle CEP content change window close non-empty content periodic
tuple-driven time-driven batch-driven
STREAM content change window close non-empty content periodic
tuple-driven time-driven batch-driven
StreamBase content change window close non-empty content periodic
tuple-driven time-driven batch-driven
22
1 1,0tt ω ββ
−−
1 ,tt β ω β ωβ
− −
1 ,tt ω β ω− −
1 1,0tt ω ββ
−−
Summary • SECRET uncovers the execution model heterogeneity of
SPEs, paving the way for: – tools for building, debugging, and optimizing streaming
applications on a given SPE – tools for porting streaming applications across SPEs – capability-based query processing across integrated SPEs – a formal basis for standardization
• SECRET is extensible to other types of inputs, queries, and SPEs.
• More information: – http://www.systems.ethz.ch/research/SECRET/
23
SECRET in Action (work in progress) Query Equivalence for SPEs
24
• When is a query Q1 under SPE query execution semantics S1 is “equivalent” to another query Q2 under SPE query execution semantics S2?
• Main challenges: – Si expressed in SECRET => explore the SECRET design space – Equivalence definition? – Dependence to input knowledge – Completeness vs. Practicality
SECRET in Action (work in progress) Example Use Case: Query Translation
25
Query Translator
SECRET in Action (work in progress) Example Use Case: Federated Query Processing
26
Future Directions Streaming Data is part of Big Data
• Streaming Data is the Velocity dimension of Big Data – With stream processing, we can react to Big Data as it happens
and tackle problems in a more incremental way.
• For ISP, we need to consider multiple dimensions together – streaming data source integration: Velocity + Variety – SPE-DBMS Integration: Velocity + Volume – SPE-SPE Integration: Velocity + “V a r i e t y”
• Lots of interesting challenges and opportunities
27
Managing Big Data in the Cloud An Example Problem
28
Big Processing Power: High number of CPU cores
available in the cloud
Big Data: Large amounts of data
stored on the client
Bottleneck: Limited network bandwidth between the client and the cloud
Managing Big Data in the Cloud The “Stream As You Go” Approach [DMC’12, SSDBM’12]
• Key idea: – Data is streamed into the cloud. – Incremental processing starts right away. – Results are streamed back to the client as they become available.
• Benefits: – hides the data transfer latency, reducing the total round-trip time – enables in-memory processing, saving from data access time and cloud
storage costs – enables pipelined parallelism (in addition to partitioned parallelism),
leading to early results and shorter completion time
• Stream-as-you-go is a good fit for data- and compute-intensive cloud applications that are incrementally-processable. – E.g.: DNA sequence analysis
29
Managing Big Data in the Cloud The “Stream As You Go” Approach [DMC’12, SSDBM’12]
• Implementation & Evaluation – DNA sequence analysis
(read alignment (Bowtie) + SNP detection (SOAPsnp))
– 1.4 GB E. Coli dataset – IBM InfoSphere Streams – Amazon EC2 – Compare against:
• Hadoop-based Crossbow • UNIX-based Trivial
30
PoC in progress for testing with 200 GB human genome dataset from German Cancer Research Institute (DKFZ)
top related