when rv meets cep (rv 2016 tutorial)

When RV Meets CEP...When RV Meets CEP...

A NEW TUTORIAL BY SYLVAIN HALLÉ

RUNTIMEVERIFICATION

COMPLEX EVENTPROCESSING

CRSNGNSERC

(C) 2016 LABORATOIRE D'INFORMATIQUE FORMELLE, UNIVERSITÉ DU QUÉBEC À CHICOUTIMI, CANADA

PRESENTED AT THE INTERNATIONAL CONFERENCE ON RUNTIME VERIFICATION, MADRID, SPAIN, SEPTEMBER 27TH, 2016

WORK FUNDED BY THE CANADA RESEARCH CHAIR IN SPECIFICATION, TESTING AND VERIFICATION OF SOFTWARE SYSTEMS

Canthese

two fieldsbenefitfromeach

other?

TODAY'SPROGRAM

Part IRV vs. CEP

Part IIA Fistful of Processors

Part IIIThe Monitor Strikes Back

PART ONE

RV vs. CEP

public static voidmain(String[] args) { Map m = new HashMap(); ... Iterator i = m.iterator(); while (i.hasNext()) { ... }}

public static voidmain(String[] args) { Map m = new HashMap(); ... Iterator i = m.iterator(); while (i.hasNext()) { ... }}

HashMap.new = HashMap$1263HashMap$1263.iterator() = Iterator$32045HashMap$1263.contains(MyObject$26) = falseIterator$32045HashMap$1263.hasNext = true...

Methodcalls

Any call to next() must be immediately preceded by a call to hasNext() that returns true.

Two calls to remove() must be separated by at least one call to next().

Calls to an iterator's methods are forbidden once its parent collection has been modified by calls like put() or remove().

start(vase,3,15).bid(vase,15).start(ring,5,30).endOfDay.bid(ring,32).bid(ring,33).bid(vase,18).sell(vase).


Event name


Parameters


For a given item, any bid must be higher than the previous one.

Any bid must be higher than the minimum price.

No bid can be placed before a start event.

Server

Ajaxwebapp

Server

Ajaxwebapp

<message> <name>CartAdd</name> <CartId>123</CartId> <Items> <ItemId>42</ItemId> <ItemId>23</ItemId> </Items></message>

<message> <name>CartAddResponse</name> <CartId>123</CartId></message>

TheBeepStore

GO

Sign inor regi ster

What is this?

Login

Ask for account

Contact us

Fault parameters

Search: Your Cart

Fault parameters

Don’t check Results’s typeIn the detailed search form, sends an ItemSearch message withoutchecking that the Results element is an integer.

"Add to cart" enabled if item present in cartMakes the "Add to cart" button available for items that are already in theuser's cart.

Message schemas

Cart manipulations

Highlightsdocumentation

Disables theverification

1. The element must be an integer between 1 and 20.

2. The element is mandatory only if is present,otherwise it is forbidden.

3. The request cannot be resent if its response issuccessful.

4. must follow a successful Logi nResponse.

5. There can be at most one active cart ID per session key.

6. You cannot add the same item twice to the shopping cart.

Page

Page Resul t s

Logi n

Car t Cr eat e

The lock graph of a set of tasks should not contain cycles

A communication window has three phases: prep, active and cleanup

Every command must be responded by either a success or a failure

RV

untime

erification

System

System

Instrumentation

System

Instrumentation

Trace

System

Instrumentation

Trace

Events

System

Instrumentation

Trace

Events

Tracevalidation

System

Instrumentation

System

Runtime monitoringInstrumentation

System

Runtime monitoring

Overhead

Instrumentation

hasNext

next

hasNext

Option #1 Finite-State Machines

hasNext

next

hasNext

Option #1 Finite-State Machines

...and extensions

1 2 3groupStart(t)

init(p)∀p

QEA: Reger et al., TACAS 2015

groupEnd(t2)

phaseStart

4run(p)

finish(p)

t2−t < 480

A call to next must be followed by a callto hasNext

No CartCreate request can occurbefore a LoginResponse message

A received order must eventuallybe shipped

Three successive login attempts shouldtrigger an alarm

G (next → X hasNext)

¬ CartCreate U hasNext

G (receive → F ship)

G ¬(fail ∧ (X (fail ∧ X fail)))

Option #2 Linear Temporal Logic

G (∃ retAddrVal ∈ ./return-address : ( (./instruction = call) ∧ (¬ ((F( ((./instruction = mov)∧ (./output/type = general-register)) → (∃ regA ∈ ./output/name : (F (( ((./instruction = mov) ∧ (./output/type = general-register)) ∧ (./input/type = litteral) ) → (∃ regB ∈ ./output/name : ( ∃ constAddr ∈ ./input/value : (F (((./instruction = cmp ) ∧ (./output/type = regA)) → (∃ loc ∈ ./location : (F((((./instruction = mov ) ∧(./output/type = general-register)) ∧(./output/name = regA ) ) ∧((./input/name = regB ) ∧ (./input/type = ptr)))))) ))))) ))) U ((./instruction = return) ∧ (./fonction-returned = retAddrVal)) )) ))

...and extensionsLTL-FO+, Hallé et al.IEEE Trans. Serv. Comput. 2012

νZ.(∨a ∈ A

a ⊤ ∧ ∧a ∈ A

[a] Z )

Option #4 mu-calculus

(hasNext+ next)*

Option #3 Regex

class SeqMonitor extends EvrMonitor { val SeqStart, SeqDone = fact def seq name(s:String) = words(s)(2)

"start_seq" −− EVR(’id -> "EVR_SEQ_START", ’sclk -> ’S, ’msg -> ’M) 7−-> { val w = words(’M.s) val seq name = w(15).slice(1, w(15).length−2) insert (SeqStart(seq name, ’S.d)) }

"end_seq_ok" −− EVR(’id -> "EVR_SEQ_SUCCESS", ’sclk -> ’E, ’msg -> ’M) & ’SeqStart(’name, ’S) --> { if (seq name(’M.s) == ’sname.s) { replace (SeqStart)(SeqDone(’name.s, ’S.d, ’E.d, "OK")) } }

"end_seq_fail" −− EVR(’id -> "EVR_SEQ_FAILURE", ’sclk -> ’F, ’msg -> ’M) & ’SeqStart(’name, ’S) --> { if (seq name(’M.s) == ’name.s) { replace (SeqStart)(SeqDone(’name.s, ’S.d, ’F.d, "FAIL")) } }

"print" −− SeqDone(’name, ’S, ’E, ’stat) --> { updateCSV(’name.s, ’S.d, ’E.d, ’ stat .s) remove(SeqDone) }} Option #5 LogFire

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

10/2/29 APPL 1212.210/2/29 MSFT 1031.610/2/29 GOGL 341.110/3/29 MSFT 1024.410/3/29 APPL 1179.010/3/29 GOGL 362.210/4/29 APPL 1164.310/4/29 MSFT 1025.110/4/29 GOGL 345.410/3/29 MSFT 1024.410/3/29 APPL 1179.010/3/29 GOGL 362.210/3/29 MSFT 1024.410/3/29 APPL 1179.010/3/29 GOGL 362.2

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$


Timestamp

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$


Stock symbol

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$


Price

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

What is the closing price of MSFT for the first five trading days?

Select all days after the 100th where MSFT closed over $50.

On every 5th day, calculate the average closing price for MSFT for the last 5 days.

What are the stocks that closed higher than MSFT for a given day?

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

What is the closing price of MSFT for the first five trading days?

Select all days after the 100th where MSFT closed over $50.

On every 5th day, calculate the average closing price for MSFT for the last 5 days.

What are the stocks that closed higher than MSFT for a given day?

Snapshot query

Landmark query

Sliding query

Join query

4948

4949

6.3132

4948

4949

6.3132

Testresult

4948

4949

6.3132

Drugprescriptions

4948

4949

6.3132

Meeting

4948

4949

6.3132

MSH|^~\&|SOURCE|383018129|PRIORITY HEALTH|382715520|2007100914484648||ORU^R01|0129938170710091448|P|2.3|PID|1|1034157|012993817||LASTNAME^FIRSTNAME||19520101|M|||1234 MAIN^^DEARBORN HEIGHT^MI^48127||||||||PID|1||94000000000^^^Priority Health||LASTNAME^FIRSTNAME||19400101|F|PD1|1|||1234567890^DOCLAST^DOCFIRST^M^^^^^NPI|OBR|1|||80061^LIPID PROFILE^CPT-4||20070911||||||||||OBX|1|NM|13457-7^LDL (CALCULATED)^LOINC|49.000|MG/DL| 0.000 - 100.000|N|||F|OBX|2|NM|2093-3^CHOLESTEROL^LOINC|138.000|MG/DL|100.000 - 200.000|N|||F|OBX|3|NM|2086-7^HDL^LOINC|24.000|MG/DL|45.000 - 150.000|L|||F|OBX|4|NM|2571-8^TRIGLYCERIDES^LOINC|324.000|

HL7HL7

Notify me when two out of three successive data points lie more than 1 standard deviation from the mean on the same side of the mean line.

Notify me when two out of three successive data points lie more than 1 standard deviation from the mean on the same side of the mean line.

Trend query

TOLL

TollIdTollLicenseStateModel

48392$3.50QF5010KWT

TOLL

Compute the toll of each booth over 3-minute intervals

CE

omplex

vent

Processing

Select AVG(closingPrice)From ClosingStockPricesWhere stockSymbol = ‘MSFT’for (t = ST; t < ST+50, t+= 5) { WindowIs(ClosingStockPrices, t - 4, t);}

Select c2.*FROM ClosingStockPrices as c1, ClosingStockPrices as c2WHERE c1.stockSymbol = ‘MSFT’ and c2.stockSymbol!= ‘MSFT’ and c2.closingPrice > c1.closingPrice and c2.timestamp = c1.timestampfor (t = ST; t < ST +20 ; t++ ){ WindowIs(c1, t - 4, t); WindowIs(c2, t - 4, t);}

Two queries inthe language ofTelegraphCQ

SELECT * FROM FILTER {cnt >= 10}( (SELECT *, 1 AS cnt FROM FILTER {contains(summary,’iPod’) = 1}(webfeeds)) FOLD {, $.cnt < 10 AND DUR < 1 DAY, $.cnt + 1 AS cnt} (SELECT * FROM FILTER {contains(summary,’iPod’) = 1}(webfeeds)) )PUBLISH ipod_popularity

A query in the language of Cayuga

from inputtrace#window.length(10)select stockSymbol, timestamp, avg(closingPrice) as avg10insert into avg10trace;

from inputtrace#window.length(20)select stockSymbol, timestamp, avg(closingPrice) as avg20insert into avg20trace;

@info(name = 'query1')from avg10trace#window.length(10) join avg20trace#window.length(20)on avg10trace.timestamp == avg20trace.timestampselect avg10trace.stockSymbol as stockSymbol, avg10trace.timestamp, avg20trace.timestamp as ts2, avg10trace.avg10 as avg10, avg20trace.avg20 as avg20having avg10 > 50 or avg20 > 50insert into output;

SiddhiQL...

CEPRVevent traces

"properties"

tuple-ish

overhead

event streams

"queries"

tuples*

throughput

=When

CEPRVoutputs T/F

result cannot be reused in another property

mostly variants of FSM and logic

good for sequential patterns

not quite

When

outputs event streams

result can be reused as input of another query

mostly "extensions" of SQL

not quite

good for data manipulation

CEPRVBeyond &

CEPRVBeyond &

Both RV and CEP assume events to be "tuples":associative maps from strings to single scalar values

start(vase,3,15).bid(vase,15).start(ring,5,30).

3/10/29 MSFT 1059.23/10/29 APPL 365.43/10/29 GOGL 1120.1

HashMap.new = HashMap$1263HashMap$1263.hasNext =true


48392$3.50QF5010KWT

CEPRVBeyond &



3/10/29 MSFT 1059.23/10/29 APPL 365.43/10/29 GOGL 1120.1



48392$3.50QF5010KWT

And what if eventsare not tuples?

CEPRVBeyond &



3/10/29 MSFT 1059.23/10/29 APPL 365.43/10/29 GOGL 1120.1



48392$3.50QF5010KWT

And what if eventsare not tuples?And what if eventsare not tuples?

We make themfit!

CEPRVBeyond &

Suppose each event is a set:

{6,5,3,9}{5,4,10}...

We can "fit" it into a tuple:

a6

a=4 ∨ b=4 ∨ c=4 ∨ d=4

c3

d9

Imposes upperbound on set size

But then 4 ∈ S becomes:

b5

CEPRVBeyond &

CEPRVBeyond &

Both RV and CEP rely on monolithicspecification/query languages

Everything must be writtenin that one language (FO-LTL,QEA, SiddhiQL, etc.)

The language mustaccommodate every possibleuse case

In general, no support forextensions

Part 2:A Fistful ofProcessors

Event stream query engine developedbased on the previous observationsAims at borrowing strengths from bothRV and CEP (and beyond)Key concepts: composability, modularity,extensibilityOpen source, developed in Javahttp://liflab.github.io/beepbeep-3

peeB peeB 3

EventsEvents

An event is an element e taken fromsome set E, called the event type.

No restriction on the type! In BeepBeep, events can be any Object.

Booleans Numbers

234

π

Strings

abc

Functions Sets PlotsTuples

3 8 a3 8 a

2 6 c

+⊇?

XMLdocuments

<a><a><a>

. . .

?

TracesTraces

An event trace (or event stream) is a potentiallyinfinite sequence of events of a given type:

2 0 6 34 9 . . .

Traces are symbolically denoted by:

e = e0 e1 e2 e3 ...

The set of all traces of type T is denoted as:

T*

FunctionsFunctions

A function takes 0 or more events as itsinput, and returns 1 or more events.

Functions are first-class objects; they descendfrom the class Function

1 : 1function 2 : 1

function1 : 2

function

⊇?

34

2+5i

₹

2 5

6

0 : 1function

6

ProcessorsProcessors

A processor takes 0 or more event traces asits input, and returns 0 or more event traces asits output

1 : 1 processor

2 : 1 processor

. . . . . .

ProcessorsProcessors

When a processor takes more than one inputtrace, the set of events at matching positionsin each trace is called a front.

bacd

3601

b 3

a 6

c 0

d 1

1st event

2nd

3rd

4th

. . .

. . . . . .

Synchronous processingSynchronous processing

Events are processed one front at a time.

+

Buffers collect events until a completefront can be processed.

⇒



+


⇒

5

3



+


⇒

5

3

+



+


⇒

8



+


⇒

1



+


⇒

6 1



+


⇒

6 1

4



+


⇒

6 1

4

+



+


⇒

6

5



+


⇒

6


Makes a couple of things simpler

Don't care about what event arrivedfirst or upstream computation time"Pen and paper" calculation is identicalto the real oneOtherwise, can do a lot with simpletimeouts ⇒ contained asynchrony

Motto: Don't use asychronous processing...


Makes a couple of things simpler

Don't care about what event arrivedfirst or upstream computation time"Pen and paper" calculation is identicalto the real oneOtherwise, can do a lot with simpletimeouts ⇒ contained asynchrony

Motto: Don't use asychronous processing......unless you really have to


In BeepBeep, all synchronous processorsare descendents of the SingleProcessorclass

Takes care of handling input/outputbuffersCalls (abstract) method process() whenan input front is ready to be consumedProcessor only needs to produce anoutput front from this inputMakes it easy to create your own(more on that later)

A high-level event trace can be produced bycomposing ("piping") together one or moreprocessors from lower-level traces

CompositionComposition

Each processor has its own input/outputbuffers


Any output can be connected to any input, aslong as they have the same type


Any output can be connected to any input, aslong as they have the same type


Many types can occur in the same chain

Trace ManipulationFunctions/Processors

ArchitectureArchitecture

TMF/P EMF/P

Event ManipulationFunctions/Processors

Type-agnosticStateful

Type-dependentStateless

Trace ManipulationFunctions/Processors


TMF/P EMF/P

Event ManipulationFunctions/Processors

Type-agnosticStateful

Type-dependentStateless

Not clear-cut, but general design goal


BeepBeep provides only a few built-inprocessors and functions

PaletteSet of processors and functions,centered around a particularuse case

Concretely, a JAR librarydefining new Processor andFunction objects

<?

+

<? =?

−

÷ ×f Σ f

n

{

n

n

Function Cumulative Trim

ForkDecimate Group

WindowSliceFilter

Built-in processors

Built-infunctions

.n <

<

SemanticsSemantics

Let P be a processor and abc

= a1,a2,...= b1,b2,...= c1,c2,... be traces

a,b,c : P [[ n = e1,e2,...

denotes the n-th output trace of P, giventraces a, b, c as input.

f

FunctionFunction

Applies an n-ary function f toevery front of size n

"Lifts" any function into aprocessor

a,b : f [[ n = f n(ai,bi)

f

FunctionFunction



a,b : f [[ n = f n(ai,bi)The n-th output

f

FunctionFunction




iThe i-th event

f

FunctionFunction




iThe i-th event

f+

f<0?

Pairwise sum ofevents

Is each eventnegative?

CumulativeCumulative

Applies a 2 : 1 function f toits previous value and thecurrent event

a : Σf [[ = f(x,a1), f(f(x,a1),a2), ...

+ <Sum of all

eventsHave we seen⊤ so far?

Σ fx

Σ f Σ f0 ⊥

TrimTrim

Returns the input trace,trimmed of its first n events

a : [[ = an+1, an+2, ...

n

n

TrimTrim

Returns the input trace,trimmed of its first n events

a : [[ = an+1, an+2, ...

n

n

1

f=?=? <

Σ f⊥

Does the samenumberrepeat twice ina row?

GroupGroup

Makes a set of connectedprocessors behave as a singleprocessor

GroupGroup

Makes a set of connectedprocessors behave as a singleprocessor

1

f=?=? <

Σ f⊥

ForkFork

Replicates its input on eachof its n outputs

a : Ψ [[ = a1, a2, ...n

DecimateDecimate

Outputs every n-th inputevent

a : [[ = a1, an+1, a2n+1, ...nΨ

n

FilterFilter

A n : n-1 processor; outputs the n-1 first components of afront if its last component is ⊤;otherwise, discards it

a , a , ..., a : =[[ 2k

1 n Fi {a if a =⊤

ε otherwise

ki

ni

Powerful mechanism: "anything canbe filtered" (don't care about condition)Boolean trace does not need to comefrom the same source as the inputs beingfiltered

FilterFilter

A n : n-1 processor; outputs the n-1 first components of afront if its last component is ⊤;otherwise, discards it

f<0?

Get only the negativeevents of the input

WindowWindow

Returns the output of aprocessor P on a slidingwindow of width n

a : Υ =[[ P i

Powerful mechanism: "anything canbe windowed" (don't care about function)

n

{

n ai, ... ai+n : P [[*

WindowWindow


a : Υ =[[ P i

Powerful mechanism: "anything canbe windowed" (don't care about function)

n

{

n ai, ... ai+n : P [[*

The last event

WindowWindow


n

{

3

{+

Σ f0

WindowWindow


n

{

3

{+

Σ f0

1 3 4 2

WindowWindow


n

{

3

{+

Σ f0

1 3 24

WindowWindow


n

{

3

{+

Σ f0

1 243

WindowWindow


n

{

3

{+

Σ f0

1

2

243

WindowWindow


n

{

3

{+

Σ f0

1

2

243

4

WindowWindow


n

{

3

{+

Σ f0

1

6

243

2

WindowWindow


n

{

3

{+

Σ f0

1

6

243

23

WindowWindow


n

{

3

{+

Σ f0

1

9

243

6 2

WindowWindow


n

{

3

{+

Σ f0

1

9

243

6 23 2 1

WindowWindow


n

{

3

{+

Σ f0

1

9

243

6 2

9

WindowWindow


n

{

3

{+

Σ f0

1 243 9

WindowWindow


n

{

3

{+

Σ f0

1 43 9

WindowWindow


n

{

3

{+

Σ f0

43 91

WindowWindow


n

{

3

{+

Σ f0

43 91

4

WindowWindow


n

{

3

{+

Σ f0

43 91

43

WindowWindow


n

{

3

{+

Σ f0

43 91

7 4

WindowWindow


n

{

3

{+

Σ f0

43 91

7 41

WindowWindow


n

{

3

{+

Σ f0

43 91

8 7 4

WindowWindow


n

{

3

{+

Σ f0

43 81

8 7 4

9

WindowWindow


n

{

3

{+

Σ f0

43 81 9

WindowWindow


n

{

3

{+

Σ f0

31 8 9

WindowWindow


n

{

3

{+

Σ f0

31 8 9

The sum of all 3successive events

SliceSlice

Dispatches an event e to adistinct instance of processorP according to the value ofsome function f

{a if f(ai)=kε otherwise

iπfk[a : =[i

Pf[a : =[i U πf

k[a1,...ai : [i[ [: Pk *+

SliceSlice

Dispatches an event e to adistinct instance of processorP according to the value ofsome function f

{a if f(ai)=kε otherwise

iπfk[a : =[i

Pf[a : =[i U πf

k[a1,...ai : [i[ [: Pk *+

Multiset union

f(x) = x mod 2

f(x) = x mod 2

πf0[a : =[subtrace of even numbers

πf1[a : =[subtrace of odd numbers

f(x) = x mod 2



mod 2

f(x) = x mod 2



mod 2

5

f(x) = x mod 2



mod 2

5

U+

f(x) = x mod 2



mod 2

5

U+

{9}{5}

f(x) = x mod 2



mod 2

5

{9}{5}6

f(x) = x mod 2



mod 2

5

{9}{5}

6

f(x) = x mod 2



mod 2

5

{9}{5}

6

U+

f(x) = x mod 2



mod 2

5

{9}{6,5}

6

U+

{9}{5}

f(x) = x mod 2



mod 2

5

{9}{6,5}

6

{9}{5}1

f(x) = x mod 2



mod 2

5

{9}{6,5}

6

{9}{5}

1

f(x) = x mod 2



mod 2

1

{9}{6,5}

6

{9}{5}

5

U+

f(x) = x mod 2



mod 2

1

{9}{6,1}

6

{9}{6,5}

5

U+

{5}

f(x) = x mod 2



mod 2

1

{9}{6,1}

6

{9}{6,5}

5

U+

{5}

The last odd and evennumbers seen so far

f(x) = x mod 2



mod 2

+

Σ f0

f(x) = x mod 2



mod 2

+

Σ f0

The sum of all oddnumbers and all evennumbers seen so far

{9}{6,6} {9}{6,5} {5}

Input/outputInput/output

0 : 1 processors can be used to produce anevent trace out of an external source (i.e.standard input, a file, etc.)

Ditto for 1 : 0 processors

a . . .b

a . . .b

WARP ZONE

PalettesPalettes

BeepBeep provides only a few built-inprocessors and functions

PaletteSet of processors and functions,centered around a particularuse case

Concretely, a JAR librarydefining new (reusable!)Processor and Functionobjects

XMLXML

Provides two new Functions

Ditto for the JSON library (using json-lif)

XML parser: converts String eventsinto XMLElement events (fromxml-lif library)XPath: evaluates an XPath expressionon an XMLElement (result is a set ofXMLElements)

SetsSets

New Functions:

New processor:

Multiset operations: add/removeelements, membership, union/intersection

→ →

Dropper

Outputs each element of amultiset as an event

GnuplotGnuplot

New Functions:

New processor:

Scatterplot: converts a set of (x,y) pairsinto a Gnuplot string producing a plot(ditto for Histogram)

→ →

Gnuplot Caller

Calls Gnuplot on a string andreturns its output (i.e. a binarystring)

TuplesTuples

New event type: tuple

New function: SELECTCreates an output tuple by combiningattributes from input tuples

Grammar extension: allow eSQL to use aSELECT statement, backward-compatible withSQL

3 8 a3 8 a

2 6 c

TuplesTuples

SELECT <tuple> FROM <processor-list>

* | <tuple-element> | <tuple-element> , <tuple>

<tuple-name> <rename> | <function-call> <rename>

<word>.<word> | <word>

<processor> <rename> | <processor> <rename> , <processor-list>

AS <eml-attribute>

<select> ::=

<tuple> ::=

<tuple-element> ::=

<tuple-name> ::=

<processor-list> ::=

<rename> ::=

FSMFSM

New processor: finite-state machine

→

→?

?T

/a/b

//status/text()

=? Walker

*

/a/b

//status/text()

=? Blocker

*

*

Guards on transitionsare arbitrary functionson events States output values

of any type (Moore machine)

Linear Temporal LogicLinear Temporal Logic

New processors for LTL temporal operatorsand first-order quantification (LTL-FO+)

Two semantics for a formula φ:

Boolean: the i-th output is the (2-valued)verdict of evaluating φ, starting at the i-thevent⇒ filter

Troolean: the i-th output is the (3-valued)verdict of evaluating φ up to the i-th event⇒ monitor


Consider the LTL formula F b on thetrace

In a monitoring context, we want theoutput of processor F b to be:

ac c b

F b

"Tell me whethereventually b"




ac c b

F b


c




ac c b

F b


?




ac c b

F b


?c




ac c b

F b


? ?




ac c b

F b


? ?b




ac c b

F b


⊤ ? ?




ac c b

F b


⊤ ? ?a




ac c b

F b


⊤ ⊤ ? ?



In a filtering context, we want theoutput of processor F b to be:

ac c b

F b

"Get me all the eventswhere eventually b"




ac c b

F b


c




ac c b

F b





ac c b

F b


c




ac c b

F b





ac c b

F b


b




ac c b

F b


⊤ ⊤ ⊤




ac c b

F b


a ⊤ ⊤ ⊤




ac c b

F b


⊤ ⊤ ⊤


Boolean operators are easy

X

G F

U

1

<Σ f

⊥

<

Σ f⊥

F

G

=

= =

= Σ f

?

♣


First-order quantifiers

Processor to runon each slice

→

A→ x →

E→ x

Quantifiedvariable

(Arbitrary) domainfunction

Variable is added tothe processor's contextBoolean and Trooleanversions

Let's put it alltoghether!

i.e. a few examples ofqueries from past publications

Use BeepBeep asa library in yourprogram

→

Fork f = new Fork(2);


→

→

→

→

Fork f = new Fork(2);FunctionProcessor sum = new FunctionProcessor( );

f


→

→

→

+

→

Fork f = new Fork(2);FunctionProcessor sum = new FunctionProcessor(Addition.instance);

f


→

→

→

+

→

Fork f = new Fork(2);FunctionProcessor sum = new FunctionProcessor(Addition.instance);CountDecimate decimate = new CountDecimate(n);

f

n Use BeepBeep asa library in yourprogram

→

→

→

+

→→

Fork f = new Fork(2);FunctionProcessor sum = new FunctionProcessor(Addition.instance);CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT)

f


→

→

→

+

→

→

→

Fork f = new Fork(2);FunctionProcessor sum = new FunctionProcessor(Addition.instance);CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT) .connect(fork, RIGHT, decimate, INPUT)

f


→

→

→

+

→

→

→

Fork f = new Fork(2);FunctionProcessor sum = new FunctionProcessor(Addition.instance);CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT) .connect(fork, RIGHT, decimate, INPUT) .connect(decimate, OUTPUT, sum, RIGHT);

f


→

→

→

+

→

→

→

Fork f = new Fork(2);FunctionProcessor sum = new FunctionProcessor(Addition.instance);CountDecimate decimate = new CountDecimate(n);Connector.connect(fork, LEFT, sum, LEFT) .connect(fork, RIGHT, decimate, INPUT) .connect(decimate, OUTPUT, sum, RIGHT);Pullable p = sum.getOutputPullable(OUTPUT);while (p.hasNext() != NextStatus.NO) { Object o = p.next(); ...}

f


→

→ →

^

n

→

→ →

+

→→

→

→

→

÷

→ →1 → →

+

Σ

Σ

0

0

The statistical moment of order n

→

→ →

^

n

→

→ →

+

→→

→

→

→

÷

→ →1 → →

+

Σ

Σ

0

0

The statistical moment of order n

→ →E(x)

n

As a groupprocessor

Trigger an alarm

→

→

→→

1

→ → >

Trigger an alarm when two successive events

→

→

→

1

→

→

→

→→

<?

→

→→

1

→ → >

Trigger an alarm when two successive events are more than 1

→

→

→

→

1

→ →

→

÷→

→

→

→→

<?

→

→→

1

→

→ →σ

→ >

Trigger an alarm when two successive events are more than 1 standard deviation

→

→→

→

→

1

→ →

→

-→ →

→

÷→

→

→

→→

<?

→

→→

1

→

→ →E(x)

1

→ →σ

→ >

Trigger an alarm when two successive events are more than 1 standard deviation from the mean

→

⊇?

→ → → A →→T

??...<a> <a> .../a/b

//character/id/text()

→ →

→

→

?

?T

/a/b

//status/text()

=? Walker

*

/a/b

//status/text()

=? Blocker

*

*

→→ →

/a/b

//character

→→→

A/a/b//id/text()

→T

??...<a> <a> ...

→

A/a/b//character[status=Walker]/id/text()

→ p1

→

A

→ p2

→ →→

→

/a/b

//character[status=Blocker]/id/text()

→

→→

3

→

→

→

<?

→

→

→

→

→

→→

→

f1

f2

→

→ →

→

→

→

→

/a/b

//character[id=p1]/position/x/text()

/a/b


-

|...|

<?

6

>

...

f1

/a/b


/a/b


-

|...|f2

→

→

**

*

*

Create Auction=?0

@

Last PriceDays

0:=

3@Max. Days :=Min. Price 2@:=

Days :=Days 1

+

End of Day=?

0

@

>

<?Days

Last Price

2

@:=

Bid=?

0

@

>

Min. Price<?

2

@

Last Price>?

2

@

Bid=?

0

@

>

Last Price>?

2

@

Days :=Days 1

+

End of Day=?

0

@

Max. Days

→

End of Day *1@*

→→→ →

*Sold

=?0

@

Days

Days

+ | |.÷

→→

1@+=

Sanitise=?0

@

Clean :=

Derive=?

0

@

>

∊?Clean

T??...Derive(a,b)

Clean

2@+=Clean :=

Clean

1

@

Use=?

0

@

>

∊?Clean

1

@

T * *

T

...

Fastest car in theworld


Slow on everytype of road



Only runs on aclosed circuit




Works on everytype of road





Heavy setup required





Heavy setup required Turn the key

PART THREE

The MonitorStrikes Back

import ca.uqac.lif.cep.*;

public class MyProcessor extends SingleProcessor {

public Queue<Object[]> compute(Object[] inputs) {

}

public void build(Stack<Object> s) {

}}

. . . Create output events from input . . .

. . . Instantiate processor from parse stack . . .

<processor> := . . .<number> := . . .<string> := . . .

Add new rules to any symbol fromthe basic grammar

Example: let us create a new 1 : 1 Processor that increments its input by a fixed value

We would like to use it in eSQL with thissyntax:

INCREMENT ( P ) BY value

Any expressiondefining a processor

Number

<processor> := <my_processor> ;

<my_processor> := INCREMENT ( <processor> ) BY <number> ;

Symbols already defined in basic grammar

Adds a new case to an existing rule

import ca.uqac.lif.cep.*;

public class MyProcessor extends SingleProcessor {

private int increment;

public Queue<Object[]> compute(Object[] inputs) { Queue<Vector<Object>> out = new Queue<Vector<Object>>(); Object[] v = new Object[1]; Integer i = (Integer) inputs[0] + increment; v[0] = i; out.put(v); return out; }

. . .

. . .

public void build(Stack<Object> s) { Number n = (Number) s.pop(); s.pop(); s.pop(); Processor p = (Processor) s.pop(); s.pop(); s.pop();

increment = n.intValue();

Connector.connect(p, this);

s.push(this); }}

Read contents ofparse stack

<number>BY(<processor>)INCREMENT

Set processor's statePipe it to its inputPut on parse stack

Total: 6+9=15 lines of code

. . .

public void build(Stack<Object> s) { Number n = (Number) s.pop(); s.pop(); s.pop(); Processor p = (Processor) s.pop(); s.pop(); s.pop();

increment = n.intValue();

Connector.connect(p, this);

s.push(this); }}

Read contents ofparse stack

<number>BY(<processor>)INCREMENT

Set processor's statePipe it to its inputPut on parse stack

Some pre-packaged grammar extensions:

Manipulation of name-value tuples

Set theory

Formatted input (CSV, XML, JSON)

Graphing (histograms, scatterplots, ...)

Basic signal processing (smoothing, peak detection, ...)

Create your own!

import ca.uqac.lif.cep.*;import ca.uqac.lif.cep.eml.tuples.*;

public class MyExample {

public static void main(String[] args) { Interpreter my_int = new Interpreter();

my_int.extendGrammar(TupleGrammar.class);

Pullable p = my_int.executeQuery( "\"HELLO WORLD\"");

for (int i = 0; i < 10; i++) { EmlString s = (EmlString) p.pull(); System.out.println(s); } }}

Create query interpreterLoad a grammar extension

Execute a query

Pull an output event

Example: let us create a new 1 : 1 Processor that parses XML events (using an existingXML library)

This can be done by first defining a Functionthat converts Strings into XML objects

import ca.uqac.lif.cep.*;import ca.uqac.lif.xml.*;

public static class XmlParsingFunction extends UnaryFunction<String,XmlElement> { public static XmlParsingFunction instance = new XmlParsingFunction();

private XmlParsingFunction() { super(String.class, XmlElement.class); }

public XmlElement getValue(String x) { try { return XmlElement.parse(x); } catch (XmlParseException e) { } return null; } }}

import ca.uqac.lif.cep.*;import ca.uqac.lif.xml.*;

public static class XmlParsingFunction extends UnaryFunction<String,XmlElement> { public static XmlParsingFunction instance = new XmlParsingFunction();

private XmlParsingFunction() { super(String.class, XmlElement.class); }

public XmlElement getValue(String x) { try { return XmlElement.parse(x); } catch (XmlParseException e) { } return null; } }} Total: 10 lines of code

Query Editor

+ ?

>

>

<

>

=

+

*

n

A /

A /

Playback controls

Can execute a query one event at a time (play/pause), or at a custom input rate

A /

Toggle between graphand grammar view

>

>

<

>

=

+

*

n

Palettes: drag/drop on playgroundto compose query. Conventions:

Light pipes = inputDark pipes = output

Colours = event types

Correct flow = dark → lightType discipline = matching colours

Logic

I/O

Tuples

Plumbing

Numbers

Booleans

BlobsName-value tuples

Math

Query Editor

+ ?

>

>

<

>

=

+

*

n

A /

3 5 53 2

"Magnifier" revealsrealtime content of messagequeues...

Query Editor

+ ?

>

>

<

>

=

+

*

n

A /

Threshold

Some processors have parameters;can modify them with contextual menu

Query Editor

+ ?

>

>

<

>

=

+

*

n

A /

Site C

Site BSite A

Separation of query parts onmultiple sites

BeepBeep needs you!BeepBeep needs you!

Licensed under LGPL: download anduse it

Create you own domain-specific palettes

Take your own monitor/script and wrapit into a new Processor object

For free: benefit from all existing processorsand functions already available

https://liflab.github.io/beepbeep-3

when rv meets cep (rv 2016 tutorial)

Technology