chasing bugs with the beepbeep event stream processor

Sylvain HalléUniversité du Québec à Chicoutimi

Canada

Chasing Bugswith the BeepBeep

Event Stream Processor

TAROT 2016

System

System

Instrumentation

System

Instrumentation

Trace

System

Instrumentation

Trace

Events

System

Instrumentation

Trace

Events

Tracevalidation

System

Instrumentation

System

Runtime monitoringInstrumentation

System

Runtime monitoring

Overhead

Instrumentation

Monitor

Property

Target system

Sensor

Internalstate

Event

VerdictFeedback(optional)

One trace at a time

Don't care aboutinteracting with the SUT

?

Internal state canbe queried

Properties partiallyspecify behaviour

Possibility for the monitor to interact with the SUT(enforcement monitors)

Opportunity to expressmore complex properties

(ab)*Overhead is a

concern

FAST

RICH

Part One

Use cases for monitoring

Sylvain Hallé

SHOW

Cloud computingAjax web application

Sylvain Hallé


JavaScript

NOINC

Sylvain Hallé


Bee G

Beatles

Camel

Caravan

NOINC

Sylvain Hallé


Bee G

Beatles

Camel

Caravan

<a onclick=>

"javascript: findBand(’ ’)"Beatles

NOINC

Sylvain Hallé


findBand(’Beatles’)

NOINC

Sylvain Hallé


findBand(’Beatles’)

artist beatles=

NOINC

Sylvain Hallé


document.innerHTML = findBand(’Beatles’)

artist beatles=

NOINC

Sylvain Hallé


Does not needto be a URL

Does not needto be HTML

<SearchResults>

</SearchResults>

The Beatles Rubber Soul ...

<Item>

</Item>

<Artist>

</Artist><Title>

</Title>

<Search>

</Search>

beatles

<Artist>

</Artist>

NOINC

Sylvain Hallé

Cloud computingAjax web application<Search>

</Search>

beatles

<Artist>

</Artist> XMLThe eXtensible Markup

Language

?Nested collection ofelements

?Input/output data issemi-structured

.

.

<SearchResults>

</SearchResults>

The Beatles Rubber Soul ...

<Item>

</Item>

<Artist>

</Artist><Title>

</Title>

NOINC

Sylvain Hallé

SHOW

Cloud computingConceptually...

Sylvain Hallé


NOINC

Sylvain Hallé


Web serviceWeb client

NOINC

Sylvain Hallé

Main issue

SHOW

Possible between messages sent and messages expected

mismatch

Not like traditional programming: all input-output is exchanged unverified!

Sylvain Hallé

Defining message formats

SHOW

?

Sylvain Hallé

1.

2.

...

?


NOINC

Sylvain Hallé

1.

2.

...

<ItemSearch>

</ItemSearch> beatles<Artist> </object>


NOINC

Sylvain Hallé

<ItemSearch>


<ItemSearchResponse> <Items>

</Items></ItemSearchResponse>

Help!The Beatles

<Item>

</Item> ...

<Title> </no> <Artist> </Artist>


NOINC

Sylvain Hallé

SHOW

<ItemSearch>




Help!The Beatles

<Item>

</Item> ...


XML request

XML response


Sylvain Hallé

<ItemSearch>


ItemSearch[ [string]]Artist



Help!The Beatles

<Item>

</Item> ...


XML request

XML response


NOINC

Sylvain Hallé

<ItemSearch>



ItemSearchResponse[ [ Item[ Title[string], Artist[string] ]{0,¥} ]]

Items<ItemSearchResponse> <Items>


Help!The Beatles

<Item>

</Item> ...


XML request

XML response


NOINC

Sylvain Hallé


ItemSearchResponse[ [ Item[ Title[string], Artist[string] ]{0,¥} ]]

Items

?

!


NOINC

Sylvain Hallé

SHOW


WSDL: Web Service Description Language


CartCreate[ [int], [int], [ Item[ Title[string], Artist[string] ]{0,¥} ]]

ItemsSessionKeyItems

?

?

ItemSearchResponse[ [ Item[ Title[string], Artist[string], ]{0,¥} ]]

Items

CartCreateResponse[ [int], [int], [ Item[ Title[string], Artist[string] ]{0,¥} ]]

SessionKeyCartIdItems

!

!

. . .

Sylvain Hallé

SHOW

<ItemSearch>

</ItemSearch>

beatles 1234<Artist> </Artist><Bizbiz> </Bizbiz>


Sylvain Hallé

<ItemSearch>

</ItemSearch>

beatles 1234<Artist> </Artist><Bizbiz> </Bizbiz>


ItemSearch[ [string]]Artist vs.?

NOINC

Sylvain Hallé

SHOW

<CartCreateResponse> <SessionKey> </SessionKey> <CartId> </CartId> <Items>


1234

abc

...


Sylvain Hallé

<CartCreateResponse> <SessionKey> </SessionKey> <CartId> </CartId> <Items>


1234

abc

...


CartCreateResponse[ [int], [int], [ Item[ Title[string], Artist[string] ]{0,¥} ]]

SessionKeyCartIdItems

vs.

!

NOINC

Sylvain Hallé

SHOW

What happened?

Sylvain Hallé

?

What happened?

NOINC

Sylvain Hallé

1.

2.

...

What happened?

NOINC

Sylvain Hallé

2What happened?

NOINC

Sylvain Hallé

c

What happened?

NOINC

Sylvain Hallé

2

c

What happened?

NOINC

Sylvain Hallé

What happened?

NOINC

Sylvain Hallé

?2

c

2

c

Interface contracts

All messages comply with the WSDL but...

1.

2.

...

SHOW

Sylvain Hallé

?2

c

2

c

Interface contracts

You cannot add the same itemtwice to the shopping cart


1.

2.

...

NOINC

Sylvain Hallé

?2

c

2

c

Interface contracts

???

You cannot add the same itemtwice to the shopping cart


1.

2.

...

NOINC

Sylvain Hallé

Interface contracts

???

NOINC

Sylvain Hallé

???

Interface contracts

NOINC

Sylvain Hallé

The big question

SHOW

Prevent contract

violations

LOG

CODE

LOG

CODE

...out.print("Lemming into Floater");...

LOG

CODE

...out.print("Lemming into Floater");...

Game startsLemming into Blocker...Lemming into Floater...

LOG

CODE

Game startsLemming into Blocker...Lemming into Floater...

...logger.log("Lemming into Floater", Logging.LEVEL_DEBUG);...

LOG

CODE

[10:24:31] INFO Game starts[10:24:33] WARN Lemming into Blocker...[10:25:01] DEBG Lemming into Floater...

...logger.log("Lemming into Floater, id: " + lem._id, Logging.LEVEL_DEBUG);...

LOG

CODE

[10:24:31] INFO Game starts[10:24:33] WARN Lemming into Blocker...[10:25:01] DEBG Lemming into Floater, id: 32...


x?y?

??

LOG

CODE



LOG

CODE


...String msg = "Lemming into Floater ";msg += "id: " + lem._id + ", ";msg += "x: " + lem._x + ", ";msg += "y: " + lem._y;logger.log(msg, Logging.LEVEL_DEBUG);...

LOG

CODE


[10:24:31] INFO Game starts[10:24:33] WARN Lemming into Blocker...[10:25:01] DEBG Lemming into Floater, id: 32, x: 320, y: 67...

y?

x? x?

LOG

CODE



LOG

CODE

...String msg = "Lemming into Floater ";for (Lemming lm in lemmings) { msg += "id: " + lm._id + ", "; msg += "x: " + lm._x + ", "; msg += "y: " + lm._y + "; ";}logger.log(msg, Logging.LEVEL_DEBUG);...


LOG

CODE

...String msg = "Lemming into Floater ";for (Lemming lm in lemmings) { msg += "id: " + lm._id + ", "; msg += "x: " + lm._x + ", "; msg += "y: " + lm._y + "; ";}logger.log(msg, Logging.LEVEL_DEBUG);...

[10:24:31] INFO Game starts[10:24:33] WARN Lemming into Blocker...[10:25:01] DEBG Lemming into Floater, id: 32, x: 320, y: 67 ; id: 31, x: 450, y: 43 ; id: 23, x: 229, y: 40 ; ......

Game code

XMLtemplate

T

Temporalproperty

φ

{

}

Gameloop

1

3

2

Monitor

P

Named pipe,TCP socket,

HTTP requestVerdict

XML event Buffer

Part Two

Specifying properties with logic

Sylvain Hallé

SHOW

Interface contracts

All possible sequences of all possibles messages with all possible values

Sylvain Hallé

NOINC

Interface contracts

Constraintson individualmessages

Sylvain Hallé

NOINC

Interface contracts

Constraintson sequencesConstraints

on individualmessages

Sylvain Hallé

NOINC

Interface contracts

Constraintson sequences

Data-awaresequential constraints


Sylvain Hallé

NOINC

Interface contracts

Interface contract =valid (error-free) interactions

Constraintson sequences

Data-awaresequential constraints


Sylvain Hallé

Constraints on individual messages

Examples:

Three types of constraints (I)

SHOWSHOW

<Message> <Action>ItemSearch</Action> <Results>5</Results> <Keyword>beatles</Keyword> <Page>1</Page></Message>

Sylvain Hallé


Examples:


SHOWSHOW

1. The element must be an integer between 1 and 20.Page "/M


Sylvain Hallé


SHOWSHOW

1. The element must be an integer between 1 and 20.

2. The element is mandatory only if is present,otherwise it is forbidden.

Page

Page Results

"/M



Examples:

Sylvain Hallé

Constraints on message sequences

Examples:

2

SHOW

<Message> <Action> Login </Action> ...</Message>

<Message> <Action> LoginResponse </Action> ...</Message>

<Message> <Action> CartCreate </Action> ...</Message>

Three types of constraints (II)

Sylvain Hallé


Examples:

2

SHOW

3. The request cannot be resent if its response is

successful..

Login "/

"/





X

Sylvain Hallé


Examples:

2

SHOW


successful..

4. must follow a successful LoginResponse.

Login

CartCreate

"/

"/

"/





X

Sylvain Hallé

Data-aware sequential constraints

Examples:

2

SHOW

5. There can be at most one active cart ID per session key."/

Three types of constraints (III)

<Message> <SessionKey>123</SessionKey> <CartId>789</CartId> ...</Message>


Sylvain Hallé


Examples:

2

SHOW

6. You cannot add the same item twice to the shopping cart."/


<Message> <Action>CartAdd</Action> <Items> <Item> <ItemId>567</ItemId> ...


X

Mario cannot reach the end of a level without jumpingat least once

Mario can never jump higher than 20 pixels

20

X

If Mario crouches, he cannot jump right after

X

Mario cannot collide with an enemy while he isholding a Koopa shell

% grep "Lemming into Floater" log

% grep -P "Lemming into Floater, .*? x: (\d+?), y: \1" log

% grep -P "Lemming into Floater, .*? x: (\d+?), y: \1" log | sed 's/^.*id: ([0-9]+).*$/\1/'

% grep -Pzo "(?s)Lemming into Basher, id: (\d+).*Lemming into \ Floater, id: \1 [^;]*?x: (\d+?), y: \2" log | sed -r 'N;s/^.*?id: ([0-9]+).x: ([0-9]+), y: ([0-9]+)$ \ /\1, \2, \3/'

#!/usr/bin/pythonimport rewith open('log', 'r') as logfile: log = logfile.read()bm = re.findall(r'Basher, id: (\d+)', log)fm = re.findall(r'Floater, id: (\d+), x: (\d+), y: (\d+)', log) for lem in fm: if lem[0] in bm: print lem[0] + ', ' + lem[1] + ', ' + lem[2]

#!/usr/bin/pythonimport re, sysbashers = {}for line in sys.stdin: res = re.match(r'^Lemming into (.*?), id: (\d+), x: (\d+),\ y: (\d+)', line) if res: if res.group(1) == 'Basher': bashers[res.group(2)] = 1 else: if res.group(1) == 'Floater' and res.group(2) in bashers: print res.group(2) + ', ' + res.group(3) + ', '\ + res.group(4)

* A regexp (matches the unstructured event text, or the "msg" field for CEE/Lumberjack structured events) ceelog '/DHCP/'* A field comparison (matches a CEE/Lumberjack field) ceelog 'uid == "0"' ceelog 'uid != "0"' ceelog 'trusted!uid == "0"' ceelog 'username ~ /^guest-/' ceelog 'username !~ /^guest-/'* A combination of the above ceelog 'trusted!uid == "0" && username ~ /^guest-/'

* A regexp (matches the unstructured event text, or the "msg" field for CEE/Lumberjack structured events) ceelog '/DHCP/'* A field comparison (matches a CEE/Lumberjack field) ceelog 'uid == "0"' ceelog 'uid != "0"' ceelog 'trusted!uid == "0"' ceelog 'username ~ /^guest-/' ceelog 'username !~ /^guest-/'* A combination of the above ceelog 'trusted!uid == "0" && username ~ /^guest-/'

grep{

RSYSLOG

RSYSLOG

grep

Sylvain Hallé

Expressing data constraints

Simple XPathFetches portions of an XML document according to aquery path = sequence of tags

: set of messages: set of XML query paths: set of atomic values

: ́ ® 2

Examples:(‘‘/a/b/c’’, m) = {1,2,4}(‘‘/a/b/d’’, m) = Æ

M

M

Q

Q

V

Vp

pp

SHOW

m

{

<a> <b> <c>1</c> <c>2</c> </b> <d> <c>9</c> </d> <b> <c>3</c> </b></a>

Sylvain Hallé


XPath termExpresses properties over values fetched by XPath expressions

For some message Î , path Î ,

" x : j(x) Û j(v) for every Î ( , )

$ x : j(x) Û j(v) for some Î ( , )

Examples:" x : x < 5/a/b/c

$ x :/a/b

$ x : " y : y £ x/a/b/c /a/b/c

m M

mq

mq

q Q

q

q

v

v

p

p

2

SHOW

<a> <b> <c>1</c> <c>2</c> </b> <d> <c>9</c> </d> <b> <c>3</c> </b></a>

m

{

Sylvain Hallé


2

SHOW

1. The element must be an integer between 1 and 20.


Page

Page Results

"/M


Sylvain Hallé


2

SHOW

1. " x : x > 0 Ù x < 21/Message/Page


Page Results


Sylvain Hallé


2

SHOW

1. " x : x > 0 Ù x < 21/Message/Page

2. $ x : Û $ y : /Message/Page /Message/Results


Sylvain Hallé

Linear Temporal Logic

Alphabet (A)Set of possible messages

Trace (A*)Sequence of messages

Sylvain Hallé

LTL formula = assertion on the of states in a tracesequence

a "always a" a "a in the next" a "eventually a"

a b "a until b"

GXF

W


G (a ® b)X (d cÚ e) WØFALSE TRUE

. . .A A EC CDB B

Sylvain Hallé

Well-known results:

1. For every LTL formula j, there exists a Büchi automaton Asuch that for every (infinite) trace s:

i.e. LTL describes languages

2. The alphabet symbols can be generalized to finite sets ofBoolean propositions

w-regular

Þ Let’s use XPath terms as our Boolean propositions


j

s |= j Û s Î L(A )j

Sylvain Hallé


Examples:

2

SHOW


successful..


Login

CartCreate

"/

"/

"/





X

Sylvain Hallé


Examples:

2

SHOW

3. (" a : a = LoginResponse ®/Message/Action

( " a’ : a’ ¹ Login))/Message/Action.


G

X G

CartCreate "/





X

Sylvain Hallé


Examples:

2

SHOW


( " a’ : a’ ¹ Login))/Message/Action.


G

X G

CartCreate "/





X

Xpath terms

Sylvain Hallé


Examples:

2

SHOW


( " a’ : a’ ¹ Login))/Message/Action

4. (" a : a ¹ CartCreate)/Message/Action

(" a’ : a’ = LoginResponse)/Message/Action

G

X G

W

.





X

Xpath terms

Mario cannot reach the end of a level without jumpingat least once

F action = jump

Mario can never jump higher than 20 pixels

G (action = jump → height < 20)

20

X

If Mario crouches, he cannot jump right after

G (action = crouch → X action ≠ jump)

X

Mario cannot collide with an enemy while he isholding a Koopa shell

G (action = haveShell → X action ≠ collision)

G (action = haveShell →(action ≠ collision U action = dropShell))

or better

Sylvain Hallé

Runtime monitoring

Gerth, Peled, Vardi, Wolper (PSTV 1995): construction of a Büchi automaton from a given LTL formula j

Benefit:

" ": automaton states are built as thetrace is readon-the-fly

Sylvain Hallé

Runtime monitoring

j


Benefit:


Sylvain Hallé

Runtime monitoring

s =

j


Benefit:


Sylvain Hallé

Runtime monitoring

s = a

j


Benefit:


Sylvain Hallé

Runtime monitoring

s = a

ja


Benefit:


Sylvain Hallé

Runtime monitoring

s = ab

ja


Benefit:


Sylvain Hallé

Runtime monitoring

s = ab

ja b

b


Benefit:


Sylvain Hallé

Runtime monitoring

s = aba

ja b

b


Benefit:


Sylvain Hallé

Runtime monitoring

s = aba

ja

a

a

b

b


Benefit:


Sylvain Hallé


Benefit:

" ": automaton states are built as thetrace is read

Dead end: formula is false

on-the-fly

Runtime monitoring

s = aba

ja

a

a

b

b

Sylvain Hallé

Runtime monitoring

Algorithm overview:

1. An LTL formula is decomposed into nodes of the form

sub-formulas thatmust be true now

sub-formulas that mustbe true in the next state

Sylvain Hallé

Algorithm overview:

1. An LTL formula is decomposed into nodes of the form

Example:

sub-formulas thatmust be true now

sub-formulas that mustbe true in the next state

Runtime monitoring

Sylvain Hallé

2. Negations pushed inside (classical identities + dual of U = V)

3. At the leaves, G contains atoms + negations of atoms:we evaluate them

Verdict:

! All leaves contain : formula is false! A leaf is : formula is true! Otherwise:

4. Next event: D copied into G and we continue

FALSEempty

Runtime monitoring

Sylvain Hallé

Example: G (a ® b)X

Runtime monitoring

Sylvain Hallé


G (a ® b)X ?

Runtime monitoring

Sylvain Hallé


G (a ® b)X ?

a ® bX G (a ® b)X?

Runtime monitoring

Sylvain Hallé


G (a ® b)X ?

Øa G (a ® b)X?

a ® bX G (a ® b)X?

Runtime monitoring

Sylvain Hallé


G (a ® b)X ?

a, X b G (a ® b)X?Øa G (a ® b)X?

a ® bX G (a ® b)X?

Runtime monitoring

Sylvain Hallé


G (a ® b)X ?

a, X b G (a ® b)X?

a G (a ® b), bX?

Øa G (a ® b)X?

a ® bX G (a ® b)X?

Runtime monitoring

Sylvain Hallé


a G (a ® b), bX?

Øa G (a ® b)X?

Runtime monitoring

Sylvain Hallé


s = a

a G (a ® b), bX?

Øa G (a ® b)X?

Runtime monitoring

Sylvain Hallé


s = a

a G (a ® b), bX?

Runtime monitoring

Sylvain Hallé


s = a

G (a ® b), bX?

Runtime monitoring

Sylvain Hallé


s = a

?G (a ® b), bX

G (a ® b), bX?

Runtime monitoring

Sylvain Hallé


s = a

a, X b, b G (a ® b)X?

a, b G (a ® b), bX?

Øa, b G (a ® b)X?

a ® b, bX G (a ® b)X?

?G (a ® b), bX

Runtime monitoring

Sylvain Hallé


s = a

a, b G (a ® b), bX?

Øa, b G (a ® b)X?

Runtime monitoring

Sylvain Hallé


s = a

Øa, b G (a ® b)X?

Runtime monitoring

Sylvain Hallé


s = ac

Øa, b G (a ® b)X?

Runtime monitoring

Sylvain Hallé


s = ac

No way to extend the trace:formula is false

Runtime monitoring

Sylvain Hallé

The verification can be separated in two steps


G

X

Ú

"$

$ G

F

®

1. Temporal stepDetermine termporal relationships to current message

2. Data stepEvaluate relevant XPath terms on message

Sylvain Hallé


Examples:

2

SHOW

5. There can be at most one active cart ID per session key."/




Sylvain Hallé


Examples:

2

SHOW

5. (" k : " c : /Message/SessionKey /Message/CartId

(" k’ : " c’ : /Message/SessionKey /Message/CartId

k = k’ ® c = c’))

G

G




Sylvain Hallé



2

SHOW



k = k’ ® c = c’))

G

G

Sylvain Hallé


·XPath terms and temporal operators aremixed

.

·Not just ‘‘LTL with syntactical sugar’’.

·Not just a pathological case



k = k’ ® c = c’))

G

G


2

SHOW

G

G

"

"

k

Sylvain Hallé


Examples:

2

SHOW

6. You cannot add the same item twice to the shopping cart."/




X

Sylvain Hallé


Examples:

2

SHOW

6. (" a : a = CartAdd ®/Message/Action

" i : (" a’ :/Message/ItemId /Message/Action

a’ = CartAdd ® " i’ : i ¹ i’ ))/Message/ItemId

G

X G




X

Sylvain Hallé

Quantification must be relative to the values in the current message, and not the whole set V of possible values!

Example: ‘‘In every message, the a parameter must equal the b parameter’’. Suppose V = {1,2}, and classical first-order quantification.

Runtime monitoring

" x : " y : x = ya b

(" y : 1 = y) Ù (" y : 1 = y)b b

( 1 = 1) Ù ( 1 = 2) Ù ( 1 = 1) Ù ( 1 = 2)

Contradiction

G

G G

G G G G

Sylvain Hallé

LTL-FO+

current

(Hallé & Villemaire, EDOC 2008)Extension of LTL with (limited) first-order quantification on message elements

·Boolean and LTL operators keep their original meaning·An XPath term is always meant to refer to the

message in the trace

Runtime monitoring

Sylvain Hallé

Adaptation of the runtime monitoring algorithm to handle LTL-FO+:

1. Atoms become equality tests

2. Decomposition rules for quantifiers

(and vice versa)

Runtime monitoring

Sylvain Hallé

Enforcing interface contracts at runtime

NOINC

XMLHttpRequest

·JavaScript object·Provided by the browser·All communications to monitor

already centralized: ‘‘no’’instrumentation

Sylvain Hallé


NOINC

XMLHttpRequestBB

Sylvain Hallé


NOINC

XMLHttpRequestBB

XMLHttpRequestLTL-FO+algorithm

·Wrapper around original·Provides same methods·Checks messages before

relaying them

Sylvain Hallé

Add BeepBeep to an application

myapplication.html

<html><head><title></title><script type=" " href=" "/>

</head><body>

</body></html>

My Application

...

text/javascriptmyapplication.js

?

�Include BeepBeep

Copy BeepBeep in the application's directoryhttp://beepbeep.sourceforge.net

Sylvain Hallé


myapplication.html


</head><body>

</body></html>

My Application

...


<script type="text/javascript" href="beepbeep.js"/>

?

�Include BeepBeep


Sylvain Hallé


myapplication.html myapplication.js


</head><body>

</body></html>

My Application

...



// Initializations

= ();

(){

( );}

...

req XMLHttpRequest

...

abc

... req. some_message

new

function

send

?

�Include BeepBeep


Sylvain Hallé


beepstore.html beepstore.js


</head><body>

</body></html>

My Application

...



// Initializations

= ();

(){

( );}

...

req

...

abc

... req. some_message

new

function

send

XMLHttpRequestBB

Include BeepBeep

?

�


Sylvain Hallé

Create a with LTL-FO+ formulascontract file ?


# -------------------------------------------------------# BeepBeep contract file for the Beep Store# -------------------------------------------------------

% The element Page must be an integer between 1 and 20.

% The element Page is mandatory only if Results is present, otherwise it is forbidden.

% The Login request cannot be resent if its response is successful.

; ( p /Message/Page (((p) > ({0})) ((p) < ({21}))))

; ( a /Message/Action (((a) = ({ItemSearch})) ( (( r /Message/Results ({TRUE}))

( p /Message/Page ({TRUE}))) (( p /Message/Page ({TRUE})) ( r /Message/Results ({TRUE}))))))

; ( a /Message/Action (((a) ({LoginResponse})) ( ( ( b /Message/Action ( ((b) ({Login}))))))))

G

G

GX G

[ ]

[ ]< >< >

< >< >

[ ][ ]

&

->->

&->

= ->! =

Caption: usedwhen violationsare discovered

Plain-textLTL-FO+(automaticallyparsed)

}

Sylvain Hallé


When loading the application, BeepBeep starts as a smallJava applet inside the page

TheBeepStore

GO

Sign in or register

What is this?

Login

Ask for account

Contact us

Fault parameters

Search: Your Cart

Search results for ‘Beatles’

Rubber SoulThe Beatles

Yellow SubmarineThe Beatles

�

?/?/?/?/?/?:0:0

Sylvain Hallé

BeepBeep’s visible interface

?/?/?/?/?/?:0:0

Current state of monitorfor each property

Number ofmessagesprocessed

Cumulativeprocessingtime (in ms)

T: last message made it truet: is trueF: last message made it falsef: is false?: not yet true/false

Chocolate Doom74 KLOC

Angry Bots26 KLOC

Pacman Canvas1 KLOC

Bos Wars113 KLOC

Infinite Mario Bros.6 KLOC

Pingus40 KLOC

We still have to write theproperties, so what's the point?*

We're too busy to learnanother toolA*

Using this requires a higher level ofeducation. We'll have to increase wagesA*

*Really happened

We can already do all thisA*

(The Angry Practicioner)

OF COURSE.

This is a finitely computable problem.

It can be computed in finite time by aTuring machine.

There exists a C/Python/Assembly/Haskell/JavaScript/Scala/Go/BrainFuck//Whatever

program that does that. We just need towrite it down.

So what's the big news???

Data

30,000LINES OF

GLUE CODE

UntestedYou wrote thisHardly reusable

Result

HOW.

DataComponentizedReusableWorth investing time testing

Result

HOW.

Engine

1,000LINES OF

DSL

Alan Perlis(1922-1990)

Beware of of the Turing tar-pit in which everything is possible but nothing of interest is easy. ,,

,,

G (∃ retAddrVal ∈ ./return-address : ((./instruction = call) ∧ (¬ ((F( ((./instruction = mov)

∧ (./output/type = general-register)) →(∃ regA ∈ ./output/name : (F (( ((./instruction = mov)∧ (./output/type = general-register))∧ (./input/type = litteral) ) →

(∃ regB ∈ ./output/name :( ∃ constAddr ∈ ./input/value :(F (((./instruction = cmp ) ∧

(./output/type = regA)) →(∃ loc ∈ ./location : (F((

((./instruction = mov ) ∧(./output/type = general-register)) ∧(./output/name = regA ) ) ∧((./input/name = regB ) ∧ (./input/type = ptr)))))) ))))) )))

U ((./instruction = return)∧ (./fonction-returned = retAddrVal)) )) ))

Part Three

From RV to CEP

CEP

omplex

vent

rocessing

EventsEvents

An event is an element e taken from someset E, called the event type

Booleans

B

Numbers

R

234

π

Strings

S

abc

Functions

X Y→

Sets

X2

Primitivetypes

Compositetypes

A sample logA sample log

A file (or stream) of events

[10:24:31] INFO Game starts[10:24:33] WARN Lemming into Blocker...[[10:25:01] DEBG Lemming into Floater, id: 32, x: 320, y: 67 ; id: 31, x: 450, y: 43 ; id: 23, x: 229, y: 40 ; ... ...

Each event has one or more data elements

Actual (physical) format not relevantfor us

Searching the logSearching the log

Select AVG(closingPrice)From ClosingStockPricesWhere stockSymbol = `MSFT'for (t = ST; t < ST+50, t+= 5) { WindowIs(ClosingStockPrices, t - 4, t);}

ProblemsProblems

Formal languages (e.g. logic, automata)focus on event ordering; not so good atperforming computations over events

Complex Event Processing often reducesto a thin layer over custom proceduralcode

Goal: provide a formal andnon-procedural framework forthe processing of event streams

TracesTraces

An event trace (or event stream) is a potentiallyinfinite sequence of events of a given type:

2 0 6 34 9 . . .

Traces are symbolically denoted by:

e = e0 e1 e2 e3 ...

The set of all traces of type T is denoted as:

T*

ProcessorsProcessors

A processor is a function that takes 0 or moreevent traces as input, and returns 0 or 1event trace as output

1 : 1 processor

2 : 1 processor

. . . . . .

A high-level event trace can be produced bycomposing ("piping") together one or moreprocessors from lower-level traces

CompositionComposition

Processor algebraProcessor algebraGoal: come up with a "toolbox" of basicprocessors sufficient to perform variouscomputations over traces

??

A few useful functionsA few useful functions

Identity function: returns an event if given one,or t if passed the empty event ε

ιt(x) = { t if x = εx otherwise

+ (x) = {x}Wrap function

-({x}) = xPeel function

Path function: returns subtree at endof path π

/π

SemanticsSemanticsProcessors can be defined formally bydescribing how their output trace is createdfrom their input trace(s)

e0, ..., en : φ(x0 , ..., xn)

Input trace(s)

Symbolic variables:xi refers to the i-th trace

on the left

Constants as processorsConstants as processors

Any element t of type T can be lifted as a0 : 1 processor producing the infinite tracet t t t ...

t . . .t t

The constantprocessor t e : t = t t t ...

Input/outputInput/output

0 : 1 processors can be used to produce anevent trace out of an external source (i.e.standard input, a file, etc.)

Ditto for 1 : 0 processors

a . . .b

a . . .b

MutatorMutator

Returns t, but only as many times as the number of events received so far

i.e. "mutates" input events into t

tte

Functions as processorsFunctions as processors

Any n-ary function f defined on individual events can be lifted to an n:1 processor ontraces, by applying it successively to n-uples

2 0 6. . .

3 8 1

+ 7 8 5

. . .

. . .

Functions as processorsFunctions as processors

Any n-ary function f defined on individual events can be lifted to an n:1 processor ontraces, by applying it successively to n-uples

e0, e1 : x0+x1

e00+ e10

e01+ e11

, e02+ e12

, , . . .

=

FreezeFreeze

Returns the first event received, upon everyevent received

abb. . . a a a . . .

e : x = e0 e0 e0 ...

DelayDelay

Returns every the input trace, starting from itsn-th event

abc. . . b . . .

e : x = en en+1 en+2 ...

2

n

e : xn=

c

DecimateDecimate

Returns every n-th event of the input trace

abc. . . a . . .

e : x = e0 en e2n ...

2

n

cΨ

Ψ

e : x= nie : xnΨ

i

COMPLEX PROCESSORS

WindowWindow

Simulates the application of a "slidingwindow" to a trace

Takes as arguments: another processor φand a window width n

Returns the result of φ after processingevents 0 to n-1...

Then the result of (a new instance of) φthat processes events 1 to n...

...an so on

Υ φn

Example: execution of the processoron the trace

WindowWindow

2 1 5 0Υ ++2

Υ2

2 12 1 2 12 3

2 11 5 2 11 6

2 15 0 2 15 5

2 1 5 0 3 6 5

WindowWindow

The window processor can take anyprocessor as an argument...

...i.e. the sliding window can be applied toanything.

Formally:

e : φ e : φ= n-1iΥn i

FilterFilter

Discards events from an input trace basedon a selection criterion

Takes as argument another processor φ

Evaluates φ on the trace that starts at event0; returns that event if the first eventreturned by φ is T

Same process on the trace that starts atevent 1...

...an so on

Φ φ


FilterFilter

2 1 5 0Φ∈2IN

Φ2 1 5 0 2 0

∈2IN

∈2IN2 1 5 0

FilterFilter

The filter can take any processor as anargument...

...including a processor that requires multipleinput events before outputting something

Formally:

e : φ e : φ=Φ 1 ΦΦ(e, φ) ,

Φ(e, φ) = { e0 if

no event otherwise

e : φ = T0

SpawnSpawn

Cumulative combination of a processor'soutput for every suffix of a trace

Creates one new instance of processorφ upon every new input event

Feeds each input event to all existinginstances of φ

Combines the value returned by eachinstance using function f

...and outputs it

Σ φf


SpaweSpawn

2 1 5 0Σ+

Σ+2 1 5 0 2 3 8

x

x

x

8

2 1 5 0 2 1 5 0

x1 5 0 1 5 0

x5 0 5 0

++

+

SpawnSpawn

Formally:

e :

e :

=1

Σ φf

e : φ 0 , f ( Σ φfe : φ 0 , e : φ 0 ,e : φ 0 , )Turns out to be a powerful device; dependingon φ and f, can provide many usefulprocessors...

SpawnSpawn

Count events Σ 1+

Cumulative sum Σ+

Set of all events Σ∪ +

= #= ++=∪

These processors can be freely composed

Compute the statistical moment of order n


n

Σ+

Σ+1

÷


Compute the statistical moment of order n


n

Σ+

Σ+1

÷

= #


Return sum of two successive events,only if it is greater than 5


++

Υ2Φ

> 5

All together nowAll together now


Count pairs of successive events that aremore than one standard deviation fromthe mean

E(X)

-



σ

E(X)

-

÷



σ

E(X)

-

÷ Φ

> 1



σ

E(X)

-

÷

X

Φ

> 1

Φ∧



#σ

E(X)

-

÷

X

Φ

> 1

Φ∧

AdvantagesAdvantages

No imperative constructs

No restrictions on what can be piped towhat (modulo type compatibility)

Streaming operation: outputs producedas inputs are being consumed

Implicit handling of buffering, duplication,etc.

A declarative event streamquery engine

abc. . . a . . .n cΨ

abc. . . b . . .n c

EVERY nTH OF (T)

TRIM n FROM (T)

WHERE2 1 5 0 2 0

C

C2 1 5 0

(T) WHERE condition

COMBINE (T) WITH f

Σ+2 1 5 0 2 3 8

f

f

8

2 1 5 0 2 1 5 0

f1 5 0 1 5 0

f5 0 5 0

++

+

a . . .b

a . . .b

FILE "filename"

SAVE (T) TO "filename"

WHEN @P IS A PROCESSOR:

THE SUM OF ( @P )

IS THE PROCESSOR

COMBINE (@P) WITH SUM.

Arbitrarysymbol

Grammar rule this symbolmust parse against

New grammar case

Grammar rule the caseis added toExpression the new casestands for

WHEN @P IS A PROCESSOR: THE COUNT OF ( @P ) IS THE PROCESSOR COMBINE (SELECT 1 FROM (@P)) WITH SUM.

WHEN @P IS A PROCESSOR: THE SUM OF ( @P ) IS THE PROCESSOR COMBINE (@P) WITH SUM.

WHEN @P IS A PROCESSOR: THE AVERAGE OF ( @P ) IS THE PROCESSOR SELECT (T.*) ÷ (U.*) FROM ( THE SUM OF (@P) AS T, THE COUNT OF (@P) AS U).

On every fifth trading day starting today,calculate the average closing price of MSFT for the five most recent trading days, and keepthe query standing for fifty trading days.

timestamp stockSymbol closingPrice

0 APPL 1039.30 MSFT 950.00 GOGL 433.31 MSFT 951.21 APPL 1038.3... ... ...


String line = br.readLine().trim();if (!line.isEmpty()) { String[] parts = line.split(","); if (parts[0].compareTo("ABC") != 0) { value_index++; sum += Double.parseDouble(parts[1]); if (value_index == 5) { double average = sum / 5; value_index = 0; sum = 0; return average;}}}


SELECT afd FROM ( SELECT S1.timestamp AS ts, AVG(S2.closingPrice) AS afd FROM (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S1, (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S2 WHERE (S2.timestamp - S1.timestamp) < 5 GROUP BY S1.timestamp) AS S3WHERE MOD(ts, 5) = 0;


EVERY 5TH OF ( APPLY (THE AVERAGE OF (*)) TO ( SELECT closingPrice FROM stocks) WHERE (stockSymbol) = ("MSFT")))) ON A WINDOW OF 5).

Calculate how many times the closing priceof MSFT is greater than 20 and the nextday, its closing price is less than 10.


SELECT COUNT(*) FROM (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S1, (SELECT * FROM stocks WHERE stockSymbol = "MSFT") AS S2 WHERE (S2.timestamp - S1.timestamp) = 1 AND S1.closingPrice > 20 AND S2.closingPrice < 10;


WHEN @P IS A PROCESSOR: MY PATTERN IN ( @P ) IS THE PROCESSOR (SELECT (closingPrice) LESS THAN (20) FROM (@P)) AND (NEXT (SELECT (closingPrice) GREATER THAN (10) FROM (@P))).

THE COUNT OF (( MY PATTERN IN ( (SELECT closingPrice FROM stocks) WHERE (stockSymbol) = ("MSFT"))) WHERE (*) = (true)).

LinearTemporalLogic!

import ca.uqac.lif.cep.*;

public class MyProcessor extends SingleProcessor {

public Queue<Vector<Object>> compute(Vector<Object> inputs) {

}

public void build(Stack<Object> s) {

}}

. . . Create output events from input . . .

. . . Instantiate processor from parse stack . . .

<processor> := . . .<number> := . . .<string> := . . .

Add new rules to any symbol fromthe basic grammar

<processor> := <my_processor> ;

<my_processor> := INCREMENT ( <processor> ) BY <number> ;

Symbols already defined in basic grammar

Adds a new case to an existing rule

import ca.uqac.lif.cep.*;

public class MyProcessor extends SingleProcessor {

private int increment;

public Queue<Vector<Object>> compute(Vector<Object> inputs) { Queue<Vector<Object>> out = new Queue<Vector<Object>>(); Vector<Object> v = new Vector<Object>(); Integer i = (Integer) inputs.firstElement() + increment; v.addElement(i); out.put(v); return out; }

. . .

. . .

public void build(Stack<Object> s) { Number n = (Number) s.pop(); s.pop(); s.pop(); Processor p = (Processor) s.pop(); s.pop(); s.pop();

increment = n.intValue();

Connector.connect(p, this);

s.push(this); }}

Read contents ofparse stack

<number>BY(<processor>)INCREMENT

Set processor's statePipe it to its inputPut on parse stack

Some pre-packaged grammar extensions:

Manipulation of name-value tuples

Set theory

Formatted input (CSV, XML, JSON)

Graphing (histograms, scatterplots, ...)

Basic signal processing (smoothing, peak detection, ...)

Create your own!

It works

1500 Hz performance

Rapid prototyping -400 LOC3

What vs. how

https://liflab.github.io/beepbeep-3

A few pointersvvv

Mv LeuckerE Cv Schallhartv y4KK9Hv A Brief Account ofRuntime Verificationv J. Log. and Alg. Prog. 78 y5Hv

Dv Luckhamv y4KK4Hv The Power of Events: AnIntroduction to CEPv AddisonbWesleyv

Sv HalléE Rv Villemairev y4KG4Hv Runtime Enforcementof Web Service Message Contracts with Datav IEEETrans. Services Computing 5y4Hv

Sv VarvaressosE Kv LavoieE Av Blondin MasséE SvGabouryE Sv Hallév y4KG:Hv Automated Bug Finding inVideo Games: A Case Study for Runtime MonitoringvICST 2014v

http://liflab.ca

GAME OVER

YES NO

QUESTIONS ?

chasing bugs with the beepbeep event stream processor

Technology