Download - Decision CAMP 2014 - Mariano de Maio
Agenda● What is Complex Event Processing?
○ Definitions and examples● Event Driven Architecture
○ EDA and BPM, SOA, BAM, BRMS○ CEP as a component of EDA
● Cloud Case○ Business Rule Management Systems○ Cloud-wide event streaming modes○ Handling CEP on the cloud○ Knowledge Engineering lessons○ Future improvements
What is Complex Event Processing?
● Event Definition○ A significant change in state at a particular point in time,
in a specific domain○ Events can be embedded in a situation, where
sometimes we want to react to some type of events.● Complex Event Definition
○ Non-atomic event: An abstraction, composition, or aggregation of other events called its members
Complex Event Examples
● 1929 stock market crash: Composition of individual stock trades
● Natural disasters: Aggregation of many natural events perceivable through sensors
● DDOS attack detection: Aggregation of each HTTP request basic information and temporal relation
What is Complex Event Processing?● Complex Event Processing
○ The ability to detect, correlate, abstract or aggregate simple events to react to complex events
○ Uses temporal reasoning: Discrete sequence of points in time (or intervals) to correlate simple events between each other
● Event Streaming Processing○ Events come in streams: Related to mostly filtering and
finding events within streams○ Once they started adding constraints to such events, they
started calling it CEP
Event Driven Architecture● Building applications focused on the generation of events● Proposes a set of building blocks
○ Event producers: They send new events into the Event Driven System
○ Event consumers: They listen to specific events from the Event Driven System
○ Event processing agents: Works using Complex Event Processing
○ Event channels: Connections and protocols for transmitting events
Event Driven Architecture
Characteristics of EDA● Applications are naturally centered on events● Decoupled design
○ The event producer doesn’t know the consumer○ The processing agents don’t know consumers or
producers● The application will have the possibility to react based on
events● Proposes a non invasive approach to extending applications
○ Provides scalability and fault tolerance for free○ Ideal for dynamic nature of cloud environments
EDA and SOA● SOA proposes a design based on shareable and reusable
components with well defined interfaces● Each of the services inside SOA can generate events that
can be consumed by EDA components to analyse or extend the system○ Example: system monitoring of ESBs
EDA and BPM● The Business Process activities can serve as event
producers, sending events outside the process’ scope● Business Processes can also wait to receive events from
external sources to change its internal state
EDA and BAM● Business Activity Monitoring Systems are always related with
KPIs● How we aggregate events will allow us to provide better
feeds for new indicators
EDA and BRMS● BRMS: Business Rule Management Systems● Business Rules can be used to filter, route and compose
events● Event Processing Agents can be written using rules
○ Events can trigger which rules are fired
Cloud case
Architecture Overview
Characteristics● Rule based CEP engine● Business Rule Management System (BRMS) for storing and
populating rule definitions○ Rules updated on runtime after tests pass
● Handling thousands of events from multiple sources○ Generic events○ Typed events
● Receiving events from multiple applications○ Event streaming framework based on AOP
● Event subscribers, persistent (with SLA) and non-persistent (without SLA)
Cloud vs CEP● Cloud Servers: Limited size, Unlimited amount
○ Scalability is key● Drools-based CEP: All events in one common processing
network○ Requires a large amount of memory○ Usually runs all rules inside one server
● Compromise: Creating a distributed Event Processing Network based on responsibility grouping of rules○ Architecture based on splitting CEP environment into
several servers depending on demand
Domain-splitted CEP nodes● Distributed Event Processing Network
○ Based on responsibility grouping of rules○ Broker CEP node: quick distribution of events to
corresponding domain-based nodes
Domain-splitted CEP nodes● Implemented with Drools Fusion
○ Rule engine based, with support for temporal operators○ BRMS provides editors for rules, processes, and data
■ Exposes definitions through Git and Maven
What are rules?● Basic structure to diagram requirements as isolated
conditions and their correspondent actions
rule "rule name" when
a condition is met
thenan action is taken
end
CEP Rule events: Drools Fusion● Simple event declaration
○ define a point in time event■ by default, when added
○ define an interval event● Can create new types of events on
demand● Usually immutable, but not enforced● Events can come from different
sources● Drools provides 13 temporal operators
to compare events between each other
import some.package.VoiceCall;declare VoiceCall
@role( event )@timestamp( callTime )@duration( callDuration )
end
declare StockTick@role( event )symbol: Stringprice: double@expires( 2h30m )
end
Temporal Operators
CEP Rule example● Drools provides ways to collect and accumulate different
groups of facts and events by the conditions they meetrule "5 voice calls in two hours by same client"when //we have over 4 calls from the same client within 2 hours since first call
$vc: VoiceCall($cn: clientNumber)$c: Client(clientNumber == $cn)$count:: Number(intValue > 4) from accumulate(
VoiceCall(this != $vc, clientNumber == $cn, $vc before this, this after $vc, this meets[2h] $vc
),count())then //mark his problem as urgent and requiring advanced treatment
globalHelper.sendAlert("user called over and over");globalHelper2.markCallsAsUrgent($cn);globalHelper2.setDifficulty(DifficultyLevel.ADVANCED);
end
BRMS and CEP nodes● JBoss BRMS 6
○ Open Source project. Prepared for cloud○ Manage rule groups as Maven projects stored on Git
■ Automatic CI for all environments
Handling the volume of events● Scalable Message Broker: cloud
based (can grow with necessity)● Multiple CEP nodes
○ Drools works in memory○ Domain divided environments○ n CEP nodes - m servers
● Reduced global domain for cross reference problems
● Old events: persisted in scalable cache (MongoDB)○ Discard policies more long-lived than in-memory events○ Queried when new events trigger retrieval rules
Event subscriptions● Persistent subscribers
○ SLA previously agreed○ No message loss○ Policy on SLA breaching
■ Drop messages■ Store for a while and notify
action required● Non-persistent subscribers
○ No SLA needed○ Possible message loss○ Ideal for quick project kick-off
Cloud Integration● Fuse ESB: Cloud based and scalable, but couldn’t take the
varying volume of events○ Events handled through RabbitMQ
App Integration● Java: if you use AOP based architecture
○ Events sent through an interceptor of your services○ Your applications gain the capacity of sending events
without adding any code!
BPM Engine Integration (cloud-based)● JBoss BPM Suite: jBPM6 based
○ BPMN2: Extensible definitions○ jBPM6: Extensible parsing for
runtime modification○ Added event sending to all
relevant steps of processes● Added special handling for complex
events in relevant process definitions (i.e. human task automatic management based on SLAs)
BPM Engine Integration (cloud-based)● JBoss BPM Suite and CEP engine
run on the same technology stack● Splitting based on:
○ CEP gets events from many other sources than BPM
○ Non-functional requirements● Processes delegate fast and persist
long running processes● CEP handles thousands of events
in memory (mostly) and very closely related in time.
Cloud Operational Tasks
● SLA breach handling○ Cloud plan that allows spikes in use○ DevOps lesson: Continuous monitoring is key!○ : : When SLA is breached, message increase can be
handled and actions taken■ Message discarding■ SLA changed
● All cloud components are monitored continuously○ Monitoring results can be fed back to the CEP engine
Knowledge Engineering Lessons● Knowledge Engineering: Discipline that involves
transferring, modeling, and integrating knowledge from humans into artificial intelligence systems.
● Experience on gathering information from healthcare experts led to the following maturity stages definitions:○ Stage 1: Analysts translate expert knowledge into rules○ Stage 2: Analysts teach experts basic rule structures○ Stage 3: Provide tools to experts to improve and write
rules directly■ DSL: Domain Specific Languages■ Decision Tables
BASIC RULES CEP RULES
when a condition is metthen take a specific action
detect a correlation of eventsand take a related action
Knowledge Engineering Lessons● Stage 1 and 2: Defining knowledge into structures
● Stage 3○ DSL: Providing translations from natural language to
specific technical implementation for rule executions○ Decision Tables: Ideal for highly structured rules with
specific delimiters.
Risk Factors● Environment allocation is still done manually● Memory allocation problem
○ Event after splitting in nodes by domain, some domains might require too much memory. Currently suggested for private cloud
● Nodes are not replicated○ In case of a node failure, a node is restarted○ The in-memory events are lost○ Small time window where event correlations can be
missed
Future Improvements● Automatic node generation
○ Cloud PaaS providers have APIs to create new environments
○ Based on new detected complex events, new environments could be created■ i.e. New rule domain created → Automated
environment generation● DSL Coverage: Extend usage for Business Analysts
Future Improvements● Automatic node replication
○ In-memory events replicated on extra nodes○ Rule execution conditioned to one node at a time○ Continuously update events and rule activations
between nodes○ Drools: Agenda Filters and WM Event Listeners
Briefing up● CEP can be handled in cloud environments
○ Special understanding of domain and processing size are needed to avoid crashes
○ Special operational tasks must be created■ Integration on existing enterprise administration
“accepted good practices” is possible■ Continuous Delivery is a very real possibility
● CEP interactions with enterprise architecture can provide a huge added value○ Automatic control of near to SLA breach operations○ Better detection of unexpected behaviour in event producers
questions?
please come next to the computer :)
