sybase bam overview

Sybase BAM Overview Xu, Jiang (BAM/Rules Team) March 20, 2007

Sybase Confidential Proprietary.

Sybase Confidential 2

Agenda

•  Technology Background •  Analytic Model

•  Architecture •  Main Features •  Demo


Background - Overview

•  Business Activity Monitoring (BAM) •  Complex Event Processing / Event Stream Processing

•  Two approach of CEP/ESP •  Real Time Business Intelligence

•  Two approach of RTBI


Background - BAM

"Business activity monitoring" (BAM) is Gartner's term defining how we can provide real-time access to critical business performance indicators to improve the speed and effectiveness of business operations. Unlike traditional real-time monitoring, BAM draws its information from multiple application systems and other internal and external sources, enabling a broader and richer view of business activities.


Background – ESP/CEP

“Event Stream Processing” (ESP) is software technology that allows applications to monitor streams of event data, analyze those events, and act upon opportunities and threats in real time. ESP systems often utilize, or include, event databases and event visualization tools, event-driven middleware, and event processing languages “Complex Event Processing” (CEP) is a key element of ESP that provides language elements that allows applications to express the complex patterns among events it's looking for. CEP provides constructs that include event correlation, event abstraction, event hierarchies, and the ability to express relationships between events such as causality, membership, and timing.


Two approach of CEP/ESP – SQL Based Approach I

Some people coming from RDBMS development have extended SQL to provide CEP/ESP. •  The SQL processing in traditional RDBMS is “data is static and query is dynamic”.

•  The SQL processing in CEP/ESP is “data is dynamic and query is static”.

•  Because the event data may be overflow, it is necessary to introduce “time window” to SQL


Two approach of CEP/ESP – SQL Based Approach II

SELECT I1.SourceIP As SourceIP, I1.AttackKind As AttackKind, V1.Virus As Virus FROM InIDSAlerts As I1 KEEP 30 SECONDS, InVirusAlerts As V1 KEEP 30 SECONDS WHERE I1.SourceIP=V1.SourceIP

Join

Projection

. . .

. . .

Scan Scan

. . .

I1 KEEP 30 SEC I2 KEEP 30 SEC

I1 I2


Two approach of CEP/ESP – Rule Based Approach I

Some people come from integration development have extend rule engine to provide CEP/ESP. Sybase BAM chooses this approach. The key of this approach is to add complex state management and corresponding operator to the traditional rule engine that can support complex event pattern and event correlation.


Two approach of CEP/ESP – Rule Based Approach II

Rules

States

Event Actions


Background – RT BI

“Real time business intelligence” (RT BI) is the process of delivering information about business operations without any latency. While traditional business intelligence presents historical information to users for analysis, real time business intelligence compares current business events with historical patterns to detect problems or opportunities automatically.


Two approach of RT BI

•  Event driven, Real time Business Intelligence Real time Business Intelligence systems are event driven, and use ESP/CEP techniques to enable events to be analyzed without being first transformed and stored in a date warehouse. This approach is better for BAM. •  Real time Data warehouse An alternative approach to event driven architectures is to increase the refresh cycle of an existing data warehouse to update the data more frequently. These real time data warehouse systems can achieve near real time update of data, where the data latency typically is in the rage from minutes to hours out of date. This approach is better for ETL.


Analytic Model - Overview

Fields: Abstract states definition. Key, Unbound, Bound, Aggregation

Rules: Intelligence If condition Then action

Actions: Behavior Update, Aggregation, Alert, Timer, SQL, Java Script, Purge

Timers: Scheduler If timer arrive Then action

Binder: Concrete states storage BAMDB, UserDB, RefAM


Analytic Model – Processing

Fields Key

Bound Bound Bound

Aggregate

Unbound

Rules Actions •  Update

•  Aggregate

•  SQL

•  Alert

•  Java Script

•  Timer control

• Purge

if…

if…

1.  Keys, (some) other field passed into Analytic Model

2.  Historical values found based on keys

3.  Rules applied to data

4.  Actions performed, update data

5.  Repeat 3, 4 as needed

6.  New values stored


Analytic Model – in SOA

Input Fields ----- ----- ----- ----- Output Fields ----- -----

Monitor Service

Fields • Key • Unbound • Bound • Aggregate

Rules / Actions • Update • Aggregate • Send Alert • SQL • Java Script • Timer Control • Purge

Analytic Models / Analytic Objects

Fields Rules / Actions

Fields Rules / Actions


Analytic Model - Functionality

•  Monitor services interact with multiple Analytic Models, setting key fields to define specific object instance. •  Within Analytic Object, multiple rule calls trigger actions that further update object and perform other activities. •  Any field set in one Analytic Object is then available to subsequent objects, as determined by the Monitor Service.

•  If there is implicate or explicate key fields setting between different Analytic Objects, record the cross correlation of Analytic Objects. •  Service output fields may be return result of any field from any Analytic Object.


Architecture - Overview

Monitor

Service Editor Monitor Service WSDL

Monitor Command and

Control

Monitor

Analytic Model Editor

Dashboard Business Process

External Client

SOAP, JMS, etc

Monitor Service

BAM-Defined

Database Binding

User Defined Database Binding

SCS Container Analytic Object Access Library

Rules

Timed Event

Daemon


Architecture - Components

•  BAM Engine §  Analytic Object Access Library §  BAM Rule Engine §  Timed Event Daemon §  Monitor Service WSIF Provider

•  BAM Tooling §  Analytic Model Editor §  Monitoring Service Editor

•  BAM Web GUI §  Monitoring Console §  Dashboard


Runtime Processing of BAM

Queue

SCS

JMS WSHF

Provider CSB Monitor Service WSIF

Provider

Optimus

Analytic Object Access Library DB

Timed Event Daemon


Main Features - Overview

•  Complex Event Processing Support •  Real Time Business Intelligence Support •  Comprehensive Alert Capability •  Intuitive Visualization for Monitoring and Analysis •  Metadata-Driven Design Tooling

•  Service Oriented Architecture Support •  High Volume


Main Features - Complex Event Processing Support I

•  Event-Condition-Action (ECA) model §  Event Triggering, Rule Evaluation, Execute Action

•  Event Transport/Triggering §  JMS, HTTP, Email, File, Timer

•  Event Parsing/Transformation §  XML, CWF, SOAP

•  Event Routing §  Body-based, Header-based, Endpoint-based


Main Features - Complex Event Processing Support II

•  Event States §  Stateless, Stateful, Historical

•  Event Correlation §  Correlate low-level events to high-level event §  Key correlation, Cross correlation, History correlation

•  Event Reprocess §  Take corrective action for closed loop integration

•  Complex Event Pattern Support §  Based on ECA model + Event States + Event Correlation.


Main Features - Real Time Business Intelligence Support I

•  Rule-based intelligence §  Light-weight BAM Rule Engine (BRE) §  Patent-pending Boolean Network Rule Engine (BNRE)

•  Analyzing real-time data in the context of historic information §  Reference contextual data from ASE, IQ, EII


Main Features - Real Time Business Intelligence Support II

•  Time windowed aggregation / computation §  User-defined computation expression §  Extensible Aggregator: Average, Rate, Standard Deviation §  Sliding Time Window / Fixed Time Window

•  Multi-dimensional analysis support §  Based on Event Correlation + Aggregation + Computation


Main Features – Comprehensive Alert Capability

•  Publish-subscribe model §  XML Messages Publish via JMS §  Customized Subscription

•  Multiple Delivery Target §  JMS, JMX, Email

•  Alert escalation §  Timer, On-demand

•  Alert lifecycle §  Active, Canceled, Completed, Escalated, Suppressed


Main Features - Intuitive Visualization for Monitoring and Analysis

•  Dashboard §  Visual objects of Key Performance Indicator (KPI) is changed

dynamically as events occur in real time

•  Monitoring §  Real time event is displayed in tabular forms §  Drill-down from high-level event to low-level events

•  Alerting §  View and resolve alerts


Main Features - Metadata-Driven Design Tooling

•  Based on Eclipse and EMF (Eclipse Modeling Framework) •  Fully integrated and conformed to Sybase WorkSpace


Main Features – SOA Support

•  BAM is exposed as “Monitoring Service” in Sybase Service Container


Main Features - High Volume

•  High Performance §  BAM engine can process about 2000 messages/sec on a 2

CPU machine

•  Linear Scalability §  BAM engine is linear scalability §  Single BAM DB is linear scalability with CPU number § Multiple BAM DB are linear scalability with machine number


Reference

Business Activity Monitoring http://en.wikipedia.org/wiki/Business_activity_monitoring Complex Event Processing http://en.wikipedia.org/wiki/Complex_event_processing Event Stream Processing http://en.wikipedia.org/wiki/Event_Stream_Processing Real-time Business Intelligence http://en.wikipedia.org/wiki/Real_time_business_intelligence BI 2.0: The Next Generation http://www.dmreview.com/article_sub.cfm?articleId=1066763 BAM: Event-Driven Business Intelligence for the Real-Time Enterprise http://www.dmreview.com/article_sub.cfm?articleId=8177 Data Integration—the Foundation of a Robust Enterprise Architecture http://www.informatica.com/company/featured_articles/data_integration_foundation_082004.htm

sybase bam overview

Software