event processing a perspective from oracle dieter gawlick, shailendra mishra oracle corporation...

Event Processing

A Perspective From Oracle

Dieter Gawlick, Shailendra Mishra

Oracle Corporation

March, 14 2006

2March, 14 2006 Event Processing Symposium

Agenda

• Elements• Support for event processing in Oracle’s products• Some unique contributions by Oracle• Which application are best served?• Additional important applications• Major trends• Major challenges• Questions to the research community


Elements

Publication(Publishers)

Dissemination(Disseminators)

Consumption(Consumers)

publishesmessages

deliversmessages

Applications Files/DBs FeedsSensor Data Propagations


Elements




publishesmessages

deliversmessages

MessagesReports about

EventsEvents

StateChanges

maygenerate

maygenerate

Event generation: * Procedural - no external specifications * Declaratives - In reaction to external specifications


Elements




publishesmessages

deliversmessages

The Core Element

Stream

Stream

Stream

Stream

Stream

Single or multiple stepsLocal or distributed

Message Management

Select Process

ComposeRetain

Deliver/propagate


Elements




publishesmessages

deliversmessages

Applications Files/DBs FeedsActuators Propagations


Some Notes

• Publishers create solicited and unsolicited (spam) messages– Solicited messages are based on subscriptions*– Subscriptions may be against messages, events, and state

transitions (Heisenberg events)– Consumers can also act as publishers – creates value chains

• Dissemination moves messages from publishers to consumers– Dissemination may be a multi-step, distributed process– Selection and composition can access single and multiple

streams– There is rich support for aggregation (e.g., running average)

and user functions• Everything is driven by meta-data

– Meta-data describe publishers, consumers, dissemination and data structures (for states/events/messages)

• CEP/Streams processing is focused on management of event data and as such may be considered as a subset of dissemination

* (Continuous) Queries can act as subscriptions


Support for Event Processing in Oracle’s Products

• Applications features– Business Events – a large number of application triggers

• Application Server features– Sync/Async Messaging– Event Capture– Event Correlation– Metrics & KPIs– Real time dashboards– Alerts

• Database features– Triggers– Streams/AQ– Rules support– Asynchronous client notification – Data Change Notification


Some Unique Contributions by Oracle

• JMS– Leveraging and extending Oracle stack for messaging resulting

in improved functionality and operational characteristics– Support for Abstract data-types over and above standard JMS

types

– Full support for transactions, recovery and high availability • BPEL/BAM/RFID/Alerts

– Uniform declarative framework to do event processing Every node is data stream processor that speaks SQL-ese Hierarchical, stream based views as an organizing principle Allows system to adapt over time Optimization across multiple applications Simplified programming constructs


Some Unique Contributions by Oracle

• Streams/AQ – Extension of database to provide a messaging solution with

improved functionality and operational characteristics e.g., auditing and tracking, security, reliability and availability

– Declarative specification of publication, dissemination and consumption

– Recoverability with minimal journaling• Rules technology – Rules technology evolved as key data base

technology– Support for SQL, XML, Text and spatial data – Support of a very large number (>10**6) of rules with moderate

rate of change, management of rule sets, auditing of rules and data

– Very low cost of evaluation for data materialized in data bases– Expression as data, provides a new way to manage and process

subscriptions– Much of the rules technology is shared with query processing – a

subscription is a query on future data


Which Applications are Best Served?

• Messaging for mission critical applications – e.g., exchanges, fulfillment

– Extended functionality: Extensive type support, auditing, dissemination and more Reduced development effort – ease of use

– Improved operational characteristics: High performance, scalability, reliability, security or more

• Replication– Declarative publication, dissemination and consumption– High reliability with minimal journaling

• Event notification– Declarative, automatic event creation in response to database

changes - subscribe to publish– Sophisticated dissemination


Which Applications are Best Served?

• Complex event processing – e.g., management and processing of sensor data

– Events from many source types– Declarative event composition and non events– Auditing and tracking (data and rules)– Management of rule sets with user specified

policies– Mutual filtering– Metrics and KPI– Alerts– Real Time dashboards


Which Applications are Next?

• Security Analytics– Intrusion Detection– Security log correlation for damage assessment

• XML Streams– Correlating complex XML streams– Detecting patterns in XML streams without staging.

• RSS– Declarative creation and/or consumption of RSS Streams – data bases

as publishers/consumers of choice– (Fine Grin) Security, mutual filtering– Push support

• Information Driven Architecture – e.g., SensorNet, VIRT– Event specification and optionally dissemination and consumption are

part of the data specification – Event specification and processing reflect the rules and interest of

(internal and external) communities


Major Trends*

• Moore’s law implies a new class of computers about every 10 years

– The next step enables the unambiguous use of sensors (and actuators)

• We are on the verge of a vast of improvement in the‘Spatial and temporal fidelity at which We instrument and analyze the physical world’Resulting in an avalanche of data

• We are moving towards proactive and probabilistic computing

– Computers will provide us with Valuable Information at the Right Time based on Conditions Of Interest

* This slide is based on ideas documented by Hans Mulder (Intel) and Rick Hayes-Roth (NPS)


Major Challenges

• Agree on and document a small number of use cases (3-5) – Capture functional and operational requirements– Capture desired (user) development support– Capture cost constraints for operations (IT and end

user)• Create a Terminology• Identify and characterize missing standards; e.g.,

– Continuous queries (for streams and states)– Dissemination – a generic model– Specification of publisher/consumer communities

• Specify benchmark


Questions to the Research Community

• There is a need for an extensible spatial and temporal view of the world

– What are major organizing principals?• There is a need for data models and CQ languages for streams and

state transitions– How can events (and messages) be derived from state

transitions? – How can dissemination and community models be integrated

with CQ’s?– How does one find relevant continuous queries?

• Rules technology is a core element of event processing– What is the state of the art for which patterns?– Which fundamental challenges exist?

• Dissemination is another core element of event processing– Is there a generic model and what is it?

• Why do researchers pay so little attention to evolving technologies in existing products?


AQ&Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S

event processing a perspective from oracle dieter gawlick, shailendra mishra oracle corporation...

Documents

management of event

data structures

data basesexpression

spatial data support

data stream processor

abstract datatypes

event processingevery

auditing of rules