dr. opher etzion, stsm lead architect, event processing technologies ibm software group
Post on 25-Jan-2016
29 Views
Preview:
DESCRIPTION
TRANSCRIPT
Software Group – Event Processing Technology and Architecture
| 2006 © 2006 IBM Corporation
Event Processing – Vision and RealityBPM-BAM-CEP ConferenceRegensburg, June 19, 2006 Keynote Address
Dr. Opher Etzion, STSMLead Architect, Event Processing TechnologiesIBM Software Group
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation2
Outline
Towards an event processing Community
Event Processing – market view
Terminology and architecture
Event Processing - segments
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation3
Event Processing – Market View
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation4
What drives this area ? From the bird eye’s view:
Nothing is really new We process events for many years (e.g. exception handling in OS).
However, recently: Significant amount of events – types, sources, instances Variety of application need to process events Some traditionally stand-alone applications need to be integrated with regular
information systems (simulation, real-time) Functionality requires sophistication – e.g. temporal capabilities, spatio-temporal
capabilities.These all contributed to COTS tools
It is not cost-effective to develop this functionality for a single application It is recognized as middleware level capability, thus customer preference for COTS. Drive for standards is next step
.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation5
Event Processing - Analyst view
Event-Based Application PlatformsDefinition: Application platforms that offer a programming model for event-driven computing. Business application vendors will need complex event processing capabilities as part of their evolving service-oriented architectures (SOAs) to deliver adaptability and flexibility of their platforms.Justification for Hype Cycle Position/Adoption Speed: Some leading platform vendors have proposed, but none has strategically committed to, an event model as primary for business application design. JavaBeans is the early, and nearly failed, attempt at this model. The model has an intrinsic power that makes its gradual adoption likely, but not in the near term.Business Impact Areas: Event-based application platforms see the software environment as a relationship of events, as opposed to a relationship of programs in prevailing models. Business application vendors will design and develop event-driven principles into their traditional SOAs. This evolution will make unique application styles possible.
Benefit Rating: High.Market Penetration: Less than 1 percent of target audience.Maturity: Embryonic.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation6
More advanced domain: Algorithmic Trading
Complex Event Processing for Trade FacilitationDefinition: A component of business activity monitoring, complex event processing (CEP) entails the receipt or extraction of events as they are occurring in real- or near-to-real time, processing (analysis, filtering, aggregation and cleansing) to determine if the event requires action, and producing an output that can be used to trigger an appropriate action.Justification for Hype Cycle Position/Adoption Speed: Current implementations typically involve a limited number of feeds and relatively simple filtering or analysis. CEP is one component of the enterprise nervous system concept, and to straight-through processing and real-time operations.Business Impact Areas: Applications include algorithmic trading, quality of service monitoring, best execution assessment.Benefit Rating: High.Market Penetration: Five percent to 20 percent of target audience.Maturity: Adolescent.Example Vendors: Atsmai, iSphere, Progress Software (acquired Apama), Streambase.Recommended Reading:• “Clarifying the Terms 'Event-Driven' and 'Service-Oriented Architecture'”• “Innovative Vendors in Business Event Management”
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation7
Circles of players
Event Processing as part of full-
fledged middleware
General Event Processing
products that can work with multiple
platforms
Event Processing capabilities
embedded inside solution /
applications (sold as products also)
Tibco
Oracle
Progress
StreamBase
AptSoft
RuleCore
Coral8
Aleri Labs
RedRabbit
Actimize
Leanway
G4
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation8
IBM complete Roles
Event-Driven Applications
Industry Solutions
Application Development
Tools
IBM Middleware
Development tools provider
Middleware Producer
Solution Provider
Services Provider
MiddlewareBased
EmbeddedBase (ISV) Integration
platformIBM software
General
Practice
IBM SW services
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation9
Event Processing – segments
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation10
Major Segments
EPBL - Event processing as part of business logic
EPPD - Event processing as part of problem determination
EPID - Event processing as part of information distribution
EPPS - Event processing as part of predictive systems
EPBO - Event processing as part of business observation
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation11
EPBL - Event Processing as part of business logic
Types: Application integration through event processing
Event-driven business policies/rules Application examples:
Adaptive workflows (Telco and others)
Trade management (FSS)
Automated shipping and receiving (Retail)
Just-in-time car rental allocation (Travel and Transportation)
Time-critical targeting (military)
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation12
EPBL – characteristics
Entire spectrum of mediators: transformation, routing, validation, enrichment.
Pattern detection: varies, requires rich language. Non functional requirements:
All mission critical systems requirements (integrity, transaction support, recoverability, high availability, failover, clustering, security, audit)
May have a need for high throughput
predictability is needed in some applications
Time-out management is important.
Require end-user ability to maintain rules/policies.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation13
EPBO - Event processing as part of business observation
Business activity monitoring KPI – performance management
Exception management in BPM. Sense and respond
Time-constrained Decision processes and analytics may be needed
A loop that contains actuators and monitoring the actuators results. Applications:
RT compliance/ Fraud detection/AML (FSS, Telco)
Manufacturing S&R (industrial)
Adaptive policy setting (operational resilience)
Business health assessment (cross-industry)
Promotion evaluation (Retail)
SLA management of a process (cross industries)
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation14
EPBO – charachterisitcs
Languages: Aggregation-land: trends, time series processing, statistical
measures etc…
Data-base connection – retrospective processing
Send and response may need:
o Predictabilityo Embedded analytics/decision modules
Business performance may need:
o Metrics and KPI managemento Derived data (“active data warehousing”)o Causality model.
Require business analysts interfaces.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation15
EPPD – Characteristics
Main functionality: Eliminate redundant events – filtering, remove duplication
Event correlation to determine symptoms
Problem/symptoms relations modelingBoth real-time and post-mortem (log files) May need high throughput Time interval typically short Assumption that events can be lost:
No fault tolerance is required.
Partial pattern satisfaction may be useful. Pattern language is focused on – sequence within short time-
frames, thresholds, filtering condition. Typical users: system administrators, developers, product support
team.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation16
EPPD – Characteristics
Main functionality: Eliminate redundant events – filtering, remove duplication
Event correlation to determine symptoms
Problem/symptoms relations modelingBoth real-time and post-mortem (log files) May need high throughput Time interval typically short Assumption that events can be lost:
No fault tolerance is required.
Partial pattern satisfaction may be useful. Pattern language is focused on – sequence within short time-
frames, thresholds, filtering condition. Typical users: system administrators, developers, product support
team.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation17
EPID - Event processing as part of information distribution
Enable individuals/applications to subscribe to personalized derived events rather than raw events
Customer alert systems (banking)
Content-based geo-spatial subcriptions
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation18
EPID – charachterisitcs
Personalized and specialized form of the publish/subscribe paradigm
Typically loosely-coupled from the operational system, Subscription can be by individuals, “communities of
interests” or applications – routing decisions and management of subscriptions is important
Subscription can be to derived events and not only to raw events, which requires pattern detection – trends, content filtering combined with basic relations (conjunction, disjunction, negation, sequence etc..).
Requires end-user interfaces for subscriptions in some cases, development interfaces in cases of applications’ subscriptions.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation19
EPPS - Event processing as part of predictive systems
Impact analysis: prediction of consequences of already observed events towards future events, or entity states.
May be in response to future change proposals (“change management”)
May be combined with “sense and respond” to mitigate or eliminate this impact.
Applications: Change management in dependent cases (automobile, industrial)
Impact of IT problems on business health (cross-industry)
o Includes ITSM Availability, Incident, Problem Management On-line/real-time detection of fraud, money laundering.
On-line risk management.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation20
EPPS – characteristics
Requires predictive tools embedded in event processing Requires dependency model of events and entities Requires mining tools to find causality relations requires rich causality model with possible uncertainty
handling (e.g. probabilistic engine) Rich pattern language – different type of condition
causality.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation21
Event Processing - Platform Independent Architecture and Model
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation22
Event Driven Applications
Event Processing
Event-drivenActions
Event Creation
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation23
Event Producer
Event
EventTopic
Emitter
Event Sensor
EventTopic
Event
processor
Pattern Detector
Complex event
Derived Event
Event Processing Mediator Event
Actor
Event Processing Mediator
Event Consumer
EventCloud
EventConsumer
ArchitectureGlossary
Event:
(1). Something of interest that occurred in reality (state changed, fact becomes true)
(2). The computerized message to report the event
Event Producer:
An entity (e.g. software artifact) that creates events and reports them (e.g. workflow)
Event Sensor: An entity that senses events and reports them
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation24
Event Creation
Produced or sensed Push, periodic pull, on-demand pull. challenge: Increase the “event scope”
Events extracted from video streams
Events from RSS feeds
Events extracted from textual information
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation25
Event Producer
Event
EventTopic
Emitter
Event Sensor
EventTopic
Event
processor
Pattern Detector
Complex event
Derived Event
Event Processing Mediator Event
Actor
Event Processing Mediator
Event Consumer
EventCloud
EventConsumer
Architecture
Glossary
Event Topic:
A data-type channel for transporting events of a certain type
Event Consumer:
An entity which is reported about event creation
Event Actor:
An entity which produces automatic actions in response to an event
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation26
Event-driven Action
Notify/Store – for consumer Trigger, Orchestrate – for actor (BPM event processing)
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation27
Event Producer
Event
EventTopic
Emitter
Event Sensor
EventTopic
Event
processor
Pattern Detector
Complex event
Derived Event
Event Processing Mediator Event
Actor
Event Processing Mediator
Event Consumer
EventCloud
EventConsumer
Architecture
Glossary
Context: A set of criteria to partition the space of events according to temporal (e.g. time window), spatial (e.g. space window) and partitioning entity (e.g. platinum customers)
Event Stream: A collection of events that arrive to a single consumer over a single event topic within a single context
Event cloud: A collection of all event streams that a single consumer receives
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation28
Event Producer
Event
EventTopic
Emitter
Event Sensor
EventTopic
Event
processor
Pattern Detector
Complex event
Derived Event
Event Processing Mediator Event
Actor
Event Processing Mediator
Event Consumer
EventCloud
EventConsumer
Architecture
Glossary
Event Processing Mediator:
A software artifact that gets an “event cloud” as an input, and produces a collection of events as output.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation29
Event Producer
Event
EventTopic
Emitter
Event Sensor
EventTopic
Event
processor
Pattern Detector
Complex event
Derived Event
Event Processing Mediator Event
Actor
Event Processing Mediator
Event Consumer
EventCloud
EventConsumer
Architecture
Glossary
Pattern Detector: A software artifact that obtains an event cloud as an input, and creates a “complex event” –Example: the stock quote is monotonically decreasing within 2 hours. Return the collection of stock quotes.
Complex event: An event that contains reference to all events that participate in the pattern
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation30
Event Producer
Event
EventTopic
Emitter
Event Sensor
EventTopic
Event
processor
Pattern Detector
Complex event
Derived Event
Event Processing Mediator Event
Actor
Event Processing Mediator
Event Consumer
EventCloud
EventConsumer
Architecture
Glossary
Derived Event: An event that is calculated as a function of other events.
Event Processor: A software artifact that obtains a composite event and creates collection of derived events using some function on the input events, such as: Enrichment, Transformation, Aggregation, Split.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation31
Event Processing Mediators
Event ProcessingMediators
EventValidation
EventRouting
EventTransformation
Event Enrichment
Event Pattern Detectors
Event Emitters
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation32
Event Producer
Event
EventTopic
Emitter
Event Sensor
EventTopic
Event
processor
Pattern Detector
Complex event
Derived Event
Event Processing Mediator Event
Consumer
Event Processing Mediator
Event Consumer
EventCloud
EventConsumer
Architecture
Glossary
Emitter:
A software artifact that makes routing decisions for the derived events to consumers and actors: subscribers, itinerary based or intelligent routing
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation33
Several assertions
There are several different approaches to event processing. Rule-based approach Extended SQL-based approach (stream and active database) Script oriented approach Other approaches
Our aim is to have a model-based approach that is in a “platform independent” level, have appropriate tooling, and consistent with standard approaches.
Probably there is no “one fits all” solution in the run-time level, but from the model and tooling level everything should be seamless
We should strive for open architecture with: Separation of concerns:
o each function can be implemented by various run-time artifacts, each of them satisfies a subset of the appropriate functions, and given set of non-functional requirements.
o We should own the “integration platform” for event processing: the biggest market for event processing will be in “application integration through event processing”.
o We should have our own run-time artifacts, but allow easy integration with other run-time artifacts, mainly specialized ones.
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation34
EDA as part of SOA
Enterprise Service Bus
Portal Service
WorkflowBusiness Activity
Business-to-Business Interactions
Enterprise Information System Adapter
Script, POJO, Stateless Session Bean
DistinguishedServices
DistinguishedServices
Information MgmtXML Database
Information MgmtXML Database
This is the famous SOA picture
Each service can emit events – all the time, periodically, or by request
The events are being processed Using event processing mediators (e.g. enrich)
The processed events notify or orchestrateThese building
blocks are not tightly coupled to the ESB
Event Processing can also be retrospective
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation35
Event Processing Network
EPN consists of a set of nodes (event processing mediations AKA as event processing agents, service, and event processing endpoints)
Following Separation of concerns principle
Each processing node has an “event pattern detector” part and “event emitter” part , which are the same for all mediator types.
This is a design pattern, but can also be run-time artifact, providing distributed set of lightweight run-time artifacts.
Event Pattern Detector
Event Processor
Event Processing Mediator
Event Emitter
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation36
Retrospective processing of events
We need to do all event processing in retrospect A naïve way is to select all the raw events from the database
and run them through the “memory” event processing.
This may not be a cost-effective way to do it when there is a large quantity of raw events…
Solutions: Built-in database capabilities that can optimize to this type of
processing
Converting this functionality to plain SQL (probably the short term game)
Time
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation37
Towards an Event Processing Community
Conclusion
Software Group – Event Processing Architecture and Technologies
An Evolving Thinking About Event Processing © 2006 IBM Corporation38
Activities in progress
Establishing international community: The kick-off meeting of the community has been the first event
processing symposium in March 2006
o The steering committee consists of representatives of IBM, Tibco, Oracle, Progress, SAP, Streambase, Gartner and some academic people.
o Follow-up symposium in November 2006 Several community work-groups:
o Terminologyo Reference Architectureo Interoperability
ACM SIG in construction Starting work with OMG on standards More workshops and conferences
Vendors announcements – SOA 2.0
top related