Event Driven Architectures with Apache Synapse

Paul FremantleVP, Apache Synapse

Member, ASFCTO and Co-Founder, WSO2

Why am I talking about this?• CTO and Co-Founder, WSO2

– Previously STSM in IBM WebSphere• VP, Apache Synapse• Apache Member• PMC Member:

– Incubator, WebServices,HTTPComponents, Qpid

Plan of Attack• Brief introduction to Apache Synapse• Understanding SOA approaches• Event Driven Architecture

– Why would you use it?– Options and approaches

• How Events fit into Synapse• Case Study• Futures

Why use an Event Driven Architecture?

Empowerment

Understanding the point of SOA• SOA is fundamentally about empowered

distributed systems• Giving each

department/organization/website/etc control of their own systems – Amazon is a great example of this– Went from a traditional three tier

data/app/web architecture – Enabled small teams to be effective

Two different kinds of SOA

Consumers

ESB

Services Services Services

Consumers Consumers

System

ESB

System

System

System

System

System

Wiring• In a process-led or mashup-led SOA, the

wiring is done using ESB configuration or BPEL

Scenario – Financial Security blocking

Database

legacyflat file

NEW YORK

Existing System

SynapsePoll

Record->XMLXML->XML

Send

LONDON

SynapseSplit/Iterate

DBLookup/FilterTransform to MQ

Send

Existing System

XML/JMS

Overall ESB flow

FileSystemListener

NY Sequence

Log

Out-Only

Flatpack

E4X

Send to London

LondonEndpoint

LondonSequence

Iterate

DBReport – log whole msg

Filter/Drop NOAC

Send

PushMePullYou

PushMePullYou scenario

Logical model• Every Service Provider and Consumer

understands the “canonical XML”– Of course in real life there will be adapters – These may be running in an ESB

• But ideally their OWN ESB (or part of a distributed ESB)

• The aim is that each department/group can “own” their own gunk– Not reliant on central systems to adapt

Event approach• Eventing takes this one step further:

– Its up to you to publish to the right place– Its up to you to subscribe to the right events

• You own the wiring too– Allows for situational integration

Loose coupled and Self Organizing

We know this!

Event based models

Actuators and Sensors• An actuator emits an event• A sensor accepts events

The Synapse model of Events• Loosely based on WS-Eventing

– But can be augmented with other models– Also more powerful

• Easily add mediation into the flow

• Main concept:– Event Source

• An Actuator

Advanced Web Services Core Axis© WSO2 Inc. 2006

20

WS-Eventing

Event Source

Subscribe with EPR of subscriber

Respond with EPR of SubsMgr

Subscriber

Event Notifications (any message)

SubsMgr

getStatus

renew

unsubscribe

Synapse configuration example<eventSource name="SampleEventSource">

<subscriptionManager class="o.a.s.eventing.managers.DefaultInMemorySubscriptionManager"> <property name="topicHeaderName" value="Topic"/> <property name="topicHeaderNS" value="http://apache.org/aip"/> </subscriptionManager>

</eventSource> <sequence name="PublicEventSource" >

<log level="full"/> <property name=“OUT_ONLY” value=“true” action=“SET”/><eventPublisher eventSourceName="SampleEventSource"/>

</sequence> <proxy name="EventingProxy">

<target inSequence="PublicEventSource" /> </proxy>

Eventing in Synapse

Synapse

Proxy

EventSource

EventPublisher

SubsMgr

Subscriber

Publisher Subscriber

Subscriber

subscribe

MediationSequence

Different subscription models• WS-Eventing message• Static subscription in Synapse XML• Store-dependent

– E.g. Registry model

Completely transport AND WS-Eventing agnostic

• You can have any kind of message republished to any kind of endpoint– SMTP -> Jabber– SOAP -> JMS

• You don’t HAVE to use WS-Eventing to subscribe

• You can bridge into JMS Topics too– Or bridge from JMS Topics

Registry-based subscription model• A header carries the “Topic”

– e.g. /config/hardware/server/windows/xp• Subscribers can subscribe to a specific topic, or all

sub-events • The topic space is represented as a tree in the registry

– Subscriptions are simply URLs stored as entries at a point in the tree

/config/ /software/ /hardware/ /server/ /linux/ /windows /xp/ URL1 / URL2 (etc)`

Case Study

Enterprise IT Management• Problem statement:

– Customers have multiple installed management systems• Network Management• User Management• Systems Management

– All from the same vendor!– These are not just “stock” systems – each has been

customized for each installation– Customers have to keep these systems in sync

• By data entry– Any solution needs to be flexible, extensible, modifiable

A difficult problem• Synchronizing multiple different systems• But:

– Systems have different underlying formats– Some of the systems may be more accurate

than others– Need to be able to scale to different

numbers of systems– Must be extensible / reprogrammable

Feedback!

Feedback problems

Black Box

change

update

Feedback loops

BlackBox

system

BlackBox

system

Adapter

Adapter

Event Broker

http://pzf.fremantle.org/2008/09/interesting-problem-in-event-driven.html

EDA + MDR

What is Complex Event Processing?

• Simple Event Processing– Acting on single events– e.g. a <filter> in the ESB– Is this a gold or platinum customer?

• Event Stream Processing– Looking across multiple events– Finding patterns – e.g. the CPU utilization has been more than

90% for the last 10 minutes• Complex Event Processing

– Looking across multiple event streams– e.g There has been a significant increase in overall trading

activity AND the average price of commodities has fallen 2% in the last 4 hours

Complex Event Processing in Open Source

• Esper– http://esper.codehaus.org – A Java library that can be integrated into multiple

systems– A GPLv2 project

• (with a commercial edition available)– Supports multiple query models

• Based on a SQL-like language• Grouping, aggregation, sorting, filtering and merging of

event streams • Windows based on time, length, sorted, and others

– Events can be XML, Map, Object

Complex Event Processing examples

• Event processing languages are typically based on SQL – With extensions!

select * from Ticker.win:length_batch(10)select AVG(CPU) FROM GANGLIA.win:time(30 sec)insert into WarningStream select trainStation,

avg(waitTime) as avgWait from MyWindow group by trainStation having avg(waitTime) > 60

Esper + Synapse• Separate project – SciFlex

– http://code.google.com/p/sci-flex/• Allows you to “inject” events from Synapse to

be processed by Esper• Interesting events come back to you

Summary• EDA is an interesting approach• If you design correctly, you can build a really distributed, loosely coupled system

Questions?