sa2011 event driven systems - artechra.com...•a simple idea, using a standard set of concepts •...
Post on 14-Sep-2020
1 Views
Preview:
TRANSCRIPT
Eoin Woods www.eoinwoods.info
Starting out with Event Driven Systems
Software Architect, London
October 2011
• Software architect & dev manager at UBS Investment Bank • responsible for equity swaps systems within Equity Derivatives
• Software architect for ~10 years
• Author of “Software Systems Architecture” book with Nick Rozanski
• 2nd Edition published in October 2011
• IASA and BCS Fellow, MIET, CEng
20111020
About Me
2 © Eoin Woods 2011
• Difficulties of Distributed Systems
• Event Driven Architecture
• Building Event Driven Systems
• Case Study
• Extending the Event Driven Approach
• Conclusion
20111020
Agenda
3 © Eoin Woods 2011
20111020 4 © Eoin Woods 2011
1. The network is not reliable
2. Latency is not zero
3. Bandwidth is not infinite
4. The network is not secure
5. Topology changes
6. There are many administrators
7. Transport cost is not zero
8. The network is not homogeneous
Peter Deutsche and others, from 1994
20111020
Difficulties of Distributed Systems
5 © Eoin Woods 2011
• The point of RPCs and services is to try to hide the difficulties of distributed systems
• This doesn’t work and never has • fragile connections
• difficult to evolve
• unexpected system qualities
• Better to accept the difficulties and pick an architectural style that can help
• event driven architecture is one of those styles
20111020
RPCs and Services
6 © Eoin Woods 2011
20111020 7 © Eoin Woods 2011
• According to the Event Processing Technical Society … [EDA is] An architectural style in which some of the components are event driven and communicate by means of events. [Event Driven means] The behaviour of a device, software module or other entity whose execution is in response to the arrival of events from external or internal sources [An Event is] An object that represents encodes or records an event, generally for the purpose of computer processing
Event Processing Glossary – v1.1 The Event Processing Technical Society
20111020
Event Driven Architecture (EDA)
8 © Eoin Woods 2011
• Events are a record of activity in a system or outside
• Events are characterised by their • form – e.g. a tuple, a header/body structure, an XML block … • significance – what activity they signify • relativity – how they relate to other events (by time, causality or aggregation)
David Luckham (Power of Events)
(Events are usually state changes in entities of interest to the system or record the passing of time)
20111020
Finding Events
9 © Eoin Woods 2011
• A simple idea, using a standard set of concepts • Event producers, consumers and event processing agents
• Events • raw events and derived events • events contain a set of header fields and a body (structured or
opaque)
• Event channels • Shared state
• These concepts form a standard reference model • an Event Processing Network (EPN)
• An EPN exists to detect, decide and respond
20111020
EDA - Overview
10 © Eoin Woods 2011
20111020
Real World Event Processing
11 © Eoin Woods 2011
Taxi please!
I’m coming on duty
I’m going home
You’re booked
Taxi Dispatcher (Event Processor)
• Broadcast Communication • event producers broadcast events rather than send them
• Timeliness • events are dispatched when ready, not in batches
• Asynchrony • event producers do not wait for events to be processed
• Fine Grained • events are published individually
• Ontology • clear classification hierarchy and semantics for events
• Complex Event Processing • events can be related (aggregation, causality, …)
(Gregor Holpe) 20111020
Characteristics of Event Driven Architecture
12 © Eoin Woods 2011
20111020
EDA – Types of Event Processing Agents
13 © Eoin Woods 2011
Event Processing
Agent
Filter
Transform
Translate
Enrich
Project
Aggregate
Split
Compose
Pattern Detect
20111020
EDA – Example Network
14 © Eoin Woods 2011
Order Processing
System
Shipping System
Billing System
Fulfillment System
Order Emitter
Order Consumer
Fulfillment Emitter
Shipping Combiner
Customer Details
Billing Consumer
Shipping Consumer
Shipping Enricher
“order created”
“order packed”
“order addressed”
Billing Status Emitter
“ready to ship”
“order billed”
20111020
EDA Example State Machine for Orders
15 © Eoin Woods 2011
• Simple event processing • events (typically) created for state changes or external occurrences, usually drive state machines
• often all that is needed for enterprise applications
• Event Stream Processing (ESP) • filtering and processing of streams of events (e.g. prices)
• Complex Event Processing (CEP) • complex events are those derived from other events • CEP is the process of creating & processing complex events
20111020
Types of Event Processing
16 © Eoin Woods 2011
• Very strong decoupling (+) • when done well and the rules are obeyed
• Very good scalability (+) • inherently scalable and distributed architectural style
• Very flexible extension and evolution (+) • again, if you stick to the rules
20111020
EDA – Strengths and Weaknesses
17 © Eoin Woods 2011
Can be complicated to implement (-) • a lot of state, a lot of pieces
• unfamiliar, needs tight semantics
• Causality and traceability difficult to achieve (-) • makes tracking problems difficult
• needs to be built in from start
• Monitoring is crucial to provide visibility (-) • otherwise failure isn’t evident, things just “don’t happen”
20111020
EDA – Strengths and Weaknesses
18 © Eoin Woods 2011
• Inadvertent Coupling • events sometimes morph into requests … coupling emerges
• Mixing Activities and Events • activity records aren’t events (can’t drive state changes)
• Failure Recovery • recovering from node or system failure can be complicated • what is the state of the system? how about in flight events? • design recovery early and test it, use monotonic consumers, assume replay needed
• Operational Complexity • operational independence can affect backup, recovery, DR, security (operate a “system of systems”)
20111020
EDA – Common Problems (1)
19 © Eoin Woods 2011
• Latency • processing a piece of work can take longer than with a monolithic system
• Event Semantics • just defining fields is fatal … what does the event signify?
• Accessing Reference Data • can become a new bottleneck or consistency problem
20111020
EDA – Common Problems (2)
20 © Eoin Woods 2011
20111020 21 © Eoin Woods 2011
• Define a domain model • identify state-oriented, data-oriented and reference classes
• Design state models for state-oriented classes • ensure entry and exit states exist (create and destroy) • add error conditions to state models (nested states?)
• identify actions needed on transitions as well as events
• Define the set of event processors & connect them • interaction design needs careful analysis to avoid problems
• Add the non-event driven parts of the system • services, reference data, batch processing, …
20111020
Design Approach for Event Based Systems
22 © Eoin Woods 2011
20111020
Reference Design for an Event Driven System
23 © Eoin Woods 2011
Entity Store
Event Processors
Reference Data
State Store Application Services
Event Channel(s)
• Central Event Store • build everything, use database or cache as an event store
• more a blackboard style than “real” EDA
• Use Messaging • use messaging system as event transmission / routing / filtering mechanism, possibly still use db as event store
• Use an ESB • ESB services (transformation, routing, …) can support EDA • commercial (e.g. Sonic) or open source (Mule, Service Mix)
• Use CEP / Stream Processors • adding complex events • open source or commercial (e.g. Esper, Streambase)
20111020
Technology for Building Event Based Systems
24 © Eoin Woods 2011
Simple EDA from Scratch
20111020 25 © Eoin Woods 2011
Event Store Table(s)
Event Emitters
event records
Event Transformers
Event Consumers (Processors)
event records
event records
event records
Evaluating Our Simple Implementation
20111020 26 © Eoin Woods 2011
Strengths Weaknesses
Achieves loose coupling Poor scalability due to db contention
Simple, easy to understand & build
Lots of custom event handling code
Flexible processing model No visibility of event processing network structure
No special technology needed Implementation easily diverges from the EDA style
Simple EDA Using Messaging
20111020 27 © Eoin Woods 2011
Pub/Sub Messaging
message_topic_3
message_topic_1
message_topic_2
Event Emitters Event Consumers
Event Transformers
Message Replay
Idempotent Transactional Message Consumption
Message Model and Representation
State Machines
Routing
Event Processing Core – the Event Processor
20111020 28 © Eoin Woods 2011
• Event model and representation
• Event processing model • state-machine based processing (events trigger state transitions)
• activity-model based processing (events trigger activities) • often one can be translated to the other
• Event routing
• Idempotent, transactional message consumption
• Reliable, standardised message replay
• Publish/subscribe message transport • ideally with message header filtering
20111020
Concerns for Message Based EDA
29 © Eoin Woods 2011
Event Model and Representation
20111020 30 © Eoin Woods 2011
<event> <event-header> <event-id>…</event-id> <event-time>…</event-time> <event-type>….</event-type> … </event-header> <event-body/> </event>
<event> <event-header> … <event-type>com.x.orders.dispatched</event-type> … </event-header> <event-body> <order-id>123</order-id> <dispatch-time>20110921T120413</dispatch-time> … </event-body>> </event>
• Define a clear hierarchy for the events • and make sure they are events
• clearly define what they mean!
• Define a standard set of “header” (or metadata) fields which every event will carry
• id, timestamp, fully qualified type, type of business entity it relates to, id of business entity it relates to, event source
• Define one or more clear concrete encodings • e.g. an XML schema or JSON encoding
• Strongly type the event header and body fields
• Allow for extension but control its use • e.g. XML schema’s <xs:any>, JSON schema’s any
• Don’t rely on ordering in the encoding if possible
20111020
Event Model
31 © Eoin Woods 2011
• Events need to trigger business logic • Custom code is sometimes used
• Better via a standard model (visibility, reliability, reuse)
• State Machine and Process based models are both used
20111020
Event Processing Model
32 © Eoin Woods 2011
Events
Your Code Here
State Machines
20111020 33 © Eoin Woods 2011
• State machine options: • An FSM library • Create a simple interpreter (representation + execution)
• Custom event processing business logic (custom code in each event consumer)
• Open source examples: • SCXML, Tungsten, Jolt.NET, …
• Event routing can easily become its own project • keep it simple and avoid building complicated routing software
• Simple message-topic based approach usually sufficient • events typically relate to a business object type (e.g. “new order”) • use a message topic per business object type • include the message type and business object type and key in the
message header fields
• encode a simple naming convention in your event emitter and consumer library code
• use message selectors (filters) to subscribe to events of interest for particular event types
20111020
Event Routing
34 © Eoin Woods 2011
20111020
Event Routing Using Message Topics
35 © Eoin Woods 2011
Reliable Message Consumption
20111020 36 © Eoin Woods 2011
To receive without 2PC: • receive the message • store to db if not there • then commit message read
Implement carefully, test thoroughly, log everything - create a library and reuse it!
Message Replay
20111020 37 © Eoin Woods 2011
• Reliable message replay is key to system recovery
• Replay requires: • a log of sent messages • the ability to resend messages from the log
• “Dispatcher” is used for sending events
• “Replay Dispatcher” used in recovery situations to resend missing events
Implementing Simple Message Based EDA
20111020 38 © Eoin Woods 2011
Evaluating A Message-Based Implementation
20111020 39 © Eoin Woods 2011
Strengths Weaknesses
Loose coupling Debugging and monitoring can be difficult
Good scalability High volumes can require complicated messaging configuration
Uses straightforward, well understood technology
No standard event processing agents (e.g. splitting) for “free”
Flexible, easy to extend, faithful implementation of EDA
Can be difficult to visualise the event processing network
20111020
Using an Event Bus (ESB)
40 © Eoin Woods 2011
• Enterprise Service Bus products provide useful EDA features … • message transformation services
• routing services • service hosting • multiple transports (WS-*, RMI, CORBA, .NET Remoting, MQ, …) • management tools
• … at a price • literally if you buy commercial • development & deployment complexity wherever you acquire one
from
• Look carefully before committing …
20111020
Using an ESB
41 © Eoin Woods 2011
HTTP Listener
XSLT Translator
JMS Sender
JMS topic
JMS Receiver
JMS Receiver
Custom Code Call
Custom Code Call
Database Listener
Email Sender
App DB
MSMQ Sender
Workflow Engine
• Complex Event Processing is processing event groups and patterns as well as individual events
• detection of event patterns • recognising relationships between events • creating new inferred events as a result (the “complex” events)
• Many enterprise systems don’t need CEP • but CEP adds many new interesting possibilities
• CEP is usually achieved using a “CEP Engine” • no shortage of vendors who’d like to sell you one • open source (e.g. Esper) and commercial (e.g. Apama)
20111020
Using a CEP Product
42 © Eoin Woods 2011
20111020
Using a CEP Product
43 © Eoin Woods 2011
Event Sources
CEP Engine Systems
CEP Definitions
Events Actions
Notifications Derived (“Complex”) Events
20111020 44 © Eoin Woods 2011
Case Study Starting Point
20111020 45 © Eoin Woods 2011
Front Office
Systems
Middle Office System
Confirmations Processing
Settlement Processing
Payments System
Operations Users
Trade Store
• Capital Markets domain (Fixed Income Middle Office) • existing monolithic system had served well but become difficult to
scale and evolve • a new middle office environment for FI rates products needed • didn’t want a single large system for the replacement
• Future State Architecture and Build project • decision made to build 5 independent systems
• but all 5 must cooperate to implement the trade lifecycle
• very strong desire to avoid technical or organisational coupling • avoid shared release cycle, no shared database, no point to point links
• no special purpose EAI or EDA software used • but ended up using events for integration • and created a custom UML state chart evaluation library
20111020
Case Study Background
46 © Eoin Woods 2011
• Messaging identified as a key technology • desire for independence, lack of coupling, scalability
• Previous experience pointed to this not being enough • coupling and complexity can quickly occur with ad-hoc messaging
• Event based messaging identified as the solution • the systems all broadcast trade lifecycle events • message subscriptions route events to the right processor(s)
20111020
Case Study Architectural Decisions
47 © Eoin Woods 2011
• No request/response interactions
• event emitters do not know of the existence of event consumer(s)
• No shared entity database that everything reads and writes
• Systems use state charts to track trade lifecycle
• Very large events need to use hybrid approach • events are state refs into a shared event store
20111020
Case Study Architectural Implications
48 © Eoin Woods 2011
The Replacement
20111020 49 © Eoin Woods 2011
Front Office
Systems
event-based message bus
Confirmations System
Settlements System
Payments System
End User Workbench
Trade Store
• An EDA approach resulted in • Smaller systems, better defined responsibilities
• Well defined neutral (constantly evolving) event models • No direct dependencies between systems
• Avoided coupling by using the common event models
• No global shared entity stores
• Development team independence (relatively)
• Has allowed much easier change • Systems can have own development & release cycles
• The architecture has enabled overall agility • Overall result is more effective delivery and change
20111020
Case Study Results
50 © Eoin Woods 2011
• Don’t ask for something, announce something • use events don’t make requests
• Don’t send business entities, send lifecycle events • one of the major mental shifts people need to make
• Use “failsafe” processing to spot missing/extra events • essential for troubleshooting and monitoring
• Monitoring is key and needs to be built in from the start • there is no one place to look any more
20111020
EDA’s Affect on Design
51 © Eoin Woods 2011
• Event model is key to reliability and evolution • needs strong semantics and support for event model evolution
• Monitoring and fail-safe processing need from the start • don’t drop unrecognised events, flag up unexpected events • alert when events are likely to have gone missing
• Asynchronous processing can cause problems • race conditions can occur if assumptions made about propagation
• System-wide view is necessary • no one system knows the whole story – “end to end” view needed
• Partitioning needs to be considered carefully • what information needs to be where? how to partition cheaply
• Designing and developing using events needs some practice • request/response is very ingrained in most of us
20111020
Case Study Lessons We Learned
52 © Eoin Woods 2011
20111020 53 © Eoin Woods 2011
• EDA is rarely the whole answer
• EDA has been both extended and combined with other architectural styles successfully
• Common combinations of EDA and other styles • SOA • EAI
• EDA has been specialised and extended in a number of ways • CEP • Staged Event Driven Architecture (SEDA) • Event Staging • CQRS (where it is an option for the command side of the system)
20111020
Extending the Event Driven Approach
54 © Eoin Woods 2011
• SOA is a popular architectural style, aiming to encourage loose coupling and flexible systems composed of services
• SOA and EDA can work quite well together • services can generate events • services can be called as part of processing an event
• EDA can act as the “choreography” for SOA’s services
• With a number of specific complications to address • who owns business object state? or does it need to be shared?
(breaking service encapsulation)
• does the EDA system perform any business logic? or is it all in the services?
• some services implicitly manage the state of business objects so this needs to be integrated into the EDA view
• Ad-hoc EDA/SOA integration is likely to be very complicated
20111020
SOA and EDA
55 © Eoin Woods 2011
20111020
SOA and EDA
56 © Eoin Woods 2011
• SEDA is a specific specialisation of EDA
• Designed to address the needs of Internet scale systems • scalability, reliability, runtime adaptation to load, efficiency
• Uses event channels as the architectural connectors between parts of a system
• finer grained use of EDA than typical enterprise system
20111020
Staged Event Driven Architecture (SEDA)
57 © Eoin Woods 2011
(Adapted from “SEDA: An Architecture for Well-Conditioned Scalable Internet Services” Matt Welsh et al – 2001)
20111020 58 © Eoin Woods 2011
1. The network is not reliable
2. Latency is not zero
3. Bandwidth is not infinite
4. The network is not secure
5. Topology changes
6. There are many administrators
7. Transport cost is not zero
8. The network is not homogeneous
20111020
Reviewing the Fallacies of Distributed Systems
59 © Eoin Woods 2011
(Doesn’t affect this one much)
EDA helps to avoid (most of) these traps
• Event driven approach solves many traditional problems • builds on many of the strengths of other approaches
• drives coupling out of the system • allows scalability across the integration infrastructure • provides flexible options for change and extension • provides timely consistency while avoiding shared data store
• Naturally brings its own challenges too • more sophisticated approach, needing careful design
• stick to the rules!
• event model needs to be done early and to be extensible
• asynchronous nature means that race conditions are possible • monitoring and failsafe processing must be built in • event causality and tracing can be difficult (harder debugging)
20111020
Summary
60 © Eoin Woods 2011
• EDA can be adopted in stages • simple event processing
• event stream processing • complex event processing
• Simple event processing is good for many enterprise systems
• Implementing EDA doesn’t need complicated middleware • databases, messaging and markup provide most of what you need • ESBs and CEP engines can add sophistication, but also complication
• EDA can integrate well into other architectural styles
• EDA helps to meet some of the key challenges of creating resilient, adaptable distributed systems
20111020
Summary (ii)
61 © Eoin Woods 2011
• Event Processing Technical Society • articles, Event Processing Glossary, event listings, (quiet) forums • http://www.ep-ts.com
• David Luckham’s site • articles, news, pointers to research, event listings
• http://www.complexevents.com
• Opher Etzion’s blog and presentations • http://epthinking.blogspot.com
• http://www.slideshare.net/opher.etzion
• Tim Bass’ blog and presentations • http://www.thecepblog.com
• http://www.slideshare.net/TimBassCEP
• Jack van Hoof’s blog • http://soa-eda.blogspot.com
• Brenda Michelson’s EDA Overview • http://tinyurl.com/eda-overview
20111020
More Information
62 © Eoin Woods 2011
• Some of the vendor people’s blogs • Tibco (Paul Vincent) - http://tibcoblogs.com/cep • Oracle CEP - http://blogs.oracle.com/CEP • Apama (Progress) - http://apama.typepad.com • Streambase – http://streambase.typepad.com
• Some representative books on the topic
20111020
More Information
63 © Eoin Woods 2011
20111020
Software Architecture More Generally
64 © Eoin Woods 2011
Eoin Woods www.eoinwoods.info
contact@eoinwoods.info
Questions and Comments?
65 20111020 © Eoin Woods 2011
top related