![Page 1: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/1.jpg)
ConcursusEvent Sourcing EvolvedGOTO London 2016
![Page 2: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/2.jpg)
![Page 3: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/3.jpg)
Introductions
Dominic Fox
Twitter: @dynamic_proxy
Email: [email protected]
Tareq Abedrabbo
Twitter: @tareq_abedrabbo
Email: [email protected]
Concursus
Page: https://opencredo.com/publications/concursus/
Github: http://github.com/opencredo/concursus
![Page 4: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/4.jpg)
Agenda• History
• Concepts
• Example
• Domain model
• Processing model
• Programming model
• Future directions
![Page 5: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/5.jpg)
What is Concursus?
A toolkit for processing and organising messy data in an distributed context.
![Page 6: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/6.jpg)
The Concursus TimelineObservations
Conception and design
Prototype
Open source implementation
Technical report and blogs
![Page 7: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/7.jpg)
Event Sourcing
“Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.”
http://martinfowler.com/eaaDev/EventSourcing.html
![Page 8: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/8.jpg)
What is Concursus?Problems Concursus addresses:
ü Processing events in a scalable and reliable way
ü Processing guarantees and ordering: exactly once, out of order, repeated or missed delivery, etc..
ü Building meaningful domain models to reason about and build business logic around
ü Flexibility: building additional views as needed
![Page 9: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/9.jpg)
Tendencies:
• From internet of users to internet of things
• From “presence” to “presents”
• From monoliths to microservices
Why Concursus?
![Page 10: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/10.jpg)
From Internet of Users to Internet of Things
![Page 11: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/11.jpg)
From Presence to Presents
![Page 12: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/12.jpg)
From Monoliths to Microservices
![Page 13: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/13.jpg)
“Write First, Reason Later”
2016-10-12 09:06:31.432Received at Depot
2016-10-12 09:06:32.106Received at Depot
2016-10-12 09:06:34.740Received at Depot
2016-10-12 11:35:02.163Loaded onto Truck
2016-10-12 11:40:21.032Loaded onto Truck
2016-10-12 11:38:51.204Loaded onto Truck
2016-10-12 14:12:44.021Delivery Failed
2016-10-12 15:00:31.322Delivered
2016-10-12 15:11:05.038Delivered
![Page 14: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/14.jpg)
“Write First, Reason Later”
![Page 15: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/15.jpg)
Handling Eventsü Delivery constraints
out of order, repeated, delayed or missed delivery
ü Processing guarantees
at least once or exactly once processing, idempotency
ü Ordering
partial ordering across aggregates (with reasonable assumptions)
![Page 16: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/16.jpg)
Data Processing Layersü Durable
sufficiently durable buffer for async processing (what’s happening)
ü Persistent
a permanent record of everything that has happened (what happened)
ü Transient
fast and consistent, but also disposable state (what happens)
![Page 17: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/17.jpg)
Building Blocks• Java 8 and Kotlin: APIs
• Cassandra: Persistent state (Event store)
• Kafka: Durable state (Message broker)
• Hazelcast: Transient state (cache, idempotency filters)
• Also, RabbitMQ and Redis
![Page 18: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/18.jpg)
Sources of InspirationStream processing frameworks such as Apache Storm and Spark
Google papers: Cloud dataflow, MillWheel
Apache Spark papers
The Axon CQRS framework
Domain Driven Design
Functional programming
![Page 19: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/19.jpg)
Summary
Concursus=
Event sourcing+
Stream processing+
Bounded contexts (DDD)+
Distributed computing
![Page 20: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/20.jpg)
Received atDepot
Loaded onto Truck
Delivered
Delivery Failed
![Page 21: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/21.jpg)
Domain Model: Events
Received atDepot
Loaded ontoTruck
DeliveryFailed
Received atDepot
Loaded ontoTruck
Delivered
![Page 22: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/22.jpg)
aggregateType: parcelaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-31 10:31:17.981parameters: { “depotId”: “Lewisham” }
Domain Model: Events
Received atDepot
Loaded ontoTruck
DeliveryFailed
Received atDepot
Loaded ontoTruck
Delivered
![Page 23: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/23.jpg)
aggregateType: parcelaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9
eventTimestamp: 2016-03-38 08:15:23.104parameters: { “truckId”: “J98 257” }
Domain Model: Events
Received atDepot
Loaded ontoTruck
DeliveryFailed
Received atDepot
Loaded ontoTruck
Delivered
![Page 24: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/24.jpg)
eventTimestamp: 2016-03-31T10:36:42.171ZprocessingTimestamp: 2016-03-31T10:36:48.3904Zparameters: { “deliveryAddress”: “123 SudburyAvenue, Droitwich DR4 8PQ”}
Domain Model: EventsaggregateType: parcelaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9
Received at Depot
Loaded onto Truck
Delivery Failed
Received at Depot
Loaded onto Truck
Delivered
![Page 25: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/25.jpg)
Domain Model: SummaryEvery Event occurs to an Aggregate, identified by its type and id.Every Event has an eventTimestamp, generated by the source of the event.An Event History is a log of Events, ordered by eventTimestamp, with an additional processingTimestamp which records when the Event was captured.
![Page 26: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/26.jpg)
Network
Event sources Event processors
Events arrive:• Partitioned• Interleaved• Out-of-order
Processing Model: Ordering
![Page 27: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/27.jpg)
Log is:• Partitioned by aggregate id• Ordered by event timestamp
Processing Model: Ordering
![Page 28: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/28.jpg)
CREATE TABLE IF NOT EXISTS concursus.Event (aggregateType text,aggregateId text,eventTimestamp timestamp,streamId text,processingId timeuuid,name text,version text,parameters map<text, text>,characteristics int,PRIMARY KEY((aggregateType, aggregateId), eventTimestamp,
streamId)) WITH CLUSTERING ORDER BY (eventTimestamp DESC);
Cassandra Schema
![Page 29: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/29.jpg)
CassandraEvent Store
RabbitMQ Topic
DownstreamprocessingLog
events
Publish events
Cassandra & AMQP
![Page 30: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/30.jpg)
CassandraEvent Store
RabbitMQ Topic
Downstreamprocessing
out-of-order events
ordered query results
Cassandra & AMQP
![Page 31: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/31.jpg)
CassandraEvent Store
Kafka Topic
Downstreamprocessing
Event store listener
Publish events
Log events
Cassandra & Kafka
![Page 32: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/32.jpg)
Processing Model: SummaryEvents arrive partitioned, interleaved and out-of-order.Events are sorted into event histories by aggregate type and id.Events are sorted within event histories by event timestamp, not processing timestamp.Event consumers need to take into account the possibility that an event history may be incomplete at the time it is read – consider using a watermark to give incoming events time to “settle”.
![Page 33: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/33.jpg)
Programming Model: Core Metaphor
Received atDepot
Loaded ontoTruck
DeliveryFailed
Received atDepot
Loaded ontoTruck
Delivered
![Page 34: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/34.jpg)
Received atDepot
Loaded ontoTruck
DeliveryFailed
Received atDepot
Loaded ontoTruck
Delivered
Consumer<Event>
Programming Model: Core Metaphor
![Page 35: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/35.jpg)
You give me a Consumer<Event>, and I send Events to it one at a time:
Emitting Events
![Page 36: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/36.jpg)
I implement Consumer<Event>, and handle Events that are sent to me.
Handling Events
![Page 37: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/37.jpg)
Java 8 Mapping
![Page 38: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/38.jpg)
Java 8 Mapping
![Page 39: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/39.jpg)
Java 8 Mapping
![Page 40: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/40.jpg)
Kotlin Mapping
sealed class ParcelEvent {class ReceivedAtDepot(val depotId: String): ParcelEvent()class LoadedOntoTruck(val truckId: String): ParcelEvent()class Delivered(val destinationId: String): ParcelEvent()class DeliveryFailed(): ParcelEvent()
}
![Page 41: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/41.jpg)
Kotlin Mapping
eventBus.dispatchTo(parcelId,ReceivedAtDepot(depotId = "Lewisham Depot") at start,LoadedOntoLorry(lorryId = "Truck CU50 ZCV") at
start.plus(2, DAYS))
![Page 42: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/42.jpg)
Kotlin Mapping
fun describeEvent(event: ParcelEvent): Unit = when (event) {is ReceivedAtDepot -> println("Received at depot:
${event.depotId}")is LoadedOntoTruck -> println("Loaded onto truck:
${event.truckId}")is Delivered -> println("Delivered to:
${event.destinationId}")is DeliveryFailed -> println("Delivery failed")
}
![Page 43: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/43.jpg)
Event-handling middleware is a chain of Consumer<Event>s that transforms, routes, persists and dispatches events. A single event submitted to this chain may be:■ Checked against an idempotency filter (e.g. a Hazelcast distributed cache)■ Serialised to JSON■ Written to a message queue topic■ Retrieved from the topic and deserialised■ Persisted to an event store (e.g. Cassandra)■ Published to an event handler which maintains a query-optimised view of part of the system■ Published to an event handler which maintains an index of aggregates by event property
values (e.g. lightbulbs by wattage)
Event-Handling Middleware
![Page 44: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/44.jpg)
• Kafka Streams
• Narrative threads across event histories
• Generic Attribute indexing
• State management and caching
• Improved cloud tooling
Future Directions
![Page 45: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/45.jpg)
Thank you for listeningAny questions?
![Page 46: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/46.jpg)
![Page 47: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/47.jpg)
Three Processing Schedules
1.Transient
2.Durable
3.Persistent
![Page 48: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/48.jpg)
Three Processing Schedules
1.Transient
2.Durable
3.Persistent
![Page 49: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/49.jpg)
Three Processing Schedules
1.Transient
2.Durable
3.Persistent
![Page 50: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/50.jpg)
Three Processing Schedules
1.Transient - what happens
2.Durable - what’s happening
3.Persistent - what happened
![Page 51: Concursus- Event Sourcing Evolved - GOTO Conference · Event-handling middleware is a chain of Consumers that transforms, routes, persists and dispatches events. A single](https://reader034.vdocument.in/reader034/viewer/2022042220/5ec5e0d58314ca5b1e4e0e01/html5/thumbnails/51.jpg)
“Write First, Reason Later”