goto london 2016: concursus event sourcing evolved (updated)

ConcursusEvent Sourcing EvolvedGOTO  London  2016

Introductions

Dominic Fox

Twitter: @dynamic_proxy

Email: dominic.fox@opencredo.com

Tareq Abedrabbo

Twitter: @tareq_abedrabbo

Email: tareq.abedrabbo@opencredo.com

Concursus

Page: https://opencredo.com/publications/concursus/

Github: http://github.com/opencredo/concursus

Agenda• History

• Concepts

• Example

• Domain model

• Processing model

• Programming model

• Future directions

What is Concursus?

A toolkit for processing and organising messy data in an distributed context.

The Concursus TimelineObservations

Conception and design

Prototype

Open source implementation

Technical report and blogs

Event Sourcing

“Event Sourcing ensures that all changes to application state are stored as a sequence of events. Not just can we query these events, we can also use the event log to reconstruct past states, and as a foundation to automatically adjust the state to cope with retroactive changes.”

http://martinfowler.com/eaaDev/EventSourcing.html

What is Concursus?Problems Concursus addresses:

ü Processing events in a scalable and reliable way

ü Processing guarantees and ordering: exactly once, out of order, repeated or missed delivery, etc..

ü Building meaningful domain models to reason about and build business logic around

ü Flexibility: building additional views as needed

Tendencies:

• From internet of users to internet of things

• From “presence” to “presents”

• From monoliths to microservices

Why Concursus?

From Internet of Users to Internet of Things

From Presence to Presents

From Monoliths to Microservices

“Write First, Reason Later”

2016-­‐10-­‐12  09:06:31.432Received  at  Depot

2016-­‐10-­‐12  09:06:32.106Received  at  Depot

2016-­‐10-­‐12  09:06:34.740Received  at  Depot

2016-­‐10-­‐12  11:35:02.163Loaded  onto  Truck

2016-­‐10-­‐12  11:40:21.032Loaded  onto  Truck

2016-­‐10-­‐12  11:38:51.204Loaded  onto  Truck

2016-­‐10-­‐12  14:12:44.021Delivery  Failed

2016-­‐10-­‐12  15:00:31.322Delivered

2016-­‐10-­‐12  15:11:05.038Delivered

“Write First, Reason Later”

Handling Eventsü Delivery constraints

out of order, repeated, delayed or missed delivery

ü Processing guarantees

at least once or exactly once processing, idempotency

ü Ordering

partial ordering across aggregates (with reasonable assumptions)

Data Processing Layersü Durable

sufficiently durable buffer for async processing (what’s happening)

ü Persistent

a permanent record of everything that has happened (what happened)

ü Transient

fast and consistent, but also disposable state (what happens)

Building Blocks• Java 8 and Kotlin: APIs

• Cassandra: Persistent state (Event store)

• Kafka: Durable state (Message broker)

• Hazelcast: Transient state (cache, idempotency filters)

• Also, RabbitMQ and Redis

Sources of InspirationStream processing frameworks such as Apache Storm and Spark

Google papers: Cloud dataflow, MillWheel

Apache Spark papers

The Axon CQRS framework

Domain Driven Design

Functional programming

Summary

Concursus=

Event sourcing+

Stream processing+

Bounded contexts (DDD)+

Distributed computing

Received  atDepot

Loaded  onto  Truck

Delivered

Delivery  Failed

Domain Model: Events

Received  atDepot

Loaded  ontoTruck

DeliveryFailed

Received  atDepot

Loaded  ontoTruck

Delivered

aggregateType: parcelaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9

eventTimestamp: 2016-03-31 10:31:17.981parameters: { “depotId”: “Lewisham” }

Domain Model: Events

Received  atDepot

Loaded  ontoTruck

DeliveryFailed

Received  atDepot

Loaded  ontoTruck

Delivered

aggregateType: parcelaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9

eventTimestamp: 2016-03-38 08:15:23.104parameters: { “truckId”: “J98 257” }

Domain Model: Events

Received  atDepot

Loaded  ontoTruck

DeliveryFailed

Received  atDepot

Loaded  ontoTruck

Delivered

eventTimestamp: 2016-03-31T10:36:42.171ZprocessingTimestamp: 2016-03-31T10:36:48.3904Zparameters: { “deliveryAddress”: “123 SudburyAvenue, Droitwich DR4 8PQ”}

Domain Model: EventsaggregateType: parcelaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9

Received  at  Depot

Loaded  onto  Truck

Delivery  Failed

Received  at  Depot

Loaded  onto  Truck

Delivered

Domain Model: SummaryEvery Event occurs to an Aggregate, identified by its type and id.Every Event has an eventTimestamp, generated by the source of the event.An Event History is a log of Events, ordered by eventTimestamp, with an additional processingTimestamp which records when the Event was captured.

Network

Event sources Event processors

Events arrive:• Partitioned• Interleaved• Out-of-order

Processing Model: Ordering

Log is:• Partitioned by aggregate id• Ordered by event timestamp

Processing Model: Ordering

CREATE  TABLE  IF  NOT  EXISTS  concursus.Event (aggregateType text,aggregateId text,eventTimestamp timestamp,streamId text,processingId timeuuid,name  text,version  text,parameters  map<text,  text>,characteristics  int,PRIMARY  KEY((aggregateType,  aggregateId),  eventTimestamp,  

streamId))  WITH  CLUSTERING  ORDER  BY  (eventTimestamp DESC);

Cassandra Schema

CassandraEvent Store

RabbitMQ Topic

DownstreamprocessingLog

events

Publish events

Cassandra & AMQP

CassandraEvent Store

RabbitMQ Topic

Downstreamprocessing

out-of-order events

ordered query results

Cassandra & AMQP

CassandraEvent Store

Kafka Topic

Downstreamprocessing

Event store listener

Publish events

Log events

Cassandra & Kafka

Processing Model: SummaryEvents arrive partitioned, interleaved and out-of-order.Events are sorted into event histories by aggregate type and id.Events are sorted within event histories by event timestamp, not processing timestamp.Event consumers need to take into account the possibility that an event history may be incomplete at the time it is read – consider using a watermark to give incoming events time to “settle”.

Programming Model: Core Metaphor

Received  atDepot

Loaded  ontoTruck

DeliveryFailed

Received  atDepot

Loaded  ontoTruck

Delivered

Received  atDepot

Loaded  ontoTruck

DeliveryFailed

Received  atDepot

Loaded  ontoTruck

Delivered

Consumer<Event>

Programming Model: Core Metaphor

You give me a Consumer<Event>, and I send Events to it one at a time:

Emitting Events

I implement Consumer<Event>, and handle Events that are sent to me.

Handling Events

Java 8 Mapping

Java 8 Mapping

Java 8 Mapping

Kotlin Mapping

sealed  class  ParcelEvent {class  ReceivedAtDepot(val depotId:  String):  ParcelEvent()class  LoadedOntoTruck(val truckId:  String):  ParcelEvent()class  Delivered(val destinationId:  String):  ParcelEvent()class  DeliveryFailed():  ParcelEvent()

}

Kotlin Mapping

eventBus.dispatchTo(parcelId,ReceivedAtDepot(depotId =  "Lewisham Depot")  at  start,LoadedOntoLorry(lorryId =  "Truck  CU50  ZCV")  at  

start.plus(2,  DAYS))

Kotlin Mapping

fun  describeEvent(event:  ParcelEvent):  Unit  =  when  (event)  {is  ReceivedAtDepot -­‐>  println("Received  at  depot:  

${event.depotId}")is  LoadedOntoTruck -­‐>  println("Loaded  onto  truck:  

${event.truckId}")is  Delivered  -­‐>  println("Delivered  to:  

${event.destinationId}")is  DeliveryFailed -­‐>  println("Delivery  failed")

}

Event-handling middleware is a chain of Consumer<Event>s that transforms, routes, persists and dispatches events. A single event submitted to this chain may be:■ Checked against an idempotency filter (e.g. a Hazelcast distributed cache)■ Serialised to JSON■ Written to a message queue topic■ Retrieved from the topic and deserialised■ Persisted to an event store (e.g. Cassandra)■ Published to an event handler which maintains a query-optimised view of part of the system■ Published to an event handler which maintains an index of aggregates by event property

values (e.g. lightbulbs by wattage)

Event-Handling Middleware

• Kafka Streams

• Narrative threads across event histories

• Generic Attribute indexing

• State management and caching

• Improved cloud tooling

Future Directions

Thank you for listeningAny questions?