handling eventual consistency in jvm microservices with event sourcing (javaone 2016)
TRANSCRIPT
![Page 1: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/1.jpg)
@crichardson
Handling Eventual Consistency in JVM Microservices with Event
Sourcing
Chris Richardson Founder of Eventuate.io
@crichardson
http://eventuate.io
Copyright © 2015. Chris Richardson Consulting, Inc. All rights reserved
Kenny Bastani Spring Developer Advocate
@kennybastani
http://pivotal.io
![Page 2: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/2.jpg)
@crichardson
Presentation goal
Show how event sourcing is a great foundation for a microservice
architecture
![Page 4: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/4.jpg)
@crichardson
About Kenny
@kennybastani
![Page 5: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/5.jpg)
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to maintain consistency
Event sourcing in a microservice architecture
Implementing queries with CQRS
![Page 6: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/6.jpg)
@crichardson
Traditional persistence
Order
id state ….
101 ACCEPTED
Order table
…
ID STATE …
102 … …
![Page 7: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/7.jpg)
@crichardson
But how did we get here?
Who did what and when?
What was the state of the Order last Monday at 3:01pm?
![Page 8: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/8.jpg)
@crichardson
Order example History
![Page 9: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/9.jpg)
@crichardson
Task example State
HistoryAudit
![Page 10: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/10.jpg)
@crichardson
Usually auditing, history and temporal
queries is additional code and/or
an after-thought
![Page 11: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/11.jpg)
@crichardson
Event sourcing
For each domain object (i.e. DDD aggregate):
Identify (state changing) domain events, e.g. use Event Storming
Define Event classes
For example, Order events: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
![Page 12: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/12.jpg)
@crichardson
Persists events NOT current state
Event table
Entity type Event id
Entity id
Event data
Order 902101 …OrderApproved
Order 903101 …OrderShipped
Event type
Order 901101 …OrderCreated
![Page 13: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/13.jpg)
@crichardson
Replay events to recreate state
Order
state
OrderCreated(…) OrderAccepted(…) OrderShipped(…)
Events
Periodically snapshot to avoid loading all events
![Page 14: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/14.jpg)
@crichardson
Benefits of event sourcing
Reifies state changes:
Built in, reliable audit log
temporal queries
Preserved history ⇒ More easily implement future requirements
Eliminates O/R mapping problem (mostly)
Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…
![Page 15: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/15.jpg)
@crichardson
Drawbacks of event sourcing
Requires application rewrite
Weird and unfamiliar style of programming
Events live forever ⇒ carefully evolve schema
Querying the event store can be challenging
Current state is no longer directly available
Often need to maintain views for efficiency
![Page 16: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/16.jpg)
@crichardson
Demo
![Page 17: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/17.jpg)
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to maintain consistency
Event sourcing in a microservice architecture
Implementing queries with CQRS
![Page 18: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/18.jpg)
@crichardson
The Microservice architecture tackles complexity through
modularization
![Page 19: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/19.jpg)
@crichardson
Microservice architecture
Browser
Mobile Device
Store Front UI
API Gateway
Catalog Service
Review Service
Order Service
… Service
Catalog Database
Review Database
Order Database
… Database
HTML
REST
REST
Apply X-axis and Z-axis scaling to each service
independently
![Page 20: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/20.jpg)
@crichardson
But there are challenges implementing
transactions and queries
![Page 21: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/21.jpg)
@crichardson
ACID transactions cannot be usedBEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION
Private to the Order Service
Private to the Customer Service
Requires 2PC
![Page 22: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/22.jpg)
@crichardson
2PC is not a viable option
Guarantees consistency
BUT
2PC is best avoided
Not supported by many NoSQL databases etc.
CAP theorem ⇒ 2PC impacts availability
….
![Page 23: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/23.jpg)
@crichardson
Queries can’t use joins
SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ?
Customer Service
Order Service
![Page 24: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/24.jpg)
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to maintain consistency
Event sourcing in a microservice architecture
Implementing queries with CQRS
![Page 25: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/25.jpg)
@crichardson
Event-driven architecture
![Page 26: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/26.jpg)
@crichardson
Order ManagementOrder
id : 4567 total: 343 state = CREATED
Customer Management
Customer creditLimit : 12000 creditReservations: {}
Customer creditLimit : 12000 creditReservations: { 4567 -> 343}
Order id : 4567 total: 343 state = APPROVED
Eventually consistent credit checking
Message Bus
createOrder()
Publishes:Subscribes to:
Subscribes to:
publishes:
OrderCreatedEvent
CreditReservedEvent
OrderCreatedEvent CreditReservedEvent
![Page 27: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/27.jpg)
@crichardson
How atomically update database and publish an event
Order Service
Order Database
Message Broker
insert Order
publish OrderCreatedEvent
dual write problem
?
![Page 28: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/28.jpg)
@crichardson
Failure = inconsistent system
Order Service
Order Database
Message Broker
insert Order
publish OrderCreatedEvent
X
![Page 29: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/29.jpg)
@crichardson
2PC is not an option
![Page 30: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/30.jpg)
@crichardson
How to reliably publish events when state changes?
![Page 31: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/31.jpg)
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to maintain consistency
Event sourcing in a microservice architecture
Implementing queries with CQRS
![Page 32: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/32.jpg)
@crichardson
Event sourcing = event-centric persistence
Application
Database
Event store
update
publish
X
![Page 33: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/33.jpg)
@crichardson
Event Store
Application architecture
Order 123 Customer 456
OrderCreated OrderApproved …
CustomerCreated CustomerCreditReserved …
CreateOrder UpdateOrder GetOrder
Subscribe
Order Service
CreateCustomer UpdateCustomer GetCustomer
Subscribe
Customer Service
![Page 34: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/34.jpg)
@crichardson
Request handling in an event sourced application
HTTP Handler
Event Store
pastEvents = findEvents(entityId)
Order
new()
applyEvents(pastEvents)
newEvents = processCmd(someCmd)
saveEvents(newEvents) (optimistic locking)
Order Service
applyEvents(newEvents)
![Page 35: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/35.jpg)
@crichardson
Event Store publishes events consumed by other services
Event Store
Event Subscriber
subscribe(EventTypes)
publish(event)
publish(event)
Customer
update()
Customer Service
![Page 36: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/36.jpg)
@crichardson
Event Store publishes events consumed by other services
Event Store
Event Subscriber
subscribe(EventTypes)
publish(event)
publish(event)
CQRS View
update()
Service Xyz
send notifications
…
![Page 37: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/37.jpg)
Event store = database + message broker
Hybrid database and message broker
Implementations:
Home grown/DIY
geteventstore.com by Greg Young
http://eventuate.io (mine)
Event Store
Save aggregate
events
Get aggregate
events
Subscribe to events
![Page 38: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/38.jpg)
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to maintain consistency
Event sourcing in a microservice architecture
Implementing queries with CQRS
![Page 39: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/39.jpg)
@crichardson
Find recent, valuable customers
SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ?
Customer Service
Order Service
What if event sourcing is
used?…. is no longer easy
![Page 40: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/40.jpg)
@crichardson
Use Command Query Responsibility Segregation
(CQRS)
![Page 41: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/41.jpg)
@crichardson
Command Query Responsibility Segregation (CQRS)
Application logic
Commands Queries
XPOST PUT DELETE
GET
![Page 42: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/42.jpg)
@crichardson
Command Query Responsibility Segregation (CQRS)
Command side
Commands
Event Sourcing domain objects
Event Store
Events
Query side
Queries
(Materialized) View
Events
POST PUT DELETE
GET
![Page 43: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/43.jpg)
@crichardson
Query side design
Event Store
Updater
View Updater Service
Events
Reader
HTTP GET Request
View Query Service
View Store
e.g. MongoDB
ElasticSearch Neo4J
update query
![Page 44: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/44.jpg)
@crichardson
Persisting a customer and order history in MongoDB
{ "_id" : "0000014f9a45004b 0a00270000000000", "_class" : "net.chrisrichardson…..views.orderhistory.CustomerView", "version" : NumberLong(5), "orders" : { "0000014f9a450063 0a00270000000000" : { "state" : "APPROVED", "orderId" : "0000014f9a450063 0a00270000000000", "orderTotal" : { "amount" : "1234" } }, "0000014f9a450063 0a00270000000001" : { "state" : "REJECTED", "orderId" : "0000014f9a450063 0a00270000000001", "orderTotal" : { "amount" : "3000" } } }, "name" : "Fred", "creditLimit" : { "amount" : "2000" } }
Denormalized = efficient lookup
![Page 45: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/45.jpg)
@crichardson
Persisting customers and order info using Spring Data for MongoDB...
![Page 46: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/46.jpg)
@crichardson
Persisting customers and order using Spring Data for MongoDB...
![Page 47: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/47.jpg)
Benefits and drawbacks of CQRS
Benefits
Necessary in an event sourced architecture
Separation of concerns = simpler command and query models
Supports multiple denormalized views
Improved scalability and performance
Drawbacks
Complexity
Potential code duplication
Replication lag/eventually consistent views
![Page 48: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/48.jpg)
@crichardson
Demo
![Page 49: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/49.jpg)
@crichardson
Summary
Microservice architecture functionally decomposes an application into services
Transactions and queries resist decomposition
Use an event-driven architecture based on event sourcing to maintain data consistency
Implement queries using CQRS
![Page 50: Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)](https://reader030.vdocument.in/reader030/viewer/2022021506/58742fdf1a28ab72188b758f/html5/thumbnails/50.jpg)
@crichardson
[email protected] [email protected]
http://learnmicroservices.io http://pivotal.io
Questions?