append only data stores
TRANSCRIPT
vs
Append-only data stores
onsdag 16 oktober 13
What?
onsdag 16 oktober 13
Append-only data stores
Only add, never remove or change
Can retrieve old values
onsdag 16 oktober 13
Source code
Version controlled
Keep all versions
onsdag 16 oktober 13
The problem
onsdag 16 oktober 13
Mutable state
id name email
jakr Jan Kronquist [email protected]
UPDATE customerSET email="[email protected]"WHERE id="jakr"
onsdag 16 oktober 13
Mutable state
id name email
jakr Jan Kronquist [email protected]
UPDATE customerSET email="[email protected]"WHERE id="jakr"
onsdag 16 oktober 13
Mutable state
id name email
jakr Jan Kronquist [email protected]
UPDATE customerSET email="[email protected]"WHERE id="jakr"
I didn’t get confirmation of the order last week?
onsdag 16 oktober 13
Mutable state
id name email
jakr Jan Kronquist [email protected]
UPDATE customerSET email="[email protected]"WHERE id="jakr"
I didn’t get confirmation of the order last week?
Could you reset my email address?
onsdag 16 oktober 13
The technical details
SELECT count(*) AS inSwedenFROM customerWHERE email LIKE "%@jayway.se"
SELECT count(*) AS elsewhereFROM customerWHERE email LIKE "%@jayway.com"
total = inSweden + elsewhere
onsdag 16 oktober 13
Human errors
DELETEFROM customerWHERE email LIKE "j%"
onsdag 16 oktober 13
Scalability - the easy parts
Static content
Caching
onsdag 16 oktober 13
Scalability - the easy parts
Static content
Caching
onsdag 16 oktober 13
Scalability - the easy parts
Static content
Stateless work
Caching
onsdag 16 oktober 13
Scalability - the easy parts
Static content
Stateless work
Load balancer
Caching
onsdag 16 oktober 13
State + behavior
Scalability - the hard parts
onsdag 16 oktober 13
State + behavior
Scalability - the hard parts
onsdag 16 oktober 13
Example domain
onsdag 16 oktober 13
Rock - Paper - Scissors
onsdag 16 oktober 13
http://rock-paper-scissors.com/
The future Facebook of Rock Paper Scissors
Millions of users
Many games per user
onsdag 16 oktober 13
Playing the game
Player A
Player B
Server
onsdag 16 oktober 13
Playing the game
Player A
Player B
Server
Game 123
onsdag 16 oktober 13
Playing the game
Player A
Player B
Serverpaper
Game 123
onsdag 16 oktober 13
Playing the game
Player A
Player B
Serverpaper
Player B: paper
Game 123
onsdag 16 oktober 13
Playing the game
Player A
Player B
Server
rock
Player B: paper
Player A: rock
Game 123
onsdag 16 oktober 13
Playing the game
Player A
Player B
Server
Player B: paper
Player A: rock
Game 123
Game 123winner: Player Bloser: Player A
onsdag 16 oktober 13
Playing the game
Player A
Player B
Server
Player B: paper
Player A: rock
Game 123
Game 123winner: Player Bloser: Player A
CREATED WAITING
GAME WON
GAME TIED
any move
other move(victory)
other move(tie)
onsdag 16 oktober 13
Entities
onsdag 16 oktober 13
Entities
Player+ id+ name+ email
onsdag 16 oktober 13
Entities
Game+ id+ state+ players+ moves+ winner+ loser
Player+ id+ name+ email
onsdag 16 oktober 13
Entities
Game+ id+ state+ players+ moves+ winner+ loser Move
+ move+ player
Player+ id+ name+ email
onsdag 16 oktober 13
Actions
Create game
Make move
Get game result
POST /games
POST /games/123 move=rock
GET /games/123
onsdag 16 oktober 13
Actions
Create game
Make move
Get game result
POST /games
POST /games/123 move=rock
GET /games/123
1. Form (make move)
2. Waiting
3. Game result
4. Error
onsdag 16 oktober 13
Clojure
onsdag 16 oktober 13
Clojure data structures
onsdag 16 oktober 13
Clojure data structures
Lists - singly linked(1 2 3 4 5) (fred ethel lucy) (list 1 2 3)
onsdag 16 oktober 13
Clojure data structures
Lists - singly linked(1 2 3 4 5) (fred ethel lucy) (list 1 2 3)
Vectors - indexed access[1 2 3] ["fred" 17 3.14 "bar"]
onsdag 16 oktober 13
Clojure data structures
Lists - singly linked(1 2 3 4 5) (fred ethel lucy) (list 1 2 3)
Vectors - indexed access[1 2 3] ["fred" 17 3.14 "bar"]
Maps{:name "Jan Kronquist" :age 37}
onsdag 16 oktober 13
Event Sourcingwith
Event Store
onsdag 16 oktober 13
Event Sourcing
Events that have happened
Ordered in time
Source of truth
onsdag 16 oktober 13
http://geteventstore.com
Runs on .NET and Mono
Free (store) or Commercial (HA)
APICustom TCP (c#)
ATOM over HTTP (xml/json)
FeaturesEvent streams, projections, generate new events (CEP)
€1500/year
onsdag 16 oktober 13
Event example
Server EventStore
onsdag 16 oktober 13
Event example
Server
CreateGameCommand{:game 123 :creator "player-1"}
EventStore
onsdag 16 oktober 13
Event example
Server
CreateGameCommand{:game 123 :creator "player-1"}
GameCreatedEvent{:game 123 :creator "player-1"}
EventStore
onsdag 16 oktober 13
Event example
Server
MakeMoveCommand{:game 123 :player "player-1" :move "rock"}
GameCreatedEvent{:game 123 :creator "player-1"}
MoveMadeEvent{:game 123 :player "player-1" :move "rock"}
EventStore
onsdag 16 oktober 13
Event example
Server
MakeMoveCommand{:game 123 :player "player-2" :move "paper"}
GameCreatedEvent{:game 123 :creator "player-1"}
MoveMadeEvent{:game 123 :player "player-1" :move "rock"}
MoveMadeEvent{:game 123 :player "player-2" :move "paper"}GameWonEvent{:game 123 :winner "player-2" :loser "player-1"}
EventStore
onsdag 16 oktober 13
GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2}
Current state?
onsdag 16 oktober 13
CREATED WAITING
GAME WON
GAME TIED
any move
if victory
other move
if tie
GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2}
Current state?
onsdag 16 oktober 13
CREATED WAITING
GAME WON
GAME TIED
any move
if victory
other move
if tie
GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2}GameCreatedEvent
Current state?
CREATED
onsdag 16 oktober 13
CREATED WAITING
GAME WON
GAME TIED
any move
if victory
other move
if tie
GameCreatedEvent{:game 123, :creator "player-1", :time 1} MoveMadeEvent{:game 123, :player "player-1", :move "rock", :time 2}
Current state?
WAITING
MoveMadeEvent
onsdag 16 oktober 13
Aggregates
Scope of consistency when handling a command
Game+ id+ state+ players+ moves+ winner+ loser
Move+ id+ move+ player
Player+ id+ name+ email
onsdag 16 oktober 13
Aggregates as Event Streams
event1 GameCreatedEvent{:game 1...}event2 MoveMadeEvent{:game 1....}
stream = game-1version = 2
onsdag 16 oktober 13
Aggregates as Event Streams
event1 GameCreatedEvent{:game 1...}event2 MoveMadeEvent{:game 1....}
stream = game-1version = 2
event1 GameCreatedEvent{:game 2...}event2 MoveMadeEvent{:game 2....}event3 MoveMadeEvent{:game 2....}event4 GameWonEvent{:game 2....}
stream = game-2version = 4
onsdag 16 oktober 13
Aggregates as Event Streams
event1 GameCreatedEvent{:game 1...}event2 MoveMadeEvent{:game 1....}
stream = game-1version = 2
event1 GameCreatedEvent{:game 2...}event2 MoveMadeEvent{:game 2....}event3 MoveMadeEvent{:game 2....}event4 GameWonEvent{:game 2....}
stream = game-2version = 4
curl "http://127.0.0.1:2113/streams/game-1" -d @eventdata.json -H "Content-Type:application/json" -H "ES-ExpectedVersion: 2"
onsdag 16 oktober 13
Projections
GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....}
onsdag 16 oktober 13
Projections
GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....}
for (Event events : allEvents) { if (event instanceof GameCreatedEvent) { created++; }}return created;
onsdag 16 oktober 13
Projections
GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....}
onsdag 16 oktober 13
Projections
GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....} created = 2
onsdag 16 oktober 13
Projections
GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....} created = 2GameCreatedEvent{:game 3...}
onsdag 16 oktober 13
Projections
GameCreatedEvent{:game 1...}MoveMadeEvent{:game 1....}GameCreatedEvent{:game 2...}MoveMadeEvent{:game 2....}MoveMadeEvent{:game 2....}GameWonEvent{:game 2....}
(state, event) -> { if (event instanceof GameCreatedEvent) { return state++; } rerturn state;}
created = 2GameCreatedEvent{:game 3...}
onsdag 16 oktober 13
Projections in JavascriptfromAll().when({ $init: function () { return { created: 0, completed:0 }; },
});
onsdag 16 oktober 13
Projections in JavascriptfromAll().when({ $init: function () { return { created: 0, completed:0 }; },
});
GameCreatedEvent: function(s, e) { s.created++; return s; },
onsdag 16 oktober 13
Projections in JavascriptfromAll().when({ $init: function () { return { created: 0, completed:0 }; },
});
GameCreatedEvent: function(s, e) { s.created++; return s; },
GameWonEvent: function(s, e) { s.completed++; return s; },
onsdag 16 oktober 13
Projections in JavascriptfromAll().when({ $init: function () { return { created: 0, completed:0 }; },
});
GameCreatedEvent: function(s, e) { s.created++; return s; },
GameWonEvent: function(s, e) { s.completed++; return s; },
GameTiedEvent: function(s, e) { s.completed++; return s; }
onsdag 16 oktober 13
Projections in JavascriptfromAll().when({ $init: function () { return { created: 0, completed:0 }; },
});
GameCreatedEvent: function(s, e) { s.created++; return s; },
GameWonEvent: function(s, e) { s.completed++; return s; },
GameTiedEvent: function(s, e) { s.completed++; return s; }
/projections/games-counter{created: 15,completed: 10
}
onsdag 16 oktober 13
External Projections
events
Business logic
onsdag 16 oktober 13
External Projections
events events
Business logic Projections
onsdag 16 oktober 13
External Projections
Consistent
Available
events events
Business logic Projections
onsdag 16 oktober 13
External Projections
Commands Queries
Consistent
Available
events events
onsdag 16 oktober 13
System with EventStore
onsdag 16 oktober 13
System with EventStore
Command Handler
onsdag 16 oktober 13
System with EventStore
1.command Command Handler
onsdag 16 oktober 13
System with EventStore
1.command Event Store
Command Handler
2.get events
onsdag 16 oktober 13
System with EventStore
1.command Event Store
Command Handler
2.get events
3.add events
onsdag 16 oktober 13
System with EventStore
1.command Event Store
Command Handler
2.get events
3.add events
4.updateprojections
onsdag 16 oktober 13
System with EventStore
1.command Event Store
Command Handler
2.get events
3.add events
4.updateprojections
5.get eventsor projections
Some service
onsdag 16 oktober 13
System with EventStore
1.command Event Store
Command Handler
2.get events
6.update state
3.add events
4.updateprojections
5.get eventsor projections
Some service
onsdag 16 oktober 13
System with EventStore
1.command Event Store
Command Handler
2.get events
6.update state
3.add events
4.updateprojections
7.query
5.get eventsor projections
Some service
onsdag 16 oktober 13
More info
Functional Programming with DDDhttp://skillsmatter.com/podcast/design-architecture/ddd-functional-programming
A deep look into the Event Storehttp://vimeo.com/53153270
onsdag 16 oktober 13
Datomic
onsdag 16 oktober 13
http://www.datomic.com/
Runs on JVM
Free (3 peers, local storage) or Commercial (HA, distrib)
FeaturesACID transactions, declarative query engine
€2300
onsdag 16 oktober 13
Fact
Something known to have happened or existed
onsdag 16 oktober 13
Fact
Something known to have happened or existed
2004-10-01 the email of Jan was ”[email protected]”
onsdag 16 oktober 13
Fact
Something known to have happened or existed
2004-10-01 the email of Jan was ”[email protected]”
2007-01-01 the email of Jan was ”[email protected]”
onsdag 16 oktober 13
Atomic fact (datom)
Entity
Attribute
Value
Time
onsdag 16 oktober 13
Atomic fact (datom)
Entity
Attribute
Value
Time
The person Jan Kronquist
2004-10-01
onsdag 16 oktober 13
Data perception
Facts
Add facts
Add more facts
Even more facts
time
onsdag 16 oktober 13
Data perception
Facts
Add facts
Add more facts
Even more facts
time
onsdag 16 oktober 13
Data perception
Facts
Add facts
Add more facts
Even more facts
time
query
onsdag 16 oktober 13
Data perception
Facts
Add facts
Add more facts
Even more facts
time
query
query
onsdag 16 oktober 13
Data perception
Facts
Add facts
Add more facts
Even more facts
time
query
onsdag 16 oktober 13
Data perception
Facts
Add facts
Add more facts
Even more facts
time
query
onsdag 16 oktober 13
Data perception
Facts
Add facts
Add more facts
Even more facts
time
query
onsdag 16 oktober 13
Database deconstructed
onsdag 16 oktober 13
Database deconstructed
TransactorConsistency by serializing transactions
onsdag 16 oktober 13
Database deconstructed
TransactorConsistency by serializing transactions
PeerYour code + Query engine
onsdag 16 oktober 13
Database deconstructed
TransactorConsistency by serializing transactions
PeerYour code + Query engine
Storage serviceKey/value store
Scalable reads and writes
eg DynamoDB, InfiniSpan, Riak + ZooKeeper, SQL(!)
onsdag 16 oktober 13
System with Datomic
Transactor
Distributed Storage Service
Peer A Peer A
onsdag 16 oktober 13
System with Datomic
Transactor
Distributed Storage Service
Peer A Peer A
1.command
onsdag 16 oktober 13
System with Datomic
Transactor
Distributed Storage Service
Peer A Peer A
2.read and cache
1.command
onsdag 16 oktober 13
System with Datomic
Transactor
Distributed Storage Service
Peer A
3.transact
Peer A
2.read and cache
1.command
onsdag 16 oktober 13
System with Datomic
Transactor
Distributed Storage Service
Peer A
3.transact
Peer A
4.write
2.read and cache
1.command
onsdag 16 oktober 13
System with Datomic
Transactor
Distributed Storage Service
Peer A
3.transact
Peer A
4.write
2.read and cache
Peer B
5.notify
1.command
onsdag 16 oktober 13
Datomic example
Peer Transactor
onsdag 16 oktober 13
Datomic example
Peer
CreateGameCommand{:game 123 :creator "player-1"}
Transactor
onsdag 16 oktober 13
Datomic example
Peer
CreateGameCommand{:game 123 :creator "player-1"}
[123 :creator "player-1" T1][123 :state :created T1]
Transactor
onsdag 16 oktober 13
Datomic example
Peer
CreateGameCommand{:game 123 :creator "player-1"}
[123 :creator "player-1" T1][123 :state :created T1]
Transactor
["player-1" :email "[email protected]" T0]
onsdag 16 oktober 13
Datomic example
Peer
CreateGameCommand{:game 123 :creator "player-1"}
[123 :creator "player-1" T1][123 :state :created T1]
Transactor
onsdag 16 oktober 13
Datomic example
Peer
CreateGameCommand{:game 123 :creator "player-1"}
[123 :creator "player-1" T1][123 :state :created T1]
Transactor
GameCreatedEvent{:game 123 :creator "player-1"}
onsdag 16 oktober 13
Datomic example
Peer
CreateGameCommand{:game 123 :creator "player-1"}
[123 :creator "player-1" T1][123 :state :created T1]
Transactor
onsdag 16 oktober 13
Datomic example
Peer
MakeMoveCommand{:game 123 :player "player-1" :move :rock}
[123 :creator "player-1" T1][123 :state :created T1][123_1 :move :rock T2][123_1 :player "player-1" T2][123 :moves 123_1 T2][123 :state :waiting T2]
Transactor
onsdag 16 oktober 13
Datomic example
Peer
MakeMoveCommand{:game 123 :player "player-2" :move :paper}
[123 :creator "player-1" T1][123 :state :created T1][123_1 :move :rock T2][123_1 :player "player-1" T2][123 :moves 123_1 T2][123 :state :waiting T2][123_2 :move :paper T3][123_2 :player "player-2" T3][123 :moves 123_2 T3][123 :state :won T3][123 :winner "player-2" T3][123 :loser "player-1" T3]
Transactor
onsdag 16 oktober 13
Aggregates?
Only by convention
cas (compare and set)
isComponent on relation
Game+ id+ state+ players+ moves+ winner+ loser
Move+ id+ move+ player
onsdag 16 oktober 13
What to find
Where clauses [entity attribute value]
Implicit joins
Call arbitrary function
Query language
onsdag 16 oktober 13
Query example - basic
[:find ?email :where [_ :email ?email]]
onsdag 16 oktober 13
Query example - basic
[:find ?email :where [_ :email ?email]]
[["[email protected]] ["[email protected]"] ["[email protected]"]]
onsdag 16 oktober 13
Query example - aggregation
[:find ?state (?count game) :where [?game :state ?state]]
onsdag 16 oktober 13
Query example - aggregation
[:find ?state (?count game) :where [?game :state ?state]]
[[:created 4] [:waiting 1] [:won 7] [:tied 3]]
onsdag 16 oktober 13
Query example - aggregation
[:find ?state (?count game) :where [?game :state ?state]]
[[:created 4] [:waiting 1] [:won 7] [:tied 3]]
inProgess = created + waiting
onsdag 16 oktober 13
Query example - aggregation
[:find ?state (?count game) :where [?game :state ?state]]
[[:created 4] [:waiting 1] [:won 7] [:tied 3]]
inProgess = created + waiting
fromAll().when({ $init: function () { return { created: 0, completed:0 }; },
GameCreatedEvent: function(s, e) { s.created++; return s; },
GameWonEvent: function(s, e) { s.completed++; return s; },
GameTiedEvent: function(s, e) { s.completed++; return s; }});
/projections/games-counter{
created: 15,completed: 10
}
onsdag 16 oktober 13
Query example - join
[:find ?email (count ?game) :where [?player :email ?email] [?game :winner ?player]]
onsdag 16 oktober 13
Query at any time
(q ’[:find ?email :where [_ :email ?email]] db)
onsdag 16 oktober 13
Query at any time
(q ’[:find ?email :where [_ :email ?email]] db)
(q ’[:find ?email :where [_ :email ?email]] (as-of db #inst "2013-09-02"))
onsdag 16 oktober 13
Query over time
(q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db)
onsdag 16 oktober 13
Query over time
(q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] db)
(q ’[:find ?e (count ?tx) :where [?e :email _ ?tx]] (history db))
onsdag 16 oktober 13
Java API
List<Object> results = Peer.q(someQuery, conn.db());
Future<Map> txResult = conn.transact(data_tx);
onsdag 16 oktober 13
More info
http://www.datomic.com/videos.html
onsdag 16 oktober 13
Comparison
onsdag 16 oktober 13
Comparison
Data model
Schema evolution
Queries
Recommendation
onsdag 16 oktober 13
EventStore Events
✓ Understandable by normal people
✓ Designed for integration
✓ Forces aggregate design (scalability)
✓ Capture intent
๏ Require design effort to get right
๏ Complect transaction and query
onsdag 16 oktober 13
Datomic Facts
✓ Support partial information
✓ Normalization support
✓ CRUD out of the box
๏ Static schema
onsdag 16 oktober 13
✓ New events
✓ New event attributes
✓ Event attributes rename (change deserialization)
✓ Projected state change (rebuild projection)
๏ Cross event refactoring
EventStore schema evolution
onsdag 16 oktober 13
✓ New or removed attributes
✓ New or removed entity types
✓Move attributes between entities
๏ Fact attributes rename not possible
๏ Fact attribute type change not possible
Datomic schema evolution
onsdag 16 oktober 13
✓ Projections use JavaScript
✓ Persistent
๏ Require projection
EventStore queries
onsdag 16 oktober 13
✓ Declarative
✓ Logic-based
✓ Allows calling out to your own code
๏ New language to learn
Datomic queries
onsdag 16 oktober 13
When to use Event Sourcing?
Domains where events seem natural
Different teams with different models
onsdag 16 oktober 13
When to use Datomic?
Data can be modelled using entities and attributes
History is or may be interesting
onsdag 16 oktober 13
More details
http://www.jayway.com/author/jankronquist/
https://github.com/jankronquist
http://www.slideshare.net/jankronquist
onsdag 16 oktober 13
Questions?
onsdag 16 oktober 13