writing datomic in clojure -...
TRANSCRIPT
![Page 1: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/1.jpg)
Writing Datomic in ClojureRich Hickey
Datomic, Clojure
![Page 2: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/2.jpg)
Overview
• What is Datomic?
• Architecture
• Implementation - Clojure Applied
• Summary
![Page 3: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/3.jpg)
What is Datomic?
• A new kind of database
• Bringing data power into the application
• A sound model of information, with time
• Enabled by architectural advances
![Page 4: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/4.jpg)
Why Datomic?
• Architecture
• Data Model
![Page 5: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/5.jpg)
Architectures
QueriesTransactionsConsistencyStorage
Server
AppAppApp
App
App
App AppApp
Client-Server
Server
AppAppApp
App
App
App AppApp
Client-Server
![Page 6: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/6.jpg)
Architectures
QueriesTransactionsConsistencyStorage
ServerServer
AppAppApp
App
App
App AppApp
Clustered Client-Server
ServerServer
AppAppApp
App
App
App AppApp
Clustered Client-Server
![Page 7: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/7.jpg)
Architectures
QueriesTransactionsConsistencyStorage
AppAppApp
App
App
App AppApp
ServerServerServer
Sharded Client-Server
AppAppApp
App
App
App AppApp
ServerServerServer
Sharded Client-Server
![Page 8: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/8.jpg)
Architectures
QueriesTransactionsConsistencyStorage
AppAppApp
App
App
App AppApp
ServerServerServer
Sharded Client-Server
AppAppApp
App
App
App AppApp
ServerServerServer
Sharded Client-ServerQueriesTransactionsConsistencyStorage
![Page 9: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/9.jpg)
Architectures
QueriesTransactionsConsistencyStorage
AppAppApp
App
App
App AppApp
ServerServerServer
K/V Store
AppAppApp
App
App
App AppApp
ServerServerServer
K/V StoreQueriesTransactionsConsistencyStorage
![Page 10: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/10.jpg)
Architectures
QueriesTransactionsConsistencyStorage
AppAppApp
App
App
App AppApp
ServerServerServer
K/V Store
AppAppApp
App
App
App AppApp
ServerServerServer
K/V StoreQueriesTransactionsConsistencyStorage
QueriesTransactionsConsistencyStorage
![Page 11: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/11.jpg)
Architectures
QueriesTransactionsConsistencyStorage
AppAppApp
App
App
App AppApp
ServiceStorageDistributed
K/V Store
AppAppApp
App
App
App AppApp
ServiceStorageDistributed
K/V StoreQueriesTransactionsConsistencyStorage
QueriesTransactionsConsistencyStorage
![Page 12: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/12.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
![Page 13: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/13.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
QueriesTransactionsConsistencyStorage
![Page 14: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/14.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
QueriesTransactionsConsistencyStorage
![Page 15: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/15.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
Transactor
QueriesTransactionsConsistencyStorage
![Page 16: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/16.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
Transactor
QueriesTransactionsConsistencyStorage
QueriesTransactionsConsistencyStorage
![Page 17: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/17.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
Transactor
QueriesTransactionsConsistencyStorage
QueriesTransactionsConsistencyStorage
![Page 18: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/18.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
Transactor
QueriesTransactionsConsistencyStorage
QueriesTransactionsConsistencyStorage
![Page 19: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/19.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
Transactor
QueriesTransactionsConsistencyStorage
QueriesTransactionsConsistencyStorage
![Page 20: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/20.jpg)
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
AppApp
ServiceStorageDistributed
App
App
AppAppAppApp
Datomic Architecture
Transactor
QueriesTransactionsConsistencyStorage
QueriesTransactionsConsistencyStorage
![Page 21: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/21.jpg)
The Database, Deconstructed
Traditional DB Datomic
Storage Service
App ProcessD Peer Lib
b,c,ea,d,e a,b,d
D Transactor
Indexing Trans-actions
Query
Cache
App Data
Data
Datasegments
Live Index
Data Segments
Data Segments
Server
Indexing
Trans-actions
Query
App Process
I/O
App
StringsDDL + DML
Result Sets
Storage
cache
![Page 22: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/22.jpg)
Designed for the Cloud
• Ephemeral instances, unreliable disks
• Redundancy in storage service
• Leverages reliable storage services
• e.g. DynamoDB
![Page 23: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/23.jpg)
Elastic Scaling
• More peers, more power
• Fewer peers, less power, lower cost
• Demand-driven
• No configuration
![Page 24: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/24.jpg)
Get Your Own Brain
• Query, communication and memory engine
• Goes into your app, making it a peer
• The db is effectively local
• Ad hoc, long running queries - ok
![Page 25: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/25.jpg)
Logic
• Declarative search and business logic
• The query language is Datalog
• Simple rules and data patterns
• Joins are implicit, meaning is evident
• db and non-db sources
![Page 26: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/26.jpg)
Perception
• Obtain a queue of transactions
• not just your own
• Query transactions for filtering/triggering
![Page 27: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/27.jpg)
Consistency
• ACID transactions add new facts
• Database presented to app as a value
• Data in storage service is immutable
![Page 28: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/28.jpg)
Programmability
• Transactions/Rules/Queries/Results are data
• Extensible types, predicates, etc
• Queries can invoke your code
![Page 29: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/29.jpg)
A Database of Facts
• A single storage construct, the datom
• Entity/Attribute/Value/Transaction
• Attribute definition is the only 'schema'
![Page 30: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/30.jpg)
Adaptability
• Sparse, irregular, hierarchical data
• Single and multi-valued attributes
• No structural rigidity
![Page 31: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/31.jpg)
Time Built-in
• Every datom retains its transaction
• Transactions are totally ordered
• Transactions are first-class entities
• Get the db as-of, or since, a point in time
![Page 32: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/32.jpg)
Implementation
![Page 33: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/33.jpg)
ArchitectureApp Process
Peer Lib
Query
Cache
App
Live IndexComm
TransactorAMI
Indexing Trans-actions
Data Segments
Storage Service (Dynamo DB)
SSDSSD
Data SegmentsRedundant
segment storage
![Page 34: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/34.jpg)
State• Immutable, expanding value
• Must be organized to support query
• Sorted set of facts
• Maintaining sort live in storage - bad
• BigTable - mem + storage merge
• occasional merge into storage
• persistent trees
![Page 35: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/35.jpg)
Memory Index• New persistent sorted set
• Large internal nodes
• Pluggable comparators
• 2 sorts always maintained
• EAVT, AEVT
• plus AVET, VAET
![Page 36: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/36.jpg)
Storage• Log of tx asserts/retracts (in tree)
• Various covering indexes (trees)
• Storage requirements
• Data segment values (K->V)
• atoms (consistent read)
• pods (conditional put)
![Page 37: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/37.jpg)
Index Storage
SortedDatoms
Index Root of key->dir
T42
VeAETAEVT AVET LuceneEAVT
StorageService
dirs
segs
![Page 38: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/38.jpg)
What’s in a DB Value?
EAVT
tVeAETAEVT
db atom
Lucene index
history
live Lucene
sinceTasOfT
key->ididsid->keykeys
indexbeing-indexed
db valuelive Storage
Hierarchical Cache
Roots
Memory index(live window)
Storage-backed index
![Page 39: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/39.jpg)
Process
• Assert/retract can’t express transformation
• Transaction function:
(f db & args) -> tx-data
• tx-data: assert|retract|(tx-fn args...)
• Expand/splice until all assert/retracts
![Page 40: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/40.jpg)
Process Expansion++ ++foo- -
baz++ ++bar- -
...+++ -++ +++- -
![Page 41: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/41.jpg)
Transactor
• Accepts transactions
• Expands, applies, logs, broadcasts
• Periodic indexing, in background
• Indexing creates garbage
• Storage GC
![Page 42: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/42.jpg)
Transactor Implementation
• HornetQ for transaction communication
• Extensive internal pipelining - j.u.c. queues
• Async message decompression
• transaction expansion/application
• encoding for, communication with storage
• Java interop to storage APIs
![Page 43: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/43.jpg)
Indexing
• Extensive use of laziness
• Parallel processing
• Parallel I/O
• Async, rejoins via queue
![Page 44: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/44.jpg)
Declarative Programming
• Embedded Datalog
• Takes data sources and rule sets as args
• Extended to work with scalars/collections
• Expression clauses call your code
![Page 45: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/45.jpg)
Datalog Implementation
• Data driven, in and out
• Query/Subquery Recursive (QSQR)
• Dynamic, set oriented
• DB joins leverage indexes
• Expressions use Clojure compiler
• caching of transforms at all stages
![Page 46: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/46.jpg)
Over Here• Peers directly access storage service
• Have own query engine
• Have live mem index and merging
• Two-tier cache
• Segments (on/off heap)
• Datoms w/object values (on heap)
![Page 47: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/47.jpg)
Peer Implementation• HornetQ for transaction communication
• Google Guava caches
• Java APIs for storage
• Entities are like multimaps
• key -> value(s)
• reverse attrs
![Page 48: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/48.jpg)
Consistency and Scale• Process/writes go through transactor
• traditional server scaling/availability
• Immutability supports consistent reads
• without transactions
• scale reads turning knobs on storage
• Query scales with peers
• dynamic e.g. auto-scaling
![Page 49: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/49.jpg)
Testing
• test.generative was born here
• Functional tests
• Simulation-based testing
![Page 50: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/50.jpg)
Simplicity is Agility• Key protocols extremely small (< 7 fns)
• Memory, embedded SQL, remote SQL, Infinispan, DynamoDB
• Move from our own dynamo cluster to DynamoDB:
• 2 weeks
• Support PostgreSQL, Infinispan
• 1 day each
![Page 51: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/51.jpg)
Leverage✓Read/print data
✓Embedded language
✓Runtime compilation
✓Extend standard interfaces/protocols
✓Interop
✓State model - extended
![Page 52: Writing Datomic in Clojure - gotocon.comgotocon.com/dl/goto-cph-2012/slides/clojure/datomic-in-clojure.pdf · • Bringing data power into the application ... Why Datomic? • Architecture](https://reader034.vdocument.in/reader034/viewer/2022043011/5fa42416acb92b4b7d320ea0/html5/thumbnails/52.jpg)
Summary• Clojure was made for this kind of app
• Fast enough at all levels
• Most key subsystems < 1000 lines
• A ton of concurrency, no sweat
• Leverage interop - Hornetq, Guava etc
• Startup time could be better
• Datomic is Simple