the database as a value - goto...
TRANSCRIPT
![Page 1: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/1.jpg)
The Database as a ValueRich Hickey
![Page 2: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/2.jpg)
What is Datomic?
• A new database
• Bringing data power into the application
• A sound model of information, with time
• Enabled by architectural and capacity advances
![Page 3: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/3.jpg)
Complexity
• Out of the Tar Pit
Moseley and Marks (2006)
• Complexity caused by state and control
• Close the loop - process
![Page 4: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/4.jpg)
Information
• Inform
• ‘to convey knowledge via facts’
• ‘give shape to (the mind)’
• Information
• the facts
![Page 5: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/5.jpg)
DB Complexity
• Stateful, inextricably
• Same query, different results
• no basis
• Over there
• ‘Update’ poorly defined
• Places
![Page 6: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/6.jpg)
Basis
• Calculation and decision making:
may involve multiple components
may visit a component more than once
• Broken by simultaneous change
![Page 7: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/7.jpg)
Update
• What does update mean?
• Does the new replace the old?
• Granularity? new ___ replace the old ___
• Visibility?
![Page 8: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/8.jpg)
Manifestations
• Wrong programs
• Scaling problems
• Round-trip fears
• Fear of overloading server
• Coupling, e.g. questions with reporting
![Page 9: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/9.jpg)
The Choices
• Coordination
• how much, and where?
• process requires it
• perception shouldn’t
• Immutability
• sine qua non
![Page 10: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/10.jpg)
Coming to TermsValue
• An immutable magnitude, quantity, number... or immutable composite thereof
Identity
• A putative entity we associate with a series of causally related values (states) over time
State
• Value of an identity at a moment in time
Time
• Relative before/after ordering of causal values
![Page 11: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/11.jpg)
v1
F
v2
F
v3
F
v4
Process events (pure functions)
Observers/perception/memory
States (immutable values)Identity
(succession of states)
Epochal Time Model
![Page 12: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/12.jpg)
Implementing Values
• Persistent data structures
• Trees
• Structural sharing
![Page 13: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/13.jpg)
Structural Sharing
PastNext
![Page 14: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/14.jpg)
Process events (pure functions)
Observers/perception/memory
Identity (succession of
states)
Place Model
DB Connection
Transactions
Queries
The Database Place
F F F
![Page 15: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/15.jpg)
v1
F
v2
F
v3
F
v4
Process events (pure functions)
Observers/perception/memory
States (immutable values)Identity
(succession of states)
Epochal Time Model
DB Connection
Transactions
DB Values
Queries
![Page 16: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/16.jpg)
Database State
• The database as an expanding value
• An accretion of facts
• The past doesn’t change - immutable
• Process requires new space
• Fundamental move away from places
![Page 17: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/17.jpg)
Accretion• Root per transaction doesn’t work
• Crossing processes and time
• Can’t convey/find/maintain roots
• Can’t do global GC
• Instead, latest values include past as well
• The past is sub-range
• Important for information model
![Page 18: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/18.jpg)
• Fact - ‘an event or thing known to have happened or existed’
• From: factum - ‘something done’
• Must include time
• Remove structure (a la RDF)
• Atomic Datom
• Entity/Attribute/Value/Transaction
Facts
![Page 19: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/19.jpg)
Process
• Reified
• Primitive representation of novelty
• Assertions and retractions of facts
• Minimal
• Other transformations expand into those
![Page 20: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/20.jpg)
Implementation
![Page 21: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/21.jpg)
Deconstruction
• Process
• Transactions
• Indexing
• O
• Perception/Reaction
• Query
• Indexes
• I
Server
IndexingTrans-actions
Query I/O Disk
![Page 22: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/22.jpg)
State
• Must be organized to support query
• Sorted set of facts
• Maintaining sort live in storage - bad
• BigTable - mem + storage merge
• occasional merge into storage
• persistent trees
![Page 23: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/23.jpg)
Transactions and Indexing
IndexMerging
Trans-actions
Log Data Segments
Live Index
Index Data Segments
Storage
Novelty
![Page 24: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/24.jpg)
Perception
Live Index Storage
Index Data Segments
Novelty
![Page 25: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/25.jpg)
Datomic ArchitectureApp Process
Peer Lib
Query
Cache
App
Live IndexComm
Transactor
Indexing Trans-actions
Data Segments
Data SegmentsRedundant
segment storage
Storage Service
Segment storage
![Page 26: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/26.jpg)
Memory Index
• Persistent sorted set
• Large internal nodes
• Pluggable comparators
• 2 sorts always maintained
• EAVT, AEVT
• plus AVET, VAET
![Page 27: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/27.jpg)
Storage
• Log of tx asserts/retracts (in tree)
• Various covering indexes (trees)
• Storage requirements
• Data segment values (K->V)
• atoms (consistent read)
• pods (conditional put)
![Page 28: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/28.jpg)
What’s in a DB Value?
EAVT
tVeAETAEVT
db atom
nextTasOfT
Lucene index
history
live Lucene
sinceT
index
db valuelive Storage
Hierarchical Cache
Roots
Memory index(live window)
Storage-backed index
Identity
Value
![Page 29: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/29.jpg)
Index Storage
SortedDatoms
Index Root of key->dir
T42
VeAETAEVT AVET LuceneEAVT
StorageService
dirs
segs
![Page 30: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/30.jpg)
Process
• Assert/retract can’t express transformation
• Transaction function:
(f db & args) -> tx-data
• tx-data: assert|retract|(tx-fn args...)
• Expand/splice until all assert/retracts
![Page 31: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/31.jpg)
Process Expansion
++ ++foo- -
baz++ ++bar- -
...+++ -++ +++- -
![Page 32: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/32.jpg)
Transactor
• Accepts transactions
• Expands, applies, logs, broadcasts
• Periodic indexing, in background
• Indexing creates garbage
• Storage GC
![Page 33: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/33.jpg)
Peers
• Peers directly access storage service
• Have own query engine
• Have live mem index and merging
• Two-tier cache
• Segments (on/off heap)
• Datoms w/object values (on heap)
![Page 34: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/34.jpg)
DB Simplicity
• Epochal state
• Coordination only for process
• Same query, same results
• stable bases
• Transactions well defined
• Functional accretion
![Page 35: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/35.jpg)
Other Benefits
• Communicable, recoverable basis
• Freedom to relocate/scale storage, query
• Time travel - db.asOf, db.since, db.asIf
• Queries comparing times
• Process events
![Page 36: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/36.jpg)
The Database as a Value
• Dramatically less complex
• More powerful
• More scalable
• Better information model
![Page 37: The Database as a Value - GOTO Bloggotocon.com/dl/goto-chicago-2013/gotonightslides/dbasvalue.pdf · magnitude, quantity, number... or immutable composite thereof Identity ... •](https://reader031.vdocument.in/reader031/viewer/2022022010/5b02e9ba7f8b9a8c688b593f/html5/thumbnails/37.jpg)
Thanks for Listening!