marklogic and the universal index
DESCRIPTION
Talk about how MarkLogic Server uses a inverted index (like search engines) to optimize this document oriented NoSQL databaseTRANSCRIPT
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
Beyond NoSQL with MarkLogicThe Universal IndexMarkLogic Developer CommunityNoSQL Frankfurt, 2010
Awesome document-oriented NoSQL database
and
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
nuno job
@dscape | nunojob.com
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
3
Stru
ctur
e
Ad h
ocPr
edefi
ned
Queries
Ad hocPredefined
IDMS
how??
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
Indexes!
indexes!
so… filter map reduce !?
well… sort of…
flickr.com/ayalan
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
partition2 partition3partition1
divide and conquer
level of abstraction: ease of use
consistent-hashing-like thingy
standa group of trees
makes sense to
have indexes in the same place
database
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
6
E Host 1
partition1
E Host 3
D Host 4 D Host 5 D Host 6 D Host k
partition2 partition3 partitionm
E Host 2
partition4
shared-nothing
cluster1st index resolution2nd get documents
HA&DR
AppServer
Data
Same Code-base
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
123, 127, 129, 152, 344, 791 . . .
122, 125, 126, 129, 130, 167 . . .
123, 126, 130, 142, 143, 167 . . .
123, 130, 131, 135, 162, 177 . . .
126, 130, 167, 212, 219, 377 . . .
. . .
. . .
Document References
126, 130, 167, …
Term Term List
Range Indexes
“accelerating”
“creation”
“content”
“application”
“agility”
<article>
<article> / <title>
product: MarkLogic
universal index
Geospatial
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
8
article
title
footer
paragraph
paragraph
un-ordered list
table
semi structured
get tables from computer science articles that include a title with word “content” but not the word “agility”
information
metadata
structure
parentchild
full text
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
123, 127, 129, 152, 344, 791 . . .
122, 125, 126, 129, 130, 167 . . .
123, 126, 130, 142, 143, 167 . . .
123, 130, 131, 135, 162, 177 . . .
126, 130, 167, 212, 219, 377 . . .
122, 125, 126, 129, 130, 143, 167
122, 125, 126, 129, 130, 167 . . .
Document References
126, 130, 167, …
Term Term List
Range Indexes
“accelerating”
“creation”
“content”
“application”
“agility”
<article>
<article> / <title>
product: MarkLogic
universal index
Geospatial
in kelly speak: zippy-ing
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
DirectoriesExclusive, hierarchical, analogous to file
system, map to URI
CollectionsSet-based, N:N relationship
SecurityInvisible to your app
wait a minute…
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
123, 127, 129, 152, 344, 791 . . .
122, 125, 126, 129, 130, 167 . . .
123, 126, 130, 142, 143, 167 . . .
123, 130, 131, 135, 162, 177 . . .
126, 130, 167, 212, 219, 377 . . .
. . .
. . .
Document References
126, 130, 167, …
Term Term List
Range Indexes
“accelerating”
“creation”
“content”
“application”
“data base”
<article>
<article> / <title>
product: MarkLogic
Directory: /articles/
Collection: CS
Role:Editor + Action:Read
universal index
Geospatial
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
throughput
flickr.com/kt
in memory stand(s)
durability: journal
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
john.xml
maria.xml
mary.xml
eric.xml
0 5 10 15 20 25
Series 1
Systemtimestamp
delete
create
query
update(could also be create)
mvccand the marklogic time
machine
append only database, use sys-timestampsto know which document is currently
available
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
14
too good to be true?try us out… free version
available!
flickr.com/nattu
markmail.org
pairs.demo.marklogic.com
heatmap.demo.marklogic.com
bit.ly/ml-demo
developer.marklogic.com/products
questions?
Love NoSQL databases?
Want to change the world
?
We are hiring!!
spkr8.com/t/4590
Feedback
the universal index @dscape #nosqlfrahttp://bit.ly/ml-fra
XQuery and why it’s awesome!
XML vs. JSON ?
MVCC
Scalable ACID transactions
AlertingReverse Indexes
Geospatial queries
App Server + Search + Database
High Order Functions
REST
Merging / Compaction
Co-occurrenceMeta programmingDocument databases
not coveredbut conversations are
welcome!
Relevance
Mobile Open-source, closed development?