introduction to nosql at silicon valley code camp

4. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 4 ALL RIGHTS RESERVED. A Short History of Data Application Specific Databases Size is paramount Relational Databases Size matters but break from the application silo and provide data integrity NoSQL Databases Agility Scalability Speed

6. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 6 ALL RIGHTS RESERVED. What's wrong with Relational? Nothing, it's perfect for square data ...where you know the relationships in advance ...where the schema doesn't change often ...where all the data can fit on one machine ...where a separate disk seek for every join isn't an issue

8. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 8 ALL RIGHTS RESERVED. The Rise of NoSQL 1998 - Carlo Strozzi coins term 2001 - MarkLogic founded 2003 - LiveJournal's Memcached 2003 - Google FileSystem paper 2004 - Google MapReduce paper 2006 - Google BigTable paper 2006 - Hadoop released to Apache 2007 - Amazon Dynamo paper 2009 - Eric Evans popularizes term 2010 - MongoDB (first production release)

10. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 10 ALL RIGHTS RESERVED. Memcached Developed at LiveJournal as a frontend cache for websites First released in 2003 Keep disk access at a minimum, pool memory on many machines So useful it found wide popularity, still under active development

11. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 11 ALL RIGHTS RESERVED. Memcached High Performance, Distributed Memory Object Caching System Distributed runs across many computers Memory runs without touching disk Object cache designed to hold small lumps of data High performance because it never touches disk, and the objects are small, its optimized for speed

12. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 12 ALL RIGHTS RESERVED. Memcached Client server system Servers are unaware of each other Clients determine server to use via hashing Servers keep content as an LRU cache So all data transitory

16. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 16 ALL RIGHTS RESERVED. Bigtable Created by Google in 2004 to store massive amounts of data Made public in famous 2006 paper Used throughout Google GMail, Google Maps, YouTube, Web Indexing, etc Reportedly over 100 internal projects Never shipped externally as a product but available for public use as part the AppEngine hosting API

17. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 17 ALL RIGHTS RESERVED. Bigtable Rows are composed of columns, which in turn belong to column families Column families are essentially typing, validation and expiration info Every cell is versioned via timestamp System is robust and crash resistant Can survive the crash of any machine, including the master Scale out architecture

21. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 22 ALL RIGHTS RESERVED. Hadoop First envisioned as Nutch at the Internet Archive in 2002 There were 100s of millions of webpages to index Early versions heavily influenced by Google File System, Map Reduce papers Goal: Perform work on large datasets using commodity machines Development moved to Yahoo in 2006 Open Sourced to Apache, as Hadoop A File system (HDFS) A Task Runner (MapReduce) A Task Manager (YARN) Note: Not a database

24. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 25 ALL RIGHTS RESERVED. Amazon Dynamo Created to power Amazons Web store Writing with low latency more important than consistency Techniques first made public in 2007 paper Never externally shipped but huge influence on market Used for a variety of critical portions of Amazons site Shopping cart User Session Succeeded by DynamoDB Similar name, but whole new architecture (with better consistency)

25. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 26 ALL RIGHTS RESERVED. Amazon Dynamo Distributed Key Value store "always writable low latency reads and writes, at the expense of consistency asynchronous replication on put() operations mean that get() may return a stale value updates during a network partition can result in conflicts and the application must handle them

28. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 29 ALL RIGHTS RESERVED. ACID Atomicity Everything either succeeds or fails Consistency Nothing is saved unless it passes consistency rules Isolation No two processes can interfere with each other Durability Once saved, data can not be lost due to system failure

30. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 34 ALL RIGHTS RESERVED. What happens without consistency? Absolute fastest performance at lowest hardware cost Highest global data availability at lowest hardware cost Working with one document or row at a time Writing advanced code to create your own consistency model Eventually consistent data Some inconsistent data that cant be reconciled Some missing data that cant be recovered Some inconsistent query results

33. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 37 ALL RIGHTS RESERVED. CAP Theory Consistent Always get correct answers, when you get answers Available Always get answers Partition Tolerant System can be divided in two Pick Two

41. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 45 ALL RIGHTS RESERVED. MemcacheDB Very early KV implementation (2008) KV Store based on Memcached source, with BerkleyDB persistent store Speaks the memcached protocol Development stopped (2009), but still quite popular For when you like Memcached, but want persistance

43. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 47 ALL RIGHTS RESERVED. Redis First released in 2009 Sponsored by VMWare, then Pivotal Name means Remote Dictionary Server Fully in memory key value store Whole db must reside in memory of one machine Limits scalability, at the benefit of performance Often used as a front end cache for other NoSQL databases

44. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 48 ALL RIGHTS RESERVED. Redis Not just strings as values: Lists of strings Sets of strings (collections of non-repeating unsorted elements) Sorted sets of strings (collections of non-repeating elements ordered by a floating-point number called score) Hashes where keys and values are strings

45. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 49 ALL RIGHTS RESERVED. Redis Master / slave replication - slave may be master to another slave allowing tree replication also publish/subscribe API slaves may be updated separately from master, allows inconsistencies (!) Persistent store Append only journal Flushed every 2 seconds by default

47. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 51 ALL RIGHTS RESERVED. Riak Basho was developing Sales Force automation software, came up with Riak, decided to make that their business instead First release 2011 Key Value store with CAP tunability, predictable latency but even Strong Consistency mode is not ACID Write conflicts handled at read time Handles JSON natively (via Search) Provides Map/Reduce engine in JavaScript Pluggable backend data store, can be an in memory implementation Search via Solr (though not realtime) Rental cloud service (Riak CS)

49. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 53 ALL RIGHTS RESERVED. Document vs Key Value Stores Extension of Key Value - the value is a document but also Structurally aware Indexed searches Document formats CouchDB JSON MongoDB BSON MarkLogic - JSON, XML

51. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 55 ALL RIGHTS RESERVED. MarkLogic The Only Enterprise NoSQL Database Founded in 2001 by search engine experts Multi-model database with built-in search and application services Flexible data model XML, JSON, RDF, Geospatial, text and binaries Scalable and elastic Clusters on commodity hardware at massive scale Trusted for mission-critical apps ACID, HA/DR, Security

52. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 56 ALL RIGHTS RESERVED. MarkLogic Features Built-in Search Scalability and Elasticity ACID Transactions Government-grade Security HA/DR Cloud Deployment Hadoop-ready

53. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 57 ALL RIGHTS RESERVED. MarkLogic Architecture Universal Index Index words, elements, the relationships of words and elements Many indexes (automatically) used at once In memory range indexes work like column indexes A native triple store that supports SPARQL Search on ranges, free text, field values, more Shared-nothing architecture, automatic partitioning and balancing Can also use a range index for tiered partitioning MarkLogic Connector for Hadoop

54. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 58 ALL RIGHTS RESERVED. XML vs JSON XML is more expressive, granular Namespaces Intermixed elements and text Standard query languages (XPath, XQuery) JSON is more compact, simpler (for better and worse) Programmatically accessible via standard JavaScript libraries

56. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 60 ALL RIGHTS RESERVED. MongoDB Development began in 2007 by 10gen Name from humongous Originally wanted to create a Google App Engine system 1.4 considered first production ready release, 2010 Horizontally scaling

57. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 61 ALL RIGHTS RESERVED. MongoDB Stores data in proprietary format BSON, similar to JSON with more data types Search on field, on range, or on regex Single index per query (secondary index optional) Replication of databases as master/slave, with (tunable) eventual consistency Sharding handled via a shard key, splitting by range Be sure the key is evenly distributed Client APIs in many (40+!) languages

59. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 63 ALL RIGHTS RESERVED. Column Stores Descended from Big Table approach Excellent for sparse data Column families need to be specified up front But still stored sparsely No way to list all the columns in the database Append only Updates via timestamp Deletes via tombstone marker

61. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 65 ALL RIGHTS RESERVED. Cassandra Developed at Facebook, 2008, donated to Apache Descended from Bigtable and Dynamo One of the primary Dynamo developers helped create Cassandra Focused on maximum throughput Write lots of data, fast But at the expense of consistency (tunable) Used by Twitter, Reddit, Netflix but not Facebook

62. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 66 ALL RIGHTS RESERVED. Cassandra Share nothing architecture Partitioned via hash (multiple strategies) Be careful choosing your Row Key! Async masterless replication CAP Tunable from "writes never fail" to "wait until persisted on all slaves Query with range queries, column family, CQL Hadoop support (replaces HDFS)

66. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 70 ALL RIGHTS RESERVED. Neo4J Released in 2010 Written in Java, APIs are Java centric Most popular Graph Database Powers the recommendation engines of Glassdoor, Walmart

67. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 71 ALL RIGHTS RESERVED. Neo4J Whole graph in memory scales to millions of relationships But does persist to disk Transactional Replicated for performance and robustness, master/slave Proprietary Graph query language (Cypher) Enterprise version adds clustering, sharding

69. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 73 ALL RIGHTS RESERVED. Semantics: A New Way to Organize Data Data is stored in Triples, expressed as: Subject : Predicate : Object John Smith : livesIn : London London : isIn : England Query with SPARQL, gives us simple lookup .. and more! Find people who live in (a place that's in) England "John Smith" "England" livesIn "London" isIn livesIn

70. COPYRIGHT 2014 MARKLOGIC CORPORATION. SLIDE: 74 ALL RIGHTS RESERVED. Context from the World at Large Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/ Linked Open Data Facts that are freely available In a form thats easily consumed DBpedia (wikipedia as structured information) Einstein was born in Germany Irelands currency is the Euro GeoNames Doha is the capital of Qatar Doha has these lat/long coordinates

introduction to nosql at silicon valley code camp

Technology