beyond relational databases: mongodb, redis & …...analytics/bi (bi connector on 3.2) time...
TRANSCRIPT
![Page 1: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/1.jpg)
Beyond Relational Databases: MongoDB, Redis & ClickHouse
Marcos Albe - Principal Support Engineer @ Percona
![Page 2: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/2.jpg)
Introduction
MySQL everyone?
![Page 3: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/3.jpg)
Introduction
Redis?
![Page 4: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/4.jpg)
OLAP -vs- OLTP
Image credits: 451 Research (https://451research.com/state-of-the-database-landscape)
![Page 5: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/5.jpg)
Agenda● Introduction
● Why not Relational Databases?
● Redis: Key-Value
● MongoDB: Document
● ClickHouse: Columnar
![Page 6: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/6.jpg)
Why not Relational Databases?
![Page 7: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/7.jpg)
Why not Relational Databases?
● General purpose but not optimal for all purposes
● Impedance mismatch with developers
● ACID/locking induced latency
● Storage architecture not optimal for big data
![Page 8: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/8.jpg)
Agenda● Introduction
● Why not Relational Databases?
● Redis: Key-Value
● MongoDB: Document
● ClickHouse: Columnar
![Page 9: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/9.jpg)
![Page 10: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/10.jpg)
![Page 11: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/11.jpg)
Redis highlights● Fast… very fast.
● Mature / large community
● Many data structures
● Advanced features
● Horizontally scalable / built-in HA (Sentinel / Cluster)
● BSD license / Commercial licenses available
● Client libraries for about every programming language
![Page 12: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/12.jpg)
Redis data types● Lists
● Sets
● Sorted sets
● Hashes
● Bitmaps
● Geohash
![Page 13: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/13.jpg)
● Lots of data
● High concurrency
● Massive small-data intake
● Simple data access patterns
● Session Cache / Full page cache
● Counters / Leaderboards
● Queues
Redis good use cases
![Page 14: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/14.jpg)
Redis bad Use Cases● Durability and consistency
● Complex data access patterns
● Non-PK access
● Security concerns
● Operations
![Page 15: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/15.jpg)
PROs CONs● Fast
● Highly scalable/available
● Simple access patterns
● Advanced data types
● Advanced features for KV store
○ Atomic operations
● Ubiquitous / large community
● Lower durability
● Operational complexities
● Limited access patterns
● Lack of security
● No secondary keys
![Page 16: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/16.jpg)
Agenda● Introduction
● Why not Relational Databases?
● Redis: Key-Value
● MongoDB: Document
● ClickHouse: Columnar
![Page 17: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/17.jpg)
![Page 18: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/18.jpg)
MongoDB Flexible Schema
![Page 19: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/19.jpg)
MongoDB Flexible Schema
![Page 20: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/20.jpg)
● Embedded in blogpost document (natural case)
● Embedded in user document (bad idea)
● Denormalize and keep duplicate data (VERY bad idea)
● Apply normalization and user lookup() (JOIN equivalent)
MongoDB Flexible Schema
![Page 21: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/21.jpg)
![Page 22: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/22.jpg)
Document Stores: Flexible Schema
![Page 23: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/23.jpg)
Document Stores: Flexible Schema
![Page 24: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/24.jpg)
MongoDB highlights● Sharding and replication for dummies!● Flexible schema● Multi-document transactions (new in 4.0)● Pluggable storage engines for distinct workloads.● Excellent compression options with PerconaFT, RocksDB, WiredTiger● On disk encryption (Enterprise Advanced)● Connectors for all major programming languages● Sharding and replica aware connectors● Geospatial functions● Aggregation framework
![Page 25: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/25.jpg)
● Catalogs
● Analytics/BI (BI Connector on 3.2)
● Time series
● Metadata repositories
● Prototype Development
MongoDB good use cases
![Page 26: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/26.jpg)
● Recursiveness
● Multiple views of the data
● Developer comfort
MongoDB bad use cases
![Page 27: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/27.jpg)
● Inefficiency for JOINs● Still immature internally● Attribute names bloat space● MMAP db-level locking
PROs CONs
● Fast● Easy sharding● Simple access patterns● Rich feature set ● Async connectors● Flexible schema
![Page 28: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/28.jpg)
Agenda● Introduction● Why not Relational Databases?● Redis: Key-Value● MongoDB: Document● ClickHouse: Columnar
![Page 29: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/29.jpg)
![Page 30: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/30.jpg)
Columnar Data Layout
Row-oriented Column-oriented
001:10,Smith,Joe,40000;
002:12,Jones,Mary,50000;
003:11,Johnson,Cathy,44000;
004:22,Jones,Bob,55000;
...
10:001,12:002,11:003,22:004;
Smith:001,Jones:002,Johnson:003,Jones:004;
Joe:001,Mary:002,Cathy:003,Bob:004;
40000:001,50000:002,44000:003,55000:004;
...
![Page 31: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/31.jpg)
Columnar Data Layout
Row-oriented Read Approach
What we want to read
Read Operation
Memory Page
1 2
3
4
10 Smith Bob 40000
12 Jones Mary 50000
11 Johnson Cathy 44000
![Page 32: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/32.jpg)
Columnar Data Layout
What we want to read
Read Operation
Memory Page
1 2
3
4
10 12 11 22
Smith Jones Johnson
Joe Mary Cathy Bob
Column-oriented Read Approach
![Page 33: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/33.jpg)
ClickHouse: columnar store
woot!
![Page 34: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/34.jpg)
ClickHouse highlights● True column oriented
● Compression
● Disk based (not in-memory)
● Parallelized
● Distributed
● SQL querying; ○ Improved JOINs in latter versions
● Vector engine
● Real-time updates
![Page 35: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/35.jpg)
ClickHouse highglights● Indexes● Dictionaries● Online queries● On-disk GROUP BYs● Approximated calculations● Replication ● Few, but popular connectors● Support for Tableau
![Page 36: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/36.jpg)
ClickHouse limitations● Non-standard SQL ● UPDATE/DELETE is a bulk operation (ALTER … UPDATE)● No windowing functions ● No transactions or constraints● Eventual consistency● Time data types have no milliseconds● No implicit type conversions● Impossible to do table partitioning (except by month)● Lack of enterprise operation tools
![Page 37: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/37.jpg)
● Suitable for read-mostly or read-intensive, large data repositories
● Full table / large range reads.
● Time series data
● Unstructured problems where “good” indexes are hard to forecast
● Log analysis
● Re-creatable datasets
ClickHouse good use cases
![Page 38: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/38.jpg)
● Not good for “SELECT *” queries or queries fetching most of the columns
● Not good for "small" writes; Good for bulk writes
● Not good for mixed read/write; Focus on large reads and large inserts
● Bad for unstructured data
ClickHouse bad use cases
![Page 39: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/39.jpg)
● VERY fast
● Distributed
● Compression
● Primary-keys
● On-disk GROUP BY
PROs CONs● Non-standard SQL
● Few connectors
● Driven mostly by Yandex needs
● No partitioning
![Page 40: Beyond Relational Databases: MongoDB, Redis & …...Analytics/BI (BI Connector on 3.2) Time series Metadata repositories Prototype Development MongoDB good use cases Recursiveness](https://reader034.vdocument.in/reader034/viewer/2022050203/5f954c80c314da7d5e6192bc/html5/thumbnails/40.jpg)