1 / 120 - databases · [x] document model [x] transactions operations: [x] performance...

1 / 120

2 / 120

MySQL 8.0

a Document Store with all the benefits of a transactional RDBMS

Frédéric Descamps - MySQL Community Manager - Oracle

3 / 120

Save the date !

Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.

4 / 120

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purpose only, and may notbe incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not berelied up in making purchasing decisions. The development, release and timing of any features or functionality describedfor Oracle´s product remains at the sole discretion of Oracle.


5 / 120

about me - http://about.me/lefred

Who am I ?


6 / 120

http://about.me/lefred

Frédéric Descamps@lefredMySQL EvangelistHacking MySQL since 3.23devops believerliving in Belgium 🇧🇪http://lefred.be


7 / 120

http://lefred.be/

RDBMS & MySQL

Relational Databases


8 / 120

Relational DatabasesData Integrity


9 / 120


normalization


10 / 120


normalizationconstraints (foreign keys, ...)


11 / 120



Atomicity, Consistency, Isolation, Durability


12 / 120



Atomicity, Consistency, Isolation, DurabilityACID compliant


13 / 120



Atomicity, Consistency, Isolation, DurabilityACID complianttransactions


14 / 120




SQL


15 / 120




SQLpowerfull query language


16 / 120

NoSQL & MySQL

NoSQL Databases


17 / 120

NoSQL or Document StoreSchemaless


18 / 120


no schema design, no normalization, no foreign keys, no data types, ...


19 / 120


no schema design, no normalization, no foreign keys, no data types, ...very quick initial development


20 / 120



Flexible data structure


21 / 120



Flexible data structureembedded arrays or objects


22 / 120



Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational model


23 / 120



Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational modelobjects persistance without the use of any ORM - mapping oobject-oriented


24 / 120




JSON


25 / 120




JSONclose to frontend


26 / 120




JSONclose to frontendnative in JS


27 / 120




JSONclose to frontendnative in JSeasy to learn


28 / 120

How DBAs see data


29 / 120

How DBAs see data How Developers see data{ "GNP" : 249704, "Name" : "Belgium", "government" : { "GovernmentForm" : "Constitutional Monarchy, Federation", "HeadOfState" : "Philippe I" }, "_id" : "BEL", "IndepYear" : 1830, "demographics" : { "Population" : 10239000, "LifeExpectancy" : 77.8000030517578 }, "geography" : { "Region" : "Western Europe", "SurfaceArea" : 30518, "Continent" : "Europe" }}


30 / 120

What if there was a way to provide both SQL and NoSQL on onestable platform that has proven stability on well know

technology with a large Community and a diverse ecosystem ?


31 / 120

DBMS or NoSQL ?


32 / 120

DBMS or NoSQL ?

Why not both ?


33 / 120

The MySQL Document Store !

SQL is now optional !


34 / 120

Developers:

[x] schemaless[x] rapid prototying/simpler APIs[x] document model[x] transactions

Operations:

[x] performance management/visibility[x] robust replication, backup, restore[x] comprehensive tooling ecosystem[x] simpler application schema upgrades

Business Owner:

[x] don't lose my data == ACID trx[x] capture all my data = extensible/schemaless[x] product on schedule/time to market = rapiddevelopement

A solution for all


35 / 120

MySQL Document Store

the Solution


36 / 120


37 / 120

Built on theMySQL JSONData type andProven MySQLServerTechnology

Provides a schema flexible JSON Document StoreNo SQL requiredNo need to define all possible attributes, tables, etc.Uses new X DevAPICan leverage generated column to extract JSON values into materialized columnsthat can be indexed for fast SQL searches.Document can be ~1GB

It's a column in a row of a tableIt cannot exceed max_allowed_packet

Allows use of modern programming stylesNo more embedded strings of SQL in your codeEasy to read

Also works with relational TablesProven MySQL Technology


38 / 120

X DevAPI Connectors forC++, Java, .Net, Node.js, Python, PHPworking with Communities to help them supporting it too

New MySQL ShellCommand CompletionPython, JavaScrips & SQL modesAdmin functionsNew Util objectA new high-level session concept that can scale from single MySQL Server toa multiple server environment

Non-blocking, asynchronous calls follow common language patternsSupports CRUD operations


39 / 120

Starting using MySQL DS in few minutes


40 / 120

MySQL Shell Info


41 / 120

Migration from MongoDB to MySQL DSFor this example, I will use the well known restaurants collection:


42 / 120


We need to dump the data to a file and we will use MySQL Shell with the Python interpreter to load the data.


43 / 120


We need to dump the data to a file and we will use MySQL Shell with the Python interpreter to load the data.


44 / 120

Dump and load using MySQL Shell & Pythonthis example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore

$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000; db.restaurants.�nd().shellPrint()' \ | perl -pe 's/(?:ObjectId|ISODate)$("[^"]+")$/ $1/g' > all_recs.json


45 / 120

https://www.slideshare.net/datacharmer/mysql-documentstore

Dump and load using MySQL Shell & Pythonthis example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore

$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000; db.restaurants.�nd().shellPrint()' \ | perl -pe 's/(?:ObjectId|ISODate)$("[^"]+")$/ $1/g' > all_recs.json


46 / 120

https://www.slideshare.net/datacharmer/mysql-documentstore


47 / 120

Let´s query


48 / 120

Let´s query

That´s too much records to show it here... let´s limit it


49 / 120


50 / 120

Some more examples


51 / 120

Some more examples

Let´s add a selection criteria:


52 / 120

Using IN...


53 / 120

Syntax slightly different than MongoDB


54 / 120

Syntax slightly different than MongoDB

> db.restaurants.�nd({"cuisine": "French", "borough": { $not: /^Manhattan/} }, {"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2){ "borough" : "Queens", "cuisine" : "French",


55 / 120

CRUD operations


56 / 120

CRUD operations for collections

Add a document


57 / 120


Modify a document


58 / 120


Remove a document


59 / 120

MySQL Document Store Objects Summary


60 / 120

Document Store Full ACID !It relies on the proven MySQL InnoDB´s strength & robustness:


61 / 120


innodb_�ush_log_at_trx_commit = 1


62 / 120



innodb_doublewrite = ON


63 / 120




sync_binlog = 1


64 / 120




sync_binlog = 1

transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...


65 / 120




sync_binlog = 1

transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...

We do care about your data !


66 / 120

Document Store Full ACID !

MySQL Document Store supports transactions


67 / 120

Document Store Full ACID !

MySQL Document Store supports transactions


68 / 120


69 / 120

what about my old SQL ?

The hidden part of the iceberg


70 / 120

JSON datatype is behind the scenenative datatype (since 5.7.8)


71 / 120

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4


72 / 120

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON values


73 / 120

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)


74 / 120

JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)Generated/virtual columns


75 / 120


Indexing JSON data


76 / 120


Indexing JSON dataForeign Keys to JSON data


77 / 120


Indexing JSON dataForeign Keys to JSON dataSQL Views to JSON data


78 / 120

How does the MySQL Document Store work ?


79 / 120

What does a collection look like on the server ?


80 / 120

_id Every document has a unique identifier called the document ID, which can be thoughtof as the equivalent of a table´s primary key. The document ID value can be manuallyassigned when adding a document. If no value is assigned, a document ID isgenerated and assigned to the document automatically !

Use getDocumentId() or getDocumentIds() to get _ids(s)


81 / 120

Mapping to SQL examplescreateCollection('mycollection')


82 / 120


CREATE TABLE `test`.`mycoll` ( doc JSON, _id VARCHAR(32) GENERATED ALWAYS AS (doc->>'$._id') STORED PRIMARY KEY ) CHARSET utf8mb4;


83 / 120



mycollection.add({'test': 1234})


84 / 120



mycollection.add({'test': 1234})

INSERT INTO `test`.`mycoll` (doc) VALUES ( JSON_OBJECT('_id','663807fe367ee6114e0e5458bdac28bf', 'test',1234) );


85 / 120

Mapping to SQL examples (2)mycollection.find("test > 100")


86 / 120

Mapping to SQL examples (2)mycollection.find("test > 100")

SELECT doc FROM `test`.`mycoll` WHERE (JSON_EXTRACT(doc,'$.test') > 100);


87 / 120

SQL and JSON Example


88 / 120


same as:


89 / 120


same as:

we can use it in SQL:


90 / 120

Indexes in Document StoreIt's also possible to create indexes without using SQL syntax:


91 / 120

Indexes in Document StoreIt's also possible to create indexes without using SQL syntax:


92 / 120

SQL and JSON Example (2): validation


93 / 120



94 / 120



95 / 120

SQL and JSON Example (3): explain


96 / 120

SQL and JSON Example (3): explain


97 / 120

SQL and JSON Example (4): add index


98 / 120

SQL and JSON Example (4): add index


99 / 120

SQL and JSON Example (5): arrays


100 / 120



101 / 120



102 / 120



103 / 120

NoSQL as SQL

JSON_TABLE


104 / 120

NoSQL as SQL (2)

JSON_TABLE


105 / 120

NoSQL as SQL - aggregation


106 / 120

NoSQL as SQL - aggregation (2)


107 / 120

SQL and NoSQL - JOINS


108 / 120



109 / 120



110 / 120

SQL and NoSQL - JOINS (2)


111 / 120



112 / 120



113 / 120

SQL and NoSQL - and back to JSON Document


114 / 120

NoSQL or SQLJS> session.sql("select doc->>'$.name', doc->>'$.cuisine' from restaurants where doc->>'$.cuisine'='Italian' limit 2")

+----------------------------+-------------------+| doc->>'$.name' | doc->>'$.cuisine' |+----------------------------+-------------------+| Philadelhia Grille Express | Italian || Isle Of Capri Resturant | Italian |+----------------------------+-------------------+2 rows in set (0.0287 sec)

JS> db.restaurants.�nd("cuisine='Italian'").�elds('name','cuisine').limit(2)[ { "cuisine": "Italian", "name": "Philadelhia Grille Express" }, { "cuisine": "Italian", "name": "Isle Of Capri Resturant" }]2 documents in set (0.0015 sec)


115 / 120

what do I gain ?

Conclusion


116 / 120

schemalessflexible data structureeasy to start (CRUD)

ConclusionThis is the best of the two worlds in one product !

Data integrityACID CompliantTransactionsSQL


117 / 120


118 / 120

http://lefred.be/content/top-10-reasons-for-nosql-with-mysql/


119 / 120

http://lefred.be/content/top-10-reasons-for-nosql-with-mysql/

Thank you !

Any Questions ?

share your 💕 for MySQL on social media using @mysql #DataOpsBarcelona


120 / 120

1 / 120 - databases · [x] document model [x] transactions operations: [x] performance...

Documents