1 / 112 - mysql · provides a schema flexible json document store no sql required no need to define...
TRANSCRIPT
1 / 112
2 / 112
MySQL 8.0
a Document Store with all the benefits of a transactional RDBMS
3 / 112
Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purpose only, and may notbe incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not berelied up in making purchasing decisions. The development, release and timing of any features or functionality describedfor Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
4 / 112
about me
Who am I ?
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
5 / 112
Frédéric Descamps@lefredMySQL EvangelistHacking MySQL since 3.23devops believerliving in Belgium 🇧🇪http://lefred.be
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
6 / 112
RDBMS & MySQL
Relational Databases
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
7 / 112
Relational DatabasesData Integrity
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
8 / 112
Relational DatabasesData Integrity
normalization
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
9 / 112
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
10 / 112
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, Durability
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
11 / 112
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID compliant
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
12 / 112
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID complianttransactions
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
13 / 112
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID complianttransactions
SQL
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
14 / 112
Relational DatabasesData Integrity
normalizationconstraints (foreign keys, ...)
Atomicity, Consistency, Isolation, DurabilityACID complianttransactions
SQLpowerfull query language
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
15 / 112
NoSQL & MySQL
NoSQL Databases
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
16 / 112
NoSQL or Document StoreSchemaless
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
17 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
18 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
19 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structure
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
20 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objects
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
21 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational model
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
22 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational modelobjects persistance without the use of any ORM - mapping oobject-oriented
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
23 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational modelobjects persistance without the use of any ORM - mapping oobject-oriented
JSON
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
24 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational modelobjects persistance without the use of any ORM - mapping oobject-oriented
JSONclose to frontend
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
25 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational modelobjects persistance without the use of any ORM - mapping oobject-oriented
JSONclose to frontendnative in JS
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
26 / 112
NoSQL or Document StoreSchemaless
no schema design, no normalization, no foreign keys, no data types, ...very quick initial development
Flexible data structureembedded arrays or objectsvalid solution when natural data can´t be modelized optimaly into a relational modelobjects persistance without the use of any ORM - mapping oobject-oriented
JSONclose to frontendnative in JSeasy to learn
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
27 / 112
How DBAs see data
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
28 / 112
How DBAs see data How Developers see data{ "GNP" : 249704, "Name" : "Belgium", "government" : { "GovernmentForm" : "Constitutional Monarchy, Federation", "HeadOfState" : "Philippe I" }, "_id" : "BEL", "IndepYear" : 1830, "demographics" : { "Population" : 10239000, "LifeExpectancy" : 77.8000030517578 }, "geography" : { "Region" : "Western Europe", "SurfaceArea" : 30518, "Continent" : "Europe" }}
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
29 / 112
What if there was a way to provide both SQL and NoSQL on onestable platform that has proven stability on well know
technology with a large Community and a diverse ecosystem ?
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
30 / 112
DMBS or NoSQL ?
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
31 / 112
DMBS or NoSQL ?
Why not both ?
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
32 / 112
The MySQL Document Store !
SQL is now optional !
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
33 / 112
Developers:
[x] schemaless[x] rapid prototying/simpler APIs[x] document model[x] transactions
Operations:
[x] performance management/visibility[x] robust replication, backup, restore[x] comprehensive tooling ecosystem[x] simpler application schema upgrades
Business Owner:
[x] don't lose my data == ACID trx[x] capture all my data = extensible/schemaless[x] product on schedule/time to market = rapiddevelopement
A solution for all
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
34 / 112
MySQL Document Store
the Solution
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
35 / 112
Built on theMySQL JSONData type andProven MySQLServerTechnology
Provides a schema flexible JSON Document StoreNo SQL requiredNo need to define all possible attributes, tables, etc.Uses new X DevAPICan leverage generated column to extract JSON values into materialized columnsthat can be indexed for fast SQL searches.Document can be ~1GB
It's a column in a row of a tableIt cannot exceed max_allowed_packet
Allows use of modern programming stylesNo more embedded strings of SQL in your codeEasy to read
Also works with relational TablesProven MySQL Technology
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
36 / 112
X DevAPI Connectors forC++, Java, .Net, Node.js, Python, PHPworking with Communities to help them supporting it too
New MySQL ShellCommand CompletionPython, JavaScrips & SQL modesAdmin functionsNew Util objectA new high-level session concept that can scale from single MySQL Server toa multiple server environment
Non-blocking, asynchronous calls follow common language patternsSupports CRUD operations
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
37 / 112
Starting using MySQL DS in few minutes
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
38 / 112
MySQL Shell Info
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
39 / 112
Migration from MongoDB to MySQL DSFor this example, I will use the well known restaurants collection:
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
40 / 112
Migration from MongoDB to MySQL DSFor this example, I will use the well known restaurants collection:
We need to dump the data to a file and we will use MySQL Shell with the Python interpreter to load the data.
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
41 / 112
Migration from MongoDB to MySQL DSFor this example, I will use the well known restaurants collection:
We need to dump the data to a file and we will use MySQL Shell with the Python interpreter to load the data.
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
42 / 112
Dump and load using MySQL Shell & Pythonthis example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000; db.restaurants.�nd().shellPrint()' \ | perl -pe 's/(?:ObjectId|ISODate)\(("[^"]+")\)/ $1/g' > all_recs.json
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
43 / 112
Dump and load using MySQL Shell & Pythonthis example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo --quiet --eval 'DBQuery.shellBatchSize=30000; db.restaurants.�nd().shellPrint()' \ | perl -pe 's/(?:ObjectId|ISODate)\(("[^"]+")\)/ $1/g' > all_recs.json
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
44 / 112
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
45 / 112
Let´s query
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
46 / 112
Let´s query
That´s too much records to show it here... let´s limit it
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
47 / 112
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
48 / 112
Some more examples
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
49 / 112
Some more examples
Let´s add a selection criteria:
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
50 / 112
Syntax slightly different than MongoDB
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
51 / 112
Syntax slightly different than MongoDB
> db.restaurants.�nd({"cuisine": "French", "borough": { $not: /^Manhattan/} }, {"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2){ "borough" : "Queens", "cuisine" : "French",
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
52 / 112
CRUD operations
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
53 / 112
CRUD operations for collections
Add a document
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
54 / 112
CRUD operations for collections
Modify a document
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
55 / 112
CRUD operations for collections
Remove a document
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
56 / 112
MySQL Document Store Objects Summary
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
57 / 112
Document Store Full ACID !It relies on the proven MySQL InnoDB´s strength & robustness:
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
58 / 112
Document Store Full ACID !It relies on the proven MySQL InnoDB´s strength & robustness:
innodb_�ush_log_at_trx_commit = 1
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
59 / 112
Document Store Full ACID !It relies on the proven MySQL InnoDB´s strength & robustness:
innodb_�ush_log_at_trx_commit = 1
innodb_doublewrite = ON
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
60 / 112
Document Store Full ACID !It relies on the proven MySQL InnoDB´s strength & robustness:
innodb_�ush_log_at_trx_commit = 1
innodb_doublewrite = ON
sync_binlog = 1
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
61 / 112
Document Store Full ACID !It relies on the proven MySQL InnoDB´s strength & robustness:
innodb_�ush_log_at_trx_commit = 1
innodb_doublewrite = ON
sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
62 / 112
Document Store Full ACID !It relies on the proven MySQL InnoDB´s strength & robustness:
innodb_�ush_log_at_trx_commit = 1
innodb_doublewrite = ON
sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
We do care about your data !
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
63 / 112
Document Store Full ACID !
MySQL Document Store supports transactions
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
64 / 112
Document Store Full ACID !
MySQL Document Store supports transactions
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
65 / 112
what about my old SQL ?
The hidden part of the iceberg
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
66 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
67 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
68 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON values
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
69 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
70 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)Generated/virtual columns
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
71 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)Generated/virtual columns
Indexing JSON data
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
72 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)Generated/virtual columns
Indexing JSON dataForeign Keys to JSON data
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
73 / 112
JSON datatype is behind the scenenative datatype (since 5.7.8)JSON values are stored in MySQL tables using UTF8MB4Conversion from "native" SQL types to JSON valuesJSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...)Generated/virtual columns
Indexing JSON dataForeign Keys to JSON dataSQL Views to JSON data
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
74 / 112
How does the MySQL Document Store work ?
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
75 / 112
What does a collection look like on the server ?
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
76 / 112
_id Every document has a unique identifier called the document ID, which can be thoughtof as the equivalent of a table´s primary key. The document ID value can be manuallyassigned when adding a document. If no value is assigned, a document ID isgenerated and assigned to the document automatically !
Use getDocumentId() or getDocumentIds() to get _ids(s)
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
77 / 112
Mapping to SQL examplescreateCollection('mycollection')
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
78 / 112
Mapping to SQL examplescreateCollection('mycollection')
CREATE TABLE `test`.`mycoll` ( doc JSON, _id VARCHAR(32) GENERATED ALWAYS AS (doc->>'$._id') STORED PRIMARY KEY ) CHARSET utf8mb4;
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
79 / 112
Mapping to SQL examplescreateCollection('mycollection')
CREATE TABLE `test`.`mycoll` ( doc JSON, _id VARCHAR(32) GENERATED ALWAYS AS (doc->>'$._id') STORED PRIMARY KEY ) CHARSET utf8mb4;
mycollection.add({'test': 1234})
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
80 / 112
Mapping to SQL examplescreateCollection('mycollection')
CREATE TABLE `test`.`mycoll` ( doc JSON, _id VARCHAR(32) GENERATED ALWAYS AS (doc->>'$._id') STORED PRIMARY KEY ) CHARSET utf8mb4;
mycollection.add({'test': 1234})
INSERT INTO `test`.`mycoll` (doc) VALUES ( JSON_OBJECT('_id','663807fe367ee6114e0e5458bdac28bf', 'test',1234) );
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
81 / 112
Mapping to SQL examples (2)mycollection.find("test > 100")
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
82 / 112
Mapping to SQL examples (2)mycollection.find("test > 100")
SELECT doc FROM `test`.`mycoll` WHERE (JSON_EXTRACT(doc,'$.test') > 100);
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
83 / 112
SQL and JSON Example
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
84 / 112
SQL and JSON Example
same as:
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
85 / 112
SQL and JSON Example
same as:
we can use it in SQL:
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
86 / 112
Indexes in Document StoreIt's also possible to create indexes without using SQL syntax:
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
87 / 112
Indexes in Document StoreIt's also possible to create indexes without using SQL syntax:
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
88 / 112
SQL and JSON Example (2): validation
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
89 / 112
SQL and JSON Example (2): validation
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
90 / 112
SQL and JSON Example (2): validation
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
91 / 112
SQL and JSON Example (3): explain
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
92 / 112
SQL and JSON Example (3): explain
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
93 / 112
SQL and JSON Example (4): add index
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
94 / 112
SQL and JSON Example (4): add index
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
95 / 112
SQL and JSON Example (5): arrays
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
96 / 112
SQL and JSON Example (5): arrays
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
97 / 112
SQL and JSON Example (5): arrays
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
98 / 112
SQL and JSON Example (5): arrays
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
99 / 112
NoSQL as SQL
JSON_TABLE
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
100 / 112
NoSQL as SQL (2)
JSON_TABLE
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
101 / 112
SQL and NoSQL
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
102 / 112
SQL and NoSQL
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
103 / 112
SQL and NoSQL
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
104 / 112
SQL and NoSQL (2)
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
105 / 112
SQL and NoSQL (2)
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
106 / 112
SQL and NoSQL (2)
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
107 / 112
SQL and NoSQL (3) : and back to JSON Document
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
108 / 112
what do I gain ?
Conclusion
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
109 / 112
schemalessflexible data structureeasy to start (CRUD)
ConclusionThis is the best of the two worlds in one product !
Data integrityACID CompliantTransactionsSQL
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
110 / 112
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
111 / 112
Thank you !
Any Questions ?
share your 💕 for @MySQL on social media using #mysqlinnovationday
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
112 / 112