mysql 5.7: core server changes

MySQL Core Server ChangesMorgan TockerMySQL Community Manager

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.!2

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


MySQL 5.5

MySQL Enterprise Monitor 2.3 & 3.0MySQL Enterprise Backup Security Scalability HA Audit

MySQL 5.6MySQL Workbench 6.1

MySQL Utilities

MySQL Applier for Hadoop

MySQL Workbench 5.2 & 6.0MySQL Enterprise Oracle Certifications

MySQL@Oracle: 4 Years of MySQL Innovation

MySQL Cluster ManagerWindows installer & Tools

MySQL Cluster 7.2MySQL Cluster 7.1

MySQL Migration Wizard MySQL 5.7

MySQL Cluster 7.4

MySQL Fabric

MySQL Cluster 7.3


Discussion Topics

▪ Let’s talk about MySQL 5.7! ▪ Less discussion about InnoDB, Performance Schema,

Benchmarks, Replication. ▪ Well covered in other talks.


Discussion Topics (cont.)

▪ New Features ▪ Changes to Existing Features ▪ Our Ongoing Efforts to Refactor and Improve MySQL.


Quick Recap of the MySQL Development Process


The DMR Model

▪ Up to 4 Development Milestone Releases per year (DMRs). ▪ Features are merged into main branch only when stable.

▪ High level review complete. ▪ Low level review complete. ▪ 80%+ code coverage testing goal.

▪ DMRs are intended to be of “Release Candidate” Quality.


MySQL 5.7 DMR Schedule

April 2013 DMR1 221 Bugs Fixed 29 Changes 21 Worklogs

Sep 2013 DMR2 384 Bugs Fixed 41 Changes 52 Worklogs

Dec 2013 DMR3 155 Bugs Fixed 23 Changes 27 Worklogs

April 2014 DMR4 256 Bugs Fixed 38 Changes 56 Worklogs

5.6 GA Feb 2013


Labs Releases

▪ For functionality that doesn’t meet the DMR requirements. ▪ Intended to be an earlier preview. ▪ Recent Labs Releases:

▪ Hadoop Applier for MySQL ▪ MySQL Fabric ▪ Multi-Source Replication ▪ NEW - InnoDB GIS+Optimizer GIS+Parser Refactoring! ▪ NEW - InnoDB Page/IO Compression! ▪ NEW - Multi-threaded Slaves!

labs.mysql.com


InnoDB GIS+Optimizer GIS+Parser Refactoring

▪ InnoDB R-Tree Index Support ▪ Based on Boost.Geometry

▪ Previously only Available in MyISAM

labs.mysql.com


InnoDB GIS+Optimizer GIS+Parser Refactoring (cont.)

▪ Improves readability, maintainability, and stability – Cleanly separate the parsing, optimizing, and execution stages – Allows for easier feature additions, with lessened risk !

▪ Transforms SQL Parser into a true bottom-up parser – Better support for larger syntax rules in the future – Fewer intermediate nodes in the parse tree

▪ Fewer node allocations ▪ Faster grammar

labs.mysql.com


InnoDB Page/IO Compression

▪ Transparent Page Level Compression ▪ Builds on new multiple page cleaner threads from DMR4.

▪ Compress/decompress transparently in background threads ▪ Applies to all tables, including the system tablespace and UNDO

logs ▪ Thank you to Fusion-IO!

labs.mysql.com


MySQL 5.7 DMRs To Date


Common Themes

▪ Improved support for high connection creation rate

▪ Lock Thread count impact reduced ▪ Multiple Metadata Locking (MDL)

improvements ▪ InnoDB Improvements for:

▪ Index lock contention ▪ Read Only Transactions ▪ Temporary Tables ▪ Redo Logs ▪ Multiple page cleaner threads. ▪ Scalability

▪ Parallel Replication (Intra-schema)

Performance & Scalability

▪ Overhead has been reduced. ▪ Improved to do lazy initialization on new

connections. ▪ Now instruments:

▪ Memory usage ▪ Stored programs ▪ Replication ▪ Metadata Locking (MDL) ▪ Transactions ▪ Prepared Statements

From DMR1 to DMR4

Performance Schema


Common Themes (cont.)

▪ Internal logging API ▪ Timestamps are now written with

log entries in YYYY-MM-DDThh:mm:ss.uuuuuu format.

▪ Configurable Verbosity ▪ InnoDB Refactored:

▪ Locking Code ▪ Redo Log Code (for

performance) ▪ Tablespace Class

▪ Atomics refactored ▪ Optimizer refactoring (multiple

improvements)

Deep Refactoring

▪ Password expiration support + timestamp of password last changed.

▪ mysql_secure_installation changed to C++ program.

▪ Improvements to SSL Support. ▪ Support to change AES key length

and block encryption mode

From DMR1 to DMR4

Security


DMR0

▪ Error issued when creating duplicate index. ▪ Warning when strict SQL Mode removed.

▪ Ctl-C fixed in MySQL CLI! ▪ Non-blocking SHOW SLAVE STATUS ▪ Idempotent mysqlbinlog mode. ▪ Stacked Diagnostics Area

Internal Release Only


DMR1

▪ Online DDL Now supports RENAME INDEX and VARCHAR extensions.

▪ InnoDB support for Spatial Data Types. ▪ Rewrite-db option for mysqlbinlog with RBR.

Released April 2013


DMR2

▪ Semi sync replication improvements. ▪ Multiple Triggers per event per table. ▪ EXPLAIN FOR CONNECTION ▪ EXPLAIN showing cost information.

Released September 2013


Example: Cost Information in EXPLAIN

*************************** 1. row ***************************EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "4786.00" }, "ordering_operation": {

"using_temporary_table": true, "using_filesort": true, "cost_info": { "sort_cost": "2151.00" }, "nested_loop": [ { "table": { .. ], "key_length": "3", "rows_examined_per_scan": 239, "rows_produced_per_join": 239, "filtered": 100, "using_index": true, "cost_info": { "read_cost": "6.00", "eval_cost": "47.80", "prefix_cost": "53.80", "data_read_per_join": "61K" },

..

EXPLAIN FORMAT=JSON FOR CONNECTION 2;


Example: EXPLAIN in MySQL WorkbenchVisual EXPLAIN Including Cost Information!


DMR3

▪ UNION ALL no longer requires temporary table. ▪ Filesort now packs VARCHAR and CHAR keys in temporary files. ▪ Optimizer Support for row value constructors.

▪ i.e. WHERE (a,b) in ((‘a’, ‘b’), (‘c’, ‘d’));

Released December 2013


DMR3 (cont.)

▪ InnoDB Fulltext now supports external parsers. ▪ Dynamic replication filters. ▪ Support ICP optimization for partitioned tables.

Released December 2013


Example: UNION ALL Optimization

***************** 1. row ***************** id: 1 select_type: PRIMARY table: table_a type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 Extra: NULL ***************** 2. row ***************** id: 2 select_type: UNION table: table_b type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 Extra: NULL ***************** 3. row ***************** id: NULL select_type: UNION RESULT table: <union1,2> type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: NULL Extra: Using temporary

***************** 1. row ***************** id: 1 select_type: PRIMARY table: table_a partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 filtered: 100.00 Extra: NULL ***************** 2. row ***************** id: 2 select_type: UNION table: table_b partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 filtered: 100.00 Extra: NULL

EXPLAIN SELECT * FROM a UNION ALL SELECT * FROM bMySQL 5.6: MySQL 5.7:


Example: Row-value Constructors

**************** 1. row **************** id: 1 select_type: SIMPLE table: City type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 4188 Extra: Using where 1 row in set (0.00 sec)

**************** 1. row **************** id: 1 select_type: SIMPLE table: City partitions: NULL type: range possible_keys: CountryCode,CountryCode$Name key: CountryCode$Name key_len: 38 ref: NULL rows: 3 filtered: 100.00 Extra: Using where

EXPLAIN SELECT * FROM City WHERE (CountryCode,Name) IN (('USA', 'San Francisco'), ('USA', 'Chicago'), ('CAN', ‘Toronto'))

MySQL 5.6: MySQL 5.7:

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

{San Francisco, California, 776733} {Chicago, Illinois, 2896016}

!25

Filesort Packing

ID Name CountryCode

District Population

3805 San Francisco USA California 776733

1812 Toronto CAN Ontario 688275

3795 Chicago USA Illinois 2896016

1818 Vancouver CAN British Columbia 514008

SELECT Name, District, Population FROM City WHERE CountryCode = ‘USA’ ORDER BY Name;

{San Francisco, California, 776733} {Chicago, Illinois, 2896016}

Sort Buffer

MySQL 5.7: CHAR, VARCHAR and NULLABLE fixed-size types are now packed in the sort buffer.

Using Modified Sort Algorithm


DMR4

▪ Statement Timeout. ▪ Change master without stopping SQL thread. ▪ Changed compiler from sun studio to gcc on Solaris. ▪ Transportable tablespaces for partitioned InnoDB tables.

Released April 2014


Example: Server-Side Statement Timeout

▪ SELECT MAX_STATEMENT_TIME = 109 * FROM my_table; ▪ SET SESSION MAX_STATEMENT_TIME = 100; ▪ SET GLOBAL MAX_STATEMENT_TIME = 100;

Incorporates Contributions from Davi Arnaut, the MySQL Community


Changes to Existing Functionality (Deprecation and Change)


Implementing Change

▪ Not all changes easily support old and new. ▪ Some changes do not easily support deprecation messages.

▪ i.e. GROUP BY with implicit ORDER BY ▪ Always special cases.

▪ i.e. removal of mysqlbug, msql2mysql CLI utilities.


Functionality Changed

▪ Change of Defaults ▪ Replication

▪ SQL Modes

▪ EXPLAIN PARTITIONS and EXTENDED

▪ Deprecation ▪ NULL synonym \N

▪ ENCODE/DECODE functions.

▪ SHOW PROFILES feature

!!!!

▪ Removal ▪ SHOW ENGINE INNODB MUTEX

▪ InnoDB Monitor Tables

▪ ALTER IGNORE TABLE

▪ Unused CLI Utilities

▪ INSERT DELAYED

▪ Specific SQL Modes


Changes to Replication Defaults

▪ MySQL 5.6 improved durable replication performance considerably (group commit fix)

▪ Proposal is to make replication durable by default starting from 5.7: ▪ sync_binlog = 1 ▪ master-info-repository = TABLE ▪ relay-log-info-repository = TABLE


Change SQL_Mode Defaults

mysql> SELECT id, invoice_id, description FROM invoice_line_items GROUP BY invoice_id; +----+------------+-------------+ | id | invoice_id | description | +----+------------+-------------+ | 1 | 1 | New socks | | 3 | 2 | Shoes | | 5 | 3 | Tie | +----+------------+-------------+ 3 rows in set (0.00 sec)


SQL Mode: Only Full Group By

▪ Proposal is to produce an error from MySQL 5.7. ▪ Will still be configurable via SQL Mode ONLY_FULL_GROUP_BY.

mysql> SELECT id, invoice_id, description FROM invoice_line_items GROUP BY invoice_id; ERROR 1055 (42000): 'test.invoice_line_items.id' isn't in GROUP BY


EXPLAIN PARTITIONS and EXPLAIN EXTENDED Always Enabled

mysql> EXPLAIN SELECT * FROM table_a\G ********************** 1. row ********************** id: 1 select_type: PRIMARY table: table_a partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 3 filtered: 100.00 Extra: NULL 2 rows in set, 1 warning (0.00 sec)


EXPLAIN PARTITIONS and EXPLAIN EXTENDED (cont.)

▪ Two optional flags that only add value ▪ Automated tooling only likely scenario to break. ▪ May be better suited for FORMAT=JSON


EXPLAIN PARTITIONS and EXPLAIN EXTENDED (cont.)

▪ Proposal: ▪ Enables two flags by default, and deprecates the use of the

extended syntax. ▪ Went ahead in 5.7 DMR3.


Deprecation of NULL synonym \N

mysql> SELECT NULL is \N; +------------+ | NULL is \N | +------------+ | 1 | +------------+ 1 row in set (0.00 sec)

mysql> INSERT INTO tablea VALUES (3, \N); Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM tablea WHERE b IS \N; +---+------+ | a | b | +---+------+ | 3 | NULL | +---+------+ 1 row in set (0.00 sec)


Deprecation of NULL synonym \N (cont.)

▪ Proposal: ▪ For removal in either 5.7 or 5.8. ▪ Still seeking feedback.


Deprecate ENCODE/DECODE Functions

mysql> SELECT HEX(ENCODE('MyPassword', 'Super Secret!'));+--------------------------------------------+ | HEX(ENCODE('MyPassword', 'Super Secret!')) |+--------------------------------------------+ | A0E844F97A48FE6E8C1F | +--------------------------------------------+

!mysql> SELECT DECODE(UNHEX('A0E844F97A48FE6E8C1F'), 'Super Secret!');+--------------------------------------------------------+| DECODE(UNHEX('A0E844F97A48FE6E8C1F'), 'Super Secret!') |+--------------------------------------------------------+| MyPassword |+--------------------------------------------------------+


Deprecate ENCODE/DECODE Functions (cont.)

▪ Functions are cryptographically weak. ▪ For crypto to be upgraded, storage format would become

incompatible. ▪ Overlaps with similar AES functionality already present.


Deprecate SHOW PROFILESAnd the INFORMATION_SCHEMA.PROFILING table

mysql> set profiling=1; Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> show profiles; +----------+------------+-------------------------------------------------------+| Query_ID | Duration | Query |+----------+------------+-------------------------------------------------------+| 1 | 0.00039200 | SELECT * FROM table_a UNION ALL SELECT * FROM table_b |+----------+------------+-------------------------------------------------------+2 rows in set, 1 warning (0.01 sec)


Deprecate SHOW PROFILES (cont.)And the INFORMATION_SCHEMA.PROFILING table

mysql> show profile for query 1; +----------------------+----------+ | Status | Duration | +----------------------+----------+ | starting | 0.000059 | | checking permissions | 0.000005 | | checking permissions | 0.000005 | | Opening tables | 0.000091 | | System lock | 0.000011 | | optimizing | 0.000004 | | statistics | 0.000012 | | preparing | 0.000009 | | optimizing | 0.000002 | | statistics | 0.000005 | | preparing | 0.000004 | | executing | 0.000003 | | Sending data | 0.000053 | | executing | 0.000002 | ..

..

| Sending data | 0.000029 | | optimizing | 0.000006 | | statistics | 0.000006 | | preparing | 0.000004 | | executing | 0.000002 | | Sending data | 0.000019 | | removing tmp table | 0.000007 | | Sending data | 0.000002 | | query end | 0.000007 | | closing tables | 0.000010 | | freeing items | 0.000019 | | cleaning up | 0.000016 | +----------------------+----------+ 26 rows in set, 1 warning (0.00 sec)


Deprecate SHOW PROFILES (cont.)

▪ Overlaps with Performance Schema interface. ▪ Performance Schema has more flexible configuration

▪ Instrumentation level can be increased/lowered to reduce overhead.

▪ Performance Schema has more data available.

And the INFORMATION_SCHEMA.PROFILING table


SHOW ENGINE INNODB MUTEX

mysql> show engine innodb mutex; +--------+----------------------------+-------------------+| Type | Name | Status |+--------+----------------------------+-------------------+| InnoDB | log/log0log.c:775 | os_waits=26 || InnoDB | log/log0log.c:771 | os_waits=1 || InnoDB | buf/buf0buf.c:1208 | os_waits=3219 || InnoDB | buf/buf0buf.c:1208 | os_waits=6990 || InnoDB | buf/buf0buf.c:1208 | os_waits=4619 || InnoDB | buf/buf0buf.c:1208 | os_waits=5627 |.. | InnoDB | buf/buf0buf.c:1208 | os_waits=387166 || InnoDB | fil/fil0fil.c:1559 | os_waits=1265 || InnoDB | srv/srv0srv.c:987 | os_waits=460452 || InnoDB | combined buf/buf0buf.c:900 | os_waits=38503 || InnoDB | log/log0log.c:832 | os_waits=184 || InnoDB | combined buf/buf0buf.c:901 | os_waits=77 |+--------+----------------------------+-------------------+23 rows in set (0.56 sec)


SHOW ENGINE INNODB MUTEX (cont.)

▪ To be removed in 5.7. ▪ Rationale:

▪ Overlaps with PS. ▪ Can not be disabled. ▪ Code refactored so that InnoDB can use different locks internally.


InnoDB Monitor tables

▪ Existed in a time when information_schema did not. ▪ Usage was:

CREATE TABLE innodb_monitor (a INT) ENGINE=INNODB;

/* view to the error log */

DROP TABLE innodb_monitor;


InnoDB Monitor tables (cont.)

▪ To be removed: ▪ innodb_tablespace_monitor ▪ innodb_table_monitor ▪ innodb_mem_validate

▪ To be replaced by config setting: ▪ innodb_monitor ▪ innodb_lock_monitor


InnoDB Monitor tables (cont.)

▪ Notable exception: ▪ In 5.7 DMR4 innodb_tablespace_monitor is removed,

without a replacement. ▪ This is included as part of our release notes. ▪ A resolution will be made before 5.7 GA.


ALTER IGNORE TABLE

mysql> CREATE TABLE a (id INT NOT NULL); Query OK, 0 rows affected (0.02 sec) !mysql> INSERT INTO a VALUES (2), (2), (3); Query OK, 3 rows affected (0.01 sec) Records: 3 Duplicates: 0 Warnings: 0 !mysql> ALTER TABLE a ADD PRIMARY KEY (id); ERROR 1062 (23000): Duplicate entry '2' for key 'PRIMARY'


ALTER IGNORE TABLE (cont.)

mysql> ALTER IGNORE TABLE a ADD PRIMARY KEY (id); Query OK, 3 rows affected (0.01 sec) Records: 3 Duplicates: 1 Warnings: 0 !mysql> SELECT * FROM a; +----+ | id | +----+ | 2 | | 3 | +----+ 2 rows in set (0.00 sec)


ALTER IGNORE TABLE (cont.)

▪ Rationale: ▪ Will silently drop rows for you! ▪ Has strange semantics for replication and ALTER TABLE.

▪ Proposal went ahead already: ▪ Deprecated in 5.6.17. ▪ Removed in 5.7 DMR4


Removal of unused CLI programs

▪ msql2mysql [Shell Script] ▪ mysqlaccess [Perl program] ▪ mysqlbug [Shell script] ▪ mysql_zap [Shell script] ▪ mysql_waitpid [C program] ▪ mysql_convert_table_format [Perl Program] ▪ mysql_fix_extensions [Perl Program] ▪ mysql_find_rows [Perl Program] ▪ mysql_setpermission [Perl Program]


Removal of unused CLI programs

▪ Aim: ▪ Increase supportability / remove a situation where users use

utilities that have outlived purpose. ▪ Improved cross platform experience in the case of shell scripts.


INSERT DELAYED Removal

mysql> INSERT DELAYED INTO a (id) VALUES (5);

Query OK, 1 row affected, 1 warning (0.00 sec)


INSERT DELAYED Removal (cont.)

▪ Rationale: ▪ INSERT DELAYED is slower than a normal INSERT when

statements do not need to be delayed. ▪ InnoDB has row level locking + MVCC. Reads do not block

writes (usually). ▪ Does not allow server to check for PRIMARY/UNIQUE

constraints. ▪ Errors can not be caught by the application.


Simplify SQL_MODE options

▪ Proposal is to reduce the number of mode options available. ▪ Aim is to increase usage of SQL Mode options, encourage more

users to run MySQL in a stricter way.


Simplify SQL_MODE options (cont.)

▪ Current recommendation: ▪ sql-mode=“STRICT_TRANS_TABLES, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, NO_AUTO_VALUE_ON_ZERO, NO_ENGINE_SUBSTITUTION, NO_ZERO_DATE, NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY"


Simplify SQL_MODE options (cont.)

▪ 5.7 Proposal: ▪ Remove the options ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE and NO_ZERO_IN_DATE.

▪ These behaviours be enabled by STRICT_TRANS_TABLES or STRICT_ALL_TABLES.

▪ Improve error reporting when a behaviour is influenced by an SQL mode.


We Want Your Feedback!

Smooth Transition

Ideas Feedback

Help make the transition as smooth as possible.


Other Upcoming Oracle Talks

▪ Meet the MySQL Team ▪ Same room directly after this talk!

▪ Tomorrow: ▪ MySQL Fabric ▪ Performance and Benchmarks (5.7)

mysql 5.7: core server changes

Documents

mysql fabric mysql cluster

mysql utilities mysql

mysql migration wizard

oracle andor

hadoop mysql workbench

mysql enterprise monitor

mysql development process

sole discretion of oracle