mysql 5.7 new features for developers

Zohar Elkayam www.realdbamagic.com

Twitter: @realmgic

MySQL 5.7 New FeaturesFor Developers

Who am I?•Zohar Elkayam, CTO at Brillix

• Programmer, DBA, team leader, database trainer, public speaker, and a senior consultant for over 18 years•Oracle ACE Associate • Part of ilOUG – Israel Oracle User Group•Blogger – www.realdbamagic.com and www.ilDBA.co.il

2 http://brillix.co.il

About Brillix• We offer complete, integrated end-to-end solutions based on best-of-

breed innovations in database, security and big data technologies• We provide complete end-to-end 24x7 expert remote database

services• We offer professional customized on-site trainings, delivered by our

top-notch world recognized instructors

Some of Our Customers

http://brillix.co.il4

Agenda•Optimizer, Performance and InnoDB changes•Native JSON datatype•The Performance Schema and SYS Schema•Other features we should know•What are we waiting for in MySQL 8.0?

Versions Guide•MySQL 5.7 released in October 2015•Current version 5.7.16 (released October 2016)•Over 200+ new features!

•Next major version will be MySQL 8.0 (Currently, DMR – a Development Milestone Release)

Optimizer, Performance and InnoDB Changes

How Does a Query Run?

Optimizer Changes• Parser and Optimizer refactoring • Readability, maintainability and stability • Separate parsing, optimizing, execution stages• Easier feature additions, with lessened risk

•New hint framework: easier to manage, new hints•Cost-based optimizer• Configurable and tunable: mysql.server_cost and mysql.engine_cost• API for where data resides: on disk or in cache

MySQL Optimizer

Cost-based Query Optimization•Assign cost to operations•Assign cost to partial or alternative plans•Search for plan with lowest cost (“best plan”)•Cost base optimization control• Access method• Join order• Subquery strategy

Optimizer Cost Model

Q3 Q7 Q8 Q9 Q120

MySQL 5.6MySQL 5.7

5 out of 22 queries get a much improved query plan (others remain the same)

13 Source: MySQL 5.7: 20 Years in the Making! By Geir Høydalsvik, Sr. Director, MySQL Engineering

Optimizer Cost Model: Performance ImprovementsDBT-3 (Size Factor 10, CPU Bound)

Q2 Q180

CPU bound

5.65.7

Optimizer Cost Model: Performance ImprovementsDBT-3 (Size Factor 10)

2 out of 22 queries get a significantly improved query plan (others remain the same)Q2 Q18

Disk bound

5.65.7

14 Source: MySQL 5.7: 20 Years in the Making! By Geir Høydalsvik, Sr. Director, MySQL Engineering

Adjustable Cost Constants (Experimental!)

•We can change the costs factors by changing system tables:•Use mysql.engine_cost and mysql.server_costs to change default values

•Use FLUSH command to make server aware of new values (only new connections will see updated cost constants)

update mysql.engine_cost set cost_value=2 where cost_name='io_block_read_cost';

flush optimizer_costs;

The Query Rewrite Plugin• New pre and post parse query rewrite APIs• Users can write their own plug-ins

• Provides a post-parse query plugin• Rewrite problematic queries without the need to make application

changes• Add hints• Modify join order

• Improve problematic queries from ORMs, third party apps, etc.• Eliminates many legacy use cases for proxies

InnoDB Improvements: Temporary Tables• Temp tables are no longer stored in normal system tables•Definitions can be maintained in-memory (without persisting to the disk)• Locking constraints can be relaxed since only one client can see these tables.

InnoDB Improvements: Native Partitioning•Eliminates previous limitations•Eliminates resource usage problems•No longer using the ha_partition handler• Reduce memory usage by 90%•MySQL 5.7.9 will try to upgrade old partitions to native or Use ALTER TABLE ... UPGRADE PARTITIONING command

•Transportable tablespace support (5.7.4)

InnoDB Bulk Load Performance•Bulk loading is now used when creating or rebuilding indexes•Much faster INDEX creation:• 2-3 x performance improvement for ADD/CREATE INDEX operations• 2-5% improvement for standard INSERT operations

InnoDB Page level Compression •Not the same as Table Compression (5.1 feature)•Reduces IO for better performance •The compressed data is written to disk, where the hole punching mechanism then releases empty blocks from the end of the page.• If compression fails, data is written out as-is.•Supports Zlib and LZ4 compressions

InnoDB Online Operations•Resize the InnoDB Buffer Pool online•More online ALTER operations• Rename indexes• Enlarge Varchar columns•Optimize Table (5.7.4)

•Truncate undo logs•More dynamic configuration variables

Native JSON Support

The JSON Native Datatype•New native data type: JSON• Supports all JSON internal types• Numbers, strings, bool• Objects, JSON arrays

• Supports also extended data types: date, time, datetime, and timestamp• Efficient access: optimized for read intensive workload• Performance: fast access to array cells by creating indexes•MySQL Document Store (5.7.12)

Why Not Just Use TEXT/Varchar?•Document validation: parse and validation on insert•Efficient binary format• Allows quicker access to object members and array elements• Binary format of JSON type is very efficient at searching• Storing as TEXT performs over 10x worse at traversal

•Built-in handling functions

Built-in JSON Functions•5.7 has built in functions to CREATE, SEARCH, MODIFY and RETURN JSON documents and JSON values•For a complete list of the JSON functions: https://dev.mysql.com/doc/refman/5.7/en/json-function-reference.html

Create JSON_OBJECT and JSON_ARRAY•Use JSON_OBJECT to create JSON from tables:

• JSON_ARRAY is a function that generates a JSON array from a list of values

mysql> select JSON_OBJECT('id', id, 'firstName', first_name, 'lastName', last_name) from employees_old;

+-----------------------------------------------------------------------+| JSON_OBJECT('id', id, 'firstName', first_name, 'lastName', last_name) |+-----------------------------------------------------------------------+| {"id": 1, "lastName": "Elkayam", "firstName": "Tamar"} || {"id": 2, "lastName": "Elkayam", "firstName": "Efrat"} || {"id": 3, "lastName": "Elkayam", "firstName": "Zohar"} || {"id": 4, "lastName": "Elkayam", "firstName": "Ido"} |+-----------------------------------------------------------------------+4 rows in set (0.00 sec)

Extracting a Value: JSON_EXTRACT• JSON_EXTRACT – Extracts data from the JSON•Uses JSON Path

• supports two shorthand operators:

JSON_EXTACT (column_name, “$.type”(

column_name->"$.type" (extract)column_name->>"$.type" (extract + unquote)

Searching In JSON•Use JSON_SEARCH to find paths with certain data

•Use JSON_CONTAINS and JSON_CONTAIN_PATH to know if data exists in the document•Use the extract shorthand syntax to locate the rows containing data

JSON_SEARCH (column_name, ‘one-or-all’, ‘value’(

WHERE column_name->"$.type" = ‘value’

Modifying JSON Documents•We can modify existing JSON with various functions• JSON_APPEND/JSON_ARRAY_APPEND: appends value/JSON array to a JSON documents• JSON_INSERT/JSON_ARRAY_INSERT: adds data/JSON array to a JSON• JSON_MERGE: merge JSONs• JSON_REPLACE: replace values at path• JSON_SET: replace value at path or adds it if not exist

Performance: Functional Indexes with JSON• Indexing of Documents using Generated Columns• InnoDB supports indexes on both STORED and VIRTUAL (default) Generated Columns

•New expression analyzer automatically uses the best “functional” index available

alter table employees add id numeric as (data->>"$.id");create index emp_id on employees(id);

Performance Schema andSYS Schema

Performance Schema•Performance monitoring schema• Include various table to help us with instrumentation and performance monitoring•Memory Instrumentation• Statement Instrumentation• Transactions and Locks• Additional Information: replication, stored routines, etc.

What’s New in Performance Schema•User variables, status variables (session/global)•Reduced footprint, memory usage and overhead•Total of 35 new tables – 5.6 had 52, 5.7 has 87 (with 983 instrumentations!)•But using the performance schema might be a bit too complicated….

The SYS Schema•A set of objects that helps DBAs and developers interpret data collected by the Performance Schema• Provides helper objects that answer common performance, health, usage, and monitoring questions• Introduced in 5.6 as part of an extension to MySQL (available at GitHub: https://github.com/mysql/mysql-sys)•MySQL 5.7.7+ includes the SYS schema by default

MySQL SYS Summary Views• Reference set of views solving various administrator use cases• Simple views, you can create/copy your own, sys is not “locked down”• Build upon both performance_schema and INFORMATION_SCHEMA• Both formatted and raw views are available• All raw views are prefixed with x$• Raw views are there for tools to poll• Formatted views are for humans and the command line

What Data Can We See In Summary Views?

•SYS schema comes with multiple dimensions for summary views:•User/Host summary views• IO summary views• Schema analysis•Wait and Lock wait analysis• Statement analysis•Memory views

Formatter and Helper Functions• SYS Schema also provides formatter and helper functions:• Make output human readable• Format time appropriately• Format bytes appropriately• Truncate output for large width values for CLI• Extract object names• Check instrumentation state

• Performance Schema Config Helper Procedures

Useful SYS Functions• Formatter:• format_time• format_bytes• format_path• format_statement

• Extractor• extract_table_from_file_name• extract_schema_from_file_na

me•Performance Schema States• ps_is_account_enabled• ps_is_consumer_enabled• ps_is_instrument_default_ena

bled• ps_is_instrument_default_time

d• ps_is_thread_instrumented

Short Demo for SYS Schema

Other Features and MySQL 8.0

Generated Columns (5.7.5)•Virtual Columns – functional columns calculated on the fly OR stored result•Generated column can use JSON extracts•Support Indexes – Functional Indexes•Example by Percona: https://www.percona.com/blog/2015/04/29/generated-virtual-columns-in-mysql-5-7-labs/

Server Side Timeouts (5.7.5)• Interrupting the execution of a statement when it takes longer to complete•Defined at the Global for server, per session, or for individual SELECT statements

•Example

SET GLOBAL MAX_STATEMENT_TIME=1000;SET SESSION MAX_STATEMENT_TIME=2000;SELECT /* MAX_STATEMENT_TIME=1000 */ * FROM table;

What Are We Waiting For In MySQL 8.0?• Improve with UTF8 support: default will be utf8mb4 instead of latin1•Transactional data dictionary•Common Table Expression (CTE)• Invisible mode for Indexes and other object types

What Are We Waiting For In MySQL 8.0? (cont.)

•Performance changes• Cost based optimizer statistics• Remove buffer pool mutexes• Redo log improvements for scalability•Memcache improvement to support multiple get and range scan

•Bug Fixes: for example, persistent auto increment (bug #199, created on 27 March 2003)

What Features Did We NOT Talk About?•MySQL Document Store (The X Plugin and X Protocol)•Security Improvements•GIS Improvements•Replication Improvements•More (200+ new features!): see http://www.thecompletelistoffeatures.com/for more details!

Summary•Performance, Optimizer and other InnoDB features are really important•Use JSON and other document store features to make more with MySQL•Learn how to use SYS Schema to leverage internal knowledge to your advantage

Thank YouZohar Elkayam

twitter: @realmgicZohar@Brillix.co.il

www.realdbamagic.com

mysql 5.7 new features for developers

Technology

mysql 5.6 & 5.7 & 5.7+ unlimited scaling

mysql security 5.7

the exciting mysql 5.7 replication enhancements ›...

mysql tech tour 2015 - 5.7 innodb

mysql 5.7 clustering: the developer perspective

mysql 5.7 milestone

whats new in mysql 5.7

mysql performance schema › docs ›...

mysql 5.7 reference manual - including mysql cluster ndb 7.5

mysql 5.7 gis-norvald h. ryeng

mysql 5.7 fabric: high availability and sharding€¦ ·...

mysql 5.7 whats new

mysql tech tour 2015 - 5.7 security

mysql 5.7 news

mysql 5.7: what's new, nov. 2015

rmoug mysql 5.7 new features

new awesome features in mysql 5.7

mysql 5.7: performance schema improvements

mysql 5.7 string functions

mysql 5.7 + java