howmysqlworks

MySQL stuff

About me• Before REA I worked for a hosting company for 4

years • Learnt a lot about MySQL during this period • I started as an OPS engineer at REA and did some

database stuff here • I havent done much MySQL for two years !

• In this period we’ve had • MySQL 5.6 come out • RDS MySQL improvements !

!

Today• High level overview of MySQL • Look inside MySQL server • Talk through how InnoDB performs queries • Schema changes and online schema changes

!!

Background• Second biggest relational database in the world

• Anyone know what the biggest is? !

• Used by many of the big web shops • Facebook for all feed data • Etsy for user & data • Twitter for tweet persistence • … !

• Sun bought MySQL AB in 2008, people freaked out • Oracle bought Sun in 2010, people freaked out • Big improvements over the past few years (5.5 / 5.6) !

How it works

mysqld

mysql client tcp:3306

filesocket

ServerClient

libmysql

mysql libORM

host$ mysqlhost$ mysqld_safe!host$ mysqld

Big features• Pluggable storage engines is massive flexibility

• You can use a different storage engine per table !!

• Common storage engines • InnoDB - ACID compliant store • MyISAM - one of the original storage engines -

avoid • Lots and lots of others

!

Inside mysqld

mysqld

storage engine(s) MyISAM / InnoDB / Memory / etc

tcp:3306

filesocket

Getting the query and responding

bits and bytes stuff

API

Inside mysqld

mysqld

storage engine(s) MyISAM / InnoDB / Memory / etc

tcp:3306

filesocket

API

Inside mysqld

mysqld

storage engine(s) MyISAM / InnoDB / Memory / etc

tcp:3306

filesocket

Parse query and convert to SE

API calls.

API

Inside mysqld

mysqld

Storage engines(s)

tcp:3306

filesocket

API

query cache

binary logjoin / sort buffers

Binary Log for replication Buffers: Kinda how much batching to storage API

to save cost

The bad parts - binary log

mysqld

tcp:3306

filesocket

query cache

binary logjoin / sort buffers

Because you can mix and match storage engines,

replicating state has to be done at the lowest

common denominator

Innodb Table 1

MyISAM Table 2

The bad parts - binary log

mysqld

Innodb Table 1

tcp:3306

filesocket

query cache

binary logjoin / sort buffers

Highly concurrent InnoDB needs its queries, serialized down

to replicate to slave

MyISAM Table 2

Compare to other DBs

DB server

tcp:x

filesocket?

performance knobs

transaction log

The bad parts - query cache

mysqld

Innodb Table 1

tcp:3306

filesocket

query cache

binary logjoin / sort buffers

Can only work on an entire table level &

uses LRUMyISAM Table 2

The bad parts - query cache

mysqld

Innodb Table 1

tcp:3306

filesocket

query cache

binary logjoin / sort buffers

All ‘update/insert/delete’ blocks all transactions to ensure correct state.

It actually slows !things down

MyISAM Table 2

Inside mysqld

mysqld

storage engine(s) MyISAM / InnoDB / Memory / etc

tcp:3306

filesocket

API

Where 10%* of !the work is done

Where 90% !of the work !

is done

*Finger in the air calculation

Summary• Turn off query cache! It is hurting you! • Binary log is a necessary evil for replication • Dont play with the performance knobs (sort buffer /

join buffer / etc) • Turning the 10% - Focus on the 90%

• Use InnoDB for all the tables (unless they’re system tables) !!

• So…lets look at the 90% !

!

InnoDB

mysqld

InnoDB

tcp:3306

filesocket

API

Unknown gems

mysqld

InnoDB

tcp:3306

filesocket

API

handlersocket/ memcache

Innodb

Unknown gems

mysqld

InnoDB

tcp:3306

filesocket

API

handlersocket/ memcache

Innodb

Memcache API that talks directly to InnoDB !Removes the SQL layer and is just K/V !Google: handlersocket memcache InnoDB

Key parts of InnoDB

Innodb

API

Key parts of InnoDB

Table space (whats on disk)

redo log

Innodb bufferpool

memory!

ib_log1!ib_log2!

sometable.ibd!ib_data!

Key parts of InnoDB

Table space

redo log

bufferpool

• Each block is a page • Just think of it as a row with some extra stuff like

version number • InnoDB is ACID meaning each connection has to

have its own view of the world!• Extra metadata at the top about where everything is

Key parts of InnoDB

Table space

redo log

bufferpool

• Black is the base metadata

What is bufferpool

Table space

redo log

bufferpool

• In memory version of hot tablespace pages

What is redo log

Table space

redo log

bufferpool

• An append only log file of changes

What is table space

Table space

redo log

bufferpool

• The raw files - but it may not be always up-to-date

Lets run some queries

Table space

redo log

bufferpool

• First - lets fill up our database with data

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

mysqld parses the query and sends !API calls to InnoDB

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Is red in bufferpool?

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Load red into buffer pool

perform random !read disk activity!At this point, the time to respond!

is dependent on disk speed!

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Respond to API calls to mysqld

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Parse results !second pass sort if necessary !

return to client

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red; #again

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

mysqld parses the query and sends !API calls to InnoDB

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Is red in bufferpool? - yes

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Respond to API calls to mysqld

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from red;

Parse results !second pass sort if necessary !

return to client

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from brown;

What if bufferpool is full?

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from blue;

Assume metadata knows page requirements!LRU on the bufferpool to find space and evict

Lets run some queries

Table space

redo log

bufferpool

• mysql> select * from blue;

!Load in blue as before and return

Lets run some queries

Table space

redo log

bufferpool

• mysql> update blue set A=B where ID=1;

!Is page in bufferpool? yes

Lets run some queries

Table space

redo log

bufferpool

• mysql> update blue set A=B where ID=1;

!Update the page in bufferpool and increment !page version number

Lets run some queries

Table space

redo log

bufferpool

• mysql> update blue set A=B where ID=1;

!Write the new page to the redo log!

what tablespace is up to!

Lets run some queries

Table space

redo log

bufferpool

• mysql> update blue set A=B where ID=1;

!Update metadata telling it that this is the !new state of the page!

perform sequential !write disk activity to redo log!

This is pretty fast!

what tablespace is up to!

Lets run some queries

Table space

redo log

bufferpool

• mysql> update blue set A=B where ID=1;

!Send the OK back to mysqld!

what tablespace is up to!

Lets run some queries

Table space

redo log

bufferpool

• mysql> update blue set A=B where ID=1;

!At sometime in the near future, do a semi sequential !parse changes and update the tablespace

what tablespace is up to! semi sequential write!Not that bad !

!

Lets run some queries

Table space

redo log

bufferpool

• mysql> insert into yellow values (1,2,3,4);

!Is space in bufferpool?!

Lets run some queries

Table space

redo log

bufferpool

• mysql> insert into yellow values (1,2,3,4);

!Is adjacent pages in bufferpool?!

Random read !(but not too big)

Lets run some queries

Table space

redo log

bufferpool

• mysql> insert into yellow values (1,2,3,4);

!Send OK back to client!

Key take aways

Table space

redo log

bufferpool

• Cold databases are slow! • bufferpool is empty and everything requires random disk

reads • The bigger the buffer pool you have, the quicker you will be!

(NUMBER 1 factor on performance!)!• Redo log is the second slowest part

• we can tune this

innodb_flush_logs_at_trx_commit=1

innodb_buffer_pool_size= ~80% memory!

Schema changes• Everyone hates them

• They are not transactional • They are blocking (well most)

• How do they work? !

Remember this?

mysqld

Storage engines(s)

tcp:3306

filesocket

API

query cache

binary logjoin / sort buffers

Lets add a few more

Storage engines(s)

tcp:3306

filesocket

API

table meta data

binary logTable triggers

functions / procedures

mysqld

tcp:3306

filesocket

mytbl

Schema changes

mytbl meta

mytbl

mysql> CREATE TABLE mytbl

(a INT, b CHAR (20), INDEX (a)) ENGINE=InnoDB;

Schema changes

mytbl

Schema changesmysql> ALTER TABLE mytbl

ADD COLUMN age INT;

mytbl

Schema changesmysql> ALTER TABLE mytbl

ADD COLUMN age INT;

!Wait for any open transactions to close and then !lock access to the table!

block

mytbl

Schema changesmysql> ALTER TABLE mytbl

ADD COLUMN age INT;

!Create a new ‘hidden’ table with the new schema!

block

.mytbl

mytbl

Schema changesmysql> ALTER TABLE mytbl

ADD COLUMN age INT;

!Do an internal ‘mysqldump’ / ‘mysqlrestore’!

block

.mytbl

mytbl

Schema changesmysql> ALTER TABLE mytbl

ADD COLUMN age INT;

!Do a switch-a-roo!

block

.mytbl

Schema changesmysql> ALTER TABLE mytbl

ADD COLUMN age INT;

!Unblock and send OK back to client!

mytbl

Online schema changes• Some changes can be done without ‘locking’

• Column additions • Index additions !

• Any removal cannot be done online • Unless you do it yourself

!

mysqld

mytbl

Online schema changes 5.1/5.5

mytbl meta

mysqld

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER…

pt-online-schema-change soundclouds lhm etc

mysqld

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

!Creates another table with the updated schema in the client!

mysqld

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

!Creates a set of triggers on the main table!

triggers

mysqld

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

!selects the entire table (in batches) and inserts into the new table!!!This can take a long time!

triggers

mysqld

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

triggersclient

mysql> select

!Whilst the new table is being built, selects still go to the old table!

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

triggersclient

mysql> UPDATE

!Updates cause the trigger to fire which updates the new table being built. (Inserts the row if its still hasn't been copied)!!Deletes happen the same way!

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

triggersclient

mysql> INSERT

!Inserts also happen the same way

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

triggers

!Eventually the table population finishes and a ‘RENAME table’ action is performed (which is quick)

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… .mytbl meta

.mytbl

triggers

!Eventually the table population finishes. This is the only ‘blocking’ time

block block

mytbl

Online schema changes 5.1/5.5

mytbl meta

clientmysql> ALTER… !

The schema change tool finishes

Online schema changes in 5.6• All that ‘switch-a-roo’ logic is now within mysqld itself !

Summary about schema changes• You ‘double your IO’ when you’re online schema

changes! • Binary log is outside of this - so watch out for your

slaves table locking if you don't use the right flags !

!