lessons learned from my experience during the adoption, use and growth of mysql at aol kevin pettit...

MySQL @ Aol.

1

Lessons learned from my experience during the adoption, use and growth of MySQL at AOL

Kevin Pettit

Principle DBA

Content Application Operations

2

MySQL Origins for AOLSummer 2003

Internet Business Solutions, Partner Hosting, Acquisitions and the adoption of open source products gave rise to a new kid in our Database world: MySQL

3

MySQL First Projects

2003

2004 2005 2006

4

MySQL First Steps & Tools

Company was moving to Linux from HP-UX & SUN.

Met with other DBAs on my team and created/documented install standards.

First instances were MySQL versions 3.23 and 4.0 on a 32 bit OS. 2 GB File system size limitations.

First scripts/tools were nearly all shell scripts. Simple scripts for Backups, Log file management, and Replication monitoring.

5

MySQL First Observations

MySQL seemed to lack utilities for lots of basic functions/needs

Backups were just ascii text files of DDL & Insert statements.

Alter tables (column, index) locked/blocked table!

Few Tuning or Performance Analysis tools.

MySQL slow & query (general) logs were intriguing.

6

David Axmark & Monty Widenius Jay Pipes

Brian Aker Kaj Arno

7

Netscape ISP MySQL Architecture

8

8

MySQL First Projects

2003

2004 2005 2006

9

10

11

11

12

GoldRush DB Architecture

Click to edit Master text styles

Second level

Third level

Fourth level

Fifth level

12

First 1000 MySQL Instances

13

MySQL Growth @ AOL

14

MySQL First Tools

First tool was a simple korn shell script (mydbmgr.ksh)for backup/log file management:

Renamed existing log files and ran a flush logs command to rotate log files (Binary, update, error)

Archived (tar) the Inactive log files (Binary, update, error)

Remove old backup & archived log files

For Each database:

Backup Each database to a separate file

Log Each database backup to a table

Compress each database Backup file

15

MySQL Query Rates for Netscape ISP

16

16

17

MySQL First Lessons

Scaling Reads is easy with MySQL Replication.

Be conservative about adopting major versions and with your MySQL upgrade schedule.

MyISAM tables did not scale well for high concurrency workloads or when we had a high rate of queries creating temp tables. (Table Level Locking)

Too many read slaves on master

Each Slave added load to master (reading binary log). This could impact Master performance.

18

MySQL First Lessons

InnoDB Buffer Pool size & innodb_log_file size are most important configuration variables for tuning InnoDB performance.

Tracking replication information with performance stats provides valuable clues for forensic sleuthing of What caused that.

Use MySQL General log helpful to improve understanding application interactions with the database.

Use MySQL Explain & Slow Query log to tune queries & identify index needs.

19

Query Analysis

Using the MySQL general log, an open source utility (mysqlreport http://hackmysql.com/mysqlreport), and a simple perl script (mysqlqa.pl), I created what I called a Query Analysis:

Steps to do a Query Analysis:

Gather 1-n Hours of general log data.

Use mysqlreport to generate a plain SQL text file of queries from a MySQL general log.

Use mysqlqa.pl perl script to read the queries and create a generic version of the query. EXPLAIN is ran against the first occurrence of each new query.

20

20

Query Analysis

21

22

2004-2005: Leveraged general log, mysqlreport & custom perl script.

2007 -Today: Leverage Slow Query Log (with long_query_time=0;) & Percona Toolkits pt-query-digest

mk-query-digest/pt-query-digest & Slow Query Log (with long_query_time=0) uses actual execution timings/internal counters so performance stats are much more accurate!

Check top 10-25 most expensive, frequent queries for optimization opportunities (Better indexes & alternate query design).

Query Analysis Evolution

23

MySQL Needs (Early Years)

With so many MySQL projects, need for displaying the replication tree became very important for NOC & On-Call DBA

24

MySQL Growth & Matrix Project

Goals for Operations Infrastructure Project (Matrix)

Able to rapidly deploy new projects and services

Able to quickly scale up and down the hardware capacity to meet the production traffic requirements

Able to support leading web 2.0 development platforms and technologies

Able to rapidly provision development and staging environments for product development teams

Able to manage costs and improve efficiencies

25

Project Matrix Solution

A server farm, installed and pre-configured with all the leading web 2.0 platforms

LAMP Platform - Linux, Apache, MySQL, PHP

Standard Blogging Platform Wordpress

Standard Content Mgmt Platform Drupal

Built in redundancy and scalability

220 servers deployed in 2 Data Centers Dulles and Manassas

Configured with Netscaler VIPs

Rapid provisioning and scalability of projects at multiple tiers

26

26

Project Matrix Solution

Database servers are setup with replication to enable quick scalability at multiple tiers

23 Project Slots with 5 levels of capacity

Tier 1 64 Read slaves

Tier 2 32 Read Slaves



Tier 5 perl mk-query-digest hostA.slowlog.20120120000001

# 43.5s user time, 150ms system time, 31.64M rss, 99.14M vsz

# Current date: Tues Jan 24 9:44:39 2012

# Files: hostA.slowlog.20120120000001

# Overall: 159.72k total, 10 unique, 1.85 QPS, 47.94x concurrency ________

# total min max avg 95% stddev median

# Exec time 4142444s 2s 253s 26s 113s 36s 11s

# Lock time 315s 0 31s 2ms 0 177ms 0

# Rows sent 9.86M 0 100 64.75 97.36 44.78 97.36

# Rows exam 59.37G 24 1.31M 389.78k 1.26M 472.79k 165.97k

# Time range 2012-01-19 00:01:40 to 2012-01-20 00:01:43

# bytes 79.67M 161 3.47k 523.03 621.67 187.51 592.07

# Profile

# Rank Query ID Response time Calls R/Call Item

# ==== ================== ================== ===== ======== ==============

# 1 0xD3498568EB9790DF 2852265.0000 68.9% 33743 84.5291 SELECT view_counts_tbl

# 2 0x98FFDF7DF9AB29F6 460973.0000 11.1% 38630 11.9330 SELECT aggregate_counts_tbl

# 3 0x023560745F86665A 198111.0000 4.8% 15372 12.8878 SELECT aggregate_counts_tbl

# 4 0x8ED5FEB4CEB2FFCE 193445.0000 4.7% 15098 12.8126 SELECT aggregate_counts_tbl

# 5 0x1C59BAE3E36C0AF4 156430.0000 3.8% 19835 7.8866 SELECT aggregate_counts_tbl

# 6 0x2CA0DFA413086FA2 135065.0000 3.3% 14397 9.3815 SELECT aggregate_counts_tbl

# MISC 0xMISC 146155.0000 3.5% 22641 6.4553

38

Tuning Lessons Details

# Query 1: 0.39 QPS, 33.02x concurrency, ID 0xD3498568EB9790DF at byte 78970271

# pct total min max avg 95% stddev median

# Count 21 33743

# Exec time 68 2852265s 4s 253s 85s 151s 43s 80s

# Lock time 77 243s 0 31s 7ms 0 372ms 0

# Rows sent 16 1.64M 3 100 51.11 97.36 47.23 2.90

# Rows exam 72 43.26G 1.31M 1.31M 1.31M 1.26M 0 1.26M

# Users 1 user_r

# Hosts 1 localhost

# Databases 1 statsdb

# Time range 2012-01-19 00:01:41 to 2012-01-20 00:01:31

# bytes 6 5.21M 161 163 161.99 158.58 0 158.58

# Query_time distribution

# 1ms

# 10ms

# 100ms

# 1s ###

# 10s+ ################################################################

# Tables

# SHOW TABLE STATUS FROM `statsdb` LIKE 'view_counts_tbl'\G

# SHOW CREATE TABLE `statsdb`.`view_counts_tbl`\G

# EXPLAIN

select SQL_CALC_FOUND_ROWS * FROM view_counts_tbl WHERE Application = 'movies'AND Type = 'movie'AND App1 = 'movie' ORDER BY ViewCountsDay DESC LIMIT 0,100\G

39


CREATE TABLE `view_counts_tbl` (

ÀggregateViewID` int(10) unsigned NOT NULL AUTO_INCREMENT,

ÀssetURI` varchar(255) NOT NULL,

Àpplication` varchar(30) NOT NULL DEFAULT '',

`Category` varchar(25) DEFAULT '',

`Type` varchar(25) DEFAULT '',

Àpp1` varchar(64) DEFAULT '',



ÙpdateDate` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

`ViewCountsDay` int(10) unsigned DEFAULT '0',

`ViewCountsWeek` int(10) unsigned DEFAULT '0',

`ViewCountsMonth` int(10) unsigned DEFAULT '0',

`ViewCountsAll` int(10) unsigned DEFAULT '0',

PRIMARY KEY (ÀggregateViewID`),

UNIQUE KEY ìndex` (ÀssetURI`,Àpplication`),

KEY `view_idx1` (`ViewCountsDay`),

) ENGINE=InnoDB DEFAULT CHARSET=utf8;

select SQL_CALC_FOUND_ROWS * FROM view_counts_tbl WHERE Application = 'movies AND Type = 'movie AND App1 = 'movie' ORDER BY ViewCountsDay DESC LIMIT 0,100\G

40


Before Adding Index:

mysql> EXPLAIN select SQL_CALC_FOUND_ROWS * FROM view_counts_tbl WHERE Application = 'movies' AND Type = 'movie' AND App1 = 'movie' ORDER BY ViewCountsDay DESC LIMIT 0,100\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE

table: view_counts_tbl

type: ALL

possible_keys: NULL

key: NULL

key_len: NULL

ref: NULL

rows: 1391165

Extra: Using where; Using filesort

------------------------ Slow Query Log Example Run ----------------------------

# Time: 120124 9:42:04

# User@Host: user[user] @ [xxx.xxx.xxx.xxx]

# Query_time: 12.35962 Lock_time: 0.000065 Rows_sent: 100 Rows_examined: 1701886

select SQL_CALC_FOUND_ROWS * FROM view_counts_tbl WHERE Application = 'movies' AND Type = 'movie' AND App1 = 'movie' ORDER BY ViewCountsDay DESC LIMIT 0,100;

41


After Adding Index:

alter table view_counts_tbl add KEY apptype(Application,Type,App1);


*************************** 1. row ***************************

id: 1

select_type: SIMPLE


type: ref

possible_keys: apptype

key: apptype

key_len: 365

ref: const,const,const

rows: 484390

Extra: Using where; Using filesort


# Time: 120124 9:44:47




42


After Adding Covering Index:

alter table view_counts_tbl add KEY apptype(Application,Type,App1,ViewCountsDay);


*************************** 1. row ***************************

id: 1

select_type: SIMPLE


type: ref


key: apptype

key_len: 365


rows: 275600

Extra: Using where


# Time: 120124 9:55:37




43


After Removing SQL_CALC_FOUND_ROWS from the query:

mysql> EXPLAIN select * FROM view_counts_tbl WHERE Application = 'movies' AND Type = 'movie' AND App1 = 'movie' ORDER BY ViewCountsDay DESC LIMIT 0,100\G

*************************** 1. row ***************************

id: 1

select_type: SIMPLE


type: ref


key: apptype

key_len: 365


rows: 275600

Extra: Using where


# Time: 120124 12:55:37



select * FROM view_counts_tbl WHERE Application = 'movies' AND Type = 'movie' AND App1 = 'movie' ORDER BY ViewCountsDay DESC LIMIT 0,100;

44


----------------------- Slow Query Log Example Run ----------------------------

# Time: 120124 12:55:37



select * FROM view_counts_tbl WHERE Application = 'movies' AND Type = 'movie' AND App1 = 'movie' ORDER BY ViewCountsDay DESC LIMIT 0,100;




After running the query, Run the command to get the output of SQL_CALC_FOUND_ROWS:

mysql> SELECT FOUND_ROWS();

+--------------+

| FOUND_ROWS() |

+--------------+

| 325312 |

+--------------+

45


46


47


48

Current Tools/Tech

MySQL 5.0, MySQL 5.1, MySQL 5.5,Percona Server 5.1

Maatkit utilities.

MySQL LVM Backups, Percona XtraBackup Hot Backups, Zmanda Backups.

Nagios & custom scripts for monitoring.

Replication Topology Tool: Integration with Netscaler VIP data.

49

Current Tools/Tech

50

51

52

Lessons Learned Summary

General:

Collect MySQL Statistics & Binlog/Replication data.

Monitor/Review for significant changes to statistics

Use statistics to monitor & notify you of growth concerns.

Leverage MySQL Slow query Log & Percona Toolkit utilities to identify problem queries.

Large Scale:

Automate Installation process.

Automate Routine needs.

53

Future Plans

Move to Percona Toolkit.

MySQL as an AOL Cloud service.

Explore/Evaluate:

Evaluate add-on Galera Replication & Percona XtraDB Cluster.

Automated Stats/Analysis Monitoring.

Create wrapper utilities to leverage Xtrabackups backups for automatic setup of new MySQL slaves.

Integrate/combine utility scripts to automate scale up & scale down on VMs.

54

Questions?

55

56

lessons learned from my experience during the adoption, use and growth of mysql at aol kevin pettit...

Documents

mysql replication

mysql instances

growth of mysql

mysql growth

mysql versions

mysql origins

mysql query rates

mysql general log helpful