pinba in-memory mysql storage engine for a full-stack real ... · pinba in-memory mysql storage...

Pinbain-memory MySQL storage engine for a

full-stack real-time performance analytics

Antony DovgalAlexey Rybak

Once upon a time... ...there was a dating site which didn’t have any of these:

● Monitoring● QA● Fridge● Fancy office

Once upon a time... This is what was used for monitoring:

Yes, it’s an MRTG traffic graph

The Problem✦ How many requests per second are served?

✦ How slow are they?

✦ What are the slowest scripts?

✦ What makes them slow?

✦ How servers perform?

✦ Where’s the fridge?

Yet another acronymP – PHP

I – is

N – not a

B – bottleneck

A – anymore

What is it?✦ It’s a daemon

✦ It’s a MySQL storage engine

✦ Which doesn’t really store anything

✦ At least not for long

What does it do?✦ Receives Protobuf packets by UDP from servers✦ Aggregates data in multiple ways✦ Provides SQL interface via MySQL

MySQL [pinba]> select req_count/time_interval as req_per_sec from info;

+-------------+

| req_per_sec |

+-------------+

| 66577.2100 |

+-------------+

1 row in set (0.04 sec)

Sliding Window✦ We don’t have infinite RAM (yet)

✦ So we store the data for N seconds

✦ Incoming raw data is processed on the fly

✦ Raw data older than N seconds is thrown away

Sliding WindowData

Raw Data✦ Request duration

✦ Server hostname

✦ Domain name

✦ Script name

✦ Bytes sent

✦ Rusage

✦ Timers

✦ Some more

Timers

✦ A way to measure certain parts of application

✦ Timer properties:

Time

CPU time

Tags: group=mysql, operation=insert, result=ok

Timers

Pseudo-code example:T = timer_start([group=mysql, op=insert, server=dbs10])

R = mysql_insert(“INSERT INTO..”)

timer_stop(T)

timer_add_tag(T, [result = (R ? ‘ok’ : ‘error’)])

Aggregated Data

✦ Basic reports

✦ Timer reports

✦ Request tag reports

Basic Reports

✦ info - the most basic of all

✦ report aggregated by script

✦ report aggregated by domain

✦ report aggregated by hostname

✦ and variations of all of the above

Creating a ReportCREATE TABLE `info` (

`req_count` int(11) DEFAULT NULL,

`time_total` float DEFAULT NULL,

`ru_utime_total` float DEFAULT NULL,

`ru_stime_total` float DEFAULT NULL,

`time_interval` int(11) DEFAULT NULL,

`kbytes_total` float DEFAULT NULL,

`memory_footprint` float DEFAULT NULL,

`req_time_median` float DEFAULT NULL

) ENGINE=PINBA COMMENT='info';

Example: infoMySQL [pinba]> select req_count, time_interval, req_time_median from info;

+-----------+---------------+-----------------+

| req_count | time_interval | req_time_median |

+-----------+---------------+-----------------+

| 137911 | 100 | 0.0179072 |

+-----------+---------------+-----------------+


MySQL [pinba]> select req_count/time_interval from info;

+-------------------------+

| req_count/time_interval |

+-------------------------+

| 1389.9600 |

+-------------------------+


Example: report by scriptCREATE TABLE `report_by_script_name` (


`req_per_sec` float DEFAULT NULL,

`req_time_total` float DEFAULT NULL,

`req_time_percent` float DEFAULT NULL,

`req_time_per_sec` float DEFAULT NULL,


`ru_utime_percent` float DEFAULT NULL,

`ru_utime_per_sec` float DEFAULT NULL,


`ru_stime_percent` float DEFAULT NULL,

`ru_stime_per_sec` float DEFAULT NULL,

`traffic_total` float DEFAULT NULL,

`traffic_percent` float DEFAULT NULL,

`traffic_per_sec` float DEFAULT NULL,

`script_name` varchar(128) DEFAULT NULL,

`memory_footprint_total` float DEFAULT NULL,

`memory_footprint_percent` float DEFAULT NULL,

`req_time_median` float DEFAULT NULL,

`index_value` varchar(256) DEFAULT NULL

) ENGINE=PINBA COMMENT='report1'

Example: report by scriptMySQL [pinba]> select req_count, req_per_sec, script_name, req_time_median from

report_by_script_name order by req_per_sec desc limit 5;

+-----------+-------------+---------------------------------------+-----------------+

| req_count | req_per_sec | script_name | req_time_median |

+-----------+-------------+---------------------------------------+-----------------+

| 91179 | 911.79 | /comet/init.phtml | 0.0166647 |

| 7432 | 74.32 | Web\WebAppFrame::index | 0.0855364 |

| 5603 | 56.03 | /profile.phtml | 0.0182654 |

| 2994 | 29.94 | Web\Web_Profile::actionNonAuthorized | 0.143617 |

| 2353 | 23.53 | Web\Web_BigPhoto::actionNonAuthorized | 0.068779 |

+-----------+-------------+---------------------------------------+-----------------+

Timer/tag reports

✦ aggregated by tag values (1/2/N tags)

✦ aggregated by script and tag values

✦ aggregated by script, hostname,

domain name and tag values

Timer/tag reportsMySQL [pinba]> select tag_value, hit_count, timer_value/hit_count

from tag_info_group order by timer_value desc limit 5;

+------------------------+-----------+-----------------------+

| tag_value | hit_count | timer_value/hit_count |

+------------------------+-----------+-----------------------+

| db::select | 71480 | 0.010082374235472334 |

| upload-photos | 1402 | 0.4313258281277862 |

| external_api::facebook | 794 | 0.7173030670704109 |

| add-photo | 1373 | 0.2622292804092885 |

| create-thumbs | 2425 | 0.09684005265383376 |

+------------------------+-----------+-----------------------+

5 rows in set (0.05 sec)

Timer/tag reportsMySQL [pinba]> select server_value, group_value, hit_count,

timer_value/hit_count from tag_info_group_server order by timer_value desc,

hit_count desc limit 5;

+-----------------+----------------+-----------+------------------------+

| server_value | group_value | hit_count | timer_value/hit_count |

+-----------------+----------------+-----------+------------------------+

| dphotos23 | upload-photos | 127 | 0.6577309435746801 |

| udb.mlan:9998 | hsc_connect | 105349 | 0.00042753526756620405 |

| udb.mlan:9998 | hsc_open_index | 114012 | 0.0003891547295108944 |

| dphotos30 | upload-photos | 56 | 0.7531399726867676 |

| udb.mlan:9998 | hsc_find | 98282 | 0.00042770797789420684 |

+-----------------+----------------+-----------+------------------------+

5 rows in set (0.06 sec)

Percentiles & Histograms

Numberof

requests

Request time

PercentilesCREATE TABLE `info` (


`time_total` float DEFAULT NULL,



`time_interval` int(11) DEFAULT NULL,

`kbytes_total` float DEFAULT NULL,

`memory_footprint` float DEFAULT NULL,

`req_time_median` float DEFAULT NULL,

`p95` float DEFAULT NULL,

`p99` float DEFAULT NULL

) ENGINE=PINBA COMMENT='info:::95,99';

Report Filters

✦ Min time

✦ Max time

✦ Request tags

Report Filters: max_time

Request time: any, >0.1s, >0.2s, >0.3s, >0.4s, >0.5s

Client libraries & plugins

✦ PHP extension

✦ Nginx plugin

✦ Java client

✦ Pure PHP client

✦ Ruby client

✦ Node.js client

✦ Go client

Pinba in Badoo✦ PHP requests

✦ HTTP requests (nginx)

✦ Mobile apps

✦ Browser (JS) requests

✦ MySQL queries

✦ C/Go daemons

✦ Internal queues

✦ Error logs

pinba.orgMore info at

Thank you!

The original, largest and leadingdating network

pinba in-memory mysql storage engine for a full-stack real ... · pinba in-memory mysql storage...

Documents