proxysql - high performance and ha proxy for mysql

56
@proxysql http://proxysql.com https://github.com/sysown/proxysql/ sysown.com ProxySQL High Performance & High Availability Proxy for MySQL René Cannaò [email protected]

Upload: rene-cannao

Post on 25-Jan-2017

2.090 views

Category:

Data & Analytics


11 download

TRANSCRIPT

Page 1: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

ProxySQL

High Performance & High AvailabilityProxy for MySQL

René Cannaò[email protected]

Page 2: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

GAStable and production readyversion 0.2

Page 3: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Genesis: main motivations

rewrite queriescache frequent reads

solving/finding the answer forDEVs vs DBAs

Page 4: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Open Source proxies for MySQL

MySQL Proxy

HAProxyhttp://tinyurl.com/ptjmzde

Now also MaxScale

Page 5: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

MySQL Proxy

Pros:customizableexpandableacceptable performance

Cons:CPU intensiveNot really reliableNot maintained

Page 6: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

HAProxy

Pros:very stablehigh performancemature software

Cons:Layer 7 proxy only for HTTPDoesn’t understand the MySQL ProtocolLayer 4 for other services

Page 7: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Proxy for rewrite and caching?

MySQL Proxy with:

Lua scriptExternal caching

Page 8: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

ProxySQLNetwork database proxy that sits transparently between the application and the database to:

empower the DBAsimprove operationunderstand and improve performance

Page 9: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

The right way!

Designed by a DBA for DBAsExperience in a variety of setupsExperience in very large production environments

Page 10: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

FeaturesSome of the most interesting features:

on-the-fly rewrite of queriescaching reads outside the database serverconnection pooling and multiplexingcomplex query routing and read/write splitload balancingreal time statisticsMonitoringHigh Availability and ScalabilitySeamless failoverFirewallQuery throttlingquery timeoutruntime reconfiguration

Page 11: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Hostgroups and Query Routing

All backends are grouped into hostgroups

Hostgroups have logical functionalities

Page 12: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Hostgroups example #1

HostGroup0 (HG0): Write masters

HostGroup1( HG1): Read slaves

Read/Write split

Page 13: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Hostgroups example #2

HG0: main write mastersHG1: main read slavesHG2: reporting slavesHG3: ad-hoc queries slavesHG4: data warehouse write masters HG5: data warehouse read slaves HG6: remote site serversHG7: test servers

Page 14: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Support for heterogeneous setups

Hostgroups have logical functionalities

No replication relationship between hostgroups

ProxySQL is not replication aware by design, yet it monitors replication

A query is routed to a single hostgroup

Page 15: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Basic design

HG0

HG2

HG1

ProxySQLAPP

Page 16: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Basic design , example #2

HG0

HG2

HG1

ProxySQL

APP1

APP2

APP3

Page 17: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Internals diagram

HG0

HG1

APP1 ProxySQL

Queries cache

Queries Processor

APP2

APP3 Thr

ead

#1

Thr

ead

#2

Thr

ead

#3

Page 18: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries Processor

Rewrite queriesDefines what to cacheDefines the hostgroup target

Page 19: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries rules

Complex rules to match incoming traffic.Regex on queryusernameschemaname(a lot more in the roadmap)

Rules can be chained

Page 20: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries cache

Caching on the wireInternal key/value storageIn memory onlyPattern basedExpired by timeout

Page 21: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries Cache Benchmark

Benchmark on a 4 cores server

Page 22: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries Rewrite

Rewrite on the wire

Regex match/replace

Optionally cached

Page 23: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries Rewrite Benchmark

Ref: http://tinyurl.com/oan4528

Page 24: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Users Authentication

HG0

HG1

APP1 ProxySQL

Queries cache

Query Processor

APP2

APP3 Thr

ead

#1

Thr

ead

#2

Thr

ead

#3Users Auth

Page 25: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Users Authentication

Credentials stored in the proxy

User login always possible (even without backends)

Max connections

Security enhancement in roadmap:different credentials on backends and frontendsinternal mapping

Page 26: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Hostgroup Managerand Connections Pool

HG0

HG1

APP1 ProxySQL

Queries cache

Query Processor

APP2

APP3 Thr

ead

#1

Thr

ead

#2

Thr

ead

#3Users Auth

ConnectionPool

HostgroupManager

Page 27: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Hostgroups Manager

Management of servers

Track servers status

Tightly integrated with the connections pool

Page 28: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Connections Pool

Reduced the overhead of creating new connections, and are recycled when not in use

One to many connectionsMultiplexing & maximum connectionsAuto-reconnect and automatic re-execution of queriesFailover management

Page 29: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Multiplexing

Reduce the number of connections against mysqld (configurable)

Many clients connections (tens of thousands) can use few backend connections (few hundreds)

Order by waiting time

Few edge cases not handled correctly: open a feature request if you hit any

Page 30: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Auto-reconnect and re-execution

Automatic detection of failuresGraceful handlingAuto-reconnect when possiblePause until a backend becomes available

Re-execution of queries

Page 31: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Failover management

Seamless switchover:http://www.proxysql.com/2015/09/proxysql-tutorial-seamless-replication.html

Managed by external processSwitchover in less than 1 second

Page 32: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Monitoring Module

HG0

HG1

APP1 ProxySQL

Queries cache

Query Processor

APP2

APP3 Thr

ead

#1

Thr

ead

#2

Thr

ead

#3Users Auth

ConnectionPool

HostgroupManager

Monitoring

Page 33: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Monitoring Module

It monitors backends and collects metrics

Monitors replication lag and shun hosts

Page 34: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Admin Interface

HG0

HG1

APP1 ProxySQL

Queries cache

Query Processor

APP2

APP3 Thr

ead

#1

Thr

ead

#2

Thr

ead

#3Users Auth

ConnectionPool

HostgroupManager

Admin

Monitoring

Page 35: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Admin Interface

Allows runtime configurationExports internal statuses

It uses MySQL protocolConfiguration possible from any client/tool using MySQL API

Page 36: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Admin Interface

mysql> show databases;+-----+---------+------------------+| seq | name | file |+-----+---------+------------------+| 0 | main | || 2 | disk | /tmp/proxysql.db || 3 | stats | || 4 | monitor | || 5 | myhgm | |+-----+---------+------------------+5 rows in set (0.00 sec)

Page 37: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Admin Interface

mysql> SHOW TABLES FROM main;+-------------------+| tables |+-------------------+| mysql_servers || mysql_users || mysql_query_rules || global_variables || mysql_collations || debug_levels |+-------------------+6 rows in set (0.01 sec)

Page 38: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Admin Interface

mysql> SHOW TABLES FROM stats;+--------------------------------+| tables |+--------------------------------+| stats_mysql_query_rules || stats_mysql_commands_counters || stats_mysql_processlist || stats_mysql_connection_pool || stats_mysql_query_digest || stats_mysql_query_digest_reset || stats_mysql_global |+--------------------------------+7 rows in set (0.00 sec)

Page 39: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Admin Interface

mysql> SHOW TABLES FROM monitor;+----------------------------------+| tables |+----------------------------------+| mysql_server_connect || mysql_server_connect_log || mysql_server_ping || mysql_server_ping_log || mysql_server_replication_lag_log |+----------------------------------+5 rows in set (0.00 sec)

Page 40: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries cache rules

proxysql-admin> SELECT match_pattern , negate_match_pattern neg , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NULL ORDER BY rule_id ; +---------------------+------+-------+------+ | match_pattern | neg | hs_id | ttl | +---------------------+------+-------+------+ | ^SELECT | 1 | 0 | -1 | | \s+FOR\s+UPDATE\s*$ | 0 | 0 | -1 | | .* | 0 | 1 | 30 | +---------------------+------+-------+------+

Page 41: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Queries rewrite rules

proxysql-admin> SELECT match_pattern m, replace_pattern r , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NOT NULL\G

m: ^SELECT(| DISTINCT) c FROM sbtest WHERE id BETWEEN (.*) AND (.*) ORDER BY c

r: SELECT c FROM sbtest WHERE id BETWEEN \2 AND \3hs_id: 1 ttl: 30

Page 42: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Deploy ProxySQL

Page 43: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Location, location, location

ProxySQL sits between the application and the database systems

Where is exactly the best location?

Page 44: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Basic design

HG0

HG2

HG1

APPAPP

ProxySQL

Very low latency usingUnix Domain Socket

Page 45: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Basic design

HG0

HG2

HG1

APP1

ProxySQL

APP2

ProxySQL

APP3

ProxySQL

Page 46: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

High Availability

HG0

HG2

HG1

APP1

ProxySQL

APP2

ProxySQL

APP3

ProxySQL

HG0

ProxySQL

ProxySQL

Page 47: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

HG0

HG2

HG1

APP1

ProxySQL

APP3

ProxySQL

ProxySQL

ProxySQL

Complex Potential Setups

APP2

ProxySQL

ProxySQL

HG0

HG0

HG0

HG1

ProxySQL

ProxySQL

Page 48: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Hostgroup reconfiguration

HG0

HG1

1

43

ProxySQLAPP 2

Remove host :all the connections to server2 are terminated

1

Add host :New connections to HG1 can use also server1

Page 49: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Failover

2 phases process:remove hostadd host

Fully supportedSeamless failover in under 1 second

Page 50: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Failover : 2 phases task

HG0

HG1

1

43ProxySQLAPP 2

1.Connections to an empty hostgroup are put on hold

2

2.Connections to HG0 are resumedConnections to HG1 were never stopped

1. remove host2. add host

Page 51: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Distributed failover

Multiple ProxySQL are available in a network

Failover managed by an external process that:remove host from each ProxySQL instanceadd new host into each ProxySQL instance

Manager is not part of ProxySQL.Ex: MHA or MySQL Utilities

Page 52: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

APP1

ProxySQL

APP2

ProxySQL

Distributed failover : remove host

HG0

1

HG1

32 4

MHA

Page 53: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

APP1

ProxySQL

APP2

ProxySQL

Distributed failover : add host

HG0

1

HG1

3 4

MHA

2

Page 54: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

New features in roadmapImproved security (SSL support, SQL injection

detection, frontend/backend users, etc) ;GUI;Prepared statements;Support for multiple backends (postgres, redis,

mongo) ;Enhanced monitoring and statistics;

Page 55: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Try it!

Source code on GitHub:https://github.com/sysown/proxysql

Binaries on GitHub:https://github.com/sysown/proxysql-binaries

Forum:https://groups.google.com/forum/#!forum/proxysql

Tutorial on:http://www.proxysql.com

Page 56: ProxySQL - High Performance and HA Proxy for MySQL

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

Please contact me

Thanks!

Feedback, feature requests, contribution:[email protected]@sysown.com@rene_cannao@proxysql