high performance & high availability proxy for mysql ......complex query routing and read/write...
TRANSCRIPT
![Page 1: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/1.jpg)
Santa Clara, California | April 23th – 25th, 2018
ProxySQL TutorialHigh Performance & High Availability
Proxy for MySQL
With a GPL license!
![Page 2: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/2.jpg)
2
Who we are
René Cannaò
ProxySQL Founder
Derek Downey
Director of OSDB, Pythian
![Page 3: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/3.jpg)
3
Agenda● What is ProxySQL?● Features● Fundamentals● What is new?● Internals● Admin● Clustering● Rules● Mirroring● Failover● New protocols● GTID casual reads
![Page 4: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/4.jpg)
4
Other sessions
Consistent Reads Using ProxySQL and GTID
Tuesday April 24, 2018 @3:50PM
Automatic Failovers with Kubernetes using Orchestrator, ProxySQL and Zookeeper
Tuesday April 24, 2018 @3:50PM
![Page 5: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/5.jpg)
Overview of ProxySQL
![Page 6: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/6.jpg)
6
What is ProxySQL
The MySQL data stargate
![Page 7: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/7.jpg)
7
Main motivations
empower the DBAs
improve operation
understand and improve performance
create a proxy layer to shield the database
High performance and High Availability
![Page 8: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/8.jpg)
ProxySQL FeaturesSome of the most interesting features:
● on-the-fly rewrite of queries
● caching reads outside the database server
● connection pooling and multiplexing
● complex query routing and read/write split
● load balancing
● real time statistics
● monitoring
● Data masking
● Multiple instances on same ports
![Page 9: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/9.jpg)
ProxySQL Features (2)Some of the most interesting features:
● High Availability and Scalability
● seamless failover
● firewall
● query throttling
● query timeout
● query mirroring
● runtime reconfiguration
● Scheduler
● Support for Galera/PXC and Group Replication
![Page 10: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/10.jpg)
![Page 11: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/11.jpg)
What’s new in ProxySQL 1.4.0Native Support for Group Replication
Multiple regex engines
Better performance
Support for millions of users
Support for tens of thousands of database servers
Better queries control
http://www.proxysql.com/blog/releasing-proxysql-140
![Page 12: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/12.jpg)
What’s new in ProxySQL 1.4.7
Native ProxySQL Clustering Solution
Support for ClickHouse as a backend
SQLite network
Support for millions of shards
![Page 13: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/13.jpg)
What’s new in ProxySQL 2.0 (dev)… all in 1.4.x plus:
Support for Aurora
SSL support for frontend
SSLv1.2
Native Support for Galera
Causal reads using GTID
Multi-threaded connections reset
![Page 14: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/14.jpg)
Hostgroups and Query Routing
All backends are grouped into hostgroups
Hostgroups have logical functionalities
![Page 15: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/15.jpg)
Hostgroups example #1
HostGroup0 (HG0): Write mastersHostGroup1( HG1): Read slaves
Read/Write split
![Page 16: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/16.jpg)
Hostgroups example #2HG0: main write masters
HG1: main read slaves
HG2: reporting slaves
HG3: ad-hoc queries slaves
HG4: data warehouse write masters
HG5: data warehouse read slaves
HG6: remote site servers
HG7: test servers
HG8 : mirror for traffic on HG0
HG9 : mirror for traffic on HG1
![Page 17: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/17.jpg)
Basic designHG0
HG2
HG1
ProxySQL
APP1
APP2
APP3
![Page 18: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/18.jpg)
Basic designHG0
HG2
HG1
APPAPP
ProxySQL
Very low latency usingUnix Domain Socket (?)
![Page 19: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/19.jpg)
One ProxySQL per host
![Page 20: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/20.jpg)
Silos approach
![Page 21: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/21.jpg)
Clustered ProxySQL Architecture
DB DBDB
APP
DB
PROXYSQL
DB DB
APPPROXYSQL
APPPROXYSQL
APPPROXYSQL
PROXYSQL PROXYSQL
DBMANAGER
PROXYSQL
Clustered Proxy Layer
![Page 22: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/22.jpg)
Clustered ProxySQL at scaleTested with:
● 8 app servers with 3k clients’ connections each (24k total)● 4 middle layer proxysqls processing 4k connections each from local
proxysqls (16k total)● 256 backends/shard (meaning 256 routing rules) processing 600
connections each (150k total)
Single ProxySQL was tested with up to 950k connections
At today, ProxySQL is able to process up to 750k QPS
![Page 23: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/23.jpg)
ProxySQL Internals
![Page 24: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/24.jpg)
ProxySQL Modules in 1.3.xHG0
HG1
APP1 ProxySQL
Queries cache
Query Processor
APP2
APP3
Thre
ad #
1
Thre
ad #
2
Thre
ad #
3
Users Auth
ConnectionPool
HostgroupManager
Admin
Monitoring
![Page 25: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/25.jpg)
ProxySQL Modules in 1.4.xHG0
HG1
APP1 ProxySQL
Queries cache
Query Processor
APP2
APP3Th
read
#1
Thre
ad #
2
Thre
ad #
3
Users AuthConnection
Pool
HostgroupManager
Admin
MonitoringClickHouse Server
ClickHouse Auth
Cluster
SQLite3 Server
Thre
ad #
3 id
les
Thre
ad #
2 id
les
Thre
ad #
1 id
les Purge Thread
![Page 26: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/26.jpg)
Queries ProcessorBased on Queries Rules
Defines what to cache
Defines the hostgroup target
Timeout/delay
Firewall
Mirroring
Rewrite queries
![Page 27: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/27.jpg)
Queries rulesComplex rules to match incoming traffic:
● regex on query
● regex on digest text
● username
● schemaname
● Source IP address
● Bind IP address/port
● digest
Rules can be chained
![Page 28: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/28.jpg)
Queries Cache and RewriteCaching on the wire
Internal key/value storage
In memory only
Pattern based
Expired by timeout
Rewrite on the wire
Regex match/replace on query on digest text
Optionally cached or mirrored
![Page 29: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/29.jpg)
Users Authentication
Credentials stored in the proxy
User login always possible (even without backends)
Max connections
Login credentials are encrypted
Extremely scalable, up to millions of users:
https://www.percona.com/live/e17/sessions/scaling-million-databases-000webhost-lt
![Page 30: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/30.jpg)
Hostgroups Manager
Management of servers
Track servers status
Tightly integrated with the connections pool
![Page 31: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/31.jpg)
Connections Pool
Reduced the overhead of creating new connections, and are recycled when not in use
One to many connections
Multiplexing & maximum connections
Auto-reconnect and automatic re-execution of queries
Failover management
![Page 32: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/32.jpg)
Purge Thread
Unsafe connections are reset instead of being dropped
Drastically reduce the need to establish new connections
In ProxySQL 2.0 : each worker thread perform most of the reset
![Page 33: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/33.jpg)
Auto-reconnect and re-execution
Automatic detection of failures
Graceful handling
Auto-reconnect when possible
Pause until a backend becomes available
Re-execution of queries
![Page 34: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/34.jpg)
Multiplexing
Reduce the number of connections against mysqld (configurable)
Many clients connections (tens of thousands) can use few backend connections (few hundreds)
Tracks connection status (transactions, user variables, temporary tables, etc)
Order by waiting time
![Page 35: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/35.jpg)
Multiplexing (see it in web UI)
![Page 36: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/36.jpg)
Multiplexing (see it in web)
![Page 37: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/37.jpg)
More metrics on web UI
![Page 38: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/38.jpg)
More metrics on web UI
![Page 39: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/39.jpg)
More metrics on web UI
![Page 40: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/40.jpg)
Monitoring Module
It monitors backends and collects metrics
Monitors replication lag and shun hosts
Monitors read_only (and super_read_only and innodb_read_only in 2.0) variables
(replication hostgroups)
Ping and terminates unresponsive nodes
![Page 41: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/41.jpg)
Monitoring ModuleBuilt in support for asynchronous replication:
second behind master and pt-heartbeat
Support for any type of clustering solution via external scripts
(Galera/PXC , and any heterogeneous replication setup)
In ProxySQL 1.4.0, support for Group Replication
In ProxySQL 2.0, support for Galera
![Page 42: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/42.jpg)
Admin Module
![Page 43: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/43.jpg)
Admin InterfaceAllows runtime configuration
Exports internal statuses
It uses MySQL protocol
Configuration possible from any client/tool using MySQL API
Covered during the tutorial
![Page 44: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/44.jpg)
Admin InterfaceDiskMemoryRuntime
Runtime
Memory
DiskConfig
![Page 45: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/45.jpg)
Try it!Source code on GitHub:
https://github.com/sysown/proxysql/
Forum:
https://groups.google.com/forum/#!forum/proxysql
Tutorials on:
http://www.proxysql.com
![Page 46: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/46.jpg)
Demo Environment
![Page 47: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/47.jpg)
47
DemoCode on GitHub:https://github.com/renecannao/ProxySQL-Tutorial-PLSC2018
Some hosts provided:
Username: plsc
Passwords: proxysql
![Page 48: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/48.jpg)
ProxySQL Cluster
![Page 49: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/49.jpg)
ProxySQL ClusterNew in 1.4.2 – experimental (not fully completed, subject to change)
Support for MySQL Query Rules, MySQL Users, MySQL Servers and ProxySQL Servers
A group of instances constantly check each other for changes in configuration
A configuration change applied in a proxy is pulled by the other proxies
Allow to keep multiple ProxySQL instances with the same configuration(s)
![Page 50: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/50.jpg)
ProxySQL Cluster
![Page 51: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/51.jpg)
ProxySQL ClusterConflict resolution is timestamp based• No order• Last win
No quorum (in the roadmap)
Highly scalable: Core nodes vs satellite nodes
![Page 52: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/52.jpg)
52
ProxySQL Cluster - Configmysql> SET admin-cluster_username='radmin';
mysql> SET admin-cluster_password='radmin';
mysql> LOAD ADMIN VARIABLES TO RUNTIME;
mysql> SAVE ADMIN VARIABLES TO DISK;
mysql> INSERT INTO proxysql_servers (hostname) VALUES
('proxysql1'),('proxysql2'),('proxysql3');
mysql> LOAD PROXYSQL SERVERS TO RUNTIME;
mysql> SAVE PROXYSQL SERVERS TO DISK;
mysql> SHOW VARIABLES LIKE 'admin-cluster%';+---------------------------------------------------+--------+| Variable_name | Value |+---------------------------------------------------+--------+| admin-cluster_username | radmin || admin-cluster_password | radmin || admin-cluster_check_interval_ms | 1000 || admin-cluster_check_status_frequency | 10 || admin-cluster_mysql_query_rules_diffs_before_sync | 3 || admin-cluster_mysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_users_diffs_before_sync | 3 || admin-cluster_proxysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_query_rules_save_to_disk | true || admin-cluster_mysql_servers_save_to_disk | true || admin-cluster_mysql_users_save_to_disk | true || admin-cluster_proxysql_servers_save_to_disk | true |+---------------------------------------------------+--------+12 rows in set (0.00 sec)
![Page 53: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/53.jpg)
53
ProxySQL Cluster - MonitoringStats_proxysql_servers_checksums
Stats_proxysql_servers_metrics
![Page 54: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/54.jpg)
ProxySQL ClusterDEMO
![Page 55: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/55.jpg)
MySQL Query rules
![Page 56: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/56.jpg)
56
Query rewriting overview
Sql-aware
Regex-based
flexible
![Page 57: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/57.jpg)
57
Query rewriting examples
read/write splitting
Data masking
Sharding
![Page 58: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/58.jpg)
58
Query rewriting flow
![Page 59: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/59.jpg)
Match Related Tablesmysql_users
• active• fast_forward(bypass)
mysql_query_rules• username• schemaname• flagIN/OUT• client_addr• proxy_addr• proxy_port• digest• match_digest• match_pattern• negate_match_pattern
![Page 60: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/60.jpg)
Action related fieldsflagOUTreplace_patterndestination_hostgroupcache_ttltimeoutretriesdelaymirror_flagOUTmirror_hostgroupOK_msgerror_msglogmultiplexapply
active
![Page 61: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/61.jpg)
3 layers for ProxySQL tablesDiskMemoryRuntime
Runtime
Memory
DiskConfig
![Page 62: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/62.jpg)
PrecautionsConfirm the query hits with stats_mysql_query_rulesdouble check rules and rewrite resultshold off on writing rules to disk until you're sure they are working as expectedhave a rollback plan
• load from disk, • then load from memory to runtime
test select query rules and rewrites on a slaves firsttest mutable rules on a mirror or other throw away db.
![Page 63: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/63.jpg)
63
Example - Simple R/Wmysql> SELECT username, default_hostgroup FROM mysql_users\G
*************************** 1. row ***************************
username: root
default_hostgroup: 0
1 row in set (0.00 sec)
mysql> SELECT match_digest, destination_hostgroup FROM
mysql_query_rules\G
*************************** 1. row ***************************
match_digest: ^SELECT.*FOR UPDATE
destination_hostgroup: 0
*************************** 2. row ***************************
match_digest: ^SELECT
destination_hostgroup: 1
2 rows in set (0.00 sec)
![Page 64: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/64.jpg)
Query Rewriting with ProxySQLDEMO
Click to add text
![Page 65: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/65.jpg)
Mirroring with ProxySQLClick to add text
![Page 66: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/66.jpg)
66
What is mirroring?
![Page 67: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/67.jpg)
67
What is mirroring?
![Page 68: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/68.jpg)
68
Why mirror queries?
Validate performance on a different server using different hostgroups.
Validate performance of query rewrite or schema change
Pre-fetch slave replication (Replication Booster)
![Page 69: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/69.jpg)
69
How to mirrormysql>SHOW CREATE TABLE mysql_query_rules\G
*********************** 1. row ************************
table: mysql_query_rules
Create Table: CREATE TABLE mysql_query_rules (
rule_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 0,
username VARCHAR,
schemaname VARCHAR,
flagIN INT NOT NULL DEFAULT 0,
*snip*
mirror_flagOUT INT UNSIGNED,
mirror_hostgroup INT UNSIGNED,
error_msg VARCHAR,
*snip*)
![Page 70: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/70.jpg)
70
Mirroring flow
![Page 71: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/71.jpg)
71
Mirroring - example #1
mysql> SELECT username, destination_hostgroup, mirror_hostgroup,
mirror_flagOUT FROM mysql_query_rules WHERE
username='plam_mirror'\G
****************** 1. row **********************
username: pl_mirror
destination_hostgroup: 3
mirror_hostgroup: 4
mirror_flagOUT: NULL
![Page 72: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/72.jpg)
72
Mirroring - example #2
![Page 73: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/73.jpg)
Mirroring DemoClick to add text
![Page 74: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/74.jpg)
74
Mirroring caveats
Can be dropped: mysql-mirror_max_concurrency,
mysql-mirror_max_queue_length
Can be executed out of order
No error handling of failed queries
![Page 75: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/75.jpg)
Failover with ProxySQL
![Page 76: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/76.jpg)
76
Failover highlights
improve failover time as perceived by the application
prevent errors sent to the application
perform transparent database failovers: gracefully redirecting traffic without the application knowing
existing applications do not have to be rewritten to autoreconnect since connections are not lost from failovers
![Page 77: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/77.jpg)
77
Failover scenario - without ProxySQL
![Page 78: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/78.jpg)
78
Failover scenario - without ProxySQL
![Page 79: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/79.jpg)
79
Failover scenario - with ProxySQL
![Page 80: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/80.jpg)
80
Failover scenario - with ProxySQL
![Page 81: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/81.jpg)
81
Failover scenario - with ProxySQL
![Page 82: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/82.jpg)
82
Failover scenario - with ProxySQL
![Page 83: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/83.jpg)
83
Failover without ProxySQL (10 minute outage)
https://www.nylas.com/blog/growing-up-with-mysql/
![Page 84: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/84.jpg)
84
Failover with ProxySQL (10 second outage)
https://www.nylas.com/blog/growing-up-with-mysql/
![Page 85: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/85.jpg)
85
Failover - MySQL Serversmysql> SELECT hostgroup_id, hostname, status FROM mysql_servers
WHERE hostname IN ('mysql1, 'mysql2’)\G
*************************** 1. row ***************************
hostgroup_id: 0
hostname: mysql1
status: ONLINE
*************************** 2. row ***************************
hostgroup_id: 1
hostname: mysql2
status: ONLINE
2 rows in set (0.00 sec)
![Page 86: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/86.jpg)
86
Failover - Replication Hostgroupsmysql> SELECT * FROM mysql_replication_hostgroups\G
*************************** 1. row ***************************
writer_hostgroup: 0
reader_hostgroup: 1
comment:
1 row in set (0.00 sec)
![Page 87: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/87.jpg)
87
ProxySQL Failover ProTIP
ProxySQL does not handle promotion or re-slaving
External process needed, such as MHA or Orchestrator
![Page 88: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/88.jpg)
Failover DemoClick to add text
![Page 89: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/89.jpg)
New FeaturesClick to add text
![Page 90: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/90.jpg)
ClickHouse ServerNew in 1.4.3 – experimental (not fully completed, subject to change)
Support for ClickHouse: https://clickhouse.yandex/
Fast, really fast data analytics using MySQL client
Users management
Supports only one backendSupports only DDL and SELECT (no INSERT)
![Page 91: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/91.jpg)
ClickHouse Server
![Page 92: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/92.jpg)
ClickHouse Server
![Page 93: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/93.jpg)
SQLite3 ServerNew in 1.4.3 – experimental (not fully completed, subject to change)
Support for SQLite3 : https://www.sqlite.org/
Fully transactional, in-memory
Same users as MySQL backends
A lot of features in the roadmap
![Page 94: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/94.jpg)
You-Name-It Server
What other service should become part of ProxySQL ?
![Page 95: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/95.jpg)
You-Name-It Server
What other service should become part of ProxySQL ?
![Page 96: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/96.jpg)
Causal read with GTIDFeature introduced in 2.0
Ensures that reads sent to slave returns what the client has written on master
Requires GTID and MySQL 5.7+
![Page 97: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/97.jpg)
97
ProxSQL Read / Write Split
![Page 98: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/98.jpg)
98
ProxySQL Binlog Reader
![Page 99: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/99.jpg)
99
ProxySQL Binlog Reader
![Page 100: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/100.jpg)
100
![Page 101: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/101.jpg)
We offer support
Subscriptions
Consulting
Professional services
![Page 102: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/102.jpg)
102
Rate My Session
![Page 103: High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances](https://reader033.vdocument.in/reader033/viewer/2022060907/60a15af80d123f05571e0b8e/html5/thumbnails/103.jpg)
Thank You!