con5209_correia-con5209 - mysql fabric tips and tricks - oow 2015


MySQL Fabric Tips and Tricks

Alfranio [email protected]

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3


Program Agenda

What’s MySQL Fabric?1

4

Configuring MySQL Fabric

Configuring Managed Servers

Failure Detection

Plan your Applications

2

3

4

5


Program Agenda

What’s MySQL Fabric?1

5



Failure Detection


2

3

4

5


MySQL Fabric is

An extensible and easy-to-use framework for managing a farm of MySQL Servers supporting high-availability and sharding.

6


What does all that mean?

• Management System – Manages a MySQL Farm– Distributed Framework

• Framework– Metadata Repository – Procedure Execution– Transaction Routing

• Extensible – Extensions are first-class– High-availability Groups– “Semi-automatic” Sharding

• Written in Python• Latest Release 1.5.6 GA• Open Source

– You can participate– Suggest features– Report bugs– Contribute patches

• MySQL Server 5.6– For now

7


MySQL Fabric: Goals & Features

• Decision logic in the connector – Eliminates network hop– Scale naturally: not a bottleneck

• Connector API extensions– Support transactions– Support full SQL

• Cloud integration– Support elasticity– Servers on-demand

8


Birds-eye View

• MySQL Fabric– Core Python Process –Metadata Repository (MySQL Server)

•High Availability Groups – Support for Primary-Secondary– Limited support for other HA Solutions

• Connector – Fabric-aware Connectors– MySQL Router

• Application – Designed to be fault-tolerant

9

Router is GA


MySQL Router

• Plugin-based architecture– Harness with general functionality

• Connection-based routing– Routing decision on connecting– Raw packet copy: very fast

• Fabric Information Plugin – Metadata cache– Use Fabric node to fetch metadata

10

Router is GA


Architecture

• Extensible RPC support:– MySQL– XML-RPC

• Extensible Framework – Event Driven System– Sharding, High Availability, ...

• State Store – Persistence Layer

• Executor ?

11

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 12

Crash-safe ProceduresRegular Execution


Crash-safe ProceduresFailover/Recovery Execution


Crash-safe ProceduresResume Execution


High Availability with Pacemaker

• Linux kernel Module Integrated into Oracle Linux• Synchronous replication• Only one MySQL Operational• Check instructions at examples/ocf/• Available in 1.5.6 release

15


New Architecture

• Replicated State Machine– No single point of failure– No bottleneck

• Quorum of nodes– At least three nodes

• Leader-based processing– One node handles write requests

16

New in Labs


Program Agenda

What’s MySQL Fabric?

2

17



Failure Detection


1

3

4

5


MySQL Fabric Pre-requisites

• MySQL Server 5.6.17 or later (Why? GTIDs):– State store– Managed Servers

• Python 2.6 or 2.7• Pure Connector Python 2.0.4 or later• Hybrid Connector Python 2.1.2 or later

18


• Super privilege on “fabric” database is required– GRANT ALL PRIVILEGES ON fabric.* TO 'fabric'@'%' IDENTIFIED BY 'password';

• Set host information properly

19

Accessing the State StoreConfiguring Privileges


• Super privilege on “fabric” database is required– GRANT ALL PRIVILEGES ON fabric.* TO 'fabric'@'%' IDENTIFIED BY 'password';


20

Accessing the State StoreConfiguring Privileges

TIP#1 Put MySQL Fabric and its storage in the same host.


• Configure MySQL State Store– Address– User/Password– Attempts to reconnect

• Security improved in 1.6.2

21

Accessing the State StoreConfiguration File

...[storage]address = localhost:13000user = fabricpassword = passworddatabase = fabricconnection_timeout = 6connection_attempts = 6connection_delay = 1


• Configure MySQL State Store– Address– User/Password– Attempts to reconnect

• Security improved in 1.6.2

22

Accessing the State StoreConfiguration File

...[storage]address = localhost:13000user = fabricpassword = passworddatabase = fabricconnection_timeout = 6connection_attempts = 6connection_delay = 1

TIP#2 Set connection options to the state store properly.


• Any bind address but 0.0.0.0– Causes issues with Multi-node Fabric– Causes issues with some connectors

• Don’t disable authentication

23

Accessing MySQL FabricConfiguration File

...[protocol. xmlrpc]address = 192.168.1.1:32274disable_authentication = no.....[protocol. mysql]address = 192.168.1.1:32275disable_authentication = no


$ mysqlfabric --config fabric-1.cfg user add connectors --protocol=mysqlAdd a new Fabric user=====================Username: connectorsProtocol: mysqlPassword: Repeat Password:

Accessing MySQL FabricConfigure Credentials (Password)


Select role(s) for new userID Role Name Description and Permissions-- -------------- -------------------------------------1 superadmin Role for Administrative users

+ Full access to all core Fabric functionality2 useradmin Role for users dealing with user administration

+ User administration+ Role administration

3 connector Role for MySQL Connectors+ Access to dump commands+ Reporting to Fabric

Enter comma separated list of role IDs or names: 3

Accessing MySQL FabricConfigure Credentials (Role)


$ mysqlfabric --config fabric-1.cfg user add connectors --protocol=xmlrpcAdd a new Fabric user=====================Username: connectorsProtocol: mysqlPassword: Repeat Password:

Accessing MySQL FabricConfigure Credentials (Password)

TIP#3 Set same user/password for both protocols.


Program Agenda


27



Failure Detection


1

3

4

5

2


...[servers]user = fabric_userpassword = passwordbackup_user = fabric_backup backup_password = passwordrestore_user = fabric_restore restore_password = password

Accessing Managed ServersConfiguration File




TIP#4 User will be used to access managed servers.




TIP#4 User will be used to access managed servers.

TIP#5 User will be used to establish replication among servers.


• GRANT ... ON *.* 'fabric_user'@'%' IDENTIFIED BY 'password';– SELECT Shard split/move– DELETE Shard split/move– TRIGGER Shard Checking– PROCESS Manage connections – RELOAD Replication– REPLICATION CLIENT Replication– REPLICATION SLAVE Replication– SUPER Manage connections


31

Accessing the Managed ServersGlobal Privileges for Managing


• GRANT ... ON mysql_fabric.* 'fabric_user'@'%' IDENTIFIED BY 'password'; – ALTER– CREATE– DELETE– DROP– INSERT– SELECT– UPDATE


32

Accessing the Managed ServersDatabase Privileges for Managing


High Availability Groups

• Group of Servers:– Hardware redundancy– Software redundancy

• Generic Concept: – Full support for Primary-Backup– Limited support to any HA Solution

• Can be used to register servers only:

– --update_only option– Use Fabric as a metadata repository only

33

To avoid accessing and changing a server use “update_only”.


Configure a group

34

$ mysqlfabric --config fabric.cfg group create YYZ$ mysqlfabric --config fabric.cfg group add YYZ 192.168.5.10:3306$ mysqlfabric --config fabric.cfg group add YYZ 192.168.5.11:3306...$ mysqlfabric --config fabric.cfg group activate$ mysqlfabric --config fabric.cfg group promote –slave_id 192.168.5.10:3306


Configure a group

35

$ mysqlfabric --config fabric.cfg group create YYZ$ mysqlfabric --config fabric.cfg group add YYZ 192.168.5.10:3306$ mysqlfabric --config fabric.cfg group add YYZ 192.168.5.11:3306...$ mysqlfabric --config fabric.cfg group activate$ mysqlfabric --config fabric.cfg group promote –slave_id 192.168.5.10:3306

TIP#6 Use IPs for managed server unless your DNS Service is also highly available.


Only update the state store and skip provisioning

36

$ mysqlfabric --config fabric.cfg group create Presto$ mysqlfabric --config fabric.cfg group add Presto 192.168.5.13:3306 –update_only$ mysqlfabric --config fabric.cfg group add Presto 192.168.5.14:3306 –update_only...$ mysqlfabric --config fabric.cfg group promote –slave_id 192.168.5.13:3306 –update_only

It could be used when servers are already deployed and running.


Program Agenda


37



Failure Detection


1

3

4

5

2


Built-in Failure Detector

• Group Level Failure Detector• Monitor Servers within a Group

– Support for Primary-Secondary

• On master failure– Mark master as faulty– Trigger fail-over

• On slave failure – Mark slave as faulty

• Solution is only for servers managed by Fabric

38


...[failure_tracking]failover_interval = 0detections = 3detection_interval = 6detection_timeout = 1

Built-in Failure DetectorConfiguration File

• Don’t fail over frequently: – failover_interval

• Every (interval/detections) check server’s status:

– detection_interval and detections

• Failed attempt after a timeout trying to connect to a server:

– detection_timeout

• After successive failed attempts, there is a failure:

– detections


FailoverMaster is operational


FailoverMaster Fails


FailoverNew Master is Elected


External Failure Detector

• External component monitors servers• Use the threat set of commands

– threat report_error– threat report_failure

• Report failures and suspicions• Take other problems into account:

– Disk Issues– Memory Issues

43


Avoiding thundering herds

• When to report errors?• Reporting errors at the session level can cause flooding• E.g., Orders Page

– Write customers orders to the database– Every failing order cause a report

• Use this feature carefully

44


...[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0

External Failure DetectorConfiguration File

• Related to thread report_error• Don’t fail over frequently:

– failover_interval

• Errors are considered in a time window:

– notification_interval

• Successive error notifications:– notifications and notification_clients


Re-enabling a faulty server

46

$ mysqlfabric --config fabric.cfg server set_status 192.168.5.10:3306 spare$ mysqlfabric --config fabric.cfg server set_status 192.168.5.10:3306 secondary...$ mysqlfabric --config fabric.cfg group promote –slave_id 192.168.5.10:3306


Re-enabling a faulty server

47

$ mysqlfabric –config fabric.cfg server set_status 192.168.5.10:3306 spare$ mysqlfabric –config fabric.cfg server set_status 192.168.5.10:3306 secondary...$ mysqlfabric –config fabric.cfg group promote –slave_id 192.168.5.10:3306

TIP#7 Check if the slave is the most updated and does not have errant transactions.


Program Agenda


48



Failure Detection


1

3

5

2

4


Eventually something will fail

Failures will happen and your application should be designed to handle them.

49


Catch errors and retry

50

try:conn.start_transaction()conn.execute('INSERT...')conn.execute('UPDATE...')self.__cnx.commit()

except InterfaceError as error:cur = conn.cursor()

TIP#8 Connectors/Router cannot safely retry a transaction.


Connectors cannot hide failures


• Handle Session Information in a retry logic:– Temporary Tables– Session Variables– Prepared Statements

• Check the wait_timeout server’s property

52

Good practices


from mysql.connector import connectfrom mysql.connector.fabric import MODE_READWRITE

params = {"fabric" : {

"host": "fabric.example.com", "port": 32274, "user": "connectors","password": "password"

}, "user": user, "password": password, "database": "userdb"}self._cnx = connect(**params)self._cnx.set_property(group=“YYZ", mode=MODE_READWRITE)cur = self._cnx.cursor()

TIP#9 Set property mode to either read-only or read-write.

Using the Fabric-aware ConnectorSample Application


...[fabric_cache]address = fabric.example.com:32275user = connectors

[routing:read-write]bind_port = 7002destinations = fabric+cache:///group/YYZmode = read-write

Using the RouterConfiguration File

TIP#10 Create different sections for modes and groups.

• Password to access Fabric must be provided when the router starts• The router can continue operating with stale data• Different binding addresses for different modes and groups


from mysql.connector import connect

params = {"host": “localhost”, "port": 7002, "user": user, "passwd": passwd, "database": "userdb"

}self._cnx = connect(**params)cur = self._cnx.cursor()

Using the RouterSample Application

Whether a slave or master will be used is determined by the bind address in the configuration file.


• Put MySQL Fabric and its storage in the same host.• Set up connection options to the state store properly.• Set up same [protocol.*]’s users/passwords.• [Server]’s user will be used to access managed servers.• [Server]’s user will be used to establish replication among servers.• Use IPs for managed servers unless your DNS is HA.• Check if a slave is the most updated or does not have errant transactions before a manual failover.• Connectors/Router cannot safely retry a transaction.• Select the type of server/operation correctly.

56

Summary


Classroom Training

Learning Subscription

Live Virtual Class

Training On Demand

Keep Learning with Oracle University

education.oracle.com

Cloud

Technology

Applications

Industries


Oracle University MySQL Training ServicesEnable Reliable and High-Performance Web-Based Database Applications in Your Organization

Benefits

Expert-led training to support your MySQL learning needs

Flexibility to train in the classroom or online

Hands-on experience to gain real world experience

Key skills needed for database administrators and developers

MySQL for Beginners

MySQL for Database Administrators

MySQL Performance Tuning

MySQL Cluster

MySQL and PHP - Developing Dynamic Web Applications

MySQL for Developers

MySQL Developer Techniques

MySQL 5.6 Database Administrator

MySQL 5.6 Developer

To find out more about available MySQL Training & Certification offerings, go to: education.oracle.com/mysql

Top Courses for Administrators and Developers

Top Certifications

Learn how to install and configure an open source,

ACID-compliant transactional database designed

to deliver real-time in-memory performance and

99.999% availability. Find out more at:

education.oracle.com/MySQL

MySQL Student Quotes:

• “Slides are excellent. I was so happy to get so much

detail. The information was relevant and written in a

way that was easy to understand and absorb.”

• “The prepared scripts to run for executing testing was

very nice to have.”

• “The course content was amazing and the instructor's

real-world solutions really helped.”

Premier Support customers eligible to save 20% on learning credits.

MySQL Cluster Training – Available NOW!

59

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=3


Step 1:Explore Certifications

Choose a certification to pursue based on the

technology area that interests you.

Certification.oracle.com

Step 2:Prepare for Your

Exam

1st step in preparing is to click on exam details. Review the Exam Preparation and Exam

Topics tabs. Are you Prepared?

Step 3:Register for Your

Exam

Once you feel confident that you have met any pre-requisites and can perform and apply the skills listed in the Exam Topics

tab, it's time to register.

How to Get Oracle MySQL Certified

MySQL Certifications – Available now:

• MySQL 5.6 Database Administrator Certified Professional

• MySQL 5.6 Developer Certified Professional

60

con5209_correia-con5209 - mysql fabric tips and tricks - oow 2015

Documents