flipper

FlipperA system for managing MySQL replicating master-master pairs

Proven Scaling - www.provenscaling.com

i

Table of Contents

1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Quick Start. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1 Before you begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.3 Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.4 Pointing Flipper at the Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.5 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.5.1 IP address information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.5.2 ARP sending command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.5.3 MySQL authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.5.4 SSH authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.5.5 Check configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.6 Set up IP addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.7 Using Flipper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.1 Obtaining Flipper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.1.1 Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.1.2 Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.2 Installing Flipper from svn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2.1 Where’s Perl? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2.2 Copying files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.2.2.1 Copying the script to your PATH . . . . . . . . . . . . . . . . . . . . 83.2.2.2 Installing into your home directory or other arbitary

location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.1 Metadata Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.1.1 DBI metadata storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.2 Metadata Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.2.1 Data Source Naming Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.2.2 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.2.3 Pointing Flipper at Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.3 Configuration Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.4 Masterpair and Node names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.5 Network configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.5.1 IP address configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.5.1.1 Node IP addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.5.1.2 Read and Write IP addresses . . . . . . . . . . . . . . . . . . . . . . . 124.5.1.3 Netmask and Broadcast addresses . . . . . . . . . . . . . . . . . . 12

4.5.2 Interface configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

ii

4.6 SSH and sudo configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.6.1 SSH and sudo Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.6.1.1 Restricting sudo usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.6.1.2 Restricting ssh access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.7 MySQL configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

5 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.1 Command Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.2 Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.2.1 Displaying configuration - the config command . . . . . . . . . . 155.2.2 Displaying the status of a masterpair - the status command

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.2.3 Moving an IP to a node - the set command . . . . . . . . . . . . . 165.2.4 Swapping IPs from one node to the other - the swap

command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.2.5 Moving IPs away from a node - the disable command . . . 185.2.6 Handling a failed node - the fail command . . . . . . . . . . . . . 18

6 Applications and Flipper . . . . . . . . . . . . . . . . . . 206.1 Common MySQL error messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

7 Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217.1 Known Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

8 Future Developments . . . . . . . . . . . . . . . . . . . . . 22

9 Platform Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 249.1 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

9.1.1 ARP announcements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249.1.2 RHEL/CentOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249.1.3 Debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

9.2 HP-UX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249.3 Unsupported Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

10 Configuration Reference . . . . . . . . . . . . . . . . . 2510.1 ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2510.2 write ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2510.3 read ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2510.4 netmask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2510.5 broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2510.6 path to ifconfig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2510.7 ssh ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2610.8 ssh user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2610.9 use sudo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2610.10 mysql user. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2610.11 mysql password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

iii

10.12 mysql port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2610.13 interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2610.14 read interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2710.15 write interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2710.16 send arp command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

11 Credits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

License (LGPLv3) . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Contacting Proven Scaling . . . . . . . . . . . . . . . . . . . . 32

Variable Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Concept Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Chapter 1: Overview 1

1 Overview

Flipper is a tool for managing read and write access pairs of MySQL servers (known asnodes) arranged in a master-master replication formation (known as a masterpair). Clientmachines do not connect "directly" to either node; instead, read and write access to themasterpair is managed through a pair of IP addresses: one for read, one for write. Flipperallows you to move these IP addresses between the nodes in a safe and controlled manner.A basic Flipper system looks like the following:

meta-data

nodea

nodeb

write_ip read_ip

mas

terp

air

Clients

Control mysql

ssh ssh

Flipper is written in Perl, and has been written to be as portable as possible. It does notrequire any Perl modules outside of the core. It has been tested with MySQL 5.0, butshould work with MySQL 4.0 and later.

Chapter 2: Quick Start 2

2 Quick Start

Follow the instructions here to get started with Flipper in just a few minutes.These instructions relate to the pre-release versions of Flipper.

2.1 Before you begin

Before you begin, make sure you have the following together:• Two machines to act as masters, on the same subnet, vlan, etc. and capable of floating

an IP address between them.• Two IP addresses reserved to share between the masters for reads and writes.

2.2 Installation

First of all, check out Flipper from the Proven Scaling Subversion repository to the nodeyou’ll use for managing your masterpairs.The flipper script assumes ‘perl’ will be in ‘/usr/bin’. If your Perl executable is locatedsomewhere else, either change the #! line in the ‘flipper’ script (the first line), or betterstill put a symlink in place in ‘/usr/bin/perl’ to your ‘perl’ binary, as there are plenty ofother scripts out there which rely on ‘perl’ being in ‘/usr/bin’.Copy the ‘flipper’ script to ‘/usr/bin’.Copy the ‘lib/Flipper’ directory to some place in Perl’s @INC on your system. To list thedirectories in @INC on your system, run perl -e ’print join "\n",@INC’.‘/usr/lib/perl5/site_perl’, ‘/usr/local/lib/perl5/site_perl’ on Linux, Solaris, andBSD, and ‘/Library/Perl’ on Mac OS X are good places to put the ‘Flipper’ directory.For more information on installing Flipper, See Chapter 3 [Installation], page 8.

2.3 Metadata

Flipper uses a database for storing it’s configuration information. This database can bestored in any DBMS supported by Perl’s DBI, but currently only MySQL has been tested.Create the metadata database on a node, then populate it using the schema definitioncontained in the ‘sql/metadata.sql’ file in subversion. The metadata database can resideon one of the masterpairs that you intend to manage using Flipper, or it can live on aseparate database server.Currently, Flipper requires only read access to the metadata database.

2.4 Pointing Flipper at the Metadata

Flipper uses a DSN-like string to locate it’s metadata, along with a username and passwordfor authentication.These three items of information can be passed to flipper on the command line, using the--metadata or -m switch for the DSN string, and --username or -u and --password or -pswitches for the username and password respectively.You will probably find it more convenient to add a [flipper] section to your ‘my.cnf’file (or create a ‘~/.my.cnf’ file) containing lines for the metadata DSN, username andpassword, like this:


[flipper]metadata=DBI:mysql:database=flippermetadata;mysql_socket=/tmp/mysql.sockusername=flippermetadatapassword=letmein

2.5 Configuration

Configuration has to be done manually at the moment. Future versions of Flipper will allowyou to use the script itself to manage the configuration.First, decide on a name for your first masterpair, and decide on names for the two nodes.

2.5.1 IP address information

Insert rows into the masterpair and node tables for the IP addresses of the two nodes andthe read and write IP addresses that you will be moving between the nodes. See Section 10.1[ip configuration variable], page 25, Section 10.3 [read ip configuration variable], page 25,Section 10.2 [write ip configuration variable], page 25.

INSERT INTO masterpair(masterpair, name, value)

VALUES(’<masterpair name>’, ’read_ip’, ’<read ip>’),(’<masterpair name>’, ’write_ip’, ’<write ip>’);

INSERT INTO node(masterpair, node, name, value)VALUES(’<masterpair name>’, ’<node 1 name>’, ’ip’, ’<node 1 ip>’),(’<masterpair name>’, ’<node 2 name>’, ’ip’, ’<node 2 ip>’);

You’ll also need to insert a row into the masterpair table for the netmask of the networkcontaining the nodes. See Section 10.4 [netmask configuration variable], page 25.


VALUES(’<masterpair name>’, ’netmask’, ’<network netmask>’);

2.5.2 ARP sending command

When an IP address is moved from one node to another, an ARP message needs to be sentout to inform other nodes (servers, routers etc) on the network that the IP address is nowavailable on a different physical (MAC) address.Most modern UNIX-based operating systems send an ARP message out automatically whenan IP address is brought up on an interface, and will handle the ARP messages correctlywhen they receive them.Linux does not send ARP messages out automatically, although it does handle receipt ofthem correctly. Therefore Flipper needs to be told how to send the ARP message out.On RedHat Enterprise Linux, CentOS and Fedora, the arping command, supplied with theoperating system, can be used to send ARP messages. In order to use arping, insert a rowinto the masterpair table.



VALUES(’<masterpair name>’, ’send_arp_command’,’/sbin/arping -I $sendarp_interface -c 5 -U -A $sendarp_ip’);

You may need to insert a row into the masterpair table for the broadcast address of thenetwork containing the nodes if this is required by the command you’re using to send ARPmessages.For more information, see Section 10.16 [send arp command configuration variable],page 27.

2.5.3 MySQL authentication

Flipper needs to be given authentication information so it can connect to the MySQL serversrunning on the nodes. By default, it will attempt to connect as root with no password,on port 3306. If these defaults are not suitable, insert rows into the masterpair table totell Flipper how to log into MySQL. This assumes the same authentication credentials areto be used on each node. See Section 10.10 [mysql user configuration variable], page 26,Section 10.11 [mysql password configuration variable], page 26, Section 10.12 [mysql portconfiguration variable], page 26.


VALUES(’<masterpair name>’, ’mysql_user’, ’<mysql username>’),(’<masterpair name>’, ’mysql_password’, ’<mysql password>’);

2.5.4 SSH authentication

By default, Flipper will ssh to the nodes as the user running the flipper script. To changethis, either set the Section 10.8 [ssh user configuration variable], page 26 configurationvariable or alternatively edit ‘~/.ssh/config’.


VALUES(’<masterpair name>’, ’ssh_user’, ’<ssh username>’);

By default, Flipper will use sudo for privilege escalation on the nodes, where required. Thisis required for non-superusers to execute operations such as bringing down/up IP addresses.You will need to configure sudo on the nodes to enable this. If you set Section 10.8 [ssh userconfiguration variable], page 26 to ’root’, you can turn the use of sudo off if required bysetting the Section 10.9 [use sudo configuration variable], page 26 to 0.The security implications of the various possible settings for Section 10.8 [ssh user configu-ration variable], page 26, Section 10.9 [use sudo configuration variable], page 26, and ssh andsudo configuration on the nodes are significant. You are strongly advised to make yourselfaware of them. See Section 4.6.1 [ssh and sudo security], page 13 for more information.

2.5.5 Check configuration

Once you’ve pointed Flipper at the Metadata, and configured Flipper correctly, you shouldbe able to use Flipper’s config command to display the configuration information for a


particular masterpair. For example, to display all the configuration values for a masterpaircalled ‘mike’:

$ flipper mike configMASTERPAIR: mike

write_ip: 10.6.0.67send_arp_command: arping -I $sendarp_interface -c 5 -U -A $sendarp_ipread_ip: 10.6.0.66netmask: 255.255.255.0

NODE: first_nodeip: 10.6.0.64

NODE: second_nodeip: 10.6.0.65

$

If you’ve not pointed Flipper to the Metadata database correctly, you’ll be presented withan error which should give useful information as to what the problem was:

$ flipper mike configCouldn’t connect using username ’root’ to DSN:DBI:mysql:database=wrongdb

The error from DBI was:Unknown database ’wrongdb’

Check that the DSN, username and password specified in my.cnf or as optionsto this script are all correct.$

To display a list of the masterpairs, just run flipper with no command:$ flipperThe following masterpairs exist:

mike$

2.6 Set up IP addresses

Once Flipper’s configured, it’s ready to use. The status command gives an at-a-glancereport on the status of a particular masterpair. The first time you run Flipper, the readand write IPs won’t be up, and the status command will give output similar to this:

$ flipper mike statusMASTERPAIR: mikeNODE: first_node is writable, replication running, 0s delayNODE: second_node is writable, replication running, 0s delayWARNING: No node has the read IPWARNING: No node has the write IP$

This tells us that both nodes in the masterpair ‘mike’ are up, MySQL is writable on bothnodes, replication is running on both nodes with no delay, but the read and write IPs arenot up on either node. We use Flipper’s set command to bring the IP up on the appropriate


node. In this case, we bring the read IP up on the ‘first_node’ node, and the write IP upon the ‘second_node’ node. The warnings are self-explanatory.

$ flipper mike set read first nodeWARNING: read IP is not up on second_node node.WARNING: Won’t attempt to take down read IP on second_node node.$ flipper mike set write second nodeWARNING: write IP is not up on first_node node.WARNING: Won’t attempt to take down write IP on first_node node.$ flipper mike statusMASTERPAIR: mikeNODE: first_node has read IP, is read-only, replication running, 0s delayNODE: second_node has write IP, is writable, replication running, 0s delay$

2.7 Using Flipper

This is just a quick overview of what Flipper can do. More detail can be found elsewherein this documentation. Help is available on the command line by running flipper --helpfor usage information and perldoc flipper for a brief overview.Flipper’s set command ensures an IP is running on a particular node. If it’s already runningon the node you specify, it won’t be moved. If it’s running on the other node, it will bemoved.The swap command moves IPs from one node to another. If you specify an IP, it will moveonly the IP you specify, otherwise all IPs will be moved to the other node.The disable command moves any IPs that are up on a particular node to the other node.This is useful for taking a machine out of use for maintenance. The fail command issimilar to the disable command, but is used for handling nodes which have suffered anexternal permanent or semi-permanent failure.The following example demonstrates the use of the commands. We use the status commandto display the current status:

$ flipper mike statusMASTERPAIR: mikeNODE: first_node has read IP, is read-only, replication running, 0s delayNODE: second_node has write IP, is writable, replication running, 0s delay$ flipper mike swap$ flipper mike statusMASTERPAIR: mikeNODE: first_node has write IP, is writable, replication running, 0s delayNODE: second_node has read IP, is read-only, replication running, 0s delay$ flipper mike set read first node$ flipper mike statusMASTERPAIR: mikeNODE: first_node has read IP, has write IP, is writable, replication running, 0s delayNODE: second_node is read-only, replication running, 0s delay$ flipper mike disable first node$ flipper mike status


MASTERPAIR: mikeNODE: first_node is read-only, replication running, 0s delayNODE: second_node has read IP, has write IP, is writable, replication running, 0s delay$

Chapter 3: Installation 8

3 Installation

3.1 Obtaining Flipper

3.1.1 Packages

Pre-built packages for flipper can be downloaded from the Proven Scaling Mirror. If apackage is not available for your operating environment, then don’t fear - it’s very easy todownload the source code and install it.

3.1.2 Source Code

To fetch the most recent stable version of Flipper from Subversion, simply run the followingcommand in a shell:

svn checkout svn://svn.provenscaling.com/public/flipper/trunk/ flipper

If you want to check Flipper out to a different directory, specify something other than‘flipper’ as the second parameter to svn checkout.

3.2 Installing Flipper from svn

3.2.1 Where’s Perl?

The flipper script assumes ‘perl’ will be in ‘/usr/bin’. If your Perl executable is locatedsomewhere else, either change the #! line in the ‘flipper’ script, or better still put a symlinkin place in ‘/usr/bin/perl’ to your ‘perl’ binary, as there are plenty of other scripts outthere which rely on ‘perl’ being in ‘/usr/bin’.

3.2.2 Copying files

You can either copy the ‘flipper’ script to somewhere in your PATH and the Perl mod-ules into @INC, or simply run the ‘flipper’ script from any directory (usually your homedirectory), invoke it as ./flipper and copy the Perl modules into the same directory.

3.2.2.1 Copying the script to your PATH

You will almost definitely need super-user privileges to do this.The ‘flipper’ script can be copied to ‘/usr/bin’, ‘/usr/local/bin’ or some other directoryin your PATH.Copy the ‘lib/Flipper’ directory from subversion to some place in Perl’s @INC on your sys-tem. To list the directories in @INC on your system, run perl -e ’print join "\n",@INC’.‘/usr/lib/perl5/site_perl’, ‘/usr/local/lib/perl5/site_perl’ on Linux, Solaris, andBSD, and ‘/Library/Perl’ on Mac OS X are good places to put the ‘Flipper’ directory.

3.2.2.2 Installing into your home directory or other arbitarylocation

The ‘flipper’ script can run anywhere, so you can copy it to your home directory or otherarbitary location. This can be useful if you want to install Flipper on a node on which youdon’t have super-user privileges.

http://mirror.provenscaling.com/extras/provenscaling/

Chapter 3: Installation 9

If you do this, and you’re only going to ever invoke Flipper as ./flipper, you can copythe ‘lib/Flipper’ directory from subversion to the same directory as the ‘flipper’ script.This works by virtue of the current working directory always appearing in @INC.If you’re on Mac OS X with a case insensitive filesystem, you won’t be able to put the‘Flipper’ subdirectory and the ‘flipper’ script in the same directory. The workaround isto put the ‘Flipper’ directory into ‘/Library/Perl’, as superuser privileges shouldn’t berequired for that.

Chapter 4: Configuration 10

4 Configuration

4.1 Metadata Storage

Flipper has a very flexible and extensible metadata storage system.

In theory it is possible for metadata to be stored anywhere in any arbitary format.

Currently only access to metadata stored in a database using Perl’s DBI is implemented,but others can be put in place later, such as XML or YAML.

4.1.1 DBI metadata storage

Any DBMS that’s supported by DBI can be used to store Flipper’s metadata. Currentlyonly MySQL has been tested, but if you have success or failure using another DBMS, pleaselet us know.

There are no restrictions on where the metadata database resides - it can live anywhere, aslong as Flipper can get access to read from it.

Currently, Flipper requires only read access to the metadata database. Future versions ofFlipper will require write access to the metadata database.

The metadata database contains two tables - one for information relating to masterpairs,and one for information specific to nodes. The schema is available in the sql/metadata.sqlfile in subversion.

4.2 Metadata Access

4.2.1 Data Source Naming Scheme

Flipper uses a naming scheme for it’s data source names (DSNs) which is compatible withPerl’s DBI.

When storing metadata in a DBMS and accessing it using DBI, the DSNshould be the same as would be passed in a DBI->connect call, for exampleDBI:mysql:database=flippermetadata;mysql_socket=/tmp/mysql.sock.

Other metadata storage methods can be chosen by specifying something other than DBIbefore the first colon in the DSN - for example, a future YAML storage method might bespecified using YAML:file=/etc/flipper.yaml".

4.2.2 Authentication

As well as the DSN, Flipper can use a username and password if the metadata storagemethod requires authentication to be used.

4.2.3 Pointing Flipper at Metadata

The three items of information (metadata DSN, username and password) can be passed toflipper on the command line.

Use the --metadata or -m switch to tell Flipper where to find the metadata. The --username or -u and --password or -p switches specify the username and password respec-tively.


You will probably find it more convenient to add a [flipper] section to your my.cnf file (orcreate a ~/.my.cnf file) containing lines for the metadata DSN, username and password,like this:

[flipper]metadata=DBI:mysql:database=flippermetadata;mysql_socket=/tmp/mysql.sockusername=flippermetadatapassword=letmein

Any options provided on the command line will override those specified in a my.cnf file.

4.3 Configuration Variables

There are three types of configuration variable in Flipper:• Masterpair configuration variables• Node configuration variables• Cascading variables

Configuration variables have a name and a value. Some configuration variables are compul-sory, some are optional. Some optional configuration variables have a default value whichis used if the variable is not specified.Masterpair configuration variables and node configuration variables are specific to master-pairs and nodes respectively.Cascading variables can be specified at masterpair or node level. If a cascading variable isspecified at node level, then it will be used for that node. If a cascading variable is specifiedat masterpair level, then it will be used for both nodes in the masterpairs - unless it’s alsospecified at node level, in which case the node level value will override the masterpair levelvalue.

4.4 Masterpair and Node names

Masterpair and node names can be up to 20 characters long, and must not contain spaces.The use of other characters which might confuse the shell isn’t recommended - stick toalphanumerics, underscore, hyphen and dot to be safe.Node names are used internally by Flipper only. They need not bear any resemblance toactual hostnames. You could use ‘first_node’ and ‘second_node’ for the two nodes in allyour masterpairs, or you may feel more comfortable using node names which resemble thehostnames.

4.5 Network configuration

4.5.1 IP address configuration

4.5.1.1 Node IP addresses

Each node has a base IP address. This is stored as the Section 10.1 [ip configurationvariable], page 25 at node level.Flipper will use this base IP address to connect to the node’s ssh daemon and MySQLserver. If the ssh daemon is bound to a different IP address to the base address on the node,


then you can store the IP address for ssh connections as Section 10.7 [ssh ip configurationvariable], page 26 at node level.The MySQL server must be listening on the base IP address of the node.

4.5.1.2 Read and Write IP addresses

The read and write IP addresses are used by external clients to query the data stored on themasterpair. Clients performing read operations only should connect to the read IP address.Clients performing read/write operations should connect to the write IP address. These arestored as Section 10.3 [read ip configuration variable], page 25 and Section 10.2 [write ipconfiguration variable], page 25 at masterpair level.The read and write IP addresses need not be on the same network (either in a physical orlogical sense) as the base address of the nodes. They do however need to be on the samenetwork as each other.

4.5.1.3 Netmask and Broadcast addresses

Flipper also needs to know the netmask, and in some cases the broadcast address, for thenetwork on which the read and write IP addresses reside. Although the broadcast addressis not always required, it’s advisable to always configure it.These are stored as Section 10.4 [netmask configuration variable], page 25 and Section 10.5[broadcast configuration variable], page 25 at masterpair level.

4.5.2 Interface configuration

Flipper needs to know which physical network interface it should use for the read and writeIP addresses on each node. If there is only one physical interface configured, then Flipperwill use that. If there are multiple physical interfaces on the machine, then Flipper willneed to be told which one to use.The interface is stored as Section 10.13 [interface configuration variable], page 26. Thiscan be stored at either masterpair level, or node level. If both nodes are to use the sameinterface for the read and write IPs, then store it at masterpair level, and it will be appliedto both nodes.

4.6 SSH and sudo configuration

In order to execute commands on the nodes, Flipper needs to be able to connect to the nodesusing ssh. The following configuration variables are used to control how Flipper connectsusing ssh to the nodes:• Section 10.7 [ssh ip configuration variable], page 26• Section 10.8 [ssh user configuration variable], page 26• Section 10.9 [use sudo configuration variable], page 26

The default configuration is that Section 10.7 [ssh ip configuration variable], page 26 isunset, Section 10.8 [ssh user configuration variable], page 26 is unset and Section 10.9[use sudo configuration variable], page 26 is set to 1, representing "on".As described above, if the ssh daemon is bound to a different IP address to the base addresson any node, then you can store the IP address for ssh connections as Section 10.7 [ssh ipconfiguration variable], page 26 at node level. If Flipper is to use the base IP address of


a node for ssh connections, Section 10.7 [ssh ip configuration variable], page 26 should notbe defined for that node.

Section 10.8 [ssh user configuration variable], page 26 is the user which Flipper will connectas via ssh. If this is unset, Flipper will use the current user to ssh to the nodes, unlessoverridden by a setting in ‘~/.ssh/config’.

If the Section 10.8 [ssh user configuration variable], page 26 is a non-root user, then youshould leave Section 10.9 [use sudo configuration variable], page 26 set to 1. This willenable Flipper to use sudo to execute commands which require superuser access - typicallythe commands used to bring up and take down IP aliases on the nodes.

If you have set Section 10.8 [ssh user configuration variable], page 26 to ‘root’, then youcan disable the use of sudo by changing the Section 10.9 [use sudo configuration variable],page 26 to 0.

Although Section 10.8 [ssh user configuration variable], page 26 and Section 10.9 [use sudoconfiguration variable], page 26 can both be specified at node level, we recommend thatthey are only specified at masterpair level and nodes within a masterpair are identicallyconfigured.

4.6.1 SSH and sudo Security

You should ensure that you are fully aware of the security implications of how you configureFlipper to work with ssh. The default settings of Section 10.8 [ssh user configurationvariable], page 26 and Section 10.9 [use sudo configuration variable], page 26 may not beappropriate for the security environment in your organisation.

We would recommend that Flipper be configured to connect to nodes as a specified userwith a restricted shell, allowing connections only from the appropriate IP address(es), andprivilege escalation for that user via sudo should be restricted to the required commandsonly.

4.6.1.1 Restricting sudo usage

We recommend that you restrict sudo usage to the bare minimum that Flipper requiresto operate. This is particularly important if you are using a role account (eg. an accountcalled ‘flipper’) for Flipper to use on the nodes.

In order to do this, add the following line to your sudoers file. Always edit the sudoers fileby using the visudo command.

flipper ALL = NOPASSWD: /sbin/ifconfig, /sbin/arping

This command instructs sudo to allow the ‘flipper’ user access to run the /sbin/ifconfigand /sbin/arping commands as root without a password.

If the path to ifconfig (see Section 10.6 [path to ifconfig configuration variable], page 25)is different on your system, you should change the sudoers line accordingly.

If you are not using a Section 10.16 [send arp command configuration variable], page 27in your environment, then there is no need to specify one here. If you’re using somethingother than /sbin/arping, you should change the sudoers line accordingly.


4.6.1.2 Restricting ssh access

If you use a special account solely for Flipper on the nodes, sometimes called a headless,role or anonymous account, then it is usually desirable to use a restricted shell for thataccount to stop the role account being used for anonymous actions on the server.An example restricted shell for use with Flipper can be found in ‘flippersh’. This assumes/sbin/arping is being used in your Section 10.16 [send arp command configuration vari-able], page 27 - if something else (or indeed nothing) is being used, change the restrictedshell accordingly.

4.7 MySQL configuration

Flipper needs to connect to MySQL on the nodes.In order to do this, the Section 10.10 [mysql user configuration variable], page 26, Sec-tion 10.11 [mysql password configuration variable], page 26 and Section 10.12 [mysql portconfiguration variable], page 26 configuration variables are used to specify the usernameand password that should be used to connect to MySQL, and the port on the node whichshould be connected to.MySQL should be bound to the base IP address of each node.The user that Flipper connects to on the nodes requires the SUPER privilege.We recommend that you create a user exclusively for the use of Flipper with just the SUPERprivilege. We recommend not using the MySQL root user for use with Flipper.

Chapter 5: Usage 15

5 Usage

Typically, the read IP will be up on one node, and the write IP will be up on the othernode. There’s nothing to stop you having both read and write IPs up on the same nodethough. This can be very useful in certain situations, for example if you’ve got unbalancedhardware (one node significantly more powerful or loaded than the other) or as a short termmeasure if there are connectivity issues or maintenance is being carried out.

5.1 Command Line

Flipper takes options, a masterpair, a command and sometimes parameters on it’s commandline:

flipper [<options>] [<masterpair> <command> [<parameters>]]

Options control the general operation of Flipper, for example ‘-v’ or ‘--verbose’ whichturns on verbose output and ‘-m’ or ‘--metadata’ which tells Flipper where to find it’smetadata.

The masterpair is specified so Flipper knows what it’s going to operate on. If no masterpairis specified, Flipper will display a list of the available masterpairs.

Commands tell Flipper what to do. Parameters are used by the commands. Some com-mands, such as status require no parameters. Others, such as the disable command,require paramters - in this case, the name of a node to disable.

If you give flipper an unknown command or option, you’ll be presented with some briefusage help. You can also display the usage information by running flipper --help.

5.2 Commands

5.2.1 Displaying configuration - the config command

To display the configuration of a masterpair and it’s nodes, use the config command.The masterpair name will be displayed, followed by the configuration variables for thatmasterpair. After that, for each node, the node name and configuration variables specificto that node will be displayed:

$ flipper customerdata configMASTERPAIR: customerdata

write_ip: 192.168.4.54send_arp_command: /sbin/arping -I $sendarp_interface -c 5 -U -A $sendarp_ipread_ip: 192.168.4.55netmask: 255.255.240.0

NODE: dell_2850ip: 192.168.3.234

interface: eth0NODE: hp_dl380

ip: 192.168.2.52interface: eth1$

Chapter 5: Usage 16

5.2.2 Displaying the status of a masterpair - the status command

The status command can be used to display the current status of a masterpair. It alsoacts as a healthcheck for the current operation of the masterpair - if there are any problems,warnings will be given.

Information about which nodes have which IPs, the read-only or read-write state of theMySQL server and the replication state are given. A typical ’healthy’ status output shouldlook like this:

$ flipper customerdata statusMASTERPAIR: customerdataNODE: dell_2850 has read IP, is read-only, replication running, 0s delayNODE: hp_dl380 has write IP, is writable, replication running, 0s delay$

Warnings will be displayed if there are any problems:

• Any IP is up on more than one node, or not up on any node

• Replication isn’t running

• Inconsistencies in read-only / read-write state (eg. if the server has the write IP, butis read-only)

• Problems connecting to MySQL or ssh on any node

5.2.3 Moving an IP to a node - the set command

The set command ensures that the specified IP is up on the given node. It takes twoparameters - firstly the name of an IP, and secondly, the name of a node.

If the IP is currently up on the other node to the given one, then it will be taken downon that other node and brought up on the given node. If the write IP is being moved,replication will be synced.

Nothing will happen if the specified IP is already up on the given node and an informationmessage will be displayed. If the specified IP is not up on the other node to the one given(ie. it is not up on either node), no attempt will be made to take down the IP from theother node, but if the write IP is being dealt with, replication will still be synced.

The following example demonstrates the use of the set command:

First, display status of customerdata masterpair:

$ flipper customerdata statusMASTERPAIR: customerdataNODE: dell_2850 has read IP, is writable, replication running, 0s delayNODE: hp_dl380 is writable, replication running, 0s delayWARNING: MySQL server on read IP is writableWARNING: No node has the write IP

Put the write IP on dell 2850:

$ flipper customerdata set write dell 2850WARNING: write IP is not up on hp_dl380 node.WARNING: Won’t attempt to take down write IP on hp_dl380 node.

Display status again, noticing that dell 2850 now has the write IP:

Chapter 5: Usage 17

$ flipper customerdata statusMASTERPAIR: customerdataNODE: dell_2850 has read IP, has write IP, is writable, replication running, 0s delayNODE: hp_dl380 is read-only, replication running, 0s delay

Put the read IP onto hp dl380:

$ flipper customerdata set read hp dl380

Display status again, noticing that the read IP has moved to hp dl380:

$ flipper customerdata statusMASTERPAIR: customerdataNODE: dell_2850 has write IP, is writable, replication running, 0s delayNODE: hp_dl380 has read IP, is read-only, replication running, 0s delay

Finally, put write IP on dell 2850. This has no effect, as it’s already there:

$ flipper customerdata set write dell 2850INFO: The write IP is already up on the dell_2850 node.

5.2.4 Swapping IPs from one node to the other - the swapcommand

The swap command moves a single IP, or all IPs to the other node to the one on which they’rerunning. It takes an optional parameter, which is the name of an IP. If this parameter isspecified, then only the given IP will be swapped to the other node. If this parameter isnot specified, all the IPs will be moved to the other node.

The following example demonstrates the use of the swap command:

First, display status of masterpair:

$ flipper customerdata statusMASTERPAIR: customerdataNODE: dell_2850 has read IP, has write IP, is writable, replication running, 0s delayNODE: hp_dl380 is read-only, replication running, 0s delay

Move the read IP to the other node:

$ flipper customerdata swap read

Display status again, noticing that the read IP has moved to the other node:

$ flipper customerdata statusMASTERPAIR: customerdataNODE: dell_2850 has write IP, is writable, replication running, 0s delayNODE: hp_dl380 has read IP, is read-only, replication running, 0s delay

Move all IPs to the other node:

$ flipper customerdata swap

Display status again, noticing that the IPs have moved to the other node:

$ flipper customerdata statusMASTERPAIR: customerdataNODE: dell_2850 has read IP, is read-only, replication running, 0s delayNODE: hp_dl380 has write IP, is writable, replication running, 0s delay

Chapter 5: Usage 18

5.2.5 Moving IPs away from a node - the disable command

If you wish to remove all IPs from a node, perhaps to undertake some maintenance work,then use the disable command. It requires a single parameter, which is the name of a node.All the IPs which are up on the specified node will be moved to the other node. If there areno IPs up on the specified node, nothing will happen and an informational message will bedisplayed.

The following example demonstrates the use of the disable command:

First, display status:

$ flipper mike statusMASTERPAIR: mikeNODE: first_node has read IP, is read-only, replication running, 0s delayNODE: second_node has write IP, is writable, replication running, 0s delay

Disable dell 2850, that is, move read IP to hp dl380:

$ flipper mike disable first node

Show status, noticing that the read IP has moved to hp dl380:

$ flipper mike statusMASTERPAIR: mikeNODE: first_node is read-only, replication running, 0s delayNODE: second_node has read IP, has write IP, is writable, replication running, 0s delay

Disable hp dl380, that is, move all IPs to dell 2850:

$ flipper mike disable second node

Show status again, noticing that all IPs have now moved to dell 2850:

$ flipper mike statusMASTERPAIR: mikeNODE: first_node has read IP, has write IP, is writable, replication running, 0s delayNODE: second_node is read-only, replication running, 0s delay

There are no IPs on the second node, so disable will do nothing:

$ flipper mike disable second nodeINFO: No IPs on the second_node node.

5.2.6 Handling a failed node - the fail command

The set, swap and disable commands all rely on being able to connect to ssh and MySQLon both nodes in order to carry out operations. This is to ensure that the masterpair isalways in a consistent, operable state.

If a node fails due to hardware or network failure, and there is no prospect of recovery ofthat node within an acceptable timeframe, then it is possible to use the fail command.

Under normal circumstances, the fail command acts exactly the same as the disablecommand - it will remove IPs from the given node, and bring them up on the other node,whilst ensuring the masterpair is in a consistent, operable state.

The fail command differs in that it will attempt to contact the failed node, and if it can’t,it will carry on regardless. As such, the ‘--yes’ option must be specified on the commandline to confirm that the user wishes to carry out this operation.

Chapter 5: Usage 19

If a node has suffered a problem with MySQL, but can still be contacted via ssh, then thefail command will handle moving IPs away from the failed node correctly, but will notattempt to synchronise replication.If a node cannot be contacted via ssh, then the fail command won’t be able to take downthe IPs from the failed node. There are dangers associated with this, as it could potentiallyleave a masterpair in an inconsistent state - for example, if used to fail a node which is onlytemporarily inaccessible, when that node comes back, one or more IPs will be up on bothnodes.

Chapter 6: Applications and Flipper 20

6 Applications and Flipper

6.1 Common MySQL error messages

In a typical Flipper-based multi-master system, there are a few error messages which anyclient should be able to handle in order to provide a "seamless" experience for users, despitefailover occurring. During any failover using Flipper (and indeed other systems based on asimilar model) the following errors must be handled:

‘ERROR 1290 (HY000): The MySQL server is running with the --read-onlyoption so it cannot execute this statement’

After the active master has been set read-only using SET GLOBAL read_only=1,any client which tries to run a new data-changing statement will receive thiserror.

‘ERROR 2006 (HY000): MySQL server has gone away’If the failover process occurs while a connection is idle, or if the connection iskilled by the Quiesce::KillAll module, this error will be received on the nextattempt to run a query.

If you choose to use the Quiesce::KillAll module to kill off connected users before afailover, you must handle two additional errors:

‘ERROR 2013 (HY000): Lost connection to MySQL server during query’If a connection is killed while it is currently running a query, the query will beaborted, and this error will be generated by the client.

‘ERROR 1040 (00000): Too many connections’If a connection attempt is made during the quiesce process, whilemax_connections is set to not allow new connections.

In nearly all cases, the best action to take would be to disconnect (if the connection is notalready dropped), pause, reconnect, and retry.

Chapter 7: Bugs 21

7 Bugs

There are almost definitely plenty of bugs in flipper. We would be delighted to hear fromyou if you find a bug - see See [Contacting Proven Scaling], page 32.If you find a bug, please let us know which version of Flipper and Perl you’re using (giveus the output of flipper --version), let us know the platform(s) you’re using (operatingsystem, version, distribution, hardware) and also send us the output that Flipper produceswhen reproducing the bug when invoked with the ‘--verbose’ and ‘--debug’ switches (or‘-vd’ for short).

7.1 Known Bugs

Display of replication delay causes an error when using MySQL 4.0 servers. The Sec-onds Behind Master variable is used to determine replication delay, and this is only presentin MySQL 4.1.1 and newer. It is possible to determine the replication delay on MySQL 4.0from the Time column in the processlist for the slave SQL thread.Behaviour of swap command when an IP is not up on either host is accidentally fixed totaking it down on the second of the two nodes in the masterpair (which will of course fail, asit’s not up on that node) and bringing it up on the first of the two nodes. Correct behaviourshould be to do nothing with that IP, and potentially print a warning.If both read and write IPs are on the same node, the status command complains thatMySQL on the node with the read IP is writable, which is of course very desirable in thiscase, and shouldn’t generate a warning.A node with no IPs assigned to it should default to being read-only. Currently the behaviouris undefined.

Chapter 8: Future Developments 22

8 Future Developments

The following are ideas for future releases of flipper:• Packaged builds

Provide RPMs for RHEL/CentOS, plus packages for other distributions and operatingsystems as required.

• Second Read IP

In some scenarios, the read and write IPs will want to normally reside on the samehost, with the other host being used for ’slow’ reads for reporting purposes etc. Addinga second read IP to Flipper would cater for this scenario.

• Slave Support

If there are slaves pointing at a masterpair, replication to those slaves will almostdefinitely break when the IP that the slaves are pointing at moves from one hostto another, because of differences in binary log positions. By logging the binary logpositions when the IP is moved, we can simplify the process of re-pointing slaves, atleast in non-failure-event scenarios.

• Logging

Logging of events in the metadata (or other) database, along with master replicationstatus, to facilitate the above.

• Ability to use script to manage metadata

At the moment, the only way to configure Flipper is to manipulate the metadatadatabase directly. This isn’t very user friendly and there’s no error checking. Thereshould be some facility to use a script to make configuration easier.

• Bring up appropriate IPs on reboot

A script could be written to bring up and/or ensure the appropriate IPs are available ona machine. Although it would not be difficult to make this script platform-independent,incorporating it into each operating system / distribution’s rc chain would requiresubstantially more effort.

• Automatic Failover

Although we ruled out implementing automatic failover of read and write IP addresseson machine and/or replication failure for the first releases of Flipper, and we don’tnecessarily believe it to be a good idea, we’re not ruling it out.

• HP-UX support

The network interface and IP alias routines in Flipper don’t work on HP-UX, but itwould be possible to fix this if required.

• Refactor code to OO style

Flipper’s code would be a lot more elegant if it were refactored in an object orientedstyle.

• Sanity checking of configuration

Flipper doesn’t currently carry out much sanity checking of it’s configuration - it should!• Consistency of output

The consistency of the output that Flipper produces when the ‘--verbose’ and/or‘--debug’ options are used could be much improved.

Chapter 8: Future Developments 23

• Documentation Improvements

The documentation could be improved by adding notes on how to set up master-masterreplication, and a ’cookbook’ with common use scenarios.

Chapter 9: Platform Notes 24

9 Platform Notes

9.1 Linux

9.1.1 ARP announcements

ARP annoucements (also known as "Gratuitous ARPs") are normally sent out when anIP address becomes available on a particular physical interface. Other devices (switches,routers and nodes) on the network will use these ARP announcements to maintain a map-ping of physical interface (MAC address) to IP address, so packets for a particular IPaddress get sent to the correct physical interface. This mapping is commonly known as anARP table.Linux, unlike other operating systems, does not automatically send ARP announcementswhen an IP address is brought up on an interface. Therefore, when using Flipper to managenodes running Linux, Flipper needs to send out the ARP announcements itself.The Section 10.16 [send arp command configuration variable], page 27 configuration vari-able is used to tell Flipper how the ARP announcements should be sent out.The most convenient way to send out ARP announcements differs from distribution todistribution.

9.1.2 RHEL/CentOS

The /sbin/arping command (supplied as standard with the distribution) can be used tosend the ARP announcements. The best value for Section 10.16 [send arp command config-uration variable], page 27 is /sbin/arping -I $sendarp_interface -c 5 -U -A $sendarp_ip - supply the absolute pathname because ‘/sbin’ isn’t normally in the default path.

9.1.3 Debian

The arping command as supplied by default with RHEL/CentOS can be obtained fromthe package repository.

9.2 HP-UX

Flipper’s network interface and IP alias handling code does not work on HP-UX. Therefore,it is not possible to use Flipper to manage nodes running HP-UX. It is however possible torun the flipper script on a node running HP-UX to manage a masterpair where the nodesrun Linux, FreeBSD or another supported operating system.

9.3 Unsupported Platforms

Windows, VMS and i5/OS are not supported, and are not likely to be in the future.

Chapter 10: Configuration Reference 25

10 Configuration Reference

This chapter describes the configuration variables available in Flipper.

For more general information on how the configuration system works, See Chapter 4 [Con-figuration], page 10.

Variables are either compulsory or optional. They are either relevant to nodes or masterpairsonly, or specific to the level at which they are defined.

10.1 ip

Applicable to nodes only. Compulsory - each node must have an ip variable defined.

Specifies the IP address of each node. Flipper will use this IP address to connect to MySQLon the node.

10.2 write ip

Applicable to masterpairs only. Compulsory - each masterpair must have a write ip variabledefined.

This is the IP address which will be used for write access to MySQL.

10.3 read ip

Applicable to masterpairs only. Compulsory - each masterpair must have a read ip variabledefined.

This is the IP address which will be used for read access to MySQL.

10.4 netmask

Applicable to masterpairs only. Compulsory - each masterpair must have a netmask variabledefined.

This is the netmask for the read ip and write ip. It’s normally the same as the netmaskfor the nodes’ main IPs, but doesn’t necessarily have to be the same.

10.5 broadcast

Applicable to masterpairs only. Optional.

This is the broadcast address for the network for the read ip and write ip. Not needed fornormal use but may be required for sending ARP messages.

10.6 path to ifconfig

Specific to node or masterpair, whereever defined. Optional. Defaults to ‘/sbin’.

If the ifconfig command doesn’t exist in ‘/sbin’, for example on FreeBSD where it is in‘/usr/sbin’, specify this variable to point Flipper to the correct location.


10.7 ssh ip

Applicable to nodes only. Optional.If ssh is bound to a different IP address than MySQL on a node, define a ssh ip variablefor that node, and Flipper will use that IP address for ssh rather than the one specified bythe ip variable.

10.8 ssh user

Specific to node or masterpair, whereever defined. Optional. Defaults to root.This specifies the user which will Flipper will use to ssh to the nodes. If you don’t want itto use the root user, then specify this variable.

10.9 use sudo

Specific to node or masterpair, whereever defined. Optional. Defaults to 0, which means"no".If set to 1, Flipper will use sudo for privilege escalation when carrying out operations onnodes (such as bringing up and taking down IP aliases) which require superuser privileges.

10.10 mysql user

Specific to node or masterpair, whereever defined. Optional. Defaults to root.This is the user which Flipper will use to connect to MySQL on the nodes. This is notthe user which is used to connect to the metadata database - that is specified using the--username command line switch or in ‘my.cnf’.

10.11 mysql password

Specific to node or masterpair, whereever defined. Optional. Defaults to empty string.This is the password which Flipper will use to connect to MySQL on the nodes. This isnot the user which is used to connect to the metadata database - that is specified using the--password command line switch or in ‘my.cnf’.

10.12 mysql port

Specific to node or masterpair, whereever defined. Optional. Defaults to 3306.If MySQL on the nodes runs on a non-standard port, use this variable to specify the portnumber.

10.13 interface

Specific to node or masterpair, whereever defined. Potentially compulsory. No defaultvalue.This variable specifies the interface to be used for the read and write IP addresses whichare controlled by Flipper on the nodes.If no value is configured and there’s only a single interface on a node, that single interfacewill be used.


If no value is configured and multiple interfaces exist on a node, the interface must bespecified.

10.14 read interface

Specific to node or masterpair, whereever defined. Optional. Defaults to an automaticallyassigned interface.The exact interface to bring up the read_ip on, for example eth0:98.

10.15 write interface

Specific to node or masterpair, whereever defined. Optional. Defaults to an automaticallyassigned interface.The exact interface to bring up the write_ip on, for example eth0:99.

10.16 send arp command

Specific to node or masterpair, whereever defined. Optional. No default value.Specifies the command which will be used to send ARP messages when an IP address isbrought up on a node.The following variables will be interpolated into the command:• $sendarp ip - the IP address which is being brought up• $sendarp netmask - the netmask• $sendarp broadcast - the broadcast address of the network (only available if the Sec-

tion 10.5 [broadcast configuration variable], page 25 variable has been specified)• $sendarp interface - the physical interface on which the IP address has been brought

up

Chapter 11: Credits 28

11 Credits

Proven Scaling would like to thank the following people for their contributions towardsFlipper:• Graham Wood, Jon Topper, Wayne Pascoe for hardware for testing Flipper on multiple

platforms.• Barry Leung for a timely donation of a copy of "Mastering Perl" to the cause.

License (LGPLv3) 29

License (LGPLv3)

The GNU Lesser General Public LicenseVersion 3, 29 June 2007

Copyright c© 2007 Free Software Foundation, Inc. http://fsf.org/

Everyone is permitted to copy and distribute verbatim copies of thislicense document, but changing it is not allowed.

This version of the GNU Lesser General Public License incorporates the terms and con-ditions of version 3 of the GNU General Public License, supplemented by the additionalpermissions listed below.

0. Additional Definitions.As used herein, “this License” refers to version 3 of the GNU Lesser General PublicLicense, and the “GNU GPL” refers to version 3 of the GNU General Public License.“The Library” refers to a covered work governed by this License, other than an Appli-cation or a Combined Work as defined below.An “Application” is any work that makes use of an interface provided by the Library,but which is not otherwise based on the Library. Defining a subclass of a class definedby the Library is deemed a mode of using an interface provided by the Library.A “Combined Work” is a work produced by combining or linking an Application withthe Library. The particular version of the Library with which the Combined Work wasmade is also called the “Linked Version”.The “Minimal Corresponding Source” for a Combined Work means the CorrespondingSource for the Combined Work, excluding any source code for portions of the CombinedWork that, considered in isolation, are based on the Application, and not on the LinkedVersion.The “Corresponding Application Code” for a Combined Work means the object codeand/or source code for the Application, including any data and utility programs neededfor reproducing the Combined Work from the Application, but excluding the SystemLibraries of the Combined Work.

1. Exception to Section 3 of the GNU GPL.You may convey a covered work under sections 3 and 4 of this License without beingbound by section 3 of the GNU GPL.

2. Conveying Modified Versions.If you modify a copy of the Library, and, in your modifications, a facility refers to afunction or data to be supplied by an Application that uses the facility (other than asan argument passed when the facility is invoked), then you may convey a copy of themodified version:a. under this License, provided that you make a good faith effort to ensure that, in

the event an Application does not supply the function or data, the facility stilloperates, and performs whatever part of its purpose remains meaningful, or

b. under the GNU GPL, with none of the additional permissions of this Licenseapplicable to that copy.

http://fsf.org/

License (LGPLv3) 30

3. Object Code Incorporating Material from Library Header Files.

The object code form of an Application may incorporate material from a header file thatis part of the Library. You may convey such object code under terms of your choice,provided that, if the incorporated material is not limited to numerical parameters, datastructure layouts and accessors, or small macros, inline functions and templates (tenor fewer lines in length), you do both of the following:

a. Give prominent notice with each copy of the object code that the Library is usedin it and that the Library and its use are covered by this License.

b. Accompany the object code with a copy of the GNU GPL and this license docu-ment.

4. Combined Works.

You may convey a Combined Work under terms of your choice that, taken together,effectively do not restrict modification of the portions of the Library contained in theCombined Work and reverse engineering for debugging such modifications, if you alsodo each of the following:

a. Give prominent notice with each copy of the Combined Work that the Library isused in it and that the Library and its use are covered by this License.

b. Accompany the Combined Work with a copy of the GNU GPL and this licensedocument.

c. For a Combined Work that displays copyright notices during execution, includethe copyright notice for the Library among these notices, as well as a referencedirecting the user to the copies of the GNU GPL and this license document.

d. Do one of the following:

0. Convey the Minimal Corresponding Source under the terms of this License,and the Corresponding Application Code in a form suitable for, and underterms that permit, the user to recombine or relink the Application with a mod-ified version of the Linked Version to produce a modified Combined Work, inthe manner specified by section 6 of the GNU GPL for conveying Correspond-ing Source.

1. Use a suitable shared library mechanism for linking with the Library. Asuitable mechanism is one that (a) uses at run time a copy of the Libraryalready present on the user’s computer system, and (b) will operate properlywith a modified version of the Library that is interface-compatible with theLinked Version.

e. Provide Installation Information, but only if you would otherwise be required toprovide such information under section 6 of the GNU GPL, and only to the extentthat such information is necessary to install and execute a modified version ofthe Combined Work produced by recombining or relinking the Application witha modified version of the Linked Version. (If you use option 4d0, the InstallationInformation must accompany the Minimal Corresponding Source and Correspond-ing Application Code. If you use option 4d1, you must provide the InstallationInformation in the manner specified by section 6 of the GNU GPL for conveyingCorresponding Source.)

License (LGPLv3) 31

5. Combined Libraries.You may place library facilities that are a work based on the Library side by side ina single library together with other library facilities that are not Applications and arenot covered by this License, and convey such a combined library under terms of yourchoice, if you do both of the following:a. Accompany the combined library with a copy of the same work based on the

Library, uncombined with any other library facilities, conveyed under the terms ofthis License.

b. Give prominent notice with the combined library that part of it is a work basedon the Library, and explaining where to find the accompanying uncombined formof the same work.

6. Revised Versions of the GNU Lesser General Public License.The Free Software Foundation may publish revised and/or new versions of the GNULesser General Public License from time to time. Such new versions will be similarin spirit to the present version, but may differ in detail to address new problems orconcerns.Each version is given a distinguishing version number. If the Library as you receivedit specifies that a certain numbered version of the GNU Lesser General Public License“or any later version” applies to it, you have the option of following the terms andconditions either of that published version or of any later version published by theFree Software Foundation. If the Library as you received it does not specify a versionnumber of the GNU Lesser General Public License, you may choose any version of theGNU Lesser General Public License ever published by the Free Software Foundation.If the Library as you received it specifies that a proxy can decide whether future versionsof the GNU Lesser General Public License shall apply, that proxy’s public statementof acceptance of any version is permanent authorization for you to choose that versionfor the Library.

Contacting Proven Scaling 32

Contacting Proven Scaling

If you have any questions, comments, problems, suggestions, you’ve found a bug or yousimply want to let us know that you’re using this software, we’d be delighted to hear fromyou. Please drop us an email at [email protected]. Let us know what versionof MySQL you’re using and what platform you’re on.Proven Scaling’s home on the web is at http://www.provenscaling.com/.

mailto:[email protected]

http://www.provenscaling.com/

Variable Index 33

Variable Index

Bbroadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12, 25

Iinterface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12, 26ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Mmysql_password . . . . . . . . . . . . . . . . . . . . . . . . . . 14, 26mysql_port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14, 26mysql_user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14, 26

Nnetmask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12, 25

Ppath_to_ifconfig . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Rread_interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

read_ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12, 25

Ssend_arp_command . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

ssh_ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11, 12, 26

ssh_user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12, 13, 26

Uuse_sudo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12, 13, 26

Wwrite_interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

write_ip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12, 25

Concept Index 34

Concept Index

Aarchitecture, illustration . . . . . . . . . . . . . . . . . . . . . . . 1author, contacting . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Bbugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21bugs, known . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

CCLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15command line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11configuration, displaying . . . . . . . . . . . . . . . . . . . . . 15configuration, variable reference . . . . . . . . . . . . . . . 25contacting the author . . . . . . . . . . . . . . . . . . . . . . . . 32contributors, credits. . . . . . . . . . . . . . . . . . . . . . . . . . 28credits, acknowledging contributors . . . . . . . . . . . 28

Ddebian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24disable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Eerror messages, from mysql . . . . . . . . . . . . . . . . . . . 20

Ffail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18failover, using disable command . . . . . . . . . . . . . . . 18failover, using fail command . . . . . . . . . . . . . . . . . . 18failover, using set command . . . . . . . . . . . . . . . . . . 16failover, using swap command . . . . . . . . . . . . . . . . 17flipper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Hhandling failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18HP-UX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Iillustration, Flipper architecture . . . . . . . . . . . . . . . 1installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8installation, from Subversion . . . . . . . . . . . . . . . . . . . 8installation, into home directory . . . . . . . . . . . . . . . 8installation, manually copying files . . . . . . . . . . . . . 8installation, using binary packages . . . . . . . . . . . . . 8

Kknown bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Llinux, debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24linux, rhel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Mmoving ip, failover . . . . . . . . . . . . . . . . . . . . . . . . . . . 16mysql error messages, during failover . . . . . . . . . . 20mysql, configuration . . . . . . . . . . . . . . . . . . . . . . . . . 14mysql, privileges for metadata . . . . . . . . . . . . . . . . 14

Ooverview, concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Pplatforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24platforms, HP-UX . . . . . . . . . . . . . . . . . . . . . . . . . . . 24platforms, linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24platforms, linux, debian . . . . . . . . . . . . . . . . . . . . . . 24platforms, linux, rhel . . . . . . . . . . . . . . . . . . . . . . . . . 24platforms, unsupported. . . . . . . . . . . . . . . . . . . . . . . 24

Rrpm, packages of Flipper . . . . . . . . . . . . . . . . . . . . . . 8

Ssecurity, using ssh to give user access . . . . . . . . . . 14security, using sudo to restrict superuser access

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16ssh, restricting access . . . . . . . . . . . . . . . . . . . . . . . . 14status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16status, investigating . . . . . . . . . . . . . . . . . . . . . . . . . . 16sudo, gaining superuser . . . . . . . . . . . . . . . . . . . . . . 13sudo, restricting access . . . . . . . . . . . . . . . . . . . . . . . 13sudoers, configuration for Flipper . . . . . . . . . . . . . 13svn, source repository for Flipper . . . . . . . . . . . . . . 8swap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

WWindows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

flipper

Documents