high availability for openstack
DESCRIPTION
The primary requirements for OpenStack based clouds (public, private or hybrid) is that they must be massively scalable and highly available. There are a number of interrelated concepts which make the understanding and implementation of HA complex. The potential for not implementing HA correctly would be disastrous. This session was presented at the OpenStack Meetup in Boston Feb 2014. We discussed interrelated concepts as a basis for implementing HA and examples of HA for MySQL, Rabbit MQ and the OpenStack APIs primarily using Keepalived, VRRP and HAProxy which will reinforce the concepts and show how to connect the dots.TRANSCRIPT
![Page 1: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/1.jpg)
HA for OpenStack: Connecting the dots Raghavan “Rags” Srinivas Rackspace
OpenStack Meetup, Boston on Feb. 19th 2014
![Page 2: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/2.jpg)
Rags
• Solutions Architect at Rackspace for OpenStack-based Rackspace Private Cloud
• Speaker at JavaOne, RSA conferences, Sun Tech Days, JUGs and other developer conferences
• Trying to help make OpenStack more “App Developer” friendly
![Page 3: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/3.jpg)
Agenda What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
Resources and Summary
A Peek into HA Methods
![Page 4: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/4.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
OpenStack Design Tenets • Scalability and elasticity are our main goals • Any feature that limits our main goals must be optional • Everything should be asynchronous
– a) If you can't do something asynchronously, see #2
• All required components must be horizontally scalable • Always use shared nothing architecture (SN) or sharding
– a) If you can't Share nothing/shard, see #2
• Distribute everything – a) Especially logic. Move logic to where state naturally exists.
• Accept eventual consistency and use it where it is appropriate. • Test everything
4
![Page 5: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/5.jpg)
What is HA?
HA as Nines Downtime/Year
99% (two nines) 3.65 days
99.9% 8.76 hours
99.99% 52.56 minutes
99.999% 5.26 minutes
99.9999% (six nines) 31.5 seconds
• Minimization of system downtime • Minimization of data/transaction loss • In case of multiple (or interrelated)
failures, minimization of data loss is preferred over minimization of system downtime
![Page 6: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/6.jpg)
Implementing HA
• Elimination of Single Point of Failure (SPOFs) • Redundancy of network components such as switchers and routers
• Redundancy of applications and automatic service migrations
• Redundancy of storage components
• Redundancy of facilities services such as power, AC, etc.
![Page 7: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/7.jpg)
Client
NODE 1 Replication Services
Health Check
Cluster Communication
NODE 2 Replication Services
Health Check
Cluster Communication
VIP
Components (High Level)
![Page 8: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/8.jpg)
Concepts State Description Example
Stateless
• There is no dependency between requests
• No need for data replication/synchronization. Failed request may need to be restarted on a different node.
Apache web server, Nova API, Nova Scheduler, etc.
Stateful
• An action typically comprises multiple requests
• Data needs to be replicated and synchronized between redundant services (to preserve state and consistency)
MySQL, RabbitMQ, etc.
![Page 9: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/9.jpg)
More Concepts Terminology Description
Failover Migration of a service from the “primary” to the “secondary”
Failback Migration of service back to the “primary”
Switchover Migration is initiated manually
![Page 10: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/10.jpg)
Much more concepts
Active/Passive o There is a single master
o Load balance stateless services using a VIP and a load balancer such as HAProxy
o For Stateful services a replacement resource can be brought online. A separate application monitors these services, bringing the backup online as necessary
o After a failover the system will encounter a “speed bump” since the passive node has to notice the fault in the active node and become active
Active/Active o Multiple masters
o Load balance stateless services using a VIP and a load balancer such as HAProxy
o Stateful Services are managed in such a way that services are redundant, and that all instances have an identical state
o Updates to one instance of database would propagate to all other instances
o After a failover the system will function in a “degraded” state
![Page 11: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/11.jpg)
HA for OpenStack
• OpenStack APIs (nova, cinder, etc.)
• RabbitMQ
• MySQL
• Cinder, Swift, and so on
• Heat (still Work in Progress)
• Application running on OpenStack (Application dependent)
![Page 12: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/12.jpg)
Agenda What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
Resources and Summary
A Peek into HA Methods
![Page 13: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/13.jpg)
HA on OpenStack
• Overall Philosophy (Don’t reinvent the wheel)
• Leverage time-tested Linux utilities such as Keepalived, HAProxy and Virtual IP (using VRRP)
• Leverage Hardware Load Balancers
• Leverage replication services for RabbitMQ/MySQL such as RabbitMQ Clustering, MySQL master-master replication, Corosync, Pacemaker, DRBD, Galera and so on
![Page 14: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/14.jpg)
vrrp_script rabbitmq {!
script “usr/sbin/service rabbitmq-server status" # Check the service status!
interval 5 # check every 5 seconds!
weight -2 # adjust priority by -2 if OK!
rise 2 # required number of failures for KO switch!
fall -2 # required number of successes for OK switch!
}!
Keepalived
• Based on Linux Virtual Server (IPVS) kernel module providing layer 4 Load Balancing
• Implements a set of checkers to maintain health and Load Balancing
• HA is implemented using VRRP Protocol
1 2 3 4 5 6 7
![Page 15: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/15.jpg)
HAProxy
• Load Balancing and Proxying for HTTP and TCP Applications
• Works over multiple connections
![Page 16: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/16.jpg)
HA with Keepalived, VRRP & HAProxy
Host1 Host2
Backup
Realserver1 Realserver2
Keepalived HAProxy Network Layer
Application Layer
VRRP
Application
![Page 17: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/17.jpg)
HA on Rackspace Private Cloud
HAProxy
INTERNET
Compute Node 1 Compute Node 2 Compute Node N
VIP(Keepalived, VRRP)
VMs Instantiated
Redundant Active-Passive Infrastructure services
Redundant Active-Active Infrastructure services
Controller 1 Controller 2 Active-Passive Infrastructure services
(MySQL, Rabbit) Active-Active Infrastructure services
(API services)
Heartbeat
![Page 18: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/18.jpg)
HA on Rackspace Private Cloud (switchover)
INTERNET
Compute Node 1 Compute Node 2 Compute Node N
VIP(HAProxy)
VMs Instantiated
Infrastructure services
Controller 1 Controller 2 Active-Passive Infrastructure services
(MySQL, Rabbit) Heartbeat
![Page 19: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/19.jpg)
Agenda What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
Resources and Summary
A Peek into HA Methods
![Page 20: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/20.jpg)
RabbitMQ HA Options
• Health Check without Clustering
• Clustering without Health Check
• Health Check and Clustering
![Page 21: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/21.jpg)
RabbitMQ HA
Ethernet VRID 13
192.168.236.199
Controller 1 VRID 13
IP address: 192.168.236.11
Backup (Passive) Master (Active)
Controller 2 VRID 13 IP address: 192.168.236.12
RabbitMQ RabbitMQ
RabbitMQ Clustering
![Page 22: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/22.jpg)
Agenda What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
Resources and Summary
A Peek into HA Methods
![Page 23: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/23.jpg)
MYSQL HA: MASTER/MASTER REPLICATION
![Page 24: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/24.jpg)
MySQL – Master/Master Replication
Ethernet VRID 12
192.168.236.198
Controller 1 VRID 12
IP address: 192.168.236.11
Backup (Passive) Master (Active)
Controller 2 VRID 12 IP address: 192.168.236.12
MySQL MySQL
Master/Master
![Page 25: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/25.jpg)
MySQL – Master/Master Replication simplified
![Page 26: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/26.jpg)
MYSQL HA: COROSYNC, PACEMAKER AND DRBD
![Page 27: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/27.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Pacemaker, Corosync and DRBD
27
Image from: http://dev.mysql.com/doc/refman/5.0/en/ha-drbd.html"
![Page 28: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/28.jpg)
Pacemaker Corosync DRBD
High availability and load balancing stack for the Linux
platform
Totem single-ring ordering and membership protocol
Synchronizes data at the block device
Interacts with applications through Resource Agents (RA)
UDP and InfiniBand based messaging, quorum, and
cluster membership to Pacemaker
Uses a journaling system (such as ext3 or ext4)
Pacemaker, Corosync, DRBD
![Page 29: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/29.jpg)
DRBD Service Service
FILE SYSTEM
BUFFER CACHE
DRBD
DISK SCHED
DISK DRIVER NIC DRIVER
TCP/IP
RAW DEVICE
DISK
NIC DRIVER
TCP/IP
RAW DEVICE
FILE SYSTEM
BUFFER CACHE
DRBD
DISK SCHED
DISK DRIVER
DISK NIC NIC
![Page 30: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/30.jpg)
MYSQL HA: GALERA
![Page 31: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/31.jpg)
Galera
DBMS
wsrep API
DBMS
wsrep API
DBMS
wsrep API
Galera Replication
Transparent Connections
CLIENTS • Synchronous multi-master cluster technology for MySQL/InnoDB
• MySQL patched for wsrep (Write Set REPlication)
• Active/active multi-master topology
• Read and write to any cluster node
• True parallel replication, in row level
• No slave lag or integrity issues
![Page 32: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/32.jpg)
Multi-master replication
• Based on Optimistic Concurrency Control
• In case of two transactions modifying the same row on different nodes, one of the transactions will abort
• Victim transaction will get Deadlock Error
• Application needs to handle this error
![Page 33: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/33.jpg)
Multi-master cluster looks like one big database with multiple entry points
read & write read & write read & write
MySQL
Multi-master Replication
![Page 34: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/34.jpg)
MySQL MySQL MySQL
write write
GALERA REPLICATION
Multi-master conflicts
![Page 35: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/35.jpg)
MySQL MySQL MySQL
GALERA REPLICATION
write write
Conflict detected
Multi-master conflicts
![Page 36: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/36.jpg)
MySQL MySQL MySQL
GALERA REPLICATION
OK write Deadlock error
Multi-master conflicts
![Page 37: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/37.jpg)
OpenStack and Galera
Image from http://www.severalnines.com/blog/clustering-mysql-backend-openstack"
![Page 38: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/38.jpg)
Galera on Rackspace Private Cloud/OpenStack A How To: OFFICIALLY UNSUPPORTED
1. Install Rackspace Private Cloud on 2 controllers with HA mode (Haproxy, Keepalived and VRRP is already installed)
2. Install Galera (with ws-rep) on 3 separate nodes
3. Mysqldump from controller nodes to Galera node
4. Grant privileges to OpenStack (nova, glance, etc.) and haproxy users
5. Update keepalived and haproxy and OpenStack configuration files on controller/compute
6. Stop/Uninstall MySQL services on controller nodes and restart controller nodes
![Page 39: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/39.jpg)
Agenda What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
Resources and Summary
A Peek into HA Methods
![Page 40: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/40.jpg)
A PEEK INTO HA METHODS
![Page 41: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/41.jpg)
Vendor Clustering/Replication Technique Characteristics
Rackspace Keepalived, HAProxy, VRRP, native clustering
• Automatic install on 2 controller nodes via Chef recipes
Red Hat Pacemaker, Corosync, DRBD
• Manual installation. Fewer components to install
Cisco Keepalived, HAProxy, Galera for MySQL
• Manual install, at least 3 controller nodes
HP Microsoft Windows based installation with Hyper-V
• MS SQL server and other Windows-based methods
HA methods
![Page 42: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/42.jpg)
HA on the Public Cloud
![Page 43: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/43.jpg)
Agenda What is HA?
HA of OpenStack APIs
HA of RabbitMQ
MySQL HA
Resources and Summary
A Peek into HA Methods
![Page 44: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/44.jpg)
Infrastructure Clustering/Replication Technique Characteristics
OpenStack APIs None required (Stateless) • HA also serves as scale out using HAProxy
RabbitMQ RabbitMQ Clustering • RabbitMQ Clustering is setup for single/multiple nodes
Heat TBD • Application Dependent (No standard methods yet).
MySQL Many • Discussed later slide
HA methods
![Page 45: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/45.jpg)
Clustering Method Replication Technique Characteristics
Pacemaker/Corosync/DRBD Mirroring on Block Devices • Well tested, more complex to setup. • Split brain possibility
Keepalived/HAProxy/VRRP Works on MySQL master-master replication
• Simple to implement and understand. • Works for any storage system. • Master-master replication does not work
beyond 2 nodes.
Galera Based on write-set Replication (wsrep)
• No Slave lag • Needs at least 3 nodes • “Deadlock erros” on hotspot rows. • Relatively new
Others MySQL Cluster, RHCS with DAS/SAN Storage
• Some relatively new (GTID) • Some well test • More complex setup
HA methods for MySQL
![Page 46: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/46.jpg)
Resources • OpenStack HA guide
• http://docs.openstack.org/high-availability-guide/content/ch-intro.html
• https://wiki.ubuntu.com/ServerTeam/OpenStackHA
• Other Resources
• http://www.rackspace.com/blog/implementing-high-availability-ha-for-rackspace-private-cloud/
• http://www.rackspace.com/blog/high-availability-ha-with-galera-for-rackspace-private-cloud/https://www.hastexo.com/
• http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-deployment-guide/
• http://docwiki.cisco.com/wiki/OpenStack_Havana_Release:_High-Availability_Manual_Deployment_Guide
• http://www.drbd.org/
• http://www.codership.com/
• http://www.severalnines.com/blog/clustering-mysql-backend-openstack
• https://wiki.openstack.org/wiki/BasicDesignTenets
• http://db.cs.berkeley.edu/papers/hpts85-nothing.pdf
![Page 47: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/47.jpg)
Book
![Page 48: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/48.jpg)
Summary
• In general leverage existing methods of HA
• There are several time-tested and more recent methods for implementing MySQL HA.
• Rackspace Private Cloud provides Chef cookbooks and recipes for implementing HA via Keepalived,
HAProxy and VRRP.
• Galera is gaining more popularity. Since it’s Active/Active it does scale out and is HA.
• Few steps to get from Rackspace Private Cloud to MySQL with Galera (officially unsupported).
• Corosync/Pacemaker/DRBD is recommended by Oracle/MySQL.
• OpenStack HA guide goes through all these options in more detail.
![Page 49: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/49.jpg)
![Page 50: High Availability for OpenStack](https://reader034.vdocument.in/reader034/viewer/2022042518/54b716e54a7959a2718b461f/html5/thumbnails/50.jpg)
Thank you! Raghavan “Rags” Srinivas Solutions Architect Rackspace