openstackha 130925132534-phpapp02
TRANSCRIPT
![Page 1: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/1.jpg)
1
HA in OpenStack
September 24th 2013
![Page 2: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/2.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Agenda
• What is HA
• Compute HA
• Controller HA
• Corosync, Pacemaker and DRBD
• Galera
• HAProxy, keepalived, VRRP
• Resources and Summary
2
![Page 3: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/3.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is HA?
3
![Page 4: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/4.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability
4
Minimize data loss
Minimize system downtime
![Page 5: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/5.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Stateless services
– There is no dependency between requests
– For example: Nova API, Nova Scheduler, etc.
• Stateful services
– An action typically comprises multiple requests
– For example: MySQL, RabbitMQ, etc.
5
![Page 6: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/6.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
High Availability Concepts
• Active/Passive
– Redundant instances of stateless services are load balanced
– For Stateful services a replacement resource can be brought online.
• Active/Active
– Redundant instances of stateless services are load balanced
– Stateful services are managed in such a way that services are redundant, and that all instances have an identical state.
– Updates to one instance of a database would also update all other instances.
6
![Page 7: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/7.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Compute HA
7
![Page 8: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/8.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
No vSphere Style HA with KVM
8
![Page 9: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/9.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Server Evacuation
9
Evacuation
Evacuation
• Without Shared Storage
– The instance will be booted from a new disk, but will preserve the configuration, e.g. id, name, uid, ip...etc.
• With Shared Storage
– The instance will be booted from same disk and data will be preserved
![Page 10: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/10.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Instance Migration
10
![Page 11: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/11.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
11
Virtualization vs. Cloud
• Virtualization needs care and feeding
– Name the VM
– Tune and groom regularly
– Feed it with good food and supplements
– Take to the vet when sick
• Cloud servers are disposable
– VMs are not unique
– Tune and groom apps not the cows
– Keep the cow upright
– Shoot the cow when it is sick
![Page 12: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/12.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Scale Up vs. Scale Out
12
Traditional Cloud
![Page 13: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/13.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Application HA
13
![Page 14: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/14.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Controller HA Options
14
![Page 15: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/15.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Pacemaker, Corosync and DRBD
15
• Pacemaker
– high availability and load balancing stack for the Linux platform.
– Interacts with applications through Resource Agents (RA)
• Corosync
– Totem single-ring ordering and membership protocol
– UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker.
• DRBD (Distributed Replication Block Device)
– Synchronizes Data at the block device
– Uses a journaling system (such as ext3 or ext4)
![Page 16: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/16.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Galera
• Synchronous multi-master cluster technology for MySQL/InnoDB
– MySQL patched for wsrep (Write Set REPlication)
– Active/active multi-master topology
– Read and write to any cluster node
– True parallel replication, in row level
– No slave lag or integrity issues
16
![Page 17: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/17.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA with Keepalived and HAProxy
17
![Page 18: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/18.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Keepalived, HAProxy and VRRP
• HAProxy
– Load Balancing and Proxying for HTTP and TCP Applications
– Works over multiple connections
– Used to load balance API services
• VRRP (Virtual Router Redundancy Protocol)
– Eliminates SPOF in a static default routed environment
• Keepalived
– Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing
– Implements a set of checkers to check service status and to maintain health
– Leverage the VRRP Protocol to remap VIPS in event of failure
18
![Page 19: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/19.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Sample OpenStack HA Architecture
19
.… Availability
Zone 1
Dedicated Firewalls
BO
ND
0 BO
ND
1BO
ND
0 BO
ND
1
Controller
API Services
API & HorizonCinder API
Nova SchedulerKeystone
GlanceRabbitMQ
MYSQL
Chef ServerRecipes
Load Balancers
Redundant Network Switches
StorageEMC, NetApp, or
SolidfireVols
BO
ND
2
Redundant Network Switches
Inside LB VLAN
Storage Network (private)
Fixed Network (private)
Compute 1
KVM
G2
G1
G4
G3
Compute N
KVM
G6
G5
G7
BO
ND
0 BO
ND
1
BO
ND
2
.… Availability
Zone 2
BO
ND
0 BO
ND
1
BO
ND
2
Compute 1
KVM
Compute N
KVM
G16
G15
G17
BO
ND
0 BO
ND
1
BO
ND
2
BO
ND
0 BO
ND
1
BO
ND
2
G12
G11
G14
G13
BO
ND
2
Controller
API Services
API & HorizonCinder API
Nova SchedulerKeystone
GlanceRabbitMQ
MYSQL
Chef ServerRecipes
![Page 20: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/20.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA on OpenStack
20
![Page 21: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/21.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HA on OpenStack
21
![Page 22: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/22.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
22
Comparison
Database Replication
method
Strengths Weakness/Limita
tions
Keepalived/HAPro
xy/VRRP
Works on MySQL
master-master
replication
Simple to
implement and
understand.
Works for any
storage system.
Master-master
replication does
not work beyond 2
nodes.
Pacemaker/Coros
ync/DRBD
Mirroring on Block
Devices
Well tested More complex to
setup. Split Brain
possibility
Galera Based on write-
set Replication
(wsrep)
No Slave lag Needs at least 3
nodes. Relatively
new.
Others MySQL Cluster,
RHCS with
DAS/SAN storage
Well tested More complex
setup.
![Page 23: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/23.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resources and Contact
23
![Page 24: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/24.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resources
• OpenStack
– openstack.org
– launchpad.net/openstack
– #openstack
– #openstack on webchat.freenode.net
• OpenStack HA
– http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf
– https://github.com/rcbops-cookbooks/
• MySQL HA
– http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-deployment-guide/
– http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html
– https://www.hastexo.com/
– http://www.drbd.org/
![Page 25: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/25.jpg)
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2
5RACKSPACE® HOSTING | WWW.RACKSPACE.COM
For More Information
You can reach me at:
Kenneth Hui
Open Cloud Architect
Rackspace
E-mail: [email protected]
Twitter: @hui_kenneth
Blog: http://cloudarchitectmusings.com
![Page 26: Openstackha 130925132534-phpapp02](https://reader030.vdocument.in/reader030/viewer/2022032421/55a7b5ae1a28abc8168b47f5/html5/thumbnails/26.jpg)
26
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM