automated deployment of an ha openstack cloud · automated deployment of an ha openstack cloud with...
TRANSCRIPT
![Page 1: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/1.jpg)
Automated Deployment of an HA OpenStack Cloudwith SUSE® CloudHO7695
Adam SpiersSenior Software [email protected]
Vincent UntzProject [email protected]
![Page 2: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/2.jpg)
Introduction
![Page 3: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/3.jpg)
3
Agenda
Start building a cloud!
Quick intro to SUSE® Cloud architecture
Learn about HA in OpenStack and SUSE Cloud
Build an HA cluster
Build an HA OpenStack cloud on the cluster
Break things!
![Page 4: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/4.jpg)
Workshop environment
![Page 5: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/5.jpg)
5
Workshop environment
Relax ;-)• We have plenty of time
• Whole build is also automated and idempotent
• You can take home the entire environment afterwards
(available online)
• You can run on any machine with at least 16GB RAM ‒ ... or 8GB at a push
‒ (although that comes with limitations)
![Page 6: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/6.jpg)
6
Workshop environment
• We'll build a miniature cloud on a single machine
• VirtualBox hypervisor
• 4 VMs‒ Administration Server (Crowbar)
‒ 2 Control Nodes in an HA cluster
‒ 1 Compute Node
• Vagrant for rapid deployment
![Page 7: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/7.jpg)
7
What is Vagrant?
• Not just for development• Perfect for "kicking the tyres", demoing, testing etc.• Cross-platform (Linux, MacOS X, Windows)• Providers for libvirt, VirtualBox, VMware, Hyper-V,
Docker, OpenStack, ...
"Creates and configures lightweight, reproducible, and portable development environments."
https://www.vagrantup.com/
![Page 8: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/8.jpg)
8
Vagrant inputs
• 1 or more Vagrant "box" – pre-built virtual appliances • Vagrantfile: Ruby DSL file which defines:
‒ which box(es) to use‒ virtual hardware required ‒ virtual network topology ‒ network ports to forward ‒ hypervisor-specific settings ‒ files to inject into appliance‒ commands to run in appliance
• files to inject
![Page 9: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/9.jpg)
9
Using Vagrant: crash course
• vagrant box add suse/cloud4-admin‒ https://vagrantcloud.com/suse/‒ Also possible to add local boxes
• vagrant up admin• vagrant up controller1• vagrant halt controller2• vagrant destroy compute1• https://docs.vagrantup.com/v2/getting-started/index.ht
ml
![Page 10: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/10.jpg)
10
Workshop Vagrant environment
• https://github.com/SUSE-Cloud/suse-cloud-vagrant‒ demos/HA/‒ vagrant/
‒ Vagrantfile and configs/2-controllers-1-compute.yaml
• VirtualBox pre-installed• 2 boxes pre-installed
‒ suse/cloud4-admin and suse/sles11sp3
• 4 VMs‒ admin: SUSE Cloud 4 Administration Server
‒ controller1, controller2 (will form an HA cluster)
‒ compute1
![Page 11: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/11.jpg)
11
Exercise #1: start the build!
• Start up VirtualBox GUI• cd to local copy of git repository• cd vagrant/• vagrant up• All 4 VMs will be booted in sequence:
‒ admin‒ controller1‒ controller2‒ compute1
![Page 12: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/12.jpg)
SUSE® Cloud Overview
![Page 13: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/13.jpg)
13
SUSE® Cloud
Enterprise OpenStack distribution that rapidly deploys and easily manages highly available, mixed hypervisor IaaS Clouds
• Increase business agility
• Economically scale IT capabilities
• Easily deliver future innovations
![Page 14: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/14.jpg)
14
OpenStack Distribution
Billing VM Mgmt Image Tool Portal App Monitor Sec & Perf
Cloud
Management
Orchestration(Heat)
Dashboard(Horizon)
Cloud APIs(OpenStack and
EC2)
Identity(Keystone)
Images(Glance)
Hypervisor
Compute(Nova)
Operating System
Physical Infrastructure: x86-64, Switches, Storage
OpenStack Icehouse
Object(Swift)
Network(Neutron)
Adapters Adapters
Telemetry(Ceilometer)
Block(Cinder)
SUSE Cloud Adds
RequiredServicesRabbitMQPostgresql
Inst
all F
ram
ewor
k(C
row
bar,
Che
f, TF
TP, D
NS
, DH
CP
)
SUSEManager
SUSEStudio
HypervisorXen, KVM
SUSE Linux Enterprise Server 11 SP3
SUSE Product
Rados
RBD
RadosGW
Ceph
Adapters
Highly Available Services
SUSE® Cloud 4
Physical Infrastructure: x86-64, Switches, Storage
Billing Portal App Monitor Sec & Perf
Adapters Adapters VMware, Hyper-V
Partner Solutions
![Page 15: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/15.jpg)
15
WeekHour
Why an Install Framework?
Parameters
Components
1229111
![Page 16: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/16.jpg)
16
SCARY AS HELL!
Why an Install Framework?
![Page 17: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/17.jpg)
Introduction to Crowbar
![Page 18: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/18.jpg)
18
Crowbar
![Page 19: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/19.jpg)
19
![Page 20: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/20.jpg)
20
![Page 21: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/21.jpg)
21
![Page 22: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/22.jpg)
22
• It could have been worse.
![Page 23: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/23.jpg)
24
SUSE® Cloud architecture
![Page 24: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/24.jpg)
25
SUSE® Cloud Administration Server
![Page 25: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/25.jpg)
26
SUSE® Cloud Control Node
Compute• PostgreSQL database
• Image Service (Glance) for managing virtual images
• Identity (Keystone), providing authentication and authorization for all SUSE Cloud services
• Dashboard (Horizon), providing the Dashboard, which is a user Web interface for the SUSE Cloud services
• Nova API and scheduler
• Message broker (RabbitMQ)
![Page 26: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/26.jpg)
27
SUSE® Cloud Compute Nodes
• Pool of machines where instances run
• Equipped with RAM and CPU
• SUSE Cloud Compute (nova) service
‒ Setting up, starting, stopping, migration of VMs
ComputeNodes
Compute
Compute
![Page 27: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/27.jpg)
29
sledgehammer
![Page 28: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/28.jpg)
barclamp
![Page 29: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/29.jpg)
Interlude
![Page 30: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/30.jpg)
34
Exercise #2: assign aliases to nodes
• Connect to admin node‒ vagrant ssh admin or‒ ssh [email protected] or‒ use VM console in VirtualBox
• Root password is vagrant• Type q then y to accept the beta EULA• Run setup-node-aliases.sh• Point a browser at the Crowbar web UI
‒ http://localhost:3000• Check the 4 nodes are registered, named correctly,
and in Ready state (green)
![Page 31: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/31.jpg)
High Availability and Cloud
![Page 32: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/32.jpg)
36
Why High Availability?
“I can't have my systems go down. We lose $1,000,000 for every minute that we're down, and upper-management gets really 'excited' when that happens.”
![Page 33: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/33.jpg)
37
High Availability for OpenStack
What might we want to protect?
• Admin server‒ core infrastructure: DNS, NTP, provisioning capabilities
• Controller node‒ OpenStack services
• Compute nodes‒ Hypervisor‒ VM instances (i.e. guests in the cloud)
![Page 34: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/34.jpg)
38
Component failure impact
• Admin server‒ New cloud nodes require manual addition and configuration‒ Currently no ability to rediscover existing nodes on restart‒ No impact on currently operating cloud
• Control node‒ Cannot start or stop guest instances‒ No ability to rediscover existing nodes or guest VMs on restart‒ No impact on currently deployed instances
![Page 35: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/35.jpg)
39
Pets vs. cattle metaphor• Pets are given names like
mittens.mycompany.com• Each one is unique, lovingly hand-
raised and cared for• When they get ill, you spend money
nursing them back to health
• Cattle are given names like vm0213.cloud.mycompany.com
• They are almost identical to other cattle• When one gets ill, you shoot it and get
another one
![Page 36: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/36.jpg)
40
Component failure impact (continued)
• Compute node‒ Loss of VMs on that node‒ Recovery is by restart and re-provisioning of physical server‒ Can be mitigated through application design
• VM instances‒ Loss of workload‒ Recovery is by booting a replacement instance (cattle)‒ Can be mitigated through application design
![Page 37: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/37.jpg)
41
Component failure assessment
• Control Node‒ Highest priority‒ Recovery realistically requires complete cloud restart
• Compute Node & VM instances‒ Application level recovery is normal practice for existing clouds‒ Not existing “enterprise” expectation, but workaround exists for
new workloads
• Admin Server‒ Least impact on deployed system‒ Operation can continue with no impact on end users
![Page 38: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/38.jpg)
42
Status Quo of HA in OpenStack
• Community is now mostly converged on a standard architecture for an HA control plane involving Pacemaker and HAproxy‒ SUSE® was first vendor to release a supported implementation
of this, via an update to SUSE Cloud 3 (May 2014)
• No one has yet implemented a full solution for HA of compute nodes and VM guests‒ However community discussion in the last month has
generated proposals which look quite promising.
• HA for storage and network nodes is also still ongoing work
![Page 39: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/39.jpg)
High Availability in SUSE Cloud
![Page 40: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/40.jpg)
44
HA in SUSE® Cloud (high level)
• Administration Server‒ No longer a SPoF (Single Point of Failure)‒ Can have multiple DNS / NTP servers‒ Backup / restore script for cold or warm standby
• Control Plane‒ Run services in a cluster
‒ to ensure availability of data and service
‒ Some OpenStack services are stateless‒ Some can run active/active, e.g. API endpoint services‒ The load balancer still needs protecting‒ Database and message queue need shared storage
![Page 41: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/41.jpg)
45
HA Control Plane in SUSE® Cloud
![Page 42: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/42.jpg)
46
HA Control Plane in SUSE® Cloud
![Page 43: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/43.jpg)
47
HA Control Plane in SUSE® Cloud
• Fully automated cluster setup through Pacemaker barclamp
• Simple, intuitive web UI• Allows choice of cluster size and quantity• Supports multiple strategies for STONITH and storage• Uses SLE HAE components
‒ Pacemaker, HAproxy, DRBD
• Architecture consistent with OpenStack community recommendations
![Page 44: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/44.jpg)
48
HA Control Plane in SUSE® Cloud
• Active/passive for PostgreSQL and RabbitMQ‒ choice of replicated (DRBD) or shared storage
• Active/active for other services via HAproxy load balancer‒ HAproxy itself is active/passive
‒ Innovative approach to Neutron L3 agent
![Page 45: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/45.jpg)
49
Setting Expectations
• Not fault tolerance
• Small outage of services is tolerated
• Automated recovery within small number of minutes
• “4 nines” availability (99.99% = ~53 mins/year)
• Maybe even five 9s achievable (4.32 mins/year) depending on context
• Some manual intervention may be necessary to repair a degraded (but still functioning) cluster
![Page 46: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/46.jpg)
50
Simple cluster architecture
Pacemaker Cluster
Control Node 1 Control Node 2
DRBDPostgreSQL
RabbitMQ
Keystone
Glance
Nova
Dashboard
Cinder
Neutron
Telemetry
Orchestration
![Page 47: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/47.jpg)
51
Recommended architecture
Services Cluster
Node 1 Node 2
Pacemaker Cluster
Control Node 1 Node
DRBDPostgreSQL
RabbitMQ
KeystoneGlanceNova
DashboardCinder
Neutron
Node 3
Orchestration
Keystone
GlanceNova
Dashboard
Cinder
Database Cluster
Node 1 Node 2
DRBD or shared storagePostgreSQL
Network Cluster
Node 1 Node 2 Node 3
NeutronRabbitMQ
Telemetry
![Page 48: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/48.jpg)
Building a Pacemaker Cluster
![Page 49: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/49.jpg)
53
Cluster fencing
• Every HA cluster needs an out-of-band fencing mechanism. This is not optional!
• Simplistically, if cluster communications break down, consensus is lost, and multiple nodes may contend for the same data/service (“split brain” syndrome)
• Solution: STONITH (Shoot The Other Node In The Head)• Popular fencing devices include IPMI, IBM RSA, HP iLO,
Dell DRAC• We'll use SBD (Storage-Based Death) which allows
sending “poison pill” messages via shared block storage device.
![Page 50: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/50.jpg)
54
Exercise #3: build a Pacemaker cluster
• In VirtualBox GUI, view settings for controller1 and controller2‒ Observe locations of extra disks (SBD and DRBD)‒ Which disk is shared?
• Point a browser at the Crowbar web UI‒ http://localhost:3000
• Follow instructions for deploying a Pacemaker cluster‒ SBD device is /dev/sdc
• On admin node:‒ Run tail -f /var/log/crowbar/chef_client/*
![Page 51: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/51.jpg)
55
Exercise #4: check cluster health
• Wait for chef-client to complete on both controller nodes• Pacemaker proposal should finish applying and go green• Connect to controller1 node
‒ vagrant ssh controller or‒ connect to admin node and ssh controller1, or‒ use VM console in VirtualBox
• Root password (vagrant) and EULA as before• Run crm_mon and check cluster has two nodes online• Visit Hawk web UI on https://localhost:7630
![Page 52: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/52.jpg)
56
Exercise #5: build HA cloud
• Follow remaining instructions for deploying the remaining barclamps
• This will take quite a long time (at least 30 minutes)• If you are feeling lazy, you can use a tool to this
automatically:‒ crowbar batch --timeout 1200 \
build HA-cloud.yaml
• Watch Hawk web UI and crm_mon output as Crowbar/Chef automatically add resources to the cluster
![Page 53: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/53.jpg)
57
Exercise #6: simulate failures
• Follow instructions for testing cluster failover
![Page 54: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/54.jpg)
58
Exercise #7: recover degraded cluster
• Follow instructions for recovering a degraded cluster
![Page 55: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/55.jpg)
59
If you made it this far, well done!
![Page 56: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/56.jpg)
Corporate HeadquartersMaxfeldstrasse 590409 NurembergGermany
+49 911 740 53 0 (Worldwide)www.suse.com
Join us on:www.opensuse.org
60
![Page 57: Automated Deployment of an HA OpenStack Cloud · Automated Deployment of an HA OpenStack Cloud with SUSE ® Cloud HO7695 Adam Spiers Senior Software Engineer aspiers@suse.com Vincent](https://reader030.vdocument.in/reader030/viewer/2022040100/5e8859e466f5b43032702a96/html5/thumbnails/57.jpg)
Unpublished Work of SUSE LLC. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.