rule breaker! - openstack · rule breaker! upgrading an openstack cloud while skipping a release...
TRANSCRIPT
Rule breaker!Upgrading an OpenStack Cloud while skipping a release
Rick Salevsky Nanuk KrinnerSUSE Cloud Engineer SUSE Cloud [email protected] [email protected]
Introduction
3
Speakers• Rick Salevsky
– SUSE Cloud Engineer– Focus deployment solutions
• Nanuk Krinner– Cloud developer at SUSE– Systems Management Engineer
• SUSE OpenStack Cloud
4
Agenda
• Why and What
• Upgrade strategies
• How we skipped a release
• Where we want to go
Why and What
6
Why upgrading?• Security Fixes
• Stability improvements
• Performance improvements
• Closely follow upstream development
• New features
7
Problems while upgrading?• Downtime
• Preparation
• Testing
• Adapting workflows
• Bugs
• Data loss
8
Customer demands• Reduced downtime
• Live upgrade
• Possible to roll back
• Clear documentation of what is happening
• Upgrading while skipping one or more releases
9
Upgrade marathon
Release Evaluating the release
Planning the upgrade
Testing the upgrade
Fine tuningIntegrating new features Upgrade
10
Upgrade marathon
Release Evaluating the release
Planning the upgrade
Testing the upgrade
Fine tuning
Maybe not this time?
Integrating new features Upgrade
11
• OpenStack User Survey April 2016No upgrade at all
Upgrade strategies
13
Official upgrade process• Always upgrade to next release
• Release cycle of 6 months
• Upgrades are required regularly
• High maintenance cost– Unexpected changes break upgrade– Lot’s of manual effort– Suffer the upgrade pain regularly– Staffing Source:
http://www.openstack.org/brand/openstack-logo/logo-download/
14
Continuous Deployment• Risky
• Needs a lot of development manpower
• Extensive testing required
• Always latest greatest
• No big upgrade, incremental changes
• Not enterprise readySource: https://wiki.jenkins-ci.org/display/JENKINS/Logo
15
Start from scratch• Roll out a fresh deployment• Lots of duplicated work
– Set up projects, users, images…
• Get rid of outdated artifacts• Run a parallel installation
– Move workload from old cloud to new cloud
• Redundant hardware required
16
Many self-made solutions
• Own deployment solutions
• Scenario-specific solutions
How we skip a release
18
High level overview• Upgrading from Juno to Liberty
• Multistep process
• Cloud is not fully functional
• Upgrading OS along with OpenStack
19
The idea• Orchestrated reinstallation
• Ignoring OpenStack Kilo release
• Migration handling
• Config file management
• Still Downtime and Disruptive
• No extra hardware required
20
Requirements• Orchestration mechanism
• Configuration management
• New OpenStack Packages are available
• Enough disk space on Controller– Duplicated database
• Shared disk for nova-compute data
21
Preparation• Stop configuration management
• Update OpenStack configs
• Check which migrations are needed
22
Backup data• Disable (not stop) all OpenStack Services
• Shutdown OpenStack on non DB nodes
• Backup OpenStack database
• Backup other important data if wanted
• Finalize OpenStack Shutdown
23
Setup new OpenStack Cloud• Reinstall Nodes with new OS if required
• Install new OpenStack Packages
• Start configuration management
• Start database service
• Restore backed up data
24
Migrating OpenStack Services• Run all migrations as documented for a upgrade
• Special commands need porting
• Juno to Liberty exceptions– Nova
25
Migrating Nova Service• Migrate to last Kilo migration level
– Last kilo migration = 290– ‘nova-manage db sync --version 290’
• Migrate Flavor data– Porting from Kilo to Liberty was required– ‘nova-manage db migrate_flavor_data’
• Migrate to Liberty migration level– ‘nova-manage db sync’
26
Finalizing Upgrade• Start all OpenStack Services
• Check if everything is running
27
Issues• Configuration File migration
• Migrations
• All or nothing
• Predefined Upgrades
28
Do’s Dont’s• Backups
• Test new configurations• Hope everything runs
29
Do’s and Dont’s
Where we want to go
31
Outlook• Seamless Upgrade
• No downtime of important services
• Reverting upgrades
• Better orchestration
32
Call for Action• Config files (automatically) upgradeable
• Uniform configuration files
• Migrating existing data from every point
• Rollback option
• Non-disruptive upgrades
• Integrating oslo version objects in every project
33
Thank you.
Questions?
35
Real world issues• Skipped or delayed upgrades
• Small operator teams
• Downstream code changes
• Ignoring recommended path
• Services depending on the
cloud
OpenStack User Survey, April 2016