building and running openstack on power8 and running...mitaka controller node runs all public api...

39
BUILDING AND RUNNING OPENSTACK ON POWER8 LANCE ALBERTSON Oregon State University Open Source Lab @ramereth 0

Upload: others

Post on 03-Jul-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

BUILDING AND RUNNING OPENSTACK ONPOWER8

LANCE ALBERTSONOregon State University Open Source Lab

@ramereth

0

Page 2: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

SUMMARYPOWER8 OverviewPOWER at OSUOSLBuilding a RHEL-based P8 platform with OpenstackArchitecture porting issuesProblems we have encountered with OpenStackOpenstack deployment with ChefOSL Wrapper cookbookNext Steps

0

Page 3: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

DISCLAIMER

POWER8 OVERVIEW

Page 4: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

POWER8 OVERVIEW

DESIGNDesigned to be a massively multithreaded chipDesigns are available for licensing under the OpenPOWER FoundationLittle-Endian & Big-EndianSeveral non-IBM companies building P8 hardware

Tyan, Rackspace (OpenCompute-based) & Google

Page 5: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

OPENPOWER ABSTRACTION LAYER (OPAL)OPAL is the new Open Source firmware for POWER8Acts as an on-system HMCEnables the machine to boot similar to PC serversLinux Kernel and loads the boot loader PetitbootPetitboot provides a shell environment for debugging and setupPetitboot will use kexec and boot into the system kernel

POWER AT OSUOSL

Page 6: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

POWER AT OSUOSL

HISTORYProviding PPC64 compute resources since 2005Close collaboration with IBM LTCPOWER5, POWER7 and now POWER8OSL managed LPAR deployment to make it easier on projectsPre-P8 Projects:

Debian, Gentoo, Fedora, PostgreSQLLinux Foundation, Haskell, GoLangMozilla, OpenSUSE, LLVM, GCC

Page 7: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

POWER8 AT OSUOSLGoal is to provide on-demand PPC64/PPC64LE compute resources to FOSSprojectsAssist with ppc64/ppc64le porting & testingExpose OSU students to OpenStack and POWER8Collaboration with IBM engineers on architecture issuesCreate a vanilla Openstack cluster for FOSS projects

Page 8: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

PROJECTS RUNNING ON OUR P8 CLUSTERCloudFoundry, Docker, CentOS, CouchDBHaskell, Glibc, JXcore, LLVM, NodeJSOpenJDK, GoLang, oVirt, libjpeg-turboBLCR, Gentoo

BUILDING A RHEL-BASED P8 PLATFORM WITH OPENSTACK

Page 9: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

BUILDING A RHEL-BASED P8 PLATFORM WITH OPENSTACK

SUPPORTED OS PLATFORMSPowerKVM

Ubuntu

RHEL

Page 10: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

DECISION TO USE RHELLittle community support at the time and opportunity to help the communityWe use CentOS internally as our primary OS & more familiar with the RHELeco-systemRHEL has the RDO OpenStack distribution that is well supportedChef support with OpenStack needed some helpI love challenges!

Page 11: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

OPENSTACK ARCHITECTURE (OLD)Started in 2014IcehouseController node

Runs all public API services, dashboardDB hosted on a shared bare-metal systemX86_64 CentOS 6 VM running on Ganeti+KVM

Compute node(s)Nova compute and networkingFlat networkingPPC64 Fedora

Page 12: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

OPENSTACK ARCHITECTURE (NEW)Deployed 2016 (deployed last week)MitakaController node

Runs all public API services, dashboardDB hosted on a shared bare-metal systemX86_64 CentOS 7 VM running on Ganeti+KVM

Compute node(s)Nova ComputeNeutron Networking

LinuxbridgeProvider and Tenant networking using VXLAN

PPC64LE CentOS 7.2

Page 13: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

COMPUTE NODESDid initial development on Fedora 19Fedora 20 PPC64 base system (old)Fedora 21 versions of a few packagesCentOS 7.2 PPC64LE base system (new)

ARCHITECTURE PORTING ISSUES

Page 14: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

ARCHITECTURE PORTING ISSUES

CHEFNo PPC64/PPC64LE Chef clientNeeded to build our own chef-clientOmnibus

Bootstrap build envBuild dependency issuesArchitecture configuration issues in Omnibus

Chef has stable ppc64/ppc64le builds today

Page 15: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

PACKAGE SUPPORTSupport for P8 was bleeding edge and new features were added weeklyBuilt versions of latest packages from Fedora rawhide packages:

qemulibvirtkernel

Internal repo for these custom packages:

Kernel required a few custom options to be enabledRuntime setup: Disable SMT

http://�p.osuosl.org/pub/osl/repos/yum/openpower/centos-7/ppc64le/

Page 16: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

GUEST OS IMAGESFew OS supported ppc64/ppc64le or provided guest images pre-builtVariety of tools which are platform specificMissing support for cloud-initInitially started creating images manually with qemu directly

Page 17: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

PACKER -- MULTI PLATFORM SUPPORTWe needed Go to use PackerGoLang support was literally in the worksFinally built our own packer binary last Nov!

WIP Packer Templates:http://�p.osuosl.org/pub/osl/openpower/rpms/

https://github.com/osuosl/bento/tree/ramereth/ppc64

Page 18: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

ARCHITECTURE ISSUESOPAL firmware bugspre-P8 machines were very buggyIPMI console would sometimes stop workingRandom lockupsIncluded HW RAID, but no cached write-back support

PROBLEMS WE HAVE ENCOUNTERED WITH OPENSTACK

Page 19: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

PROBLEMS WE HAVE ENCOUNTERED WITH OPENSTACK

LEARNING AND UNDERSTANDING OPENSTACKLots of moving piecesNeutron networking is complex and a moving targetDeciding on the proper design architecture for our use case

Page 20: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

BUGS AND "FEATURES"Interaction between libvirt and nova-compute was buggy at timesSome bugs were just Icehouse itself, others were architecture specificLearning how to deploy Openstack and making (gasp) mistakes!Iptables issues between Chef and OpenstackProvider networks configures dnsmasq as an open resolverSSL API endpoints

Page 21: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

STABILITYRabbitmq would constantly need to be restartednova-compute services would randomly stop workingRunning Fedora on compute and CentOS on controller made things ...interesting

CENTOS 7 PPC64LE ON OPENSTACK

Page 22: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

CENTOS 7 PPC64LE ON OPENSTACK

RHEL / CENTOS SUPPORTIntroduced in 7.1 and fully supported in 7.2CentOS community was still bootstrapping and testingWe built our own pre-release CentOS 7.2 for testingUsing ppc64le on compute nodes

Page 23: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

RDOCommunity for deploying Openstack on CentOS, Fedora and RHELRepositories built against each PlatformEach release of OpenStack separated

Page 24: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

RHEV (RED HAT ENTERPRISE VIRTUALIZATION)Updated KVM packagingPart of the Virt SIG of CentOSUsed SRPMs to build ppc64le versions in a location repoOne patch needed to work around bug

OPENSTACK DEPLOYMENT WITH CHEF

Page 25: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

OPENSTACK DEPLOYMENT WITH CHEF

WHY CHEF?Primary CM tool used at the OSLProvides a lot of testing capability on deploymentCan use the full power of the Ruby language for configuring the cluster

Page 26: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

CHEF OPENSTACKSet of cookbooks that will deploy the various services of OpenstackPart of the OpenStack umbrellaCommunity drivenDid a major refactor of the code for Mitaka release

Page 27: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

OSL OPENSTACKCreated a wrapper cookbook (osl-openstack)

OSL site specific configurationSplit recipes out by upstream cookbook nameContains ppc64le specific changesCurrently only tested on CentOS 7

https://github.com/osuosl-cookbooks/osl-openstack

OSL WRAPPER COOKBOOK

Page 28: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

OSL WRAPPER COOKBOOK

RECIPES/DEFAULT.RB

Default configuration for clusterInclude local yum reposInclude command clientsLogic around endpoints

recipes/default.rb

Page 29: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

RECIPES/IDENTITY.RB

Just includes recipesSome wrapper, some upstreamAllows us to test just Keystone by itself

recipes/identity.rb

Page 30: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

RECIPES/CONTROLLER.RB

Pulls in all wrapper recipes needed to build a controllerAllows for us to split things out eventually if we want to

recipes/controller.rb

Page 31: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

TESTING AND DEVELOPMENTUnit Testing

ChefSpecRSpec

Integration TestingTest KitchenServerSpec

Chef ProvisioningDeploy VMs as controller/computeDeploy on bare-metal for a test cluster

Page 32: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

UNIT TESTINGEnsure the Chef code is doing what it’s supposed to doEasily test Architecture-specific logicVerify configuration files contain proper settingsExamples:

spec/default_spec.rbspec/compute_controller.rbspec/linuxbridge_spec.rb

Page 33: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

UNIT TESTING (OUTPUT)$ rspec spec/default_spec.rb

osl­openstack::default  includes cookbook base::ifconfig  includes cookbook selinux::permissive  includes cookbook yum­qemu­ev  includes cookbook openstack­common  includes cookbook openstack­common::logging  includes cookbook openstack­common::sysctl  includes cookbook openstack­identity::openrc  includes cookbook openstack­common::client  includes cookbook openstack­telemetry::client  setting arch to x86_64    does not add OSL­Openpower repository on x86_64  setting arch to ppc64    add OSL­openpower­openstack repository on ppc64  setting arch to ppc64le

Page 34: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

TEST KITCHEN & SERVERSPECTest Kitchen

Test CLI tool which allows you to execute the configured code on one ormore platformsIntegrates with testing frameworksMust have tool for Chef usersConfigured via

ServerSpecRSpec tests for configured serversIntegration testsEnsures things actually happen on the systemExample:

.kitchen.yml

test/integration/default/serverspec/default_spec.rb

Page 35: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

TEST KITCHEN (LIST)$ kitchen listInstance                            Driver     Provisioner  Verifier  Transport  Last Actiondefault­centos­72                   Openstack  ChefZero     Busser    Rsync      Not Createdmon­centos­72                       Openstack  ChefSolo     Busser    Rsync      Not Createdmon­controller­centos­72            Openstack  ChefSolo     Busser    Rsync      Not Createdops­messaging­centos­72             Openstack  ChefZero     Busser    Rsync      Not Createdidentity­centos­72                  Openstack  ChefZero     Busser    Rsync      Not Createdimage­centos­72                     Openstack  ChefZero     Busser    Rsync      Not Creatednetwork­centos­72                   Openstack  ChefZero     Busser    Rsync      Not Createdlinuxbridge­centos­72               Openstack  ChefZero     Busser    Rsync      Not Createdcompute­controller­centos­72        Openstack  ChefZero     Busser    Rsync      Not Createdcompute­centos­72                   Openstack  ChefZero     Busser    Rsync      Not Createddashboard­centos­72                 Openstack  ChefZero     Busser    Rsync      Not Createdblock­storage­centos­72             Openstack  ChefZero     Busser    Rsync      Not Createdblock­storage­controller­centos­72  Openstack  ChefZero     Busser    Rsync      Not Createdtelemetry­centos­72                 Openstack  ChefZero     Busser    Rsync      Not Createdcontroller­centos­72                Openstack  ChefZero     Busser    Rsync      Not Created

Page 36: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

TEST KITCHEN (TEST)$ kitchen test default­­­­­> Starting Kitchen (v1.8.0)­­­­­> Cleaning up any prior instances of <default­centos­72>­­­­­> Destroying <default­centos­72>...       Finished destroying <default­centos­72> (0m0.00s).­­­­­> Testing <default­centos­72>­­­­­> Creating <default­centos­72>...       OpenStack instance with ID of <a25fa410­5caf­4f96­bddb­1e6daddd06d9> is ready....

Chef Client finished, 115/198 resources updated in 03 minutes 12 secondsFinished converging <default­centos­72> (3m41.31s).­­­­­> Setting up <default­centos­72>...Finished setting up <default­centos­72> (0m0.00s).­­­­­> Verifying <default­centos­72>...Preparing files for transfer...

NEXT STEPS

Page 37: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

NEXT STEPS

INFRASTRUCTURE NEXT STEPSAdd Nagios checks (DONE!)Continue to fix bugs and other issues as they come upRebuild old Icehouse cluster as Mitaka (no upgrade)Add support for object storageUpdate documentationAdd support for non-live migrationMellanox networking

Page 38: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

PROJECT EXPERIENCEImprove and streamline on boarding processExpand cluster’s disk storage capacityImprove stability of the clusterAdd more projects!Submit your request:

http://osuosl.org/services/powerdev/request_hosting

Page 39: BUILDING AND RUNNING OPENSTACK ON POWER8 and running...Mitaka Controller node Runs all public API services, dashboard DB hosted on a shared bare-metal system X86_64 CentOS 7 VM running

QUESTIONS?Lance Albertson

[email protected]

@ramereth

– http://osuosl.org http://lancealbertson.com

Links:

http://github.com/ramereth/presentation-openstack-power8https://github.com/osuosl-cookbooks/osl-openstackhttp://osuosl.org/services/powerdev/request_hostinghttp://�p.osuosl.org/pub/osl/repos/yum/openpower/centos-7/ppc64le/http://�p.osuosl.org/pub/osl/openpower/

Attribution-ShareAlike CC BY-SA ©2016