continuously-integrated puppet in a dynamic environment
DESCRIPTION
This talk will show how we deploy Puppet without a Puppetmaster on an autoscaling Amazon Web Services infrastructure. Key points of interest: - Masterless Puppet - Use of Jenkins for Puppet manifest testing and environment promotion (test->staging->production) - Puppet integration with Amazon CloudFormation Sam Bashton Director, Bashton Ltd After working for a number of Internet Service Providers, Sam founded Bashton Ltd in 2004. Focussing exclusively on Linux and Open Source software, Sam and his team provide consultancy, support and 24/7 infrastructure management for a number of high-traffic websites. A serial early adopter, Sam has travelled the world providing training and consultancy and generally spreading the Open Source message. Sam lives in Manchester, UK.TRANSCRIPT
![Page 1: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/1.jpg)
CONTINUOUSLYINTEGRATED PUPPET IN ADYNAMIC ENVIRONMENT
SAM BASHTON, BASHTON LTD
![Page 2: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/2.jpg)
![Page 3: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/3.jpg)
CONTINUOUSLYINTEGRATED PUPPET IN ADYNAMIC ENVIRONMENT
MASTERLESS PUPPET: WHYAND HOW
SAM BASHTON, BASHTON LTD
![Page 4: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/4.jpg)
ABOUT MELinux guy since Slackware, floppy disks and root + bootUsing Puppet since 2007Run a company in Manchester, North West EnglandWe provide outsourced ops for other companies
![Page 5: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/5.jpg)
OUR FULLY MANAGEDENVIRONMENTS
Primarily transactional websites (e-commerce)Majority (70%+) on Amazon Web Services (AWS)Majority using CentOS
![Page 6: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/6.jpg)
HOW WE WORKSimple is better than complexComplexity is worth adding only if it provides obviousfunctional benefits
Re-usabilityResilience
![Page 7: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/7.jpg)
WHY DID WE PICK AWS?Featureset and toolset massively in advance of anyother cloud provider, public or private#1 customer reason for switching to AWS? The ability toscale on demand
![Page 8: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/8.jpg)
![Page 9: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/9.jpg)
TOOLS WE USE FORBUILDING AND MANAGINGDo one thing and do it well
CloudFormation - Amazon tool to manageinfrastructurePuppet - Manage system configurationPulp - centralised repository, manages packagerevisionsJenkins
![Page 10: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/10.jpg)
HOW WE USE PUPPETNo PuppetmasterPuppet manifests, hieradata and modules distributedto all machines via RPMAll machines boot with a common, blank image and getconfigured at first boot
![Page 11: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/11.jpg)
WHAT'S WRONG WITHMASTER BASED PUPPET?
Pets vs Cattle
Puppet designed for a world of servers as petsWe do not live in that world
![Page 12: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/12.jpg)
PUPPET DESIGNED FOR PETSMany assumptions in Puppet presume that yourservers are petsSome of these work against us when managing a herd
![Page 13: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/13.jpg)
MANUAL CERTIFICATESIGNING
Clearly unsuitable when machines are automaticallyprovisioned
![Page 14: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/14.jpg)
POTENTIAL WORKAROUNDS:AutosignUse/write another automated certificate generationmechanism
Possibly tied in with autoscaling
![Page 15: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/15.jpg)
NO MECHANISM FORCLEANING OLD HOSTS
Likely to have host-names reused, causing machines tofail to configurePuppetmaster will fill with certificates for machines thatran for a few hours and went away again
![Page 16: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/16.jpg)
POTENTIAL WORKAROUNDS:Use UUID certificatesAgree not to look in the certificate directoryWrite mechanism for cleaning up old certificates
![Page 17: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/17.jpg)
HOSTS CONFIGURED BASEDON HOSTNAME
Our machines have names like ip-172-26-5-123How does Puppet know what type of machine this is?
![Page 18: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/18.jpg)
POTENTIAL WORKAROUNDSUse an external node classifierUse some mechanism for giving a better hostname, egweb-172-26-5-123 and use regex for node names
![Page 19: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/19.jpg)
PUPPETMASTER IS A SINGLEPOINT OF FAILURE
If the Puppetmaster fails, we can no longer autoscaleupIn particular, this could be a problem if there isavailability zone failure
![Page 20: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/20.jpg)
POTENTIAL WORKAROUNDSClustered Puppetmasters
![Page 21: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/21.jpg)
WORKAROUND RECAPUse/write alternative certificate management softwareWrite an external node classifier / mechanism forsetting hostname appropriatelyCluster multiple Puppetmasters
![Page 22: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/22.jpg)
WHAT WE DID INSTEADDecided using a Puppetmaster was trying to fit a squarepeg into a round holeInstead, decided to run Puppet without a master
![Page 23: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/23.jpg)
APPLYING LOCAL PUPPETMANIFESTS
puppet apply --modulepath=/etc/puppet/modules example.pp
![Page 24: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/24.jpg)
DISTRIBUTING MANIFESTSUse RPMDistribute full set of manifests/modules to eachmachineApply only the manifest relevant to that machine
![Page 25: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/25.jpg)
PACKING PUPPETMANIFESTS IN RPM
Build an RPM containing everything under /etc/puppetMake files readable only by root
![Page 26: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/26.jpg)
APPLY PUPPET MANIFESTSHave an RPM %postinst command apply the Puppetconfig
This isn't as straightforward as running the puppetapply from %postinstPuppet needs to install packages via yum, but yum isrunning installing the Puppet packageInstead, we work around with a dirty hack: have the%postinst create an at script which checks if yumhas finished and then runs the puppet apply
![Page 27: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/27.jpg)
RPM INSTALLATION ANDMANAGEMENT
How do we get these RPMs on our machines?
![Page 28: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/28.jpg)
PULPWe were already using PulpProvides yum repository managementUsed for managing security updates and deployingapplication code
http://pulpproject.org/
![Page 29: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/29.jpg)
WHAT IS PULPRepository managerAllows us to easily audit what packages and versionsare installed whereAllows us to push package installations
Uses qpid message queue
Has concept of 'content distrubtion servers' for easyreplication and clustering
![Page 30: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/30.jpg)
HOW WE USE PULPPuppet contains details of what packages should beinstalledPulp manages which version of the package should beinstalledPulp allows us to clone repos and copy packagesbetween them for easy qa->stage->live environmentmanagement
![Page 31: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/31.jpg)
DEPLOYINGCONFIGURATION AS CODEAllows us to reuse our existing code deploymentinfrastructureManage configuration deployment from Jenkins
![Page 32: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/32.jpg)
HOW WE DEPLOY CODEEverything managed via the Jenkins continuousintegration serverJenkins uses Pulp to install code on remote machines
![Page 33: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/33.jpg)
DETAILS ON HOW WEDEPLOY CODE
Jenkins fetches code from source control (git)An RPM is builtTests are runIf tests pass, the RPM is added to the relevant Pulprepository RPM installed on the target machine(s)
![Page 34: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/34.jpg)
DEPLOYMENT LIFE-CYCLEJenkins also manages deployment life-cycleRPMs are installed on stagingPromoted Builds plugin then used to install the sameRPMs on live once testing is complete
![Page 35: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/35.jpg)
PUPPET DEPLOYMENTPROCESS
Puppet manifests are checked into gitLint tests via Jenkins pulls in modules with librarian-puppet, thenbuilds an RPMDeployment to test environments, functional tests forwider code-base run
Jenkins Warnings plugin
![Page 36: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/36.jpg)
PUTTING IT INTOPRODUCTION
Once suitable tests (automated and manual) have beencarried out, we promote Puppet config into productionWe use the Jenkins 'Promoted Builds' plugin for this
![Page 37: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/37.jpg)
JENKINS: PROMOTION
![Page 38: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/38.jpg)
EXCEPT..How does a machine get from a bare image to the statewhere we can push packages to it from Pulp?How does a machine know what type of machine it is?How do we find other resources, eg databasehostname?
![Page 39: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/39.jpg)
CLOUDFORMATIONAmazon tool for specifying infrastructureEverything* we provision inside AWS is provisioned viaCloudFormationJSON templates
* Everything except for the things Amazon doesn't exposevia CloudFormation..
![Page 40: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/40.jpg)
CLOUD-INITWorks with multiple cloud typesSorts out things like SSH keys, allows us to configurehost namesAlso allows us to provide a bash script to run on startup
![Page 41: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/41.jpg)
PROVISIONING A BAREINSTANCE
cloud-init automatically manually adds the pulp repowhich contains Pulp, Puppet and our Puppetmanifests/modulesInstalls appropriate RPMsPuppet runs, subscribing the machine to the relevantPulp repos, and installing packages in the usual Puppetway
![Page 42: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/42.jpg)
HOW DOES IT KNOW WHATTYPE OF MACHINE IT IS?
We tell it!Use an environmental variable $HOSTTYPESimply run
puppet apply \--modulepath=/etc/puppet/modules ${HOSTTYPE}.pp
![Page 43: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/43.jpg)
EXTRA FACTSCustom facter factsAlso specified in an environmental variable
Data comes from within the CloudFormation templateOn our list of things to look at:
FACTER_HOSTENVIRONMENT=liveFACTER_STACKNAME=customer-web-live
https://github.com/fanduel/hiera-cloudformation
![Page 44: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/44.jpg)
OTHER RESOURCESWe either:
Provide details as a facter factF̀ACTER_DBHOST=xyz
Also use this approach to limit distribution ofsecure details, eg DB passwords
Discover via the EC2 APIEg Varnish servers discover web backends bycalling API and finding hosts tagged appropriately
![Page 45: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/45.jpg)
FREE WINS!
![Page 46: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/46.jpg)
FREE WINS!Greater control over the timing of Puppet runsImproved visibility - for ops and devsConfiguration changes now have to be deployed totesting/staging first
![Page 47: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/47.jpg)
MORE FREE WINS!Puppet configs now have a versionEasy to find config version on the machine itselfConfig changelogs accessible on every machine
(Git changelog added to RPM)
![Page 48: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/48.jpg)
THE DOWNSIDESPuppet manifests and modules on all machines
Potentially a security issue?Mitigated by CloudFormation holding most sensitivedata
![Page 49: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/49.jpg)
ALTERNATIVEIMPLEMENTATIONS
Don't want to use Pulp?Could do basically the same thing with yum s3 plugin
Use mcollective to push package updateshttps://github.com/jbraeuer/yum-s3-plugin
![Page 50: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/50.jpg)
FUTURE IMPROVEMENTSBuild AMIs using Packer instead of configuring at boottime
Decrease time to autoscaleWould probably still need to run Puppet at first bootto configure machine specific settings
![Page 51: Continuously-Integrated Puppet in a Dynamic Environment](https://reader033.vdocument.in/reader033/viewer/2022060107/554bb764b4c9053a298b4a10/html5/thumbnails/51.jpg)
QUESTIONS? COMMENTS?Sam Bashton
Twitter: @bashtoni
(Psst.. )http://www.bashton.com/jobs/