story of puppet @ebay global classifieds group (ecg)
DESCRIPTION
"Story of Puppet @eBay Global Classifieds Group (eCG)" by Vladimir Lazarenko of eBay at Puppet Camp Amsterdam 2013.TRANSCRIPT
Story of Puppet @eBay Classifieds Group or how we came to the setup we use
Some bio or “Who is Vladimir” ! Sysadmin for eighteen years
! Currently in Solutions team of eCG’s marktplaats.nl
! All-round nerd
! Hobbyist programmer
Puppet at eBay Classifieds: ! 9 countries
! 13 platforms
! 1 codebase
! 1 change management process
Puppet at eBay Classifieds: ! We started adopting puppet in 2008
! Since then we had about 4 different iterations of “this is bloated and doesn’t work, let’s think more and re-do”
! Currently we believe we achieved a workable scenario where we: ! Enable teams to share and re-use puppet code
! Enforce the same code style and standards
! Allow for peer-reviews of the code, also enforce them
“If it’s not in Puppet – it does not exist.”
What we are going to talk about: • Different iterations of puppet implementation at eCG
• Go in-depth on the structure of the current one
• Process around the codebase
• Tools
“Bucket of recipes”
Puppet at eBay Classifieds (1st implementation): ! A bunch of modules without visible structure (server-role based)
! Relations between modules on any levels (includes, instantiations)
! One Subversion repository, where everyone works on trunk
What were the lessons learned?
ü As puppet adoption grew, modules for different platforms started clashing (same, yet a bit different functionality)
ü As number of modules and their dependencies grew people were afraid of making changes - dependency tree was very hard to follow
ü Resulted in tons of duplicate code
ü Change management was very risky – you commit something and it’s live
“Buckets of recipes”
Puppet at eBay Classifieds (2nd implementation): ! What did we improve?
! Release management with Subversion branching
! Better module grouping
What were the lessons learned?
ü Waiting on releases is killing. If you need a change - you need it now, not in 2 or 3 days
ü Provokes people to lock machines to their own puppet environment
ü Surprises at times of the releases. For some people it was daytime, for others it was middle of the night
Puppet at eBay Classifieds (3rd implementation): ! What did we improve?
! Switched to Git
! Completely redesigned modules structure (more on this later)
! Introduced extlookup for data separation vs configuration logic
! Release more often
What were the lessons learned?
ü When release process is human-dependent it will be a bottleneck
ü Extlookup is nice, but in many places unreadable
Puppet at eBay Classifieds (4th implementation): ! What did we improve?
! Introduced Gerrit for code reviews
! Implemented automatic puppet-lint, template syntax and PEP8 checkers
! Introduced automated quorum votes for peer reviews
! Release as soon as your code passed checks and YOU need it live
! Further improved module tree structure
! Replaced extlookup with hiera
! Introduced puppetdb
So how does it look now?
Shared
eCG
Platform
Config
! Four module levels
! One include direction
! No includes on the same level
Shared modules
Shared
eCG
Platform
Config! Shared modules are very basic, deploy one piece
of software in its standard configuration (no customization)
! Standalone. Includes no other modules
! Installs needed packages, basically
! Parameterized to allow configuration override ! Examples: apache, ntpd, activemq, mysql, postgres
eCG modules
Shared
eCG
Platform
Config! eCG modules add our specific eCG-wide
approved customizations. Rule: it must play well with majority of teams to make it
! Mostly include shared modules
! Never include each other
! Override default shared module configuration with eCG specific one via parameters
! May be parameterized to allow for configuration tweaks or overrides ! Examples: puppetmaster, nagios checks, collectd
plugins
Platform modules
Shared
eCG
Platform
Config
! Platform level introduces company-specific modules and configuration customizations
! May include eCG modules
! May include shared modules
! Never include each other
Config modules
Shared
eCG
Platform
Config
! Config level introduces server-role specific modules
! Are the starting point for our servers configuration
! May include platform, eCG or shared modules
! Never include each other
! Node definition must include config module
Why did we do it like this?
Shared
eCG
Platform
Config
! Easy starting point. Look up a node, see its class, open the class – you see on one page how your server is configured on a global level
! If you need to know more, dive deeper
! We enable code sharing, we do not enforce it ! If you see a module in shared you can use – use
it. If you can’t use it, extend it in a backwards-compatible way or write other extension on an appropriate level
Hiera
! Great way to separate code from data
! Company-based structure
! Lookup path based on facts to allow company data separation while keeping the same source tree :datadir: /var/lib/hiera/%{environment}/%{companyprefix}
! Inside company dirs a variety of fact-based lookups to allow for fine-grained data structures
PuppetDB
! Much faster catalog compilations for huge catalogs (nagios server with 200k resources)
! REST API to talk to for inventory and searches
! Many brilliant ways to (ab)use it ! Stores all facts (can be used to feed CMDBs)
! Stores most recent catalog (can be used for inspections)
! POC we are running - internal Puppet naginator module is very slow, collection of exported resources takes about 15 minutes. We use PuppetDB to generate the same nagios configs in just under 8 seconds
! “If there is data, it can be used”. PuppetDB is a central data store. Love.
So all good, but how does this work in practice?
! Per-user environments on puppetmasters
! Make the change in your own env
! Test it from your own env
! Push to Gerrit
! Ask team members to review
! Merge
Code reviewPush to GerritTest the change with --environment
Local environment Merge
Gerrit
! Great code review tool (https://code.google.com/p/gerrit/)
! Acts as a Git server with fine-grained access control
! Each commit is a separate change, that requires votes for approvals
! Votes can be +1 for Code Review, +2 for change approval
! No +2 – no merge
! We use voting system with auto +2 ! Depending on platform size or number of platforms affected by the
change more votes are needed to get a change through
! Each change can be “pushed through” in case of an emergency
More Gerrit goodies
! Allows for post-push hooks ! Puppet-lint
! Ruby templates parsing
! PEP8 (we love Python)
! Each change is separately checked out by hooks and tested for syntax or other obvious errors
! Concept of one commit – one change forces people to create atomic changes that are easy to revert in case of errors
! Keeping it all in the same system, even if you decide to split Git repositories still makes work done by colleagues VISIBLE
Gerrit and puppetmasters
! Master is automatically pulled by crons on puppetmasters
! Gerrit disallows pushing directly to master
! Even if you force-merge a change, it makes your change VISIBLE and easily REVERTIBLE
! Takes about half a minute extra time
! Ensures you didn’t slack out on syntax checkers
To sum it all up
! There is no single structure that fits every company
! Puppet code is like any other code ! Write
! Use
! Refactor if needed
! Layered modules structure can work well in many cases
! Complementary tooling around Puppet is a life- (and time-) saver ! Lint(s), Gerrit, PuppetDB
Work to be done in our house
! Approval process improvements ! Better quorum schemas
! Easier ways of code style and syntax checking
! See that people don’t leave changes “hanging”
! Integrate Gerrit with corporate GitHub
! Work more with community ! Open source some generic shared modules
! Offer changes to PuppetForge modules
! POC with Vagrant for puppet code testing
Questions?
Thank you!
You can reach me by: • Mailing to [email protected] • Following me on GitHub (
https://github.com/favoretti)
Special thanks to:
• PuppetLabs and eCG for the opportunity to speak here • Teams of eCG for bearing with me all these years • You all for attending and listening
We are hiring! Visit http://ebayclassifiedsgroup.com/careers for more info.