islands: puppet at bulletproof networks
DESCRIPTION
Bulletproof Networks provides managed hosting services to some of the largest companies in Australia. Bulletproof implements strong isolation of customer environments, and this can present unique challenges when re-using Puppet code across our customer base. Additionally, the environments range in size from small to very large, and our tools + processes need to be able to handle both uses cases equally well.In this talk Lindsay + Mick will cover how Bulletproof's approach to these problems has evolved over the last 4 years, and some of the tools Bulletproof has developed and built upon to provide an awesome service to our customers.TRANSCRIPT
Islands
Who are these guys?
Mick Pollard@aussielunix
&
Lindsay Holmwood@auxesis
Puppet users for > 5 years
BULLETPROOF
IaaS &Managed Services
Movember Australian Museum Rebel Group
Blackmores Angus & Robertson Telstra
Perisher BlueScope Steel Woolworths
DMG Radio Clive Peters Deloitte
Clemenger budget.gov.au Nissan
AOC Nova Sydney Airports
Whirlpool Smooth Theiss
Borders Fosters Country Road
Midas Australian Geographic Sensis
Tourism Victoria ABS AusPost
DET FWA Vodafone
Using Puppet since 2008
Unique challenges
Strong isolation
Network segregationwith VLANs
Central Puppetmasterisnʼt an option
Thorough change control
Rapid growth
How do we use Puppet?
Standalone systems
(puppetmaster-less)
Internal infrastructure
Full customer environments
Standalone systems
(puppetmaster-less)
Campaign driven
business
budget.gov.au
movember.com
mamamia.com.au
Reverseproxies
Nginxwith customisation
Rump
More detail in
John Ferlitoʼstalk at 14.00
Internal infrastructure
Vanilla
One Puppetmaster
Ubuntu(Lucid or Precise)
Full customer environments
Every customer has their own puppetmaster
“Islands of Puppet”
Copypasta
Configuration drift
apache
apache
apache
customer-a
apache
customer-a
apache
customer-a
customer-b
apache
customer-a
customer-b
apache
customer-a
customer-b
customer-c
apache
customer-a
customer-b
customer-c
apache
customer-a
customer-b
customer-c
customer-d
Poor code share
What if customers
edit the code?
How dowe maintain
common code?
Commonalities
Mix of Puppet
versions
0.25 (as provided by
Ubuntu)
2.7 (as provided by Puppet Labs)
Mix of Operating Systems
lucid precise
2.7 internal infrastructure some customers
0.25 most customers
Passenger > webrick
--no-daemonize
Default behaviour is orthogonal to
change control
We don't want systems to change
without control
All changes initiated byan engineer
nodes + roles
node 'stlyqy-lvs02.cust.bulletproof.net' { server { $fqdn: }
include snmp::server::lvs include sysctl::lvs
include keepalive::lvs
include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan
include ript}
node 'stlyqy-lvs02.cust.bulletproof.net' { server { $fqdn: }
include snmp::server::lvs include sysctl::lvs
include keepalive::lvs
include network::conntrack::modules include network::conntrack::hashsize include network::bonding::activebackup include network::type::bonded_vlan
include ript}
define server($collectd_client_report_to='collectd.bulletproof.net') { include motd include augeas include apt include utils include puppet::client
include ssh::server include ssh::authorized_keys include ntp::client include postfix::satellite
include ruby::dev include ruby::rubygems
include bzr::client include git::common include git::github
include snmp::server include vmware::tools
include apparmor::disable collectd::client { "${fqdn}": report_to => $collectd_client_report_to }}
app_serverdatabase_serverfile_servermanagement_servermemcache_servermonitor_serverproxy_serverpuppetmaster_serverredis_serversingle_serversphinx_serverstatic_server
Heira is the future
We useCapistrano
ssh in-a-parallel-for-loop
Why cap and not mcollective?
We deploy everything with cap
Monitoring configurationFirewall configuration
Web applicationsInternal tools
Consistent deployment tool across all projects
Principle of least surprise
Engineers learn1 tool
Puppet is no different to the
rest of our stack
How do we usecap + Puppet?
Puppet changes
cap puppet:go ROLES=lvs options="--noop"cap puppet:go ROLES=lvs
Smoke tests
$ cap puppet:go options="--noop"
# ...
infmon hosts serves a Nagios page over https
Finished in 1.67 seconds1 example, 0 failures
Works out all roles that hosts within a
run belong to
Runs tests tagged with those roles
Fast feedback on
change success
Bootstrapping
cap node:bootstrap HOSTFILTER=lvs-08.bp.net
Takes VM in unknown state
Brings into known state for Puppet 2.7 run
Limitations
Singling out hosts is tricky
Re-using data across commands
requires... creativity
servers = []
run "mysql -e \"SHOW MASTER STATUS;\" | tail -n 1" do |channel, type, data| hostname = channel[:host] filename = data.split(/\s+/).first position = data.split(/\s+/).last servers << { :hostname => hostname, :filename => filename, :position => position }end
ROLESand
HOSTFILTER
TLDR;
There are edge cases
It does the job
How have we tried to solve
them?
First iteration
Modules
modules/apache/
| manifests/init.pp
| files/
| templates/
| lib/
| README.markdown
Stored on GitHub
Drink from the firehose
puppet-module-tool
github.compuppetlabs/puppet-module-tool
gem install puppet-module
Modulefile
puppet-module build
Turns
modules/apache/
| manifests/init.pp
| files/
| templates/
| lib/
| README.markdown
into
bulletproofnetworks-apache-1.3.0.tar.gz
Puppet forge
Public by default
Not great if you don't want to open source all your secret sauce
But puppet-module-tool is interesting...
...can we fake the forge?
Pain points
Arduous release
workflow
Better suited for infrequent
changes
High barrier of entry for customers to submit patches
Sharing bugfixes & improvements
requires significant refactoring
Limited reporting on customer lag
Second iteration
Bundler
Gemfile
#!/usr/bin/env ruby
source :rubygems
gem 'capistrano', '2.9.0'gem 'capistrano-ext', '1.2.1'gem 'colorize'gem 'puppet', '2.7.13'gem 'puppet-module'
group :test do gem 'rspec', '2.8.0' gem 'mechanize' gem 'puppet-lint'end
Can we reuse it?
Tim Sharpe@ GitHub
(@rodjek)
Messy working prototype
pre-alpha quality
More research required
Librarian
“A framework for bundlers”
librarian-chefmanages Chef repositories
by Jay Feldblumgithub.com
applicationsonline/librarian
librarian-puppet
“You can all stop using git submodules now”
gem install librarian-puppet --pre
github.comrodjek/librarian-puppet
Bundler-like behavior
for Puppet
Puppetfile
#!/usr/bin/env ruby
forge "http://forge.puppetlabs.com"
mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"
#!/usr/bin/env ruby
forge "http://forge.puppetlabs.com"
mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"
mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"
#!/usr/bin/env ruby
forge "http://forge.puppetlabs.com"
mod "puppetlabs/razor"mod "puppetlabs/ntp", "0.0.3"
mod "apt", :git => "git://github.com/puppetlabs/puppetlabs-apt.git" :ref => 'feature/master/dans_refactor'
mod "stdlib", :git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"
One canonical module
modules/ in
.gitignore
librarian-puppet outdated
tells you what needs to be updated
librarian-puppet update
updates your modules
Demo!http://aussielunix.github.com/jenkins-appliance/
Policy
Always use a module'smaster
Make changes to modules as
usual
git commit &&
git push the module
Make module changes generic
by default
Only branch a module when:
1. something is superclient specific
2. there is an unmissable deadline
3. testing ideas
Set a reminder to merge changes into
master
Use pull requests on GitHub for
dangerous changes
Take aways:
Puppet modulesare pretty neat
Keep feedback loops short
Code shareis king
Do you have similar
problems?
How do you solve them?
Thank you!