puppet camp sydney feb 2014 - a build engineering team’s journey of infrastructure as code

Monday, 10 February 14

@peterleschev

Husband, Father of 3 & Atlassian

Build Engineering Team Lead

Peter Leschev


A Build Engineering Team’s Journey of

Infrastructure as Code


• Build platform & services used internally within the company• 60k builds per month• 35k automated tests for JIRA

Build Engineering today @ Atlassian


• 600 build agents (own hardware + EC2 instances)• include SCM clients, JDKs, JVM build tools, databases, headless

browser testing, python builds, NodeJS, installers & more

• Maintain 20 AMIs of various build configurations• 6 Bamboo Servers• maven.atlassian.com / 6 Nexus instances • Monitoring - opsview / graphite / statsd

Build Engineering today @ Atlassian


Infrastructure as Code

= Puppet + SCM ?


• Manually maintained snowflakes• Started using puppet

3 years ago...


Production rollout

puppetmaster

build agents


Production rollout failure

puppetmaster

build agents


HIGH

NONE

Lifecycle of an infra change

Confidence of Change

Dev Rollout Soak in Prod


http://atlassian.com/git

https://bitbucket.org/




https://bitbucket.org

https://bitbucket.org

Style in Pull Requests


• Automated style checking• Setup automated build that runs checks & posts results• Still need to implement a ratchet build

Puppet Lint https://github.com/rodjek/puppet-lintTim Sharpe

@rodjek


https://github.com/rodjek/puppet-lint

https://github.com/rodjek/puppet-lint

HIGH

NONE



Dev Code review Rollout Soak in Prod

initial + Code review


• Coding on Puppet Master• Culture of manually modifying

production - Configuration Drift

• Impact on Builds

Using Staging for Development

puppetmaster

build agentsbuild agents

staging puppet environment


• Easily spin up Infrastructure locally on your laptop• Disposable / reproducible environments• Machine provisioning via Virtual Box / VMWare / AWS• Configuration applied via Shell Scripts / Puppet / Chef• Develop and test infrastructure changes locally

Vagrant http://www.vagrantup.com/Mitchell Hashimoto

@mitchellh


http://www.vagrantup.com/


Vagrant

Vagrantfile

vagrant basebox

http://www.vagrantup.com/Mitchell Hashimoto

@mitchellh




Vagrant

Spins up a local VM to a known state

Destroy the VM when done

Make some puppet changes and then run:

to apply your changes

SSH into your VM using:

to check your changes

http://www.vagrantup.com/Mitchell Hashimoto

@mitchellh




HIGH

NONE




initial + Code review + Vagrant


• Vagrant basebox differences with production machines• Originally using publicly available vagrant baseboxes

• Installed packages biggest differences

• Generating a basebox manually was a painful process

Vagrant != Production


VeeweeAutomated

Vagrant basebox generationhttps://github.com/jedi4ever/veewee

Patrick Debois@patrickdebois

Ubuntu installation iso vagrant baseboxVeewee definitions.rbpreseed.cfgpostinstall.sh

+


https://github.com/jedi4ever/veewee


Veeweehttps://github.com/jedi4ever/veewee

AutomatedVagrant basebox generation

Patrick Debois@patrickdebois




• Latest basebox generated in CI & published to fileshare• No need to generate baseboxes locally

Basebox generation via CI


• VirtualBox Guest additions• Reduced to a minimal

There are still differences!


Common Preseed / Postinstall

preseed.cfg postinstall.sh

+

custom ISOsvagrant basebox PXEBoot


Packer http://packer.ioMitchell Hashimoto

@mitchellh


http://packer.io

http://packer.io

HIGH

NONE


Confidence in Change


initial + Code review + Vagrant + Veewee


Developing locally

Rolling out to production

Broken build agents!

Rolling out to staging


• Behaviour Driven Development

Cucumber


Cucumber & Vagrant

Vagrant

Custom Provisioner

Virtual Box

VM

puppet apply

cucumber *.features

via ssh


• Requires cucumber dependencies to be installed on tested VM

• Tests run within the VM making testing firewall rules harder

Disadvantages


HIGH

NONE




initial + Code review + Vagrant + Veewee + Cukes


But it works on my machine!– Every Developer”“


• ‘From scratch’ provisioning• Confidence that you can rebuild in disaster

Continuous Integration


The Pets: you give nice names,

you stroke them, and when they get ill,

you nurse them back to health,

taking a long time over it

– Tim Bell, CERN”

“

The Cattle: you give them numbers.

When they get ill, you shoot them


HIGH

NONE



Dev Code review CI & Rollout Soak in Prod

initial + Code review + Vagrant + Veewee + Cukes + CI


Provisioning from scratch is slow


Spread out CI

provision VM1

provision VM2

provision VM3

provision VM4

provision VM1

provision VM2 provision VM3

provision VM4Moved from sequentialto parallel provisioning


There are so many MacPros you can steal


The onesI have my eye on....


Profiling Puppet Runs

Add “--evaltrace” to puppet apply

+ =Collect and show the longest occurrences of:“Evaluated in ([\d\.]+) seconds”


Profiling Cucumber runs

http://itshouldbeuseful.wordpress.com/2010/11/10/find-your-slowest-running-cucumber-features/




• Provision locally & for CI• Faster & different class of problems found• Matches production state

Delta Provisioning

‘from scratch’ provision delta provision

provision VM1

export VM1 fileshare

import VM1 box

provision VM1

on success


HIGH

NONE




initial + Code review + Vagrant + Veewee + Cukes+ CI + Delta CI


Infrequent Releases


• Puppet runs impacted running builds• Disabling all the build agents

• Performing the roll out

• git clone / librarian-puppet / symlink update on puppetmaster

• Manually kick off puppet on all the build agents

• Enabling all the build agents

• Set of Puppet environments for every bamboo server

Painful Puppet Rollouts


Graceful Service restarts

+Bamboo Agent JVM process watches for touch file & shutdowns when Idle(written as a Bamboo Plugin)


• BEFORE - Multiple puppet envs for each Bamboo Server• jbac_staging

• jbac_production

• cbac_staging

• cbac_production

• etc

• AFTER - Changed to use ‘staging’ & ‘production’ only

Puppet Environments


• BEFORE: Manually on puppetmaster• git clone the puppet tree

• run librarian-puppet to pull external modules

• Update staging / production symlink

• AFTER: Bamboo build which performs the above steps automatically

Updates on Puppetmaster


Less Human interaction +

More automation=

Higher Confidence


Less Human Effort =

Increased frequency of releases


HIGH

NONE




initial + Code review + Vagrant + Veewee+ Cukes + CI + Delta CI + Frequent releases


Should I be scared?– Peter Leschev, 3 months ago”“

I’m scared!– Peter Leschev, 3 years ago”“


Hipchat integration


HIGH

NONE




initial + Code review + Vagrant + Veewee+ Cukes + CI + Delta CI + Frequent releases+ Notification


HIGH

NONE




before after



or

Finding & fixing problems sooner rather

than later


Commit Graph


Snowflakes

Pets

Cattle

Stateless Machines


We’re still on the Journey

Come join us!

atlassian.com/jobs


Questions?


Thank you!


puppet camp sydney feb 2014 - a build engineering team’s journey of infrastructure as code

Technology

condence of change initial

setup automated build

production broken build

cfgvagrant basebox

shvagrant basebox

available vagrant baseboxes

puppet changes

scratch provisioning