continuously testing infrastructure - beyond module testing - puppetconf 2014

Continuously Testing Infrastructure

Puppet Conf, San Francisco, 2014

Gareth Rushgrove

Beyond Module Testing

@garethr

Gareth Rushgrove

Gareth Rushgrove

Gareth Rushgrove

Not talking about

Finished software

Gareth Rushgrove

Testing individual modules

Gareth Rushgrove

puppet-lint, puppet-syntax, rspec-puppet, beaker

Gareth Rushgrove

Gareth Rushgrove

Am talking about

Experiments

Gareth Rushgrove

Testing images and containers

Gareth Rushgrove

Test driving infrastructure as a service

Gareth Rushgrove

Testing with PuppetDB

Gareth Rushgrove

Testing images and containers

1

Gareth Rushgrove

Packer builds images based on a JSON template

Gareth Rushgrove

Gareth Rushgrove

It has some Puppet integration too

Gareth Rushgrove

Gareth Rushgrove

But how do we know the image works?

Gareth Rushgrove

Lets add some tests!

Gareth Rushgrove

Gareth Rushgrove

shaunduncan/packer-provisioner-host-command

Gareth Rushgrove

serverspec.org

Gareth Rushgrove

Gareth Rushgrove

Gareth Rushgrove

Gareth Rushgrove

Serverspec also supports port, file, ppa, selinux, user, group, lxc, iptables, cron and more

Gareth Rushgrove

Only publish the image if the tests pass

Gareth Rushgrove

Run tests automatically with a continuous integration system

Gareth Rushgrove

Gareth Rushgrove

Gareth Rushgrove

garethr/packer-serverspec-example

Gareth Rushgrove

Gareth Rushgrove

Same approach works with containers too

Gareth Rushgrove

Gareth Rushgrove

garethr/docker-spec-example

Gareth Rushgrove

Test drive your IaaS

2

Test driven development

Gareth Rushgrove

First the developer writes an automated test case that defines a desired improvement or new function

Gareth Rushgrove

Then produces the minimum amount of code to pass that test

Gareth Rushgrove

And finally refactors the new code

Gareth Rushgrove

Gareth Rushgrove

First the developer writes an automated test case that defines a desired improvement or new function

Your infrastructure should!have an API

Gareth Rushgrove

What if we write assertions against!that API?

Gareth Rushgrove

Aside: Clojure

2.1

Gareth Rushgrove

Great for building DSLs

Gareth Rushgrove

Don’t worry, you could write the examples in any language

Gareth Rushgrove

Policy driven development

Gareth Rushgrove

I don’t want to launch too many nodes, they’re expensive

Gareth Rushgrove

Policy

Gareth Rushgrove

I don’t want any stopped nodes, they are costing me money

Gareth Rushgrove

Policy

Gareth Rushgrove

Large nodes are really expensive, so limit their usage

Gareth Rushgrove

Policy

Gareth Rushgrove

We should be backing up every node

Gareth Rushgrove

Policy

Gareth Rushgrove

I only want nodes in London and !San Francisco

Gareth Rushgrove

Policy

Gareth Rushgrove

All our nodes should be named environment-name

Gareth Rushgrove

Policy

Gareth Rushgrove

garethr/digitalocean-expect

Gareth Rushgrove

Gareth Rushgrove

Now we have the tests, we can provision some infrastructure

Gareth Rushgrove

Aside: Provisioning with Puppet

2.2

Gareth Rushgrove

Gareth Rushgrove

puppetlabs/gce_compute

Gareth Rushgrove

Gareth Rushgrove

Gareth Rushgrove

garethr/digitalocean

Gareth Rushgrove

Gareth Rushgrove

bobtfish/aws_api

Gareth Rushgrove

Testing with PuppetDB

3

Aside: PuppetDB

3.1

puppetlabs/puppetdb

Gareth Rushgrove

PuppetDB can store a lot of data about your infrastructure

Gareth Rushgrove

The most recent facts from every node

Gareth Rushgrove

The most recent catalog for every node

Gareth Rushgrove

A wide range of metrics

Gareth Rushgrove

Gareth Rushgrove

I want to run the same operating system on all hosts

Gareth Rushgrove

Policy

Gareth Rushgrove

Security enforcing packages should be installed everywhere

Gareth Rushgrove

Policy

Gareth Rushgrove

I want to limit how many puppet resources I’m using

Gareth Rushgrove

Policy

Gareth Rushgrove

We should avoid heavy I/O load on the database by maintaining a high catalog duplication rate

Gareth Rushgrove

Policy

Gareth Rushgrove

garethr/puppetdb-expect

Gareth Rushgrove

Testing based on PuppetDB

3.2

PuppetDB is a great source of context for tests

Gareth Rushgrove

Generate serverspec tests from PuppetDB data

Gareth Rushgrove

Automatically detect hosts, and generate commands

Gareth Rushgrove

Gareth Rushgrove

Match puppet resources to serverspec resources

Gareth Rushgrove

Gareth Rushgrove

For instance on a Puppet Enterprise master

Gareth Rushgrove

Gareth Rushgrove

Run serverspec tests on all puppet managed hosts

Gareth Rushgrove

Gareth Rushgrove

garethr/serverspec-puppetdb

Gareth Rushgrove

Conclusions

Is this monitoring?

Gareth Rushgrove

We’re still moving towards infrastructure as code

Gareth Rushgrove

Infrastructure as code rather than infrastructure from code

Gareth Rushgrove

Taking about policy as code might help communicate intent

Gareth Rushgrove

Questions?And thanks for listening