cooking perl with chef: real world tutorial with jitterbug

Cooking Perl with ChefReal World Tutorial with Jitterbug

Copyright © 2012 by David A. GoldenThis work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License

Tutorial Version 1.0http://perlchef.com/

Abstract

This tutorial provides a command-by-command walk-through for deploying the Jitterbug continuous integration server using the Chef configuration management tool.

Prerequisites

This tutorial was written with Unix operating systems in mind and was tested using Ubuntu Linux 12.04 LTS. You will need to be familiar with Perl, CPAN, git and common Unix tools like ssh and rsync. You will need an Internet connection to download files during this tutorial.

The tutorial is written using a Linode virtual machine for deployment. You will need a Linode account or else must have a real/virtual machine available to which you can deploy a fresh operating system.

You will also need to install the Pantry tool, which is available on CPAN. Install it using your preferred CPAN client.

Note that you won't need Chef installed on your own machine. You only need it on the virtual machine you are deploying to and this bootstrap will be covered in Step 1 of the tutorial.

I assume that you have already seen the "Cooking Perl with Chef" overview presentation and "Hello World" tutorial before you attempt this one, so I will not explain some Chef fundamentals again here. If you have not seen these, please follow the links on http://perlchef.com/ before continuing.

Overview

Our goal is to deploy the Jitterbug continuous integration server (http://lumberjaph.net/jitterbug/) to a small Linode virtual machine using Chef, Pantry and the Perl Chef cookbooks. This Jitterbug server will be ready for testing Perl/CPAN modules using Github's webhook integration.

This tutorial is broken up into five distinct steps:

• Provision Linode, deploy SSH keys and install Chef• Set up Pantry and third-party cookbooks• Adapt Jitterbug for Chef and Carton• Specify the configuration for the server• Deploy and test

http://creativecommons.org/licenses/by-sa/3.0/

http://lumberjaph.net/jitterbug/

http://perlchef.com/

Cooking Perl with Chef: Real World Tutorial with Jitterbug

We will deploy these components as our "stack":

• Nginx web server• Jitterbug web application (behind the nginx proxy)• Jitterbug builder (the task worker process)• Postfix mail server (to send failure reports)• Perl 5.14.2

We will also configure a firewall to restrict everything except ports 22 and 80.

You can contact the author with praise, critique or questions at [email protected] or as @xdg on various social networks.

Step 1. Provision a Linode, deploy SSH keys and install Chef

This tutorial uses Linode (http://linode.com/). If you don't already have a Linode account you should create one now. If you don't want to use Linode, you should be able to adapt this provisioning step for another virtual machine provider you prefer.

Add a linode 512 on a month-to-month (MTM) plan. (Linode will pro-rate you a refund when delete the linode, so you'll only pay for a bit of usage if you only plan to use it for this demo.) Use only 10,000 MB for the main drive and deploy Ubuntu 12.04LTS. Remember the root password you assign; you'll need that to deploy SSH keys (and then you can stop using the password or disable it). When that is complete, boot the linode.

If you manage your own domain somewhere, setup a DNS A record mapping a server name to the public IP address of the linode. Set linode's reverse DNS to map back to that hostname. If you don't set up your own DNS, you'll need to use the default linode-provided hostname instead.

NOTE: When you see "jitterbug.example.com" in the tutorial, use your own hostname instead!

On your own computer (not the linode), you need to create a directory to hold your configuration information. You should keep your configuration under version control, so this tutorial shows how to do that with git.

$ mkdir /tmp/jitterbug-config$ cd /tmp/jitterbug-config$ git init

Next, you'll want to create SSH keys you'll use to connect to the jitterbug server. You will be prompted for a passphrase for the private key, and it's a good idea to use one.

$ ssh-keygen -f jitterbug-key


Then, check the keys into git.

$ git add jitterbug-key*$ git commit -m "added jitterbug SSH keys"

Now that the keys have been created, deploy them to the server (using the root password).

$ ssh-copy-id -i jitterbug-key [email protected]

Add the key to your SSH agent and try it out.

$ ssh-add jitterbug-key$ ssh [email protected]

Once you're logged into remote machine, you'll need to install the Chef client on it. These steps are adapted from the Opscode Chef wiki:

$ echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" > \/etc/apt/sources.list.d/opscode.list

$ mkdir -p /etc/apt/trusted.gpg.d$ gpg --keyserver keys.gnupg.net --recv-keys 83EF826A$ gpg --export [email protected] > \

/etc/apt/trusted.gpg.d/opscode-keyring.gpg$ apt-get update$ apt-get upgrade -y$ apt-get install -y opscode-keyring$ DEBIAN_FRONTEND=noninteractive apt-get install -y chef$ /etc/init.d/chef-client stop$ update-rc.d -f chef-client remove$ chef-solo --version

The version you see should be at least 10.4. Note that the regular Chef client is disabled because we'll be using chef-solo instead.

Once Chef is installed, log out of the server.

$ exit

At this point, you should duplicate the disk drive and keep it as a base image with Chef already bootstrapped in case you need to start over for any reason (or for future projects. On the Linode dashboard, "edit" the drive and click "duplicate" to make a copy.

Step 2. Set up Pantry and third-party cookbooks

We use the Pantry tool to manage configuration and deploy with chef-solo. Start off by initializing the current directory for Pantry.

$ pantry init


Next, you need to download several cookbooks with Chef deployment recipes. Some of these will come from the Opscode community site, some will come from Opscode Github repositories, and some will come from the Perl Chef Github repository. Some will be used directly as we configure the Jitterbug server and others are dependencies.

Here is the list of cookbooks with an explanation of their purpose

• apt – ensures apt-get update is run before further configurations• carton – used to deploy the Jitterbug application• firewall – common firewall framework code• hostname – set the hostname• nginx – deploy and configure Nginx• ntp – deploy ntpd• ohai – common framework for Ohai plugins (used by Chef)• perlbrew – deploy Perl intepreters• postfix – deploy Postfix• runit – used by carton for persistent services• ufw – firewall configuration

I find it helpful to stage third-party cookbooks in a separate directory first before copying them to the Pantry cookbooks directory. This lets me keep the Pantry cookbooks directory under version control that is specific to my own configuration, while letting me browse third-party cookbook code repositories independently. (git submodules could be used for this, but that gets complex.)

Start by creating another directory for staging cookbooks.

$ mkdir /tmp/jitterbug-src$ cd /tmp/jitterbug-src

One handy trick is to get all the cookbooks in a similar directory structure, organized by source and under a cookbooks directory. Some repositories are like this already, others need to be cloned to specific locations. Afterwords, we can write a simple script to rsync them all to the right place in the Pantry directory.

First, we'll get all the Opscode cookbooks, including a custom version from my own repository with some bug fixes that haven't been merged upstream yet.

$ mkdir -p opscode/cookbooks$ cd opscode/cookbooks$ git clone git://github.com/opscode-cookbooks/apt.git$ git clone git://github.com/opscode-cookbooks/firewall.git$ git clone git://github.com/opscode-cookbooks/nginx.git$ git clone git://github.com/opscode-cookbooks/ntp.git$ git clone git://github.com/opscode-cookbooks/ohai.git$ git clone git://github.com/opscode-cookbooks/postfix.git$ git clone git://github.com/dagolden/runit.git -b CHEF-154$ git clone git://github.com/opscode-cookbooks/ufw.git$ cd ../..


The next cookbooks don't need extra subdirectories, because they already have a "cookbooks" directory in the repository. We'll actually download a Jitterbug cookbook now, as well, even though in Step 3 we'll walk through it as if we were creating it from scratch.

$ git clone git://github.com/dagolden/perl-chef.git$ git clone git://github.com/dagolden/jitterbug.git -b carton-chef

The hostname cookbook does not appear to have a repository, so we'll download a tarball instead. (Apologies for the smaller font, but I wanted to keep "curl ..." all on one line.)

$ mkdir -p community/cookbooks$ cd community/cookbooks$ curl -L http://community.opscode.com/cookbooks/hostname/versions/0_0_2/downloads | tar xz$ rm hostname.tgz$ cd ../..

Now that we have all the cookbooks we need, we need to copy them over to the Pantry cookbook directory. Create this little shell script to make it easy:

$ cat > copy-cookbooks.sh#!/bin/bashfor d in *; do if [[ -d $d ]]; then rsync -av --exclude=.git $d/cookbooks/ /tmp/jitterbug-config/cookbooks fidone

Run that script, then go to the Pantry directory and check everything in.

$ chmod +x copy-cookbooks.sh$ ./copy-cookbooks.sh$ cd /tmp/jitterbug-config$ git add cookbooks$ git commit -m "imported cookbooks"

Step 3. Adapt Jitterbug for Chef and Carton

In Step 2, we downloaded a cookbook for Jitterbug, but let's pretend it didn't exist and we had to create it. In this section, I'll describe how I did that. If you're not interested in learning how to make a cookbook, you can skip ahead to Step 4. (If you're really hard-core, you can delete the jitterbug cookbook you downloaded, and recreate it using these instructions.)

To create the Jitterbug cookbook, I started by forking the project on Github (git://github.com/franckcuny/jitterbug.git) and creating a new branch in my own repository:

$ cd ~/git$ git clone git://github.com/dagolden/jitterbug.git$ git checkout -b carton-chef


To adapt Jitterbug for Chef deployment with Carton, I need a carton.lock file to hold dependency information and a Chef cookbook. The cookbook needs these files:

• attributes/default.rb – configuration attributes• recipes/default.rb – a deployment recipe• files/default/jitterbug.db – an empty SQLite database with the Jitterbug schema• templates/default/config.yml.erb – Jitterbug's configuration file• templates/default/jitterbug.conf.erb – app-specific Nginx configuration file

If I were creating a cookbook to upload to Opscode's community site, I'd also need to write a metadata.rb file and a README.rdoc file, but I didn't do that for this tutorial.

Pantry has a command for creating a blank cookbook under the cookbooks directory. I could have run that in the Pantry directory, but in this case, I ran it in the Jitterbug branch so I could share it later.

$ pantry create cookbook jitterbug

That creates several directories under cookbooks/jitterbug and touches some empty files to fill in. For the files under the attributes and recipe directories, I then copied and adapted my Hello World tutorial recipe.1 For the Nginx configuration file, I created one based on how Fletcher Nichol wrote one for Jenkins.2 I still find cookbook creation to be a bit of a black-art, so this is a pretty common pattern for me. I find things similar to what I want to do, use that as a base, and tweak it to my needs. The other cookbook files, I had to create from scratch.

Creating the carton.lock file was straightforward. I used Perlbrew to install Perl 5.14.2 (which is what I plan to deploy with), activated it, and installed the Carton module from CPAN. Then creating the carton.lock file was just:

$ carton install

Next, I modified the attributes/default.rb file to contain the configuration attributes I need for the recipe and template files. Here's what it looks like:

# perlbrew to execute with (should be a legal perlbrew target)default['jitterbug']['perl_version'] = 'perl-5.14.2'

# Install directory, repo and tagdefault['jitterbug']['deploy_dir'] = '/opt/jitterbug'default['jitterbug']['deploy_repo'] = 'git://github.com/dagolden/jitterbug.git'default['jitterbug']['deploy_tag'] = 'carton-chef'

# Service user/group/portdefault['jitterbug']['user'] = "jitterbug"default['jitterbug']['group'] = "jitterbug"default['jitterbug']['port'] = 3000

# Jitterbug configdefault['jitterbug']['db_dir'] = "/var/lib/jitterbug"default['jitterbug']['conf_dir'] = "/etc/jitterbug"default['jitterbug']['on_failure_subject_prefix'] = "[jitterbug] FAIL "default['jitterbug']['on_failure_to_email'] = ""

1 https://github.com/dagolden/zzz-hello-world 2 https://github.com/fnichol/chef-jenkins/blob/master/recipes/proxy_nginx.rb

https://github.com/fnichol/chef-jenkins/blob/master/recipes/proxy_nginx.rb

https://github.com/dagolden/zzz-hello-world


default['jitterbug']['on_failure_cc_email'] = "[email protected]"default['jitterbug']['on_failure_from_email'] = "[email protected]"default['jitterbug']['on_pass_subject_prefix'] = "[jitterbug] PASS "default['jitterbug']['on_pass_to_email'] = ""default['jitterbug']['on_pass_cc_email'] = "[email protected]"default['jitterbug']['on_pass_from_email'] = "[email protected]"

I decided to deploy as the "jitterbug" user and group, so I'll need to remember to configure that user later in the deployment recipe.

After defining attributes, it was time to create the templates for configuration files. The Jitterbug repository already contains some sample configuration files, config.yml and example.yml, so I copied the example.yml file to cookbooks/jitterbug/templates/default/config.yml.erb, tweaked it, and replaced some of the sample configuration values with entries from the attributes/default.rb file. I only did a partial replacement to demonstrate it for the tutorial. In theory, every one of the config values could be parameterized. Here is the resulting file:

layout: "main"logger: "console"appname: "jitterbug"

builds_per_feed: 5template: "xslate"engines: xslate: path: - "<%= node['jitterbug']['deploy_dir'] %>" type: text cache: 0

jitterbug: reports: dir: /tmp/jitterbug build: dir: /tmp/build build_process: builder: ./scripts/perlchef-capsule.sh builder_variables: on_failure: jitterbug::Emailer on_failure_to_email: "<%= node['jitterbug']['on_failure_to_email'] %>" on_failure_cc_email: "<%= node['jitterbug']['on_failure_cc_email'] %>" on_failure_from_email: "<%= node['jitterbug']['on_failure_from_email'] %>" on_failure_subject_prefix: "<%= node['jitterbug']['on_failure_subject_prefix'] %>" on_failure_header: on_failure_footer: on_pass: jitterbug::Emailer on_pass_to_email: "<%= node['jitterbug']['on_pass_to_email'] %>" on_pass_cc_email: "<%= node['jitterbug']['on_pass_cc_email'] %>" on_pass_subject_prefix: "<%= node['jitterbug']['on_pass_subject_prefix'] %>" on_pass_from_email: "<%= node['jitterbug']['on_pass_from_email'] %>" on_pass_header: on_pass_footer: reuse_repo: 1 options: email_on_pass: 0

plugins: DBIC: schema: skip_automake: 1 pckg: "jitterbug::Schema" connect_info: - "dbi:SQLite:dbname=<%= node['jitterbug']['db_dir'] %>/jitterbug.db"


The Nginx configuration file, templates/default/jitterbug.conf.erb is also straightforward:

server { listen 80; server_name <%= node[:fqdn] %>;

location / { proxy_pass http://127.0.0.1:<%= node['jitterbug']['port'] %>; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; }

error_log <%= node[:nginx][:log_dir] %>/jitterbug-error.log; access_log <%= node[:nginx][:log_dir] %>/jitterbug-access.log;}

To create the empty SQLite database, I first ran Jitterbug's schema deployment tool, then moved the resulting database to cookbooks/jitterbug/files/default/jitterbug.db:

$ carton exec -I lib -- scripts/jitterbug_db --config config.yml --deploy$ mv jitterbug.db cookbooks/jitterbug/files/default/jitterbug.db

During deployment, this file will get deployed as the starting database, but only if it doesn't already exist. This is a naively simple way to deploy an SQLite database. A more sophisticated recipe might get the deployment database from a backup location to help with future disaster recovery deployment and we'd seed the backup location with the empty database for first deployment.

Next, I need the deployment recipe in recipes/default.rb to tie all these components together. Because it's long, I'll explain it it pieces, but you can see the whole thing in the location cloned during Step 2.

The first part of the recipe includes dependency cookboks, ensures that some required OS packages are installed, and creates a jitterbug user.

include_recipe 'carton'include_recipe 'perlbrew'include_recipe 'nginx'

package 'git-core'package 'libxml2-dev'package 'libexpat-dev'package 'zlib1g-dev'

user node['jitterbug']['user'] do home '/home/jitterbug'end

The next part of the recipe uses git to check out the jitterbug application from the repository. The destination, repostory and source tag are all attributes. We also tell Chef to notify the applications (defined later) to restart if anything has changed:

git node['jitterbug']['deploy_dir'] do repository node['jitterbug']['deploy_repo'] reference node['jitterbug']['deploy_tag'] notifies :restart, "carton_app[jitterbug]" notifies :restart, "carton_app[jitterbug-builder]";end


Next, we deploy the Nginx template:

template "#{node[:nginx][:dir]}/sites-available/jitterbug.conf" do source "jitterbug.conf.erb" owner 'root' group 'root' mode '0644'

if File.exists?("#{node[:nginx][:dir]}/sites-enabled/jitterbug.conf") notifies :restart, 'service[nginx]' endend

Then we deploy the Jitterbug configuration file into a specified directory:

directory node['jitterbug']['conf_dir'] do owner node['jitterbug']['user'] group node['jitterbug']['group']end

template "#{node['jitterbug']['conf_dir']}/config.yml" do source "config.yml.erb" owner node['jitterbug']['user'] group node['jitterbug']['group'] mode '0644' notifies :restart, "carton_app[jitterbug]"; notifies :restart, "carton_app[jitterbug-builder]";end

Next, the database is deployed, also into a specific directory. Note the "action :create_if_missing" line – that ensures we don't overwrite an existing database if we re-run the configuration. The extra "file" resource stanza ensures the database has the right user/permissions, even if it does exist.

directory node['jitterbug']['db_dir'] do owner node['jitterbug']['user'] group node['jitterbug']['group']end

cookbook_file "#{node['jitterbug']['db_dir']}/jitterbug.db" do source "jitterbug.db" mode "0644" owner node['jitterbug']['user'] group node['jitterbug']['group'] action :create_if_missingend

file "#{node['jitterbug']['db_dir']}/jitterbug.db" do mode "0644" owner node['jitterbug']['user'] group node['jitterbug']['group'] action :touchend

With the configuration files and database deployed, the final step is to deploy two application services. The first is the Jitterbug web application and the second is the Jitterbug task worker, which actually does the testing.


carton_app "jitterbug" do perlbrew node['jitterbug']['perl_version'] command "#{node['jitterbug']['deploy_dir']}/jitterbug.pl -p #{node['jitterbug']['port']}" cwd node['jitterbug']['deploy_dir'] user node['jitterbug']['user'] group node['jitterbug']['group'] environment ({ 'DANCER_CONFDIR' => node['jitterbug']['conf_dir'] }) action [:enable, :start]end

carton_app "jitterbug-builder" do perlbrew node['jitterbug']['perl_version'] command "perl #{node['jitterbug']['deploy_dir']}/scripts/builder.pl -c #{node['jitterbug']['conf_dir']}/config.yml" cwd node['jitterbug']['deploy_dir'] user node['jitterbug']['user'] group node['jitterbug']['group'] action [:enable, :start]end

Finally, the Jitterbug Nginx configuration is enabled.

nginx_site "jitterbug.conf" do enable true notifies :reload, 'service[nginx]'end

Then I made sure all this work was checked into git and pushed up to Github, so it was ready for the tutorial.

Step 4. Specify the configuration for the server

Now that you have all the cookbooks you'll need, it's time to create some roles and then apply the roles and recipes to the server node. (We could do everything without roles, but I want to show how you might use them.)

The first role is a "base" role that we would want to apply to any node. It does some basic housekeeping, enables a firewall, and turns on NTP. (Make sure you're back in the Pantry directory before you continue.)

$ cd /tmp/jitterbug-config$ pantry create role base$ pantry apply role base -r apt -r ohai -r hostname -r ufw -r ntp

Note that we don't apply any firewall rules in the role, we merely ensure that the firewall is enabled (by default only port 22 is allowed). We'll override that later in a "web" role to open up port 80.

The base roles can have attributes we want everywhere. For example, we can make sure that Perlbrew always builds in parallel and without tests.

$ pantry apply role base -d perlbrew.install_options="-j 5 -n"

This doesn't cause perlbrew to run on a node, it just sets some default attributes for any node that does actually configure perlbrew.


Next, we'll create a "web" role that deploys Nginx and opens up port 80 in the firewall.

$ pantry create role web$ pantry apply role web -r nginx

We can also set an attribute that disables the default Nginx web page:

$ pantry apply role web -d nginx.default_site_enabled=false

Unfortunately, the data structure the ufw recipe expects for firewall data can't be specified on the Pantry command line. You'll need to tell Pantry you want to manually edit the role JSON file and add the necessary data structure directly.

Start by editing the file:

$ pantry edit role web

Then, edit the "default_attributes" data to add a section for the firewall configuration. The final result should look like this:

{ "json_class" : "Chef::Role", "run_list" : [ "recipe[nginx]" ], "chef_type" : "role", "override_attributes" : {}, "default_attributes" : { "firewall" : { "rules" : [ { "http" : { "port" : 80 } } ] }, "nginx" : { "default_site_enabled" : false } }, "name" : "web"}

Since Jitterbug wants to send out email reports when test fail, we need to configure a mail client. Again, we'll create an "mx" (mail exchange) role, add postfix to that role, and configure postfix to be a master (i.e. sends mail directly).

$ pantry create role mx$ pantry apply role mx -r postfix -d postfix.mail_type=master

We don't create a firewall rule for the "mx" role, because we're only sending mail and not receiving it. (If we needed to receive mail, we'd have to open up port 25.)


Now that the "base", "web" and "mx" roles have been created, the next step is to create a node and configure it for those roles and the Jitterbug recipe we created.

$ pantry create node jitterbug.example.com$ pantry apply node jitterbug -R base -R web -R mx -r jitterbug

Note that once the node is created with the fully qualified name, you only need to specify a unique substring when Pantry operates on a node name. (You could even just say "j" instead of "jitterbug", since that's the only node.)

The "hostname" recipe requires us to set an attribute for the desired hostname, so we add that:

$ pantry apply node jitterbug -d set_fqdn=jitterbug.example.com

We also need to configure Jitterbug itself. For this tutorial, we'll just configure the addresses used for email. We'll use some shell loops to avoid repetitive typing. Replace "[email protected]" with your own email address:

$ for i in pass failure; do \ for j in cc from; do \ pantry apply node jitterbug \ -d jitterbug.on_${i}_${j}[email protected]; \ done; \ done

You can look at the resulting node file to be sure everything was set correctly:

$ pantry show node jitterbug{ "set_fqdn" : "jitterbug.example.com", "jitterbug" : { "on_pass_from_email" : "[email protected]", "on_failure_cc_email" : "[email protected]", "on_failure_from_email" : "[email protected]", "on_pass_cc_email" : "[email protected]" }, "run_list" : [ "role[base]", "role[web]", "role[mx]", "recipe[jitterbug]" ], "name" : "jitterbug.example.com"}

Once all the configuration is done, we want to check everything into git.

$ git add .$ git commit -m "jitterbug node configured"


Step 5. Deploy and test

With all the configuration work done, deployment is easy:

$ pantry sync node jitterbug

Now comes the hard part... waiting for it to finish.

Unfortunately, we have to let it work for a while as each component is installed and configured. Generally, this is the time to go do something else while you wait.3

Once it's done, switch to a browser and enter the hostname you configured. You should see an empty dashboard like this:

Congratulations! Your Jitterbug server has been deployed. Now it's time to test it with a Perl module repository from Github.

Browse to Github and go to one of your repositories with a Perl module in it. For example, I used the repo for my own IO::Prompt::Tiny at https://github.com/dagolden/io-prompt-tiny. This simple module is actually a good torture test for Jitterbug because it's built with Dist::Zilla, so Jitterbug has to install the full Dist::Zilla dependency tree in order to be able to test commits to the repository.

Under the "Admin" tab, the Service Hooks menu option lets you entire a WebHook URL "http://jitterbug.example.com/hook/" like this:

3 If deployment crashes out while building Perl, just try it again. I've seen some rare transient errors I've yet to diagnose, but the nice thing about idempotent deployment is that you can just try again and see what happens.

http://p3rl.org/Dist::Zilla

https://github.com/dagolden/io-prompt-tiny


After you "Update Settings", go back to the WebHook and click "Test Hook". Now switch back to your Jitterbug dashboard and refresh the page.

You should see your new repository being tested:

Refresh your dashboard every so often until the pending build task is gone. The first time might take a long time as prerequisites get installed.4 You should then be able to click into the repository link and see a "PASS" or "FAIL" notice for that commit:

4 If it seems like it's taking too long, you can ssh into the box and tail the file deep in the /tmp/jitterbug directory to see what's going on.


Now, every time you push a commit to that repository, Github will push a task to your Jitterbug server and tests will be run.

We're done — we just deployed a Jitterbug continuous integration server using Chef and Pantry.

Happy cooking!

cooking perl with chef: real world tutorial with jitterbug

Technology

need chef

y chef

linode http

chef real world tutorial

chef fundamentals

perl chef cookbooks

jitterbug page

provision linode