continuous integration for the world - xpdays benelux 2009

Patrick Debois

Continuous Integration for

the world

Wednesday 25 November 2009

Patrick Debois


My name is Patrick Debois and I come from Belgium


Iʼm a freelancer always on the look out for new interesting things


I have a sysadmin background


Now for the past few years, Iʼve been looking together with other sysadmins with great interest to the Agile Developers community at what we could learn from Agile development


And this year I wanted to give you my own version of Continuous Integration

Continuous Integration is the practice of integrating early and often, so as to avoid the pitfalls of "integration hell". The ultimate goal is to reduce timely rework and thus reduce cost and time. When done well, continuous integration has been shown to achieve these goals.


Let’s start with the definition on wikipedia.

http://www.martinfowler.com/articles/continuousIntegration.html

http://en.wikipedia.org/wiki/Continuous_Integration

Traditional CI Overview

http://www.javaworld.com/javaworld/jw-12-2008/images/CIOverview.jpgWednesday 25 November 2009

Different parts: Source Control, Build, Test, Package, Deploy/Publish

Important to remember

http://www.javaworld.com/javaworld/jw-12-2008/images/CIOverview.jpg

http://www.javaworld.com/javaworld/jw-12-2008/images/CIOverview.jpg

Traditional CI Flow

Courtesy Chris Read / ThoughtWorksWednesday 25 November 2009

Courtesy Chris Read / ThoughtWorkshttp://www.slideshare.net/ChristopherRead/continuous-integration-build-pipelines-and-continuous-deployment

CI Maturity Model

http://www.anthillpro.com/blogs/anthillpro-blog/2009/05/05/1241542860000.htmlWednesday 25 November 2009

Yes some call it insane.

http://www.anthillpro.com/blogs/anthillpro-blog/2009/05/05/1241542860000.html

http://www.anthillpro.com/blogs/anthillpro-blog/2009/05/05/1241542860000.html

Build Pipeline Flow

Courtesy Chris Read / ThoughtWorksWednesday 25 November 2009

http://www.slideshare.net/ChristopherRead/continuous-integration-build-pipelines-and-continuous-deployment

the flow - it it passes from Integration to UATthen to the next environment

Huge problem is managing all these environments and keeping them aligned

Definition of Done


We need to go beyond the traditional testing and extend it to the live system

Agility in Production• redefine key concept

• « the software works »

• « the software works in production »

• = beyond silos


http://www.lifeinamatrix.com/wp-content/uploads/2009/01/life-in-a-matrix-9-breaking-the-silos.jpg

Who is part of the team

• developers

• testers

• analysts

• project managers

• usability experts

• architects


But usually we forget

• the sysadmins

• the guys from operations


non-functionalrequirements

“Equal rights amendment”

Create value vs Protect value


“Enterprise” Triangle

Iron Triangle


Usually only the left triangle is discussed, but it complements the second


Project

Iron Triangle




Project Operations

Iron Triangle



Traditional CI limits

• Resistance to change

• « if it ain't broken don't fix it »

• Mostly hear as

• « if it works don’t touch it »

• Some even consider it best practice !

• « don’t put things into prod on a friday »


Why such a resistance

• development = create value

• operations = protect value

• => operations must limit risks

• change = risk

• so we limit the changes

• QED!


If it’s hard do it more often


We need to deploy more often. First times will fail, but eventually we will get better at this.The same for systems rebuilding.

Minimize Risks :integration of production• Add to traditional CI

• deployment of VM, OS,packages, middleware

• Production related tests

• infrastructure

• *ilities...

• performance


CI for systems


now managing a system is becoming more like writing software. So we will draw the parallel in the next part of the presentationfor Source , Build, Test

CI for systems (2)


packaging a system, deploying a system

Remember there is no silver bullet


Typical System Installation in CI

• Download and Install a distribution

• Download and Add some packages

• Run some configuration scripts

• Start software deployment testing


Problems?

• It’s a one shot operation

• Systems tend to become application friendly

• Is this the same as production? Patches, Versions, ...

• Usually allow all permissions


Birth / Adoption

Life

Palliative

Death

Life of System


Projects tend to think most of the birth of the software.But what about the system? We need to care about the running in production, in phase out and shutdown of a system

Examples of systems integrations in CI software

Hudson - PXE Hudson - Vmware

Not interested in the birth


Current integration of virtual machines in CI software

Evolution of operations

In the beginning there were the dinosaurs, slow and with large delays


Often IT is viewed as this

Operations evolution (2)

• Then came the monkeys with automated installations and automated deployment


Then they automated the installation

Clone system

• there is no such things a golden image

• image sprawl

• Cloning = Adoption of a System


And then tried to clone things

Test in a clone of the production environment

• Large effort

• similar to maintaining a ‘fork’



So let’s build it from the same source!

If we want to integrate let’s define what makes up the ‘source’ of our systems

http://jserv.sayya.org/misc/use_source_luke.png

Virtual Machines API• Systems virtualization :

Vserver, Zones, OpenVZ/Virtuozzo, chroots

• Hardware Emulation: VMWare, Xen, Qemu, VirtualBox, KVM


First part of the source

Language Bindings

• java libvirt / virsh

• ruby-libvirt

• virtualbox SOAP Interface

• vixie

• vmware API


Example Virtualbox SOAP

require 'soap/wsdlDriver'require 'pp'

WSDL_URL="vboxwebService.wsdl"

soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driversoap.wiredump_dev=STDERR#soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driver("vboxService", "vboxServicePort")#pp soap.methodsvbox=soap.IWebsessionManager_logon({:username => '', :password => ''})puts "Sessions"+vbox.returnvalversion=soap.IVirtualBox_getVersion({:_this => vbox.returnval})puts version.returnvaldisks=soap.IVirtualBox_getHardDisks({:_this => vbox.returnval})diskids=disks.returnvaldiskids.each do |diskid| type=soap.IHardDisk_getType({:_this => diskid }) size=soap.IHardDisk_getLogicalSize({:_this => diskid }) location=soap.IMedium_getLocation({:_this => diskid }) puts diskid+"-"+type.returnval+"-"+size.returnval+location.returnvalend


Example libvirt

require 'libvirt'

conn = Libvirt::open("qemu:///system") puts conn.capabilities File::open("domain.xml") do |f| conn.createDomainLinux(f.readlines, nil) end

dom = conn.lookupDomainByName("mydomain") dom.suspend dom.resume puts dom.xmlDesc


libvirt abstracts a lot of these types Xen, Qemu , Virtualbox, Vmware

Control the machine

• Physical machine

• IPMI, Light Out Module, Wake On LAN

• Virtual machine

• use of an API


Network provisioning

• DNS

• DHCP

• PXE

• TFTP Boot


Example Cobbler API# Set the distribution of the machine# The distribution was previously imported by importing an installation DVD# Cobbler will detect the possible kernels to boot# And this will also link the TFTP and PXE file necessary to Bootcobbler profile add --name=#{profile} --distro=centos53-i386

# Add the new machine with an IP and Mac address# By the IP Address it knows in which reverse DNS cobbler system add --name=#{name} --ip=#{ip} --mac=#{mac_address}"# Set the DNS domain of the machine (this determines in which zone file it is create)cobbler system edit --name=#{name} --profile=#{profile} --dns-name=puppet1.#{domainname}"

# For linux machines, the kernels are provided with an option # ks=kickstart so that it will start the kickstart installationcobbler system edit --name=#{name} --kickstart=/var/lib/cobbler/kickstarts/#{kickstart}cobbler system edit --name=#{name} --name-servers-search='#{domainname}'

# This finally commits all the changescobbler sync


Unattended OS Installs

AutoYastPreseed JumpstartJET Kickstart

.....


Example Kickstart# install_type MUST be firstinstall_type initial_install# start with the minimal required number of packagescluster SUNWCXallcluster SUNWCapache deletecluster SUNWCpcmc deletecluster SUNWCpcmcx deletecluster SUNWCthai deletecluster SUNWClp deletecluster SUNWCnis deletecluster SUNWCppp delete# format the entire disk for Solarisfdisk all solaris all# define how the disk is partitionedpartitioning explicitfilesys rootdisk.s0 6144 /filesys rootdisk.s1 1024 swapfilesys rootdisk.s7 free /state/partition1# install systems as standalonesystem_type standalone# specify patches to installpatch 119281-06 nfs 172.16.64.194:/export/patches# specify packages to installpackage SPROcc add nfs 172.16.64.194:/export/packages


It’s there under your nose

• After your manual install (system DEV)

• There is often a left over of silent install scripts

• anaconda

• silent install of oracle

• answer files


OS minimal

Just Enough Operating System


Instead of a full installation with the kickstart template. Use a minimal install/base install and then continue with config mgt.

configuration management

• chef

• puppet

• cfengine 3

• bcfg2

• lcfg

• smartfrog HP


Handles also live of the system.

Sample Recipeclass httpd { package { httpd: ensure => latest }

configfile { "/etc/httpd/conf/httpd.conf": source => "/httpd/httpd.conf", mode => 644, require => package["httpd"] }

group { apache: gid => 48 }

user { apache: comment => "Apache", uid => 48, gid => 48, home => "/var/www", shell => "/sbin/nologin" }

service { httpd: running => true, subscribe => [ file["/etc/httpd/conf/httpd.conf"], package["httpd"] ] }}


Shareable recipesFormal description of what needs to happen instead of custom scripts

• auto documentation

• confidence

• idempotent!

What a config mgt system can do for you

sed ‘s/the/then/g’


Managing Config Files API

• augeas - RedHat/Fedora

set /files/etc/hosts/01/ipaddr 192.168.0.1set /files/etc/hosts/01/canonical pigiron.example.comset /files/etc/hosts/01/alias[1] pigironset /files/etc/hosts/01/alias[2] piggysave


An extension for managing config files

Scriptable Appserver

• glassfish, Weblogic Scripting

• Oracle silent installs


Further automation

Example AppServer API# create an cluster asadmin create-cluster --user admin --host hostname -port 4848 cluster1

# create instance 1 asadmin create-instance --user admin --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8687:IIOP_LISTENER_PORT=3330:IIOP_SSL_LISTENER_PORT=4440:IIOP_SSL_MUTUALAUTH_PORT=5550:HTTP_LISTENER_PORT=1110:HTTP_SSL_LISTENER_PORT=2220" instance1

# create instance 2 asadmin create-instance --user admin --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties “JMX_SYSTEM_CONNECTOR_PORT=8688:IIOP_LISTENER_PORT=3331:IIOP_SSL_LISTENER_PORT=4441:IIOP_SSL_ MUTUALAUTH_PORT=5551:HTTP_LISTENER_PORT=1111:HTTP_SSL_LISTENER_PORT=2221" instance2

# start the cluster asadmin start-cluster --user admin --host hostname --port 4848 cluster1


Executing scripts• dsh, func, clusterssh, pssh, fabric

• or system calls in ruby/java/..

• exit code = important!


For the final tweeks, we run shell scripts to make

Make your build self-testing


Test Driven Automation

• Checks vs Tests

• Monitoring vs Testing

• Administration vs Automation


checks = automation of things to verify - similar to monitoringtests goes beyond check (human interpretation)

Red green refactor (change of config, change of patch, ....)

TDD Virtual Machines


Testing if CPU, Memory , Disk are there by booting up a rescueCD and executing a script.

Example DNS TestingFeature: Mass DNS Resolution In order to present a consistent brand image As a System Administrator I want to ensure no domains point away from our main IP

Scenario Outline: Resolve a name to a number Given a hostname of <hostname> Then I should see the IP address 266.266.266.266

Examples: | hostname | | example.org | | www.example.org |


http://blog.unixdaemon.net/cgi-bin/blosxom.pl/2009/04/06#ruby-dns-testing

http://www.example.org

http://www.example.org

Apache Recipe testing


Thanks to the build doctor! http://www.build-doctor.com/

Testing DeploymentScripts Capistrano

• SMF monitoring scripts


http://pivotallabs.com/users/jdean/blog/articles/763-testing-capistrano-recipes-with-cucumber

Testing / Monitoring

Cucumber-Nagioshttp://www.slideshare.net/auxesis/monitoring-web-application-behaviour-with-cucumbernagios


http://holmwood.id.au/~lindsay/2009/02/23/web-app-integration-testing-for-sysadmins-with-cucumber-nagios/

http://www.slideshare.net/auxesis/monitoring-web-application-behaviour-with-cucumbernagios

http://www.slideshare.net/auxesis/monitoring-web-application-behaviour-with-cucumbernagios

Lots of non

functionaltests


http://www.acutest.co.uk/acutest/load-testing-services

Maintain a codesystem repository

• Virtual Machine definitions + automation scripts

• Network provisioning (IP, DNS, DHCP) + scripts

• Install Media (DVD, RPM’s) +

• Kickstart Definitions + postinstall scripts

• Config mgt recipes

• Deployment scripts


Snapshots

vs.


Git doesn’t handle well large files / rsync worksRsync snapshots (used by timemachine)

File system repositories

Similarities

• html/js/java - scripts & recipes

• RPMS - jar files

• VM’s - war files

• local repositories


Every commit should be built


Automate the Build(of the System)


Good practice. Not only 1 part of it , but all pieces together.

Builders

• maven, ant,

• buildr

• rake, raven

• make

• schmant

• lancet

• sbt, kundo, gant


There are a lot of ways to combine all this code using one of the builder tools.They know how to handle dependencies well

Funny Tests

“Truck Factor”

“10th floor test”


The 10th floor test , is a term first used by Steve Traugott from www.infrastructures.org It refers to the ability to take a random machine in your infrastructure , drop it from the 10th floor, and be able to restore your infrastructure to working within 5-10 minutes.

Truck Factor - ProjectTruck Factor (definition): "The number of people on your team who have to be hit with a truck before the project is in serious trouble"

triggers

• changes in VM definition

• changes in IP, DNS, ..

• changes in OS, packages

• changes in recipes

• changes in scripts


build ENV = PROD• Development

• Test

• Quality Assurance

• Pre-production

• Production

• Differences in hardware

• Differences in software

• Differences in configuration


Templates/ Config should be configurable with the environment

Everyone commits every day


Keep the build fast


Filesystem Snapshots

Virtual Machine Snapshots

LVM, ZFS, ...


To speed up builds , intermedia results are stored using snapshots (system + disks)

Continuous vs Nightly

• During the day, recipes only

• Nightly the whole system rebuild


Package

• koan : creates virtual images

• rpath / rbuilder

• ec2

• SUN forklift

• jumpbox


Deployment tools

• carpet

• ec2 tools AMI

• Vmware Appliance


http://www.agileweboperations.com/an-in-depth-look-at-a-carpet-appliance-the-apache-load-balancer/

Carpet= Mixture of puppet with capistrano

Everyone can see the results of the latest build


Make it easy to get the latest deliverables

deploy on the latest installations


do you checkin wars for archiving

Hot topics

• Cloud Computing

• Production virtualization

• Continuous deployment

• Required fully automated provisioning


http://dev2ops.blogspot.com/2009/05/clouds-virtualization-and-continuous.html

Concept of Pipelines


Cloud perspective


There is actually very much of similarity with the cloud model

Key integration points

• What environments to combine

• Stable dev vs. Stable prod?

• Stable dev vs. Test prod?

• Test dev vs. Test prod?

• => explosion combinations!

• select the most sensible combination

• be sure to be able to pass easily to the next environment


Problems/Pitfalls• Infrastructure and software need to be

adapted for easy put into production

• update with downtime (load-balancer, redundant servers, clusters,

• data migrations


State Transition Process

http://digiorgio.com/blog/?p=239


If N+1 is in development then N is in integration , and N-1 in staging



Parallel Streams

http://www.agilejournal.com/content/view/865/195/


Example on how to integrate between different teams.

It’s all about flow here!



Development Sysadmins


Development Sysadmins

DEVOPS


More?

Email: [email protected] Twitter: @patrickdebois

http://[email protected]

IEEE-paper: http://www.jedi.be/presentations/IEEE-Agile-Infrastructure.pdf

http://www.jedi.be/blog


mailto:[email protected]









So the next time you’re building a fun project


Remember the sysadmins and guys from operations

We reboot with pride


And remember to reboot with pride

Questions?



Thanks for listening.

continuous integration for the world - xpdays benelux 2009

Technology

integration of production

left triangle

enterprise triangleusually

value operations

ci download

traditional ci overviewhttp

thoughtworks http

ci maturity model http