deployment tactics

Post on 17-May-2015

11.180 Views

Category:

Documents

8 Downloads

Preview:

Click to see full reader

DESCRIPTION

The slides from my Deployment Tactics talk at the ThinkVitamin Code Management online conference (http://thinkvitamin.com/online-conferences/code-manage-deploy/).

TRANSCRIPT

DEPLOYMENT

TACTICS

Managing code from development to production

Ian Barber - ian.barber@gmail.com twitter.com/ianbarber | phpir.com

- Table of Contents -

1.... Change Control2.... Environments3.... Version Control 4.... The Deploy Process5.... Scripts6.... Continuous Integration7.... Remote Releases8.... Packaged Releases9.... Package Management10.. Managing Hotfixes11.. Managing Database Changes12.. Rollbacks13.. Tactical Deployment

Change control

identify need

plan change

verify

execute change

deliverclose

require release

export code

compare md5

copy to server

restart apache

report back

Change Request Form

Requested By: J. Teamlead Authorised By: S. Manager Submit Date: 2011-01-27 Change Date: 2011-02-04

Reason For Change: Resolve JIRA-1602 - Listen for new .com variants on vhostChange Request:Release tag 1.1.3 via normal processmv /etc/httpd/conf.d/fooweb.conf /etc/httpd/conf.d/fooweb.oldmv ~releases/1.1.3/conf/fooweb.conf /etc/httpd/conf.d/fooweb.conf Verification:http://foweb.com shows the same page as http://fooweb.comRollback: Re-release 1.1.2mv /etc/httpd/conf.d/fooweb.old /etc/httpd/conf.d/fooweb.conf

Environments

Production Development

staticrobustreliableoptimised

verbosedynamicunstable

experimental

Image: http://flickr.com/photos/lejoe/3763218501

The Production Environment

Image: http://flickr.com/photos/simononly/4454401446

The Staging Environment

Image: http://flickr.com/photos/unfoldedorigami/2374016430

The Integration Environment

Image: http://flickr.com/photos/drewnew/511936681

The Development Environment

VERSION CONTROL

Image: http://flickr.com/photos/robbie73/4346732208

/trunk

/branches/search

/branches/newpage /branches/...

/tags/1.1.2/branches/1.1.2

/branches/newpage

/trunk

/branches/search

/branches/...

/tags/1.1.2/branches/1.1.2

Development

Staging Production

Integration

master

devel

search feature

release1.1.1

long feature

master

devel

search feature

release1.1.1

long feature

Development

Integration

Staging

Production

The DEPLoy PROCESS

The DEPLoy PROCESS

transparent easy

scalable reliable

flexible

graceful

code config

commandspackages

SMTP config apache

vhost

support process

app update

libpng update

cache service

restart service

file perms

server

code repository

package repository

deployment controllerserver

config repository

server commands

data

server

code repository

package repository

deployment controllerserver

config repository

server commands+ data

data

BUILD SCRIPTS

#!/bin/bash # Deployment script for FooWeb Projectgit archive --format=tar \ --remote=git://repo.com/myrepo/myrepo.git \ HEAD -o fooweb.tartar -xf fooweb.tar /var/wwwservice httpd restart

#!/bin/bash # Deployment script for FooWeb Projectsvn export svn://localhost/fooweb-service/trunk releasecd release && mkdir buildcp -r web/* build/javac -cp /usr/share/java/servlet-api-2.5.jar -d build/WEB-INF/classes src/com/fooweb/service/*.javacd build && jar cvf ../fooweb.war * && cd ../# assumes autoDeploy is truecp fooweb.war /var/lib/tomcat6/webapps

BUILDS TOOLS

code

assets

tests

build

testresults

docs

release

buildtools

<?xml version="1.0" encoding="UTF-8"?><project name="FooWeb"><property name="install" location="/var/lib/tomcat6/webapps" /><property name="svn.repo" value="svn://localhost/fooweb-service/trunk" />

<!--A "clean" target to delete compiled files--><target name="clean"> <delete dir="build" /> <delete dir="release" /> <delete file="fooweb.war" /></target>

<!-- Checkout, mkdir and compile--><target name="build"> <exec executable="svn"> <arg line="export ${svn.repo} release" /> </exec> <mkdir dir="build"/> <copy todir="build"> <fileset dir="release/web" /> </copy> <javac srcdir="release/src" destdir="build/WEB-INF/classes/"> <classpath> <pathelement path="/usr/share/java/servlet-api-2.5.jar"/> </classpath> </javac></target>

<!-- Build our WAR file -->

<!-- Build our WAR file --><target name="war" depends="build"> <war destfile="fooweb.war" webxml="build/WEB-INF/web.xml"> <fileset dir="build"/> <classes dir="build/WEB-INF/classes"/> </war></target>

<!-- Copy our file --><target name="deploy" depends="war"> <copy file="fooweb.war" todir="${install}" /></target>

</project>

$ sudo ant deployBuildfile: build.xmlbuild: [exec] Exported revision 8. [mkdir] Created dir: /tmp/build [copy] Copying 2 files to /tmp/build [copy] Copied 3 empty directories to 1 empty directory under /tmp/build [javac] Compiling 1 source file to /tmp/build/WEB-INF/classeswar: [war] Building war: /tmp/fooweb.wardeploy: [copy] Copying 1 file to /var/lib/tomcat6/webapps

BUILD SUCCESSFUL Total time: 2 seconds

<project name="Fooweb" default="build"> <target name="build" depends="phpunit" />

<target name="init"> <mkdir dir="${basedir}/build/logs" /> </target>

<target name="phpunit" depends="init"> <exec executable="phpunit" dir="${basedir}/tests" failonerror="on"> <arg line=" --log-junit '${basedir}/build/logs/phpunit.xml' --coverage-clover '${basedir}/build/logs/clover.xml' --coverage-html '${basedir}/build/logs/coverage'" /> </exec> </target>

<target name="phpcpd" depends="init"> <exec executable="phpcpd" dir="${basedir}/application" failonerror="on"> <arg line=" --log-pmd '${basedir}/build/logs/php-cpd.xml' ." /> </exec> </target></project>

REMOTE RELEASES

Image: http://flickr.com/photos/scragz/309353618

server

deployment controller

server

ssh / scpid_rsa.pub

authorized_keys authorized_keys

ssh-keygen -t rsauser “deploy”

ssh

/ scp

from fabric.api import * # Development environmentdef dev(): env.user = 'deployer' env.roledefs = { "web" : ['localhost'], "db" : ['localhost'], }

# Production environmentdef production(): env.user = 'deployer' env.roledefs = { "web" : ['primary.fooweb.com', 'secondary.fooweb.com'], "db" : ['backend.fooweb.com'], }

Fabrichttp://fabfile.org

# Package up release - run localdef prepare_deploy(): local('svn export svn://localhost/fooweb/trunk release') with cd('release'): local('tar cvzf ../fooweb.tar.gz .') local('rm -rf release') # Restart web server @roles('web')def restart_webserver(): sudo('/etc/init.d/apache2 restart')

# Deploy to remote servers@roles('web')def deploy(): prepare_deploy() # in case of already existing with settings(warn_only=True): run('mkdir /tmp/release') run('rm -rf /tmp/release/*') put("fooweb.tar.gz", '/tmp/release') with cd('/tmp/release'): run("tar xvzf fooweb.tar.gz") run("rm -rf fooweb.tar.gz") run("mv * /tmp/test") restart_webserver(); local("rm -rf fooweb.tar.gz");

$ fab dev deploy

[localhost] run: svn export svn://localhost/fooweb/trunk release[localhost] run: tar cvzf ../fooweb.tar.gz .[localhost] run: rm -rf release[localhost] run: mkdir /tmp/release[localhost] err: mkdir: cannot create directory `/tmp/release': File exists

Warning: run() encountered an error (return code 1) while executing 'mkdir /tmp/release'

[localhost] run: rm -rf /tmp/release/*[localhost] put: fooweb.tar.gz -> /tmp/release/fooweb.tar.gz

[localhost] run: tar xvzf fooweb.tar.gz[localhost] run: rm -rf fooweb.tar.gz[localhost] run: mv * /tmp/test[localhost] sudo: /etc/init.d/apache2 restartPassword for ianbarber@localhost: [localhost] out: * Restarting web server apache2[localhost] out: ... waiting ...done.[localhost] run: rm -rf fooweb.tar.gz

Done.Disconnecting from localhost... done.

$ fab production deploy[localhost] run: tar cvzf ../fooweb.tar.gz .....[primary.fooweb.com] run: rm -rf /tmp/release/[localhost] run: tar cvzf ../fooweb.tar.gz .....[secondary.fooweb.com] run: mkdir /tmp/release....Disconnecting from secondary.fooweb.com...doneDisconnecting from primary.fooweb.com... done

set :application, "fooweb"set :repository,"svn://localhost/fooweb/trunk"

set :scm, :subversionset :scm_username, "deployment" set :scm_password, "s3kkr3tp4a55" set :scm_checkout, "export"set :keep_releases, 4 set :normalize_asset_timestamps, falseset :deploy_to, "/usr/local/#{application}"

role :web, "primary.fooweb.com"role :web, "secondary.fooweb.com"role :db, "backend.fooweb.com"

namespace :deploy do task :migrate do # nothing end

task :restart do sudo "/etc/init.d/apache2 restart" endend

namespace :fooweb do task :perms do sudo "chmod -R a+w #{deploy_to}" endend

after "deploy:setup", "fooweb:perms"

$ cap deploy:setup * executing `deploy:setup' * executing "sudo mkdir -p /usr/local/fooweb [...]" servers: ["primary","secondary", "backend"] [backend] executing command [...] command finished triggering after callbacks for deploy:setup * executing `fooweb:perms' * executing "sudo chmod -R a+w /usr/local/fooweb" servers: ["primary","secondary","backend"] [primary] executing command [...] command finished

$ cap deploy * executing `deploy' * executing `deploy:update' ** transaction: start * executing `deploy:update_code' executing locally: "svn info svn://localhost/fooweb/trunk -rHEAD"/usr/bin/svn * executing "svn checkout -q -r17 svn://localhost/fooweb/trunk /usr/local/fooweb/releases/20110116192456 && (echo 17 > /usr/local/fooweb/releases/20110116192456/REVISION)" servers: ["primary.fooweb.com"] [primary.fooweb.com] executing command[....] * executing `deploy:finalize_update' * executing "chmod -R g+w /usr/local/fooweb/releases/20110116192456" servers: ["primary.fooweb.com"] [primary.fooweb.com] executing command command finished * executing "rm -rf /usr/local/fooweb/releases/20110116192456/log /usr/local/fooweb/releases/20110116192456/public/system /usr/local/fooweb/releases/20110116192456/tmp/pids &&\\\n mkdir -p /usr/local/fooweb/releases/20110116192456/public &&\\\n mkdir -p /usr/local/fooweb/releases/20110116192456/tmp &&\\\n ln -s /usr/local/fooweb/shared/log /usr/local/fooweb/releases/20110116192456/log &&\\\n ln -s /usr/local/fooweb/shared/system /usr/local/fooweb/releases/20110116192456/public/system &&\\\n ln -s /usr/local/fooweb/shared/pids /usr/local/fooweb/releases/20110116192456/tmp/pids" servers: ["primary.fooweb.com"] [primary.fooweb.com] executing command command finished * executing `deploy:symlink' * executing "rm -f /usr/local/fooweb/current && ln -s /usr/local/fooweb/releases/20110116192456 /usr/local/fooweb/current" servers: ["primary.fooweb.com"] [primary.fooweb.com] executing command command finished ** transaction: commit * executing `deploy:restart' * executing "sudo -p 'sudo password: ' /etc/init.d/apache2 restart" servers: ["primary.fooweb.com"]

/usr/local/fooweb/!"" current -> releases/20110116192316!"" releases#   !"" 20110116190608#   #   !"" application#   #   !"" log -> /usr/local/fooweb/shared/log#   #   !"" public#   #   !"" REVISION#   #   !"" tmp#   !"" 20110116192316#   #   !"" application#   #   !"" log -> /usr/local/fooweb/shared/log#   #   !"" public#   #   !"" REVISION#   #   !"" tmp $"" shared

PACKAGEDRELEASES

Image: http://flickr.com/photos/halfbisqued/2353845688

Fooweb

FoowebMail

FoowebService

Symfony 1.3

PHP 5.2.12

Tomcat 6.0Any SMTP Server

Java 1.6

!"" application#   !"" controllers#   #   $"" home.php#   $"" library#   $"" Foow#   $"" Router.php!"" fooweb.spec!"" public#   $"" index.php$"" vhosts $"" fooweb.conf

Summary: Fooweb ApplicationVendor: FoowebName: foowebVersion: 1.0Release: 1Source0: fooweb-%{version}.tar.gzLicense: BSDGroup: FoowebBuildArch: noarchBuildRoot: %{_tmppath}/%{name}-%{version}-buildrootRequires: php%descriptionThis is the Fooweb web application

%prep%setup

%installmkdir -p $RPM_BUILD_ROOT/var/www/foowebmkdir -p $RPM_BUILD_ROOT/etc/httpd/conf.d/cp -r application $RPM_BUILD_ROOT/var/www/foowebcp -r public $RPM_BUILD_ROOT/var/www/foowebcp vhosts/fooweb.conf $RPM_BUILD_ROOT/etc/httpd/conf.d/

%cleanrm -rf $RPM_BUILD_ROOT

%files%dir /var/www%dir /var/www/fooweb%config /etc/httpd/conf.d/fooweb.conf/var/www/fooweb/*

~$ mkdir buildroot buildroot/tmp~$ cat .rpmmacro %packager Fooweb Release Manager%_topdir ~/buildroot%_tmppath ~/buildroot/tmp~$ cd ~/tags~/tags$ tar cvzf fooweb-1.0.tar.gz fooweb-1.0

~$ rpmbuild -ta fooweb-1.0.tar.gz ~$ rpm -qip ~/rpmbuild/RPMS/noarch/fooweb-1.0-1.noarch.rpm Name : fooweb Relocations: (not relocatable)Version : 1.0 Vendor: FoowebRelease : 1 Build Date: Thu 13 Jan 2011 12:26:24 AM PSTInstall Date: (not installed) Build Host: ubuntu.localdomainGroup : Fooweb Source RPM: fooweb-1.0-1.src.rpmSize : 781 License : BSDSignature : (none)Summary : Fooweb ApplicationDescription : This is the Fooweb web application

$ mkdir /var/www/repo$ cd /var/www/repo$ mkdir centox/5/fooweb/{SRPMS,X86_64,i386,noarch}$ cp ~rpmbuild/RPMS/noarch/* centos/5/fooweb/noarch$ cp ~rpmbuild/SRPMS/* centos/5/fooweb/SRPMS$ createrepo -v centos/5/fooweb/noarch/

centos/5/fooweb/noarch/!"" fooweb-1.0-1.noarch.rpm$"" repodata !"" filelists.xml.gz !"" other.xml.gz !"" primary.xml.gz $"" repomd.xml

$ cat /etc/yum.repos.d/fooweb.repo [fooweb_noarch]name = Fooweb Private Repositorybaseurl = http://fooweb.com/repo/centos/5/fooweb/noarchenabled = 1gpgcheck = 0gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fooweb

$ yum updatefooweb_noarch 100% |========| 951 B fooweb_noarch/primary 100% |========| 701 B fooweb_noarch 1/1Setting up Update ProcessNo Packages marked for Update

$ yum info foowebAvailable PackagesName : foowebArch : noarchVersion : 1.0Release : 1Size : 3.3 kRepo : fooweb_noarchSummary : Fooweb ApplicationLicense : BSDDescription: This is the Fooweb web application

<target name="buildrpm" depends="init"> <tar destfile="build/rpm/SOURCES/fooweb.tar.gz" compression="gzip"> <tarfileset dir="${basedir}" prefix="fooweb-1.0"> <include name="*/**" /> <exclude name="build/**" /> </tarfileset> </tar> <copy file="${basedir}/fooweb.spec" tofile="${basedir}/build/rpm/SPECS/fooweb.spec" /> <rpm command="-ba" specFile="fooweb.spec" topDir="${basedir}/build/rpm" cleanBuildDir="true" failOnError="true" /></target>

PACKAGE MANAGEMENT

Image: http://flickr.com/photos/southerncalifornian/2129676744

$ sudo aptitude install puppetmaster 0 packages upgraded, 10 newly installed, 0 to remove and 76 not upgraded.Need to get 3,233kB of archives. After unpacking 13.7MB will be used.

$ sudo aptitude install puppet0 packages upgraded, 5 newly installed, 0 to remove and 76 not upgraded.Need to get 587kB of archives. After unpacking 1,892kB will be used.

Puppethttp://puppetlabs.com

Puppet master

primary fooweb.com

intfooweb.com

seconday fooweb.com

staging fooweb.com

backend fooweb.com

/etc/puppet!"" auth.conf!"" fileserver.conf!"" manifests#   $"" site.pp!"" modules#   !"" apache2#   #   $"" manifests#   #   $"" init.pp#   $"" fooweb#   !"" files#   #   $"" fooweb.conf#   $"" manifests#   $"" init.pp!"" puppet.conf$"" templates

class fooweb { package { "fooweb": ensure => latest, }" file { "/etc/apache2/sites-enabled/fooweb.conf": owner => root, group => root, mode => 0444, source => "puppet:///files/fooweb/files/fooweb.conf", notify => Service["apache2"] }}

modules/fooweb

manifests/init.pp

node "ubuntu.localdomain" { include fooweb include apache2}

class apache2 { service { apache2: ensure => running }

manifests/site.pp

modules/apache2 manifests/init.pp

# puppet agent -o -v --no-daemonizeinfo: Caching catalog for ubuntu.localdomaininfo: Applying configuration version '1295514488'notice: /Stage[main]/Fooweb/Package[fooweb]/ensure: ensure changed 'purged' to 'latest'notice: /Stage[main]/Fooweb/File[/etc/apache2/sites-enabled/fooweb.conf]/ensure: defined content as '{md5}d41d8cd98f00b204e9800998ecf8427e'info: /Stage[main]/Fooweb/File[/etc/apache2/sites-enabled/fooweb.conf]: Scheduling refresh of Service[apache2]notice: /Stage[main]/Apache2/Service[apache2]: Triggered 'refresh' from 1 eventsnotice: Finished catalog run in 3.33 seconds# ls /etc/httpd/conf.d/fooweb.conf /etc/httpd/conf.d/fooweb.conf# ls /var/www/fooweb/application public

MANAGING HOTFIXES

Image: http://flickr.com/photos/moogan/8206134

/trunk

/branches/1.1.3

/tags/1.1.2 /tags/1.1.3

package

run db changes

run db backup

copy code

make code active

package

run db changes

run db backup

copy code

make code active

MANAGING Database ChanGES

Image: http://flickr.com/photos/theplanetdotcom/4878814847

CREATE TABLE `blogpost` ( `id` int(11) auto_increment NOT NULL PRIMARY KEY, `title` VARCHAR(255), `timestamp` DATETIME, `content` TEXT);

--//@UNDO

DROP TABLE `blogpost`;DBDeploy

http://dbdeploy.com

ALTER TABLE `blogpost` ADD `author` varchar(255) NULL;

--//@UNDO

ALTER TABLE `blogpost` DROP `author`;

CREATE TABLE changelog ( change_number BIGINT NOT NULL, delta_set VARCHAR(10) NOT NULL, start_dt TIMESTAMP NOT NULL, complete_dt TIMESTAMP NULL, applied_by VARCHAR(100) NOT NULL, description VARCHAR(500) NOT NULL, PRIMARY KEY(change_number, delta_set));

$ wget http://dbdeploy.googlecode.com/files/dbdeploy-dist-3.0M2-distribution.zip

$ java -cp mysql-connector-java.jar:dbdeploy-cli-3.0M2.jar com.dbdeploy.CommandLineTarget -D com.mysql.jdbc.Driver -d mysql -o delta.sql -u jdbc:mysql://localhost/foowebdb -U root -P ******dbdeploy 3.0M2Reading change scripts from directory dbdeploy.Changes currently applied to database: (none)Scripts available: 1, 2To be applied: 1, 2

-- START CHANGE SCRIPT #1: 1-create-blogposts.sql

CREATE TABLE `blogpost` ( `id` int(11) auto_increment NOT NULL PRIMARY KEY, `title` VARCHAR(255), `timestamp` DATETIME, `content` TEXT);

INSERT INTO changelog (change_number, complete_dt, applied_by, description)VALUES (1, CURRENT_TIMESTAMP, USER(), '1-create-blogposts.sql');

COMMIT;

-- END CHANGE SCRIPT #1: 1-create-blogposts.sql

-- START CHANGE SCRIPT #2: 2-add-author.sql

ALTER TABLE `blogpost` ADD `author` varchar(255) NULL;

INSERT INTO changelog (change_number, complete_dt, applied_by, description) VALUES (2, CURRENT_TIMESTAMP, USER(), '2-add-author.sql');

COMMIT;

-- END CHANGE SCRIPT #2: 2-add-author.sql

<?xml version="1.0" encoding="UTF-8" standalone="no"?><databaseChangeLog [....]> <changeSet author="ianbarber" id="1"> <createTable tableName="blogposts"> <column autoIncrement="true" name="id" type="int(11)"> <constraints nullable="false" primaryKey="true" /> </column> <column name="title" type="varchar(255)" /> <column name="body" type="text" /> <column name="author" type="varchar(255)"/> <column name="date" type="timestamp" /> </createTable> </changeSet> Liquibase

http://liquibase.org

<?xml version="1.0" encoding="UTF-8" standalone="no"?><databaseChangeLog [....] > <include file="v000/master.xml" /></databaseChangeLog>

<?xml version="1.0" encoding="UTF-8" standalone="no"?><databaseChangeLog [....]> <include file="v000/create-blog-posts-1.xml" /></databaseChangeLog>

update.xml

v000/master.xml

$ liquibase --changeLogFile=update.xml updateLiquibase Home: /opt/liquibaseINFO 1/18/11 1:32 PM:liquibase: Successfully acquired change log lockINFO 1/18/11 1:32 PM:liquibase: Reading from `DATABASECHANGELOG`INFO 1/18/11 1:32 PM:liquibase: Reading from `DATABASECHANGELOG`INFO 1/18/11 1:32 PM:liquibase: ChangeSet v000/create-blog-posts-1.xml::1::ianbarber ran successfully in 101msINFO 1/18/11 1:32 PM:liquibase: Successfully released change log lockLiquibase Update Successful

class CreateProjects < ActiveRecord::Migration def self.up create_table :projects do |t| t.column :name, :string t.column :description, :text t.column :template, :string t.column :created_at, :datetime t.column :updated_at, :datetime end end

def self.down drop_table :projects endend

ROLLING BACK

Image: http://flickr.com/photos/roolrool/4758613588

<?xml version="1.0" encoding="UTF-8" standalone="no"?><databaseChangeLog [....]> <changeSet author="ianbarber" id="2"> <addColumn tableName="blogposts"> <column name="commenter" type="varchar(255)" /> </addColumn> </changeSet></databaseChangeLog>

$ liquibase --changeLogFile=update.xml updateLiquibase Home: /opt/liquibaseINFO 1/18/11 2:38 PM:liquibase: ChangeSet v000/add_commenter-2.xml::2::ianbarber ran successfully in 136msLiquibase Update Successful

$ liquibase --changeLogFile=update.xml rollbackCount 1Liquibase Home: /opt/liquibaseINFO 1/18/11 2:39 PM:liquibase: Successfully acquired change log lockINFO 1/18/11 2:39 PM:liquibase: Reading from `DATABASECHANGELOG`INFO 1/18/11 2:39 PM:liquibase: Rolling Back Changeset:v000/add_commenter-2.xml::2::ianbarber::(Checksum: 3:cc45ae1014b26f8b35cb70a5fc39a1ae)INFO 1/18/11 2:39 PM:liquibase: Successfully released change log lockLiquibase Rollback Successful

TACTICAL DEPLOYMENTS

Image: http://flickr.com/photos/romainguy/230416692

namespace :deploy do namespace :web do task :disable, :roles => :web do on_rollback { rm "#{shared_path}/system/maintenance.html" }

require 'erb' deadline, reason = ENV['DATE'], ENV['WHY'] maintenance = ERB.new( File.read("./templates/maintenance.erb" )).result(binding)

put maintenance, "#{shared_path}/system/maintenance.html", :mode => 0644 end end

# DATE="16:00 MST" WHY="a database upgrade" cap deploy:web:disable

if (-f $document_root/system/maintenance.html) { rewrite ^(.*)$ /system/maintenance.html last; break;}

caches & proxies sessions links

migrate

warm redirect

sys admins QA

developers

devops

Image: http://flickr.com/photos/rossharmes/4153769740

Feature Flags

0%

25%

50%

75%

100%

Day 1 Day 2 Day 3 Day 4

With Feature Without Feature

Gradual Ramp

Dark Launches

THanks!

Deployment Tactics: Managing code from development to

production

Ian Barber - ian.barber@gmail.com twitter.com/ianbarber | phpir.com

top related