An Almost Complete Continuous Delivery Pipeline Including Configuration Management

Ulf Månsson @ulfmanssonRecorded Future AB

Why almost complete?

You never complete

It's not easy

Recorded Future

We wanted

● Quick feedback

● Build all artifacts

● Upload all build artifacts to s3

● Trust in deploying every successful build

● Test everything

What do we mean test everything?

● Unit tests

● Integration tests

● End-to-end test

● UI test

● Performance test

● Cookbook tests

● Deployment tests

● Configuration tests

No champagne

Long journey

● The “build master” left the company

● Lacking focus

● Very long build times

● Use the build pipe line plugin

Continues delivery embrace devops

● Everyone should dare to touch stuff

● Developers touch ops code

● Operation touch application code

● Everyone should be able to deploy

operations@recordedfuture

● Quite big team

● Infra code

● Build pipe line

● Service and on premise installations

● End to end tests

● Performance tests

● Amazon operation costs

Our system

● A number of major infrastructure components – ElasticSearch, RabbitMQ etc

● About 45 different processes

● Development in Java, Python, Scala and Ruby

● External API and web servers

● Developed by 5 different teams in Sweden, US and Ukraine

Subversion

For good and bad

All code is code

Keep the code in the same repository

Configurations also in subversion

Treat all code in the same way, it doesn’t matter if it’s application code, infra code, test code or whatever

Feature toggling to our help

● No branching

● One trunk

● Took some time train developers

● Now we try to train infra coders as well – Feature toggling in Chef

● Read more by Jez Humble http://continuousdelivery.com/2011/07/on-dvcs-continuous-integration-and-feature-branches/

Jenkins

and his slaves...

Build Flow Plugin ­ the thing!

out.println "Start"

revision=build.properties["environment"]["SVN_REVISION"]

out.print "Original revision for this build is "

out.println revision

parallel (

{build("master-build-flow",SVN_REVISION: revision)},

{ ignore(FAILURE) {build("master-infrastructure_test",SVN_REVISION: revision)}},

{build("master-fast_track_upload_cookbooks-flow",SVN_REVISION: revision)},

{ ignore(FAILURE){build("master-infrastructure_lint",SVN_REVISION: revision)}}

)

build( "master-fast_track_deploy-flow", SVN_REVISION: revision)

build( "master-fast_track_integration_test_harvesting-flow", SVN_REVISION: revision)

parallel (

{ build("master-fast_track_test_ff-flow", SVN_REVISION: revision) },

{ build("master-fast_track_test_ie-flow", SVN_REVISION: revision) }

)

Build Graph View Plugin

Jenkins text finder

puts "The build is UNSTABLE" if failed

S3

● Store artifacts

● Simple

● Control

● Reliable

● Accessible

Test tools

● Test kitchen

● Server spec

● Maven

● Cucumber

● Rspec

● Load impact

● Selenium

Build tools

Then: Only Maven

Now: Mix of Rake and Maven

Environment cookbook

# Cookbook Name:: env

# Recipe:: fast_track_1  (test environment)

include_recipe 'env::prod'

node.default[:rabbitmq][:version] = "3.1.5"

# Interactive

node.default['rfwebapp']['interactive']['java_opts'] = '­Xms1G ­Xmx1G'

In env::default include_recipe "env::#{node.chef_environment}"

Why environment cookbook

● We can version handle it

● We are sure that the test environment will get the same settings as production environment

● On premise installation use chef-solo

● When we upload the cookbooks the environment settings will be uploaded as well

Our master build pipe line

Master­flow

● Only the build-flow plugin, describes the build

● Triggered by subversion polls

● Co-ordinate the sub build processes

● Only one at a time running

Master build flow

● Compiles all application code

● Run unit tests

● Builds application artifacts

● Uploads artifacts to S3

Upload cookbooks

● Upload cookbooks to chef server for the build test environment - fast_track

Infrastructure lint

● Food critic of course

● Tailor – nope

● Rubocop – probably as next step

Fast track...

● All-in-one box

● Will deploy,configure and launch all processes

● Configure infrastructure components

● Spin-up EC2 instance

Fast track cont...

● Use Chef server

● Deploy all processes with Chef

● Instance used for following tests

● Perfect case for Docker

Infrastructure test

● Test-kitchen tests

● Mainly server-spec

● Run in lxc with vagrant-lxc

● Run our “default” suite in test-kitchen

● Disabled in the Master-flow at the moment

End­to­end test

● Cucumber tests

● Populate system with data

● Tests via our API

● Check process log database to ensure all steps has been run

Browser tests IE and Firefox

● Run via Selenium

● Test browser functionality

● Upload screen snapshots from the tests to s3

Supporting tools

Bygge – an overview

Logstash

● Collect all log from the fast track run

● Used for problem solving

● Easy to access with a link from Bygge

● We also use it in production

Sensu ­ monitoring

● Almost same monitoring as in production of each fast truck run

● Easy to access monitoring dashboard and see errors

● Will fail builds when we have checks that fails

Lessons learned

Opscode jenkins cookbook

● Really good

● Providers for slaves, jobs, plugins

● Updated

Run parallel

Many and smaller build steps

From 150 minutes to 45 minutes in 3 days

● SSD instances for Jenkins

● SSD test instances

● Build slaves

● Parallel execution

● Pre upload

Infrastructure

Jenkins

Logstash

Sensu

Fast track instancesFast track instancesFast track instances

I would like to share experience