Download - From Virtual Machines to Containers
From Virtual Machines to ContainersAchieving Continuous Integration, Build Reproducibility, Isolation and Scalability
London, UK, June 2015
1
#jenkinsconf
Agenda
• Introduction• The Dark Age• The Promising Present• Lessons Learned• The Bright Future
#jenkinsconf
2014201320122011201020092008
IncorporationCamunda Services GmbHBerlin, Germany
IncorporationCamunda Inc.San Francisco, US
BPM Consulting BPM Software Vendor
● 30 Fulltime Employees ● No external funding ● Avg. 50% Turnover Growth per Year
Introduction - About Camunda@camundaBPM
Introduced Hudson
Adapted Jenkins
#jenkinsconf
Introduction - What is Camunda BPM
• Camunda BPM is an open source platformfor workflow and business process automation
• Integrates with:• 7 Application Server (11 different versions)• 6 Databases (17 different versions)• 1 Development & 4 Maintained Versions
#jenkinsconf
• Every Camunda BPM version is tested against:• 187 combinations of DBs and App Servers• 11 JDKs• ~ 400 Jobs per version
• Bi-Annual release of a new Camunda BPM version• Support for Enterprise Customer (24/7 + Fix Time)
Introduction - Why CI is Important
#jenkinsconf
The Dark Age - The Numbers
• 1 Jenkins Master with alot of plugins• 8 Jenkins Slaves VMs• ~1000 Jobs total configured / manually managed:
• 4 Camunda Versions• Community Projects• Websites• Maintenance
#jenkinsconf
The Dark Age - Isolation Problem
• Unit and Integration Tests need a database/application server
• Only 1 Instance per Database• All Jobs use the same Databases• Every half-year a new Version (~400 Jobs) using
same Databases
#jenkinsconf
The Dark Age - No Scalability
• Total Executors: 12• 1 Jenkins -> 4 Executors• 8 static heterogenous slave VMs, each with 1
Executor• Jobs tied to slaves through labels• Slaves “restrict” database access by allowing no
other build to run
#jenkinsconf
The Dark Age - Maintenance Problems
• Upgrading Jenkins or any plugin• Supporting a new Database vendor / version• Supporting a new App Server version• Creating jobs for new Camunda BPM version• Disaster recovery
#jenkinsconf
The Dark Age - The Other Problems
• Slow feedback cycle for developers• Developers cannot reproduce CI environments• QA engineers use and maintain separate bloated
test build setup
#jenkinsconf
The Present - What we achieved
• Configuration & Infrastructure as Code• Isolated and Reproducible Jobs• Scalable CI Infrastructure
#jenkinsconf
The Present - Infrastructure as Code
1. Every configuration is checked into SCM2. Every application/test runs in a Docker
Container3. Every Docker image is build
automatically
#jenkinsconf
The Present - Infrastructure as Code
1. Every Configuration is checked into SCM
• Docker for• Applications• Test Environments
• JobDSL for• Jenkins Jobs
#jenkinsconf
The Present - Infrastructure as Code
2. Every application/test runs in a Docker Container
Images:• Application (Jenkins, Nexus …)• Test Env. Images (DB + SSH)• Build Env. Images
• DIND, QEMU + Packer.io
#jenkinsconf
The Present - Infrastructure as Code
3. Every Docker Container is build automatically
• Own Jenkins for Docker/KVM Images• KVM Images build in Docker Container
with Packer + QEMU• KVM Images bundled in Docker Image
#jenkinsconf
The Present - The Current Flow
camunda-ci
camunda
Camunda BPMPlatform
InfrastructureJenkins
CI Jenkins
#jenkinsconf
The Present - Isolation
One Jenkins per Concern:
• CI• Release• Infrastructure• Community and other Projects• Marketing
#jenkinsconf
The Present - Isolation & Reproducibility
• Every Jobs runs in an One-Shot Docker Container• No Interference between Jobs• The Database Settings are well documented• Every Docker Image is stored in a private registry• Developers/QA can use the Docker Images for local
testing
#jenkinsconf
The Present - Scalability
• Jenkins uses Docker-Plugin with one Docker Cloud running on Docker Swarm
• Docker images are added through Groovy scripting• Running on Commodity Hardware
• 1 Infrastructure Host (Jenkins, Nexus, …)• 6 Docker Hosts as 1 Swarm
#jenkinsconf
The Present - Advantages
• Easy to add new Databases/Test Environments• New Release = New Branch of JobDSL Repository• Fully parallelized Job Execution• Accountable Configuration History• Testable Infrastructure• Minimize Administration Overhead
#jenkinsconf
The Present - Conclusion
• 2 People + 3 Months of Work• A fully scalable, isolated and reproducible CI
Infrastructure• Faster Feedback• Happy Developers and Product Owner
#jenkinsconf
• Automate as much as you can• Jenkins config• Jobs config• Environment creation
• Design to scale to support the business agility
Lessons learned - Architecture
#jenkinsconf
Lessons learned - Testing
Test everything:• Jenkins configuration
• required plugin features• Job generation• Docker images• Scalability• Disaster recovery
#jenkinsconf
Lessons learned - Job DSL
• Unit-test the job generation• Write JobGenerator classes to abstract the
common build logic of most jobs out of the box• Use XML diffing to compare previously generated
jobs with new ones
#jenkinsconf
Lessons learned - Docker
https://github.com/rancherio/vm/
#jenkinsconf
• Pin your plugin versions• Be prepared to contribute to plugin development or
maintain a branch yourself• Choose the right plugin for the job
Our Top 3 plugins:JobDSL, Docker-Plugin, Build-Failure-Analyzer
Lessons learned - Plugins
#jenkinsconf
Lessons learned - Control
• Control as much as possible• Third party binaries vs package manager• explicit versions• own mirrors for important packages
#jenkinsconf
The Future
• Public Community Jenkins• Internal Webapp for Developers and QA to start
Environments• Jenkins is deploying Jenkins• Back to the Datacenter• Centralized Logging and Monitoring (ELK)
#jenkinsconf
Resources
• Camunda CI repositoryhttps://github.com/camunda-ci
• Job DSL gradle examplehttps://github.com/sheehan/job-dsl-gradle-example
• Camunda BPM Docker imagehttps://registry.hub.docker.
com/u/camunda/camunda-bpm-platform/