delivery pipeline as code: using jenkins 2.0 pipeline

RADIOLOGY WORKFLOW SOLUTIONS

© 2015 medavis GmbH. All rights reserved.

DevOps KarlsruheSlawa Giterman - July 2016

Delivery Pipeline as Codeusing Jenkins 2.0 Pipeline


Slawa Giterman

Java Developer (10+ years) A lot of experience with DevOps:

Continuous Integration & Build Tools Delivery Pipeline Automated acceptance testing

About myself


medavis GmbH - Software solutions for radiological workflows (MRT, CT, nuclear medicine, etc.)

Browser clients (Java, JEE, Vaadin) Fat clients (Windows) Integration with multiple 3rd party medical

services (HIS, HL7, DICOM, etc.)

About my company


Life before Jenkins Pipeline Pipeline as Code Jenkins Pipeline DSL How to use existing plug-ins in pipeline Lessons learned

Agenda


"Old" Jenkins Pipeline Plug-Ins Since 2011 there have been multiple Jenkins plug-ins

which supported pipelines, e.g. Build Pipeline Plugin or Build Pipeline Plugin (pictured below)


"Old" Jenkins Pipeline Plug-Ins They presented multiple Jenkins jobs as if they were

part of the same pipeline

From Jenkins point of view these pipelines were just collections of jobs which triggered each other


Decision Points and Fork / Join Logic It was necessary to use plug-ins to describe decision points

or fork / join workflows

Maintenance cost for complex pipelines was too high and rose exponentially with the number of steps


Jenkins Job DSL Plug-In A partial solution is to use Jenkins Job DSL plug-in

It allows to create jobs using Groovy scripts instead of complex manual configurations (see an example on the next slide)

But Groovy scripts can only be used during job creation not during job executions


Jenkins Job DSL Plug-In Examplevoid createMavenJob(String jobName, String svnUrl) { job(jobName) { svn { location(svnUrl) { credentials(SVN_CREDENTIALS_ID) } }... steps { maven { goals('clean deploy') mavenOpts('-Dserver.username=$username') } }


GoCD, Travis CI, Shippable Absence of the “full-fledged” pipeline support in

Jenkins led multiple teams to switch to alternative products like GoCD, Travis CI, Shippable, etc.


Jenkins Pipeline This led Cloudbees to start working on Jenkins

Pipeline Project (Originally called Jenkins Workflow)

Development started – May 2014

Release 1.0 – November 2014

Renamed to Jenkins Pipeline – January 2016

Jenkins 2 (weekly: 2.0) – April 2016

Jenkins 2 (LTS: 2.7.1) – July 2016


Jenkins Pipeline – New Project Dialog To create a new Pipeline using Jenkins 2 select

New Job -> Type: Pipeline


Jenkins Pipeline Project Configuration Then instead of configuring dozens of combo- and

checkboxes just configure your project SCM (Git, SVN, etc.)


Pipeline as Code

node('java-build') {

checkout scm

sh 'mvn -B clean deploy'

}

Definition of your build / pipeline will be stored together with your code as a Jenkinsfile


Simple Maven Projectstage 'Compile'


checkout scm

sh 'mvn -B clean deploy'

junit testResults: 'build/test-results/*.xml'

}

stage 'Automated tests'

node('qa-environment') { …

See step definitions on the next slide


Jenkins Job DLS Steps Jenkinsfile describe pipeline using Jenkins Pipeline DSL

Jenkins Pipeline DSL consist of steps like:

stage: starts a new pipeline stage (see next slide)

node: executes steps on an agent (node) with the corresponding name / label

checkout: checkouts code from SCM (e.g. Git or SVN)

sh: executes command line or shell script

bat: executes Windows command line or script

junit: publishes JUnit test results


Stage View UI Each pipeline consists of stages, e.g. Compile Stage, QA

Stage, Release Stage, etc. Each stage has one or multiple steps


Pipeline DSL: Working with Files

//Write string to filewriteFile file: 'status.log', text: 'Status: OK', encoding: 'UTF-8'

//Example: read XML file to string and then //get artifact version using regexpom = readFile('pom.xml')matcher = (pom =~ '<version>(.+)</version>')version = matcher ? matcher[0][1] : null

//Read file to stringresults = readFile file: 'results.log‚ encoding: 'UTF-8


Pipeline DSL: Pass files between stages

//"Archive" an artifact file to be used laterstash name: 'artifact', includes: 'package-a.zip'

//Stash multiple files:stash name: 'testResults', includes: '**/test-results/*.xml'

//Get the artifact file e.g. on another nodeunstash 'artifact'


Parallel Execution of Stepsparallel( //Both deployments are executed in parallel deployToEnv1: { node('environment-1') { deploy(…) } }, deployToEnv2: { node(environment-2') { deploy(…) } })//Execution continues after the longer step finishes


Use existing plugins in pipeline It is possible to use existing Jenkins Plug-ins if they added

Pipeline support

Majority of popular Jenkins plug-ins support Pipeline already (see list here)

E.g. Email-Ext plug-in:

emailext( subject: "Build #$buildNumber, $status", recipientProviders:'CulpritsRecipientProvider' …)

https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md


Docker Support

docker.image('.../java8-maven').inside { checkout svn sh 'mvn -B clean install'}

Jenkins Pipeline also provides Docker support See Orchestrating Workflows with Jenkins and Docker

https://www.cloudbees.com/blog/orchestrating-workflows-jenkins-and-docker


Pipeline DSL API Reference See Pipeline DSL API Reference

Or just use Snippet Generator in Jenkins

https://jenkins.io/doc/pipeline/steps/


Manual steps in Pipeline Jenkins Pipeline supports manual steps, i.e. steps which

require user interaction, e.g. tester should choose which environment to use for testing:


Manual steps – Confirmation dialog

//Clicking on Release will start the next phase and //on Abort will abort the pipeline executioninput message: 'Release to production?', ok: 'Release'

Confirmation dialogs are also supported


Multibranch Pipeline Jenkins can auto discover branches (Job type: Multibranch

Pipeline)

If a new branch is created, a corresponding job will be created for it. As soon as the branch is merged (deleted) the job will be deleted as well.


Reuse pipeline modules Use Scripts in other languages (e.g. Schell

script, Python, Ruby, etc.) if necessary How to test Pipeline scripts? New Jenkins GUI: Project Blue Ocean

Part 2: Lessons learned


How it worked before pipeline

Our First Pipeline


Delivery pipeline - an automated implementation of build, deploy, test, and release process

Automate the full process from Code Commit to Product Release

Our First Pipeline


Pipeline Code Reuse

stage 'Compile'


checkout scm

bat 'mvn -B clean deploy‘


}

… 200 more lines …

If the whole pipeline will be described in a single Jenkinsfile script, it will be too long and its maintenance will be too expensive


Pipeline Code Reuse To solve the problem use full Groovy potential (see the

following slides for examples) :

1. Extract methods to avoid copy-pasting and achieve code reuse

2. Organize code in modules by extracting methods into separate files

3. Reuse pipeline modules between multiple pipelines by storing them into a separate VCS repository (e.g. in Git or Subversion)


1. Extract Methodsvoid runPhase(String phase){ //extract code as method


checkout scm

bat "mvn -B $phase" //phase is a variable


}

}…

stage 'Compile'

runPhase('deploy') //just call the extracted method


2. Extract Modules

In Jenkinsfile load and reuse the extracted modules:

def maven = load 'maven.groovy'

stage 'Compile'

maven.runPhase('deploy') //method runPhase() from file

//maven.groovy will be called

Copy methods to separate files, e.g. maven.groovy:

void runPhase(String phase){

…}


3. Reuse Shared Modules

Jenkinsfile:

//checkout shared modules from VCS firstcheckout 'http://svn/main/pipeline-framework/trunk'

//then just load modules as beforedef maven = load 'maven.groovy'

stage 'Compile'maven.runPhase('deploy')

Check-in all modules (like maven.groovy, deployment.groovy, etc.) to VCS and then reuse them in all pipelines:


VCS vs. Jenkins Global Library1. You can specify which version of shared modules to use, e.g. trunk/master, or stable branch, etc.: checkout 'http://svn/main/pipeline/trunk'

checkout 'http://svn/main/pipeline/tags/1.0.0.2'

checkout 'http://svn/main/pipeline/branches/stable'

2. As an alternative to VCS, add modules to the Jenkins Global Library (currently does not support versioning):ssh://user@jenkins:3421/workflowLibs.git

https://github.com/jenkinsci/workflow-cps-global-lib-plugin/blob/master/README.md




Reuse Existing non-Groovy scripts If you prefer to use a language other than Groovy (e.g.

Python, Ruby, Shell script, etc.) or already have multiple non-Groovy scripts which implement pipeline step:

Implement in Groovy just the Pipeline orchestration logic (e.g. decisions, parallel execution, etc.)

Implement steps using your preferred language

Call these scripts from Pipeline using sh (Linux) or bat (Windows) DSL steps (see the next slide for an example)


Example: Reusing PowerShell Scripts

2. Call the script from Jenkins Pipeline:bat 'copy2share.ps source destination username password'

1. PowerSchell script copy2share.ps (or shell, python, ruby, etc.):Map-Networkdrive $drive $destination $username $passwordCopy-Item $source $destination

We decided to reuse a PowerShell script which copies artifacts to shared folders


How to Test Pipeline If Pipeline becomes complex and has multiple steps, then

testing it become time consuming as well

Currently Jenkins Pipeline does not have any support for testing or Dry-Run mode (see feature request)

To solve the problem we implemented a simple unit-testing module (testing.groovy) for the pipeline using xUnit principles

Methods like assertEquals() or assertTrue() were added. If assertion fails, an exception is thrown (see the next slide)

https://issues.jenkins-ci.org/browse/JENKINS-33925


Assert Methods in testing.groovyvoid assertEquals(expected, actual, msg = null){ if (actual == expected) { return } else { throwException(expected, actual, msg) }}

void assertTrue(actual, msg = null)

void assertContains(subString, string, msg = null)

…


Test Example

testing.test("Get Maven artifact coordinates") { given: //just a label to structure code (see also Spock) def pomXml = ''' <groupId>de.medavis</groupId> <artifactId>project-a</artifactId> <version>1.0.0.1</version> ''' when: def coordinates = maven.getCoordinates(pomXml)

then: testing.assertEquals('de.medavis', coordinates.groupId) testing.assertEquals('project-a', coordinates.artifactId) testing.assertEquals('1.0.0.1', coordinates.version)}

Sample test of maven.getCoordinates() function:

http://spockframework.github.io/spock/docs/1.1-rc-1/all_in_one.html#_blocks


How to Test Pipeline We also implemented a simple test reporting: if an

exception was thrown in test, it is marked as failed:


New UI –Blue Ocean Jenkins will soon get a new UI optimized for pipelines:

Blue Ocean – see the following slides

It can be installed from the experimental update center: http://updates.jenkins-ci.org/experimental/update-center.json


Blue Ocean: Parallel Execution


Blue Ocean: Console Output