pimp your continuous delivery pipeline with jenkins workflow (w-jax 14)

©2014 CloudBees, Inc. All Rights Reserved

Pimp your Continuous Delivery Pipeline with Jenkins workflow Cyrille Le Clerc, CloudBees


About Me

Cyrille Le Clerc

@cyrilleleclerc

Open Source

CTO

DevOps, Infra as Code, Con9nuous Delivery

Product Management


About you

Jenkins? CI? CD? Code Jenkins plugin?


Agenda

•  Con9nuous Delivery •  Jenkins & Con9nuous Delivery •  Workflows in Jenkins today •  The new Jenkins Workflow Engine •  Jenkins Workflow Syntax Card •  Possible future


Continuous Delivery


Continuous Delivery

•  “Reliable So+ware Releases through Build, Test, and Deployment Automa=on”

•  DONE = Shippable into Produc9on


Continuous Integration Steps

Con$

nuou

s Integra$o

n Source Code

Quality Analysis

Compile &Unit Tests

Binaries


Continuous Delivery Steps

Con$

nuou

s Delivery

DEV

DEV

OPS

QA

Source Code

Quality Analysis

Compile &Unit Tests

Binaries

Perfs Environment

Func Test Environment

UAT Environment

Production Environment

GO LIVE!

Release!

Staging Environment


Continuous Delivery Pipeline


Compile &Unit Tests

Perfs Tests

Selenium Tests Deploy UAT

Sonar Analysis

Deploy Staging Deploy Production


Jenkins: Hub of Continuous Delivery

•  Connect the dots

•  It is the process Source Code

Perfs Environment

Func Test Environment

UAT Environment

Binaries

Quality Analysis

Compile &Unit Tests


CD Pipeline è Workflow


Compile &Unit Tests

Perfs Tests


Sonar Analysis



What we need

•  Complex pipelines involving mul9ple stages •  Non-‐sequen$al logic such as loops, forks … •  Long-‐running builds must survive outages •  Interac$ons with humans including pauses, input •  Restartable builds in case of transient errors •  Reusable defini$ons to avoid duplica9on •  Comprehensible scripts with one clear defini9on


Workflow until today

•  Job chaining

•  Build Flow Plugin

•  Build Pipeline Plugin

•  Build Delivery Pipeline


Workflow until today

•  Many atomic jobs •  Hard to share variables/state between jobs •  Limited logic •  Mix build triggers, parameterized build …


Build Flow Plugin

•  Did have scriptability and extensibility •  Did not address configura9on “sprawl” •  Disjointed view of what really ran •  No ability to survive restarts •  Almost good enough but could not go further


Jenkins Workflow Engine


Simplifying our sample workflow

Compile &Unit Tests

Perfs Tests


Sonar Analysis



What we need

•  Complex pipelines involving mul9ple stages •  Non-‐sequen$al logic such as loops, forks … •  Long-‐running builds must survive outages •  Interac$ons with humans including pauses, input •  Restartable builds in case of transient errors •  Reusable defini$ons to avoid duplica9on •  Comprehensible scripts with one clear defini9on


New Workflow Syntax


Key features

•  En9re flow is one concise Groovy script –  For loops, try-‐finally, fork-‐join …

•  Can restart Jenkins while flow is running •  Allocate slave nodes and workspaces

–  As many as you want, when you want

•  Stages throYle concurrency of builds •  Human input/approval integrated into flow •  Standard project concepts: SCM, ar9facts, plugins


Groovy DSL vs. Graphical Workflow

•  Familiar control flow construc9on •  Familiar “tools” for building abstrac9ons

Func9ons, classes, variables, …

•  Workflow in version control •  As opposed to:

Graphical workflow designer


Project setup

•  One workflow is defined as a job •  Single script for all steps •  Build triggers & parameters like regular projects •  Each workflow execu9on is a regular Jenkins build displayed in regular Jenkins views

•  Graphical visualiza9on of actual build possible Not of visualiza9on job defini9on but of build execu9on


Non-sequential logic

try {} catch {}


Interaction with humans


Restartable build / checkpoint


Visualization


Stages

•  Semaphore •  Visualiza9on


Comprehensible scripts


Demo time!


Workflow Syntax Card



•  stage: Enter a new stage •  node: Allocate node •  ws: Allocate workspace •  parallel: Execute sub-‐workflow in parallel •  retry: Retry the body up to N 9mes •  catchError: Catch error and con9nue •  input: Input / human interac9on •  load: Evaluate a Groovy source file into the workflow script



•  step: General build step •  sh: Shell script •  bat: Windows batch script



•  archive: Archive ar9facts •  unarchive: Copy archived ar9fact into the workspace •  echo: Print message •  dir: Change directory •  readFile: Read file from workspace •  writeFile: Write file to workspace



•  git: Git •  svn: Subversion •  scm: General SCM •  tool: install a tool •  build: Build a job



•  checkpoint: capture the state of the execu9on so that it can be restarted later


Possible Future


Possible Future: probably OSS

•  More plugin interoperability •  Mul9-‐branch project (& SCM API) integra9on •  More advanced SCM •  In-‐IDE edi9ng support •  Concise syntax


Possible Future: probably Jenkins Enterprise

•  More visualiza9ons •  Validated merge integra9on •  Deployment, incl. blue/green with rollback •  Freestyle and/or Build Flow import •  Templates integra9on


Get the code

•  hYps://gist.github.com/cyrille-‐leclerc/796085e19d9cec4a71ef – workflow.groovy

pimp your continuous delivery pipeline with jenkins workflow (w-jax 14)

Technology

rights reservedwhat

cloudbees2014 cloudbees

today2014 cloudbees

manyatomicjobs2014 cloudbees

sample workflow2014

staging environment

code jenkins plugin