building a continuous delivery with gradle and jenkins

© 2014 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Continuous Delivery with Gradle and Jenkins

By Peter Niederwieser

Releases don’t have to be painful

Con4nuous Delivery

Deliver so8ware fast and frequently

#1 Every commit can result in a release


#2 Automate everything!



#3 Automated tests are essen4al



#3 Automated tests are essen4al

#4 Done means released

Build pipeline

Automated manifesta4on of delivery process

Build quality in!

Establish automated quality gates

Compile/Unit Tests

Compile/Unit Tests

Integra4on Tests

Compile/Unit Tests

Integra4on Tests

Code Analysis

Test

Compile/Unit Tests

Integra4on Tests

Code Analysis

Package/Deploy

Test

Compile/Unit Tests

Integra4on Tests

Code Analysis

Package/Deploy

Acceptance Tests

Test

Compile/Unit Tests

Integra4on Tests

Code Analysis

Package/Deploy

UAT

Acceptance Tests

Test

Compile/Unit Tests

Integra4on Tests

Code Analysis

Package/Deploy

UAT Prod

Acceptance Tests

! But how?

The “revolu4onary” sample applica4on

Internet To Do applicationBrowser Data

store

Writes

Reads

Mul4-‐project dependencies

Model

Web Repository

dependdepend

depend

Project hierarchytodo

model

repository

build.gradle

settings.gradle

web

build.gradle

build.gradle

build.gradle

gradle

wrapper

xyz.gradle

gradlew

gradlew.bat


model

repository

build.gradle

settings.gradle

web

build.gradle

build.gradle

build.gradle

gradle

wrapper

xyz.gradle

gradlew

gradlew.bat

Define project-‐specific behavior

Project hierarchy

include 'model'include 'repository'include 'web'

settings.gradle

Defines which projects are taking part in the build

todo

model

repository

build.gradle

settings.gradle

web

build.gradle

build.gradle

build.gradle

gradle

wrapper

xyz.gradle

gradlew

gradlew.bat


model

repository

build.gradle

settings.gradle

web

build.gradle

build.gradle

build.gradle

gradle

wrapper

xyz.gradle

gradlew

gradlew.bat

Always use Wrapper to execute the build!


model

repository

build.gradle

settings.gradle

web

build.gradle

build.gradle

build.gradle

gradle

wrapper

xyz.gradle

gradlew

gradlew.bat

Externalize concerns into script plugins and organize them in dedicated directory


model

repository

build.gradle

settings.gradle

web

build.gradle

build.gradle

build.gradle

gradle

wrapper

xyz.gradle

gradlew

gradlew.bat

Externalize concerns into script plugins and organize them in dedicated directory

Examples: !! Versioning strategy ! Integra4on and func4onal test setup ! Deployment func4onality ! ...

Project ar4facts

rootproject

modelproject

repositoryproject

webproject

JAR

JAR

WAR

Project ar4facts

rootproject

modelproject

repositoryproject

webproject

JAR

JAR

WAR

Deployable ar4fact

Stages in build pipeline

Acceptance stage

Functional tests

PublishBinaries

Commit stage

DeployBinaries

UAT

DeployBinaries

Production

IntegrationTests

Code Analysis

AssembleDistribution

CompileUnit Tests

RetrieveBinaries

DeployBinaries

Asserts that system works at a technical level


Acceptance stage

Functional tests

PublishBinaries

Commit stage

DeployBinaries

UAT

DeployBinaries

Production

IntegrationTests

Code Analysis


CompileUnit Tests

RetrieveBinaries

DeployBinaries

Asserts that system works on a func4onal/non-‐func4onal level


Acceptance stage

Functional tests

PublishBinaries

Commit stage

DeployBinaries

UAT

DeployBinaries

Production

IntegrationTests

Code Analysis


CompileUnit Tests

RetrieveBinaries

DeployBinaries

Trigger manuallyTrigger manually

Commit stage: Compile/unit tests

Rapid feedback (< 5 mins)

Run on every VCS check-‐in

Priority: fix broken build

PublishBinaries

Commit stage

CompileUnit Tests

IntegrationTests

Code Analysis


Commit stage: Integra4on tests

Longer running tests

Require environment setup

Harder to maintain

PublishBinaries

Commit stage

CompileUnit Tests

IntegrationTests

Code Analysis


Separate tests in project layout

todo

model

repository

web

src

integTest

java

main

java

test

java

Produc4on Java sources

Unit test Java sources

Separate tests in project layout

todo

model

repository

web

src

integTest

java

main

java

test

java

Integra4on test Java sources

Produc4on Java sources

Unit test Java sources

Separate tests with SourceSets

sourceSets { integrationTest { java.srcDir file('src/integTest/java') resources.srcDir file('src/integTest/resources') compileClasspath = sourceSets.main.output + configurations.testRuntime runtimeClasspath = output + compileClasspath } } !task integrationTest(type: Test) { description = 'Runs the integration tests.' group = 'verification' testClassesDir = sourceSets.integrationTest.output.classesDir classpath = sourceSets.integrationTest.runtimeClasspath testResultsDir = file("$testResultsDir/integration") }



Set source and resources directory




Set compile and run4me classpath



Custom test results directory





Custom test results directory



gradlew integrationTest

Database integra4on tests

stopDatabasetest start

Databasebuild

Schemaintegration

Test check build...


apply from: "$rootDir/gradle/databaseSetup.gradle" !integrationTest.dependsOn startAndPrepareDatabase integrationTest.finalizedBy stopDatabase !check.dependsOn integrationTest


Databasebuild

Schemaintegration

Test check build...




Databasebuild

Schemaintegration

Test check build...

Separate complex setup logic into script plugin




Databasebuild

Schemaintegration

Test check build...

Separate complex setup logic into script plugin

Integrate tasks into build lifecycle

Picking the “right” code coverage tool


Cobertura Offline bytecode instrumenta4on



Offline bytecode instrumenta4on



Source code instrumenta4on




Source code instrumenta4on


On-‐the-‐fly bytecode instrumenta4on

On-‐the-‐fly bytecode instrumenta4on

No modifica4on to source or bytecode

Code coverage with JaCoCo

apply plugin: "jacoco"!task jacocoIntegrationTestReport(type: JacocoReport) { sourceSets sourceSets.main executionData integTest}



Apply JaCoCo plugin



Apply JaCoCo plugin

Also report code coverage for integra4on tests

Genera4ng coverage reports

test integrationTest

... ... .exec



... ... .exec

jacocoTestReport

jacocoIntegrationTestReport

.exec .html



... ... .exec

jacocoTestReport

jacocoIntegrationTestReport

.exec .html

.html

Commit stage: Code analysis

PublishBinaries

Commit stage

CompileUnit Tests

IntegrationTests

Code Analysis


Perform code health check

Fail build for low quality

Record progress over 4me

Sta4c code analysis tools

Checkstyle

FindBugs

apply plugin: 'pmd' pmd { ignoreFailures = true } tasks.withType(Pmd) { reports { xml.enabled = false html.enabled = true } }

apply plugin: 'jdepend’ jdepend { toolVersion = '2.9.1' ignoreFailures = true }

gradlew check

Measure quality over 4me with Sonar

Sonardatabase

Gradle

Sonar Runner

root

model

repo.

web

analyzes

publishes reads / writes

Gradle project

Applying the Sonar Runner plugin

apply plugin: "sonar-runner"!sonarRunner { sonarProperties { property "sonar.host.url", "http://my.server.com" property "sonar.jdbc.url", "jdbc:mysql://my.server.com/sonar" property "sonar.jdbc.driverClassName", "com.mysql.jdbc.Driver" property "sonar.jdbc.username", "Fred Flintstone" property "sonar.jdbc.password", "very clever" }}!subprojects { sonarRunner { sonarProperties { property "sonar.sourceEncoding", "UTF-8" } }}

gradlew sonarRunner

http://my.server.com

Commit stage: Assemble distribu4on

Exclude env. configura4on

Include build informa4on

Choose versioning strategy

PublishBinaries

Commit stage

CompileUnit Tests

IntegrationTests

Code Analysis


Versioning strategy

1.0-‐SNAPSHOT 1.0

during development when released

…the Maven way

Change version with Maven Release plugin

Versioning strategy

1.0.134 1.0.134

during development when released

…the Con4nuous Delivery way

1.0.134Jenkins build numberProject version number

Versioning strategy…implemented with Gradle

allprojects { apply from: "$rootDir/gradle/versioning.gradle" }

todo

model

repository

build.gradle

settings.gradle

web

gradle

versioning.gradle Contains version implementa4on


ext.buildTimestamp = new Date().format('yyyy-MM-dd HH:mm:ss') version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER) class ProjectVersion { Integer major Integer minor String build ProjectVersion(Integer major, Integer minor, String build) { this.major = major this.minor = minor this.build = build } @Override String toString() { String fullVersion = "$major.$minor" if(build) { fullVersion += ".$build” } fullVersion } }



Jenkins Build Number



Jenkins Build Number

Builds version String representa4on

Packaging the deployable ar4fact

project(':web') { apply plugin: 'war' task createBuildInfoFile << { def buildInfoFile = new File("$buildDir/build-info.properties") Properties props = new Properties() props.setProperty('version', project.version.toString()) props.setProperty('timestamp', project.buildTimestamp) props.store(buildInfoFile.newWriter(), null) } war { dependsOn createBuildInfoFile baseName = 'todo' from(buildDir) { include 'build-info.properties' into('WEB-INF/classes') } } }



Creates file containing build informa4on




Include build info file Into WAR distribu4on




Include build info file Into WAR distribu4on

gradlew assemble

Commit stage: Publish binaries

Version ar4fact(s)

Use binary repository

Publish once, then reuse

PublishBinaries

Commit stage

CompileUnit Tests

IntegrationTests

Code Analysis


Publishing the deployable ar4fact

1.0.34

1.0.32 1.0.33 1.0.34

Defining build configura4onbinaryRepository { url = 'http://mycompany.bin.repo:8081/artifactory' username = 'admin' password = 'password' name = 'libs-release-local' } environments { test { server { hostname = 'mycompany.test' port = 8099 context = 'todo' username = 'manager' password = 'manager' } } uat { server { hostname = 'mycompany.uat' port = 8199 context = 'todo' username = 'manager' password = 'manager' } } ...


Common configura4on


Environment-‐specific configura4on

Common configura4on


Environment-‐specific configura4on

Common configura4on Read creden4als

from gradle.proper4es

Read creden4als from gradle.proper4es

Read creden4als from gradle.proper4es

Reading build configura4on

def env = project.hasProperty('env') ? project.getProperty('env') : 'test' logger.quiet "Loading configuration for environment '$env’." !def configFile = file("$rootDir/gradle/config/buildConfig.groovy") def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL()) ext.config = parsedConfig

Initializationphase

Congurationphase

Executionphase



Initializationphase

Congurationphase

Executionphase

Assign configura4on to extra property



Initializationphase

Congurationphase

Executionphase

Assign configura4on to extra property

gradlew –Penv=uat ...

Using the Maven Publishing plugin

apply plugin: 'maven-publish'

def fullRepoUrl = "$config.binaryRepository.url/$config.binaryRepository.name” publishing { publications { webApp(MavenPublication) { from components.web } } repositories { maven { url fullRepoUrl credentials { username = config.binaryRepository.username password = config.binaryRepository.password } } } }




Build repository URL from configura4on





Assign publica4on name and component





gradlew publish

Assign publica4on name and component

Acceptance stage: Retrieve binaries

Request versioned ar4fact

Store in temp. directory

Acceptance stage

DeployBinaries

Functional Tests

RetrieveBinaries

Downloading the deployable ar4fact

1.0.34

1.0.32 1.0.33 1.0.34

1.0.34

Test

UAT

Prod

Task for downloading ar4fact

repositories { maven { url fullRepoUrl }}!configurations { war}!dependencies { war "$project.group:$project.name:$project.version"}!task downloadBinaryArchive(type: Copy) { from configurations.war into "$buildDir/download" }

Task for downloading ar4fact

repositories { maven { url fullRepoUrl }}!configurations { war}!dependencies { war "$project.group:$project.name:$project.version"}!task downloadBinaryArchive(type: Copy) { from configurations.war into "$buildDir/download" }

Target loca4on for downloaded ar4fact

gradlew downloadBinaryArchive

Acceptance stage: Deploy binaries

Deployment on request

Make it a reliable process

Acceptance stage

DeployBinaries

Functional Tests

RetrieveBinaries

Use process for all envs

Deploying to mul4ple environments

Test

UAT

Prod

–Penv=prod

–Penv=uat

–Penv=test

Deployment with the Cargo plugin

cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote

cargoUndeployRemote { onlyIf appContextStatus } cargo { containerId = 'tomcat7x' port = config.server.port deployable { file = downloadedArtifact context = config.server.context } remote { hostname = config.server.hostname username = config.server.username password = config.server.password } }




Download ar4fact from binary repository and undeploy exis4ng one





Only undeploy if URL context exists






Use environment-‐specific configura4on






Use environment-‐specific configura4on

gradlew –Penv=uat cargoDeploy

Acceptance stage: Func4onal tests

Test UI permuta4ons

Test important use cases

Acceptance stage

DeployBinaries

Functional Tests

RetrieveBinaries

Run against different envs

Execu4ng func4onal tests

def functionalTestReportDir = file("$testReportDir/functional") def functionalTestResultsDir = file("$testResultsDir/functional") def functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")] !task functionalTest(type: Test) { testClassesDir = sourceSets.functionalTest.output.classesDir classpath = sourceSets.functionalTest.runtimeClasspath testReportDir = functionalTestReportDir testResultsDir = functionalTestResultsDir systemProperties functionalCommonSystemProperties systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/" }



Reuse setup proper4es



Build URL from env. configura4on




gradlew –Penv=test remoteFunctionalTest

Build URL from env. configura4on


Let’s bring Jenkins into play!

Test

UAT

Prod

Deployment Acceptance Tests

Publish

Download

Trigger Build

Pull Source Code

Model pipeline as series of jobs

Model pipeline as series of jobs

Triggered job when change to SCM is detected

Ini4al Jenkins Build Job

Build Name Seker Plugin

JaCoCo Plugin

Parameterized Trigger Plugin

Gradle Plugin

Always use the Wrapper!

Gradle Plugin

Always use the Wrapper!

Gradle Plugin

Run clean task to remove exis4ng ar4facts

Clearly iden4fy a build through an expressive build name


Use Jenkins environment variable

Clearly iden4fy a build through an expressive build name


Next job to trigger if build is stable


Next job to trigger if build is stable

Defines build number parameter provided to subsequent jobs


Clone Workspace SCM Plugin

Archive all files


Archive all files

Only archive if build was successful


JaCoCo Plugin

Point to separated test results

JaCoCo Plugin


JaCoCo Plugin

Point to JaCoCo files as well as source and class files


Fail build of quality gate criteria are not met

JaCoCo Plugin

Point to JaCoCo files as well as source and class files

Clone Workspace SCM Plugin Build Name Seker Plugin

Reuse ini4al workspace


Reuse ini4al workspace

Reuse ini4al build number


Build Pipeline Plugin

Define the target environment


Define the target environment

Downstream job that requires manual execu4on



Visualiza4on of chained pipeline jobs

> gradle qa

:askQuestions

!

BUILD SUCCESSFUL

!

Total time: 300 secs

building a continuous delivery with gradle and jenkins

Software

gradlerepo versioningstrategy

continuous delivery