juc europe 2015: hey! what did we just release?

Jenkins World Tour 2015London, UK, June 2015

1

Footer

#jenkinsconf

Hey! What Did We Just Release?

Robert McNulty

Experian Marketing Services

[email protected]

http://www.experian.com/marketing-services

2

Footer

#jenkinsconf

Agenda

● Introduction● Things As They Are● Things As They Should Be● Demonstration● Q & A

3

Footer

#jenkinsconf

About Me

• 20+ years in software development and database design

• Lead Technical Developer and Architect at Experian Marketing Services

• Lead and mentor a global team of developers➢ United States➢ Costa Rica

4

Footer

#jenkinsconf

About Me

• Not a dedicated build engineer. It’s just one of the many hats I wear at work.

• Started using Continuous Integration with CruiseControl

• Moved to Hudson after viewing a session at JavaOne by the creator, Kohsuke

• Migrated to Jenkins after the Hudson split

5

Footer

#jenkinsconf

Problem Statement

• In an out-of-the-box Jenkins parameterized build scenario, user input such as notes, Git branches, and bug tracking issues are lost once a build has completed.

• The only historical reference, to most Project Managers is in a report.

• This data cannot easily be reused for future builds. • Using Jenkins, Groovy and Neo4j, the data can be

persisted for the life of a project.

6

Footer

#jenkinsconf

Standard Email Notification

• The standard post-build email notification only runs on failure and subsequent success

• The failure message contains a portion of the console output

• The success message is not much to look at either

7

Subject: Jenkins build is back to normal : hello-world #6

See <http://localhost:8080/jenkins/job/hello-world/6/changes>

Footer

#jenkinsconf

Standard Email-ext Success Email

• Sample body content of unmodified email-ext plugin email

• This is what is contained in the body of the email message sent by an unmodified project.

• Not exactly user-friendly.

8

juc-hello-world - Build # 5 - Successful: Check console outputat http://localhost:8180/jenkins/job/juc-hello-world/5/ to view the results.

#jenkinsconf

Footer

Standard Success Email – Multiple Builds

Clicking on the link, the information that the user entered for this build can be viewed:

Easy enough, but not very helpful.

Footer

#jenkinsconf


• The information is not so easily available to subsequent builds. The user must open each build individually to see what information had been entered.

• For example, a user has run three builds and included notes for each build. If the email has just been received for build #7, clicking on the link will take the user to the build page where the user notes can be viewed.

10

Footer

#jenkinsconf


However, if the user wants to see the notes of all the builds to date, each build must be opened one at a time to view the notes for that build. What a hassle!

11

Footer

#jenkinsconf


And if the build administrator has chosen to discard old builds and limit the number of builds to keep – good luck with that.

12

Footer

#jenkinsconf

#jenkinsconf

There Is A Better Way!!

13

#jenkinsconf

Footer

Enhanced Success Email

Wouldn’t it be nice to receive an email with this formatted content instead?

#jenkinsconf

Footer

Enhanced Success Email

That looks just as good on a mobile device!

#jenkinsconf

Footer

User Interface Possibilities

Wouldn’t it be nice to save, review, reconstruct and print reports from any and all builds via a spiffy user interface?

Footer

#jenkinsconf

User Interface Possibilities

17

Footer

#jenkinsconf

#jenkinsconf

And this…

Is how… it's done!

18

#jenkinsconf

Footer

Components Used for this Jenkins Job

Main Components Jenkins Plugins

Git Git plugin Build User Vars plugin

Groovy Groovy plugin EnvInject plugin

Gradle Gradle plugin Workspace Cleanup plugin

Artifactory Artifactory plugin Export Parameters plugin

Database (Neo4j) Email-ext plugin Job Exporter plugin

Credentials plugin Deploy plugin

Multiple SCMs plugin

#jenkinsconf

Footer

Database – Why Neo4j?

● Previous versions used MySQL● Adding projects with different build data requirements resulted in a non-

cohesive database, sparse cells● In Neo4j, a node can contain an arbitrary number of properties● The need for join tables is eliminated

#jenkinsconf

Footer

Database – Graph Model

● Neo4j is an open-source, NoSQL graph database implemented mainly in Java and Scala.

● The fundamental units that form a graph are nodes and relationships.

● Nodes are often used to represent entities

● Relationships organize the nodes by connecting them

#jenkinsconf

Footer

Database - Schema

Nodes Relationships

Build

SubProject OF_PROJECT

User BUILT_BY

Server DEPLOYED_TO

Property HAS_PROPERTY

Tag TAGGED_AS

● This is the base graph model the Jenkins application uses:

#jenkinsconf

Footer

Database – Node Properties

SubProject Server User Tag Property

name host name name name

title port email version value

artifactId userId

artifactExt

artifactDir

gitRepoName

buildTasks

context

Nodes and relationships can hold any number of individual attributes (key-value pairs)

Footer

#jenkinsconf

Cypher Query Language

• Neo4j uses Cypher as a query language.• Queries can be execute interactively via the built-in UI

OR via the REST API• Cypher borrows its structure from SQL — queries are

built up using various clauses such as: MATCH (This is the most common way to get data from the graph) WHERE ORDER BY RETURN

24

#jenkinsconf

Footer

Database – Create SubProject Node

MATCH (p1:Project) WHERE p1.name = "juc"CREATE (s1:SubProject { name: "juc-hello-world", title: "Jenkins User Conference Test", artifactId: "juc-hello-world", artifactExt: "war", artifactDir: "build", gitRepoName: "juc-hello-world", buildTasks: "clean war", context: "/hello-world"}),(p1)-[:HAS_SUBPROJECT]->(s1)

1.Get the Project node (p1)

2.Create lookup node SubProject (s1)

3.Add properties to node

4.Create a relationship between the Project and the SubProject

#jenkinsconf

Footer

Database – Query SubProject Node

• Neo4j provides a web interface on port 7474

• Queries can be executed interactively and the resulting graphs displayed

• For the SubProject node previously created, the following graph is displayed

• MATCH (s:SubProject)<-[HAS_SUBPROJECT]-(n) RETURN s,n

#jenkinsconf

Footer

Database – SubProject Node

• The node data can also be viewed in the web interface by toggling the view button

#jenkinsconf

Footer

Database – 'Build' Node

• Query the Build node inserted by the Jenkins job

MATCH (b:Build)-[]-(n)RETURN b,n

#jenkinsconf

Footer

Database – 'Build' Node

• Running the same query after running a second build yields both builds and associated nodes

MATCH (b:Build)-[]-(n)RETURN b,n

#jenkinsconf

Footer

Database – Query with Groovy

def neo4jQuery = “”” MATCH (u:User) WHERE u.id = '$userId' MATCH (s:SubProject) WHERE s.name = '$subProjectName' MATCH (v:Server) WHERE v.host = '$host' CREATE (t:Tag {name: '$tag', version: '$releaseVersion'}) CREATE (p1:Property {name: 'NOTE', seq: 1, value: '$note'}) CREATE (b:Build { number: '$buildNo', started: '${new Date()}', gitBranch: '$gitBranch' }), (b)-[:BUILT_BY]->(u), (b)-[:OF_PROJECT]->(s), (b)-[:DEPLOYED_TO]->(v), (b)-[:TAGGED_AS]->(t), (b)-[:HAS_PROPERTY]→(p1) RETURN ID(b)“””

1.Create the CYPHER query as a regular string

2.Variable substitution is supported

3.Return the unique ID of the newly created Build node

#jenkinsconf

Footer

Database – Query with Groovy

import groovy.json.JsonBuilderimport groovy.json.JsonSlurperimport groovyx.net.http.*

def server = new RESTClient("http://localhost:7474/db/data/cypher")server.headers['Accept'] = 'application/json; charset=utf-8'server.headers['Content-Type'] = 'application/json'server.contentType = JSON

def json = new JsonBuilder()json query: neo4jQuery

def results = server.post(body: json.toString())return results

1.Create a RESTClient to the Neo4j server

2.Set http headers to send and accept JSON

3.JSON is POSTed to the server

Footer

#jenkinsconf

#jenkinsconf

Putting it All Together

32

#jenkinsconf

Footer

Configuring the Build Job

Enable Parameterized Build• This example will only define 2

parameters:➢ GIT_BRANCH

➢ NOTES

#jenkinsconf

Footer


Git Setup

Source Code Management

• In ‘Branches to build’, enter the parameter➢ $GIT_BRANCH

• The variable will be replaced by the value entered by the user, defaulting to ‘master’. This is the branch that will be checked out and built.

#jenkinsconf

Footer


Git Setup – cont.

Check out to a sub-directory

• Since we will also be checking out the Groovy code from a different repository, it is important to keep the projects separated.

#jenkinsconf

Footer


Environment Variables

Inject environment variables to the build process

• Define global properties, used in both the Jenkins job and the Groovy script, in the ‘Property content’ section.

#jenkinsconf

Footer


Inject Passwords

Inject passwords into the build as environment variables

• Passwords are not included in the regular environment variable section because they would then be visible to end-users via the logs.

• Here, we are defining the database password that will be passed to the Groovy script.

#jenkinsconf

Footer


Build Steps

Execute Shell script

• The first step is to execute inline Bash commands to save the build properties to an external file.

• This makes it easier to pass properties back and forth between the Jenkins job and the Groovy script.

• Especially useful when there are many properties

#jenkinsconf

Footer


Build StepsExecute Shell script

#!/bin/bashcd $WORKSPACEecho "APPLICATION=juc-hello-world" >> $PROP_FILEecho "GIT_BRANCH=$GIT_BRANCH" >> $PROP_FILEecho "PROP_FILE=$PROP_FILE" >> $PROP_FILEecho "SCRIPT_DIR=$SCRIPT_DIR" >> $PROP_FILEecho "WORKSPACE=$WORKSPACE" >> $PROP_FILE# -----------------------------------------USER_NOTES=`echo "$NOTES" | sed -e '$ ! s|$.*$$|\1~|' | tr '\n' ' '`echo "NOTES=$USER_NOTES" >> $PROP_FILE

#jenkinsconf

Footer


Build Stepsexport job runtime parameters

● This build step exports various build job and Jenkins parameters into a property file named hudsonBuild.properties in the project workspace.

● The external Groovy script will read this file and load several of the properties into the master properties file

Property Value

build.user.id ID of user that started this job

build.user.name User that started this job

build.user.fullName Full name of user that started this job

build.user.emailAddress Email address of user that started this job

#jenkinsconf

Footer


Workspace

View of the job workspace after the Git repositories have been cloned and the property files created

#jenkinsconf

Footer


Execute Groovy Script● The Groovy script is called twice, first , using the ‘preBuild’

action and finally using the ‘postBuild’

• ‘preBuild’ is executed prior to running the Gradle build script

• ‘postBuild’ is executed after the successful completion of the Gradle build

#jenkinsconf

Footer


Execute Groovy Script - preBuild

This script will:1. Load the properties files ($PROP_FILE)2. Query the database for the project and server information3. Append database values as new properties back to the file

#jenkinsconf

Footer


Execute Groovy Script - preBuildRC_PROPERTIES_FILE=juc.propertiesGIT_BRANCH=origin/masterARTIFACT_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-world//build/libARTIFACT_ID=hello-worldARTIFACT_TYPE=warBUILD_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-worldBUILD_NUMBER=90BUILD_USER_FULLNAME=Robert McNultyCONTEXT_PATH=/hello-worldEMAIL_SUBJECT=JUC Hello World AppGIT_REPOSITORY_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-worldGIT_REPO_NAME=juc-hello-worldPROJECT_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-world/SERVER_HOST=localhostSERVER_PORT=8080ARTIFACTORY_VERSION=1.0.0-20140519.034403-8RC_TAG=JUC_HELLO_WORLD_2_0_0_SNAPSHOT_0071USER_NOTES_HTML=<li>BUILD 5 : Initial build of hello-world project.</li><li>BUILD 6 : Fixed a problem in the doohickey.</li><li>BUILD 7 : Working on thingamajig refactoring.</li>

#jenkinsconf

Footer


Inject environment variables● After the properties have been written to the file, it is necessary

to reload the file into Jenkins so the build can use the properties

● The ‘Inject environment variables’ plugin accomplishes this task

● The properties that were retrieved from the database are now available to the Jenkins job

#jenkinsconf

Footer


Run the Gradle Build● Build the project using the Gradle plugin

➢ Note: Maven can be used in place of Gradle

● The $BUILD_TASKS and $PROJECT_DIR variables were loaded from the properties file

#jenkinsconf

Footer


Execute Groovy Script - postBuild

This script will:1. Load the properties files ($PROP_FILE)2. Get artifact information (groupId, version)3. Create unique tag name4. Aggregate notes from previous builds

5. Create HTML elements to use in report6. Create and push new branch to Git

(using tag name)7. Append new properties back to

properties file

Footer

#jenkinsconf

#jenkinsconf

Post-build Actions …and more!

48

#jenkinsconf

Footer


Post-build actions

Once the Gradle build has completed, the artifacts deployed to Artifactory and the Groovy scripts finished processing, there are still tasks that are required to successfully complete the Jenkins job

• Deploy artifact to Tomcat (if webapp)• Send the Email

#jenkinsconf

Footer


Execute Groovy Script - postBuild

● For this demo, I used the ‘Deploy Plugin’ for simplicity● Unfortunately, this plugin does not recognize the environment variables imported into Jenkins

#jenkinsconf

Footer


Deploy to Tomcat● In our production instance

of Jenkins, I deploy to Tomcat via the postBuild Groovy script

● Since our Jenkins instance runs on Linux, it is a simple matter to use ‘curl’ to access the Tomcat manager and list, deploy or undeploy applications

static boolean deploy(context, path) { def curl = new StringBuilder().with { append "curl -X PUT -F file=@$path “ append ‘http://jenkins@localhost:8180’ append “/manager/text/deploy?path=$context" }.toString() def result = curl.execute().text return result.startsWith('OK')}

#jenkinsconf

Footer


Editable Email Notification● The ‘piece de resistance’ is the email report delivered to a list of

interested recipients• The ‘email-ext plugin’ does have the ability to use the imported

environment variables• Using the HTML Content Type enables the use of basic HTML and

CSS to format the page• The ‘dynamic’ portions of the email are populated using the

properties generated by the Groovy script and imported into Jenkins

#jenkinsconf

Footer


Editable Email NotificationUse HTML and CSS in the report template

#jenkinsconf

Footer


Editable Email Notification

Use HTML and CSS in the report template

● If the build is successful, configure the ‘Success’ trigger

Footer

#jenkinsconf

#jenkinsconf

Demonstration

55

Footer

#jenkinsconf

#jenkinsconf

Questions? Answers!

56

#jenkinsconf

Please Share Your Feedback

• Did you find this session valuable?• Please share your thoughts in the

Jenkins User Conference Mobile App.• Find the session in the app and click

on the feedback area.

57

juc europe 2015: hey! what did we just release?

Technology