building a solr continuous delivery pipeline with jenkins: presented by james strassburg, direct...

O C T O B E R 1 1 - 1 4 , 2 0 1 6 • B O S T O N , M A

Building a Solr Continuous Delivery Pipeline With Jenkins James Strassburg

Senior Software Architect, Direct Supply

http://bit.ly/SolrCD

Links to Repositories

•  Overview of Continuous Delivery

•  Jenkins Delivery Pipeline as Code

•  Packaging / Deploying Solr with a Pipeline

•  Deploying Solr Configuration with a Pipeline

Objectives

Solr – Where We’re Going

Solr Configuration – Where We’re Going

•  Deployments are predictable and routine

•  Every change is deployable

•  Reduce work-in-progress (lean)

•  Poka-Yoke – mistake proofing

Continuous Delivery

Plugins: Build Pipeline, Copy Artifacts

Jenkins < 2.x

•  Required separate builds for each stage

•  Not version controlled

•  Complicated build configuration

•  More work for me ;)

Jenkins < 2.0

•  Jenkinsfile

•  Groovy script that defines your pipeline

•  Versioned in your repository

•  Vagrantfile, Dockerfile, or .travis.yml (Travisfile?)

Jenkins >= 2.x Pipelines as Code

node { stage ‘build’ sh ‘echo building...’ stage ‘test’ sh ‘echo running tests...’ stage ‘deploy’ sh ‘echo deploying...’ }

Jenkinsfile Example

BUILD

TEST

DEPLOY

•  Fork of lucene-solr repo

•  A repo for Solr configuration

•  A repo with config for a Jenkins Docker Image

Example Implementation

•  Clone my solr-jenkins-cicd repo and…

$ ./start_jenkins_and_solr.sh Then browse to http://localhost:8080 admin:admin

Starting Jenkins

•  Apply bug fixes (from JIRA) without waiting for an official release

•  Building helps you understand the code

Why Build / Package Solr

stage “Build Solr Package” env.PATH = "${tool 'ant'}/bin:${env.PATH}” checkout scm sh 'ant ivy-bootstrap’ sh 'ant -Dversion.suffix=${BUILD_NUMBER}\ create-package -f solr/build.xml’ archive 'solr/package/**/*.tgz'

Building Solr

stage "Deploy Solr - Stage Env” sh 'scripts/deploy_server.sh 10.0.0.11\ solr /usr/local/solr 8983 ~/stage_private_key'

Deploy Solr

Deploy Solr – There will be no demo

${BUILD_NUMBER}

•  Collections

•  Core properties

•  Configsets

•  Extra libs

•  solr.xml

•  solr.in.sh

Solr Configuration – Artifacts in Version Control

•  Different files for different environments

•  Deploy script takes an environment parameter

•  Similar to Chef’s data bags or Puppet’s Hiera

Environmental Differences

Environmental Differences - DIH

Configuration Deployed / Collections and DIH

•  This groovy statement will pause the pipeline until an admin approves it:

input message: 'Approve this release', ok: 'Approve', submitter: 'admin'

Manual Intervention

Manual Intervention

•  Clone/build lucene-solr repo (ant server)

•  Clone configuration repo

•  Use scripts/start_solr.sh and stop_solr.sh

Developing Configuration

Deploy Configuration Demo

•  http://bit.ly/SolrCD - Links to Repositories

•  http://bit.ly/JenkinsPipeline - Pipeline as Code

•  http://bit.ly/FowlerCD - More on CD

•  @jstrassburg

Links / Questions