Wrangling 3rd Party Installers with Puppet

Puppet Camp Portland 2014January 14, 2014

Overview● About us● Definitions● Our situation● Overall steps● What we get and

what is missing from installers

● Requirements

● Installer “state”● Initial scripting● v2.0 using Puppet● Our solution● Testing● Challenges● Future● Q&A

About us

Mike Kinneymike.kinney@gmail.comgithub.com/mkinney

● UTi Worldwide● Mentioned in the

“Book”

Spencer Krumkrum.spencer@gmail.comgithub.com/nibalizer

● UTi Worldwide● PSU CAT● Co-authorof the “Book”2nd Edition

About us

Mike Kinney● Automating

installations for few years

● New to puppet

Spencer Krum● New to these 3rd

party tools● Puppet expert

A few words about vendors

● We use a few vendors.● We will try to keep their names out of it,

though it might slip out once or twice.● The focus of this talk is how we work around

not being able to use a package management tool. (like apt/rpm)

● These patterns should be universal.

Definitions (1 of 4)● 3rd Party software:

○ Software that is bought from a software vendor, almost always closed source.

● Installer:○ The installation method described by the vendor for

the 3rd party software. Often this is a binary file like “installer.bin”.

Definitions (2 of 4)

● Quality Assurance (QA):○ Process that validates functionality of a release.

● Silent installation template file:○ File with the all installation options and values.

Sample file:<?xml version="1.0"?>

<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">

<properties>

<entry key="acceptLicense">true</entry>

</properties>

Definitions (3 of 4)● Version upgrade: When either the x or y in

version x.y.z is higher than the currently installed version.

When the product has bugs or errors, sometimes we will get out of cycle patches from the vendor:

● Engineering build: (no applicable version)○ quick (little or no) QA before releasing version

● Hotfix: (the “z” in version x.y.z)○ full QA before releasing versionNote: Some vendors use x.y.z.a

Definitions (4 of 4)

● idempotent:Only make changes if they are required.

● hiera:Helps to separate data from code (see

Chapter 12 of the “Book”)

Our situation● most of our stack uses Enterprise tools (lots

of 3rd party installers)● mostly linux (OEL 5)● lack of “root” has its challenges● lots of environments● each technology on a separate virtual

machine (means lots of VMs)● many vendor/consultants/partners

Overall Installation steps

● Download installation file (zip/tgz)● Review requirements● Pre-install steps● Run installer● Installation options● Post-install steps

3rd Party Installers

Most have:● sample silent

template file● installation binary● installation files● installation

documentation

Most are missing:● service script

(start/stop/status)● any verification

methods (post installation)

Pre-install Requirements

● Most require java (run-time)○ cannot use openjdk!○ some products bring own jre

● Most require another specific version of another 3rd party installation (example: product B v4.9.0 requires product A v8.3+)

Installation User Interfaces

● Interactive graphical user interface (xterm window)○ often the options are saved in a file and can be

converted into a silent installation file● Interactive text mode● Command line using “silent” installation

option:./installer -silent options.txt

3rd Party Installer “state”

● Different installers store installation status in different places; typically in home dir and or in the destination directory

● If vendor support wants to know about system, they often ask for files/directories showing this state (for what is installed)

Scripting options

A. Re-package:○ install once - preserve files post-install○ the internal “state” would have to be reverse

engineered○ vendor would have an excuse “you did not install per

instructions”

B. Script using the “silent” installation

Initial installation scripting (v1.0)● Initial scripting (ant/shell scripts)● One configuration file per installation per VM● Lots of redundant config● Hard to maintain● Hard to train● Put the binary installers in version control

system (sounds bad - but it works for us)● Shared /software source (home of tarballs)

v2.0 - Using puppet

● <environment>/site.pp● rootless package● /data/env/<envname> (hiera)● Combine product installs into one class

v2.0 - Using puppet (continued)● Vendor install class (defined type)● Two stages to install:

1. extract software to /tmp and run installation (see sufact hack)

2. configuration files, cron, aliases, etc.

Example of a Puppet Package

package { 'apache2':

ensure => '2.4',

}

Resource abstraction layer

Pretty Picture Here

Resource abstraction layer

Puppet is a Programming Language● Contains and hides complexity (API)● You do not really want to see how the

sausage gets made.

Note:● We are creating “anti-patterns”.● Need to ensure code is idempotent.

Example of API

class { 'ssh':

root_login => 'without-password',

}

Another API example

class { 'apache':

mods => ['index', 'rewrite'],

}

Example of Vendor::Software

class { 'vendorSoft::someSoftware':

version => 2.1,

}

Sample Vendor/Product

As an example, we are going to install Acme’s product Foo version 4.9.0.

● In the <environment>/site.pp: class { 'acme::foo490': }

Two types of Vendor Installs

1. 'untar' installsInstall consists of untarring or unziping files to the correct locations.

2. 'installer' installsInstall consists of running the vendor's installation binary with flags and a silent file.

Note: Both of these will have the same interface from puppet.

'untar' installs: Puppet class

● Ensure all dependencies● Unzip zipfile/tarball from network share to

installation location● Set values in configuration files

'install' installs: Puppet class

● Unzip zipfile/tarball from network share to temporary location

● Set values in configuration files or “silent” file● Run installer● Clean up temporary installation

Note: We were not going for how many Puppet Style guide infractions we could make, it just happened that way.

Idempotency

How do we make these idempotent?

Idempotency: untars

rootless::tardir { '/opt/app/other_place/folder':

ensure => present,

tarfile => '/big/nfs/mount/tar.tar.gz'

}

Idempotency: untars

exec { 'untar-directory':

provider => shell,

command => '/bin/tar -xvzf /big/file.tgz /the/location',

unless => 'stat /the/location/tardir',

}

define acme::acme_install(

$creates_file,

$installationFile,

$installationRoot,

$installer,

$product,

$tmpdir,

$version,

$installer_content,

$ensure = true,

$group = $::puppet_user,

$owner = $::puppet_group,

$source_dir = "/software/install/installers",

$create_tmpdir = true,

){

Sample acme::acme_install (1 of 5)

if $ensure {

# install

if $create_tmpdir {

file { "${tmpdir}-create":

path => $tmpdir,

ensure => directory,

}

}

anchor { "$name-unpack::begin": }

anchor { "$name-unpack::end": }

<snip - unzip/untar >

Sample acme::acme_install (2 of 5)

Anchor pattern

file { "${tmpdir}/silent":

ensure => file,

content => "$installer_content",

require => Anchor["$name-unpack::end"],

}

file { "${tmpdir}/${installer}":

ensure => file,

mode => '0755',

require => Anchor["$name-unpack::end"],

}

Sample acme::acme_install (3 of 5)

Silent file from template

Ensure installer is executable

What is the return value of a zero byte file that has the “execute” bit set?

$ echo > /tmp/nobytes

$ ls -al /tmp/nobytes

-rw-r--r-- 1 user wheel 1 Jan 14 12:06 /tmp/nobytes

$ chmod +x /tmp/nobytes

$ ./nobytes

$ echo $?

What is the return value from “./nobytes”?

Question

# verify that the file we are about to execute is *not* a 0 byte "executable"

exec { "verify-installer-${acme_product}":

command => "/usr/bin/[ -s ${tmpdir}/${installer} ]",

provider => 'shell',

cwd => $tmpdir,

require => File["${tmpdir}/${installer}"],

}

exec { "install-${acme_product}":

command => "source ~/.bashrc && ${tmpdir}/${installer} -silent -V responseFile=${tmpdir}/silent",

provider => 'shell',

timeout => 30000,

cwd => $tmpdir,

require => [File["${tmpdir}/silent"], File["${tmpdir}/${installer}"], Exec["verify-installer-${product}"], ],

creates => "${installationRoot}/${creates_file}",

}

Sample acme::acme_install (4 of 5)

Check

Do installation

# Check the logs for errors

exec { "${acme_product}-log-check":

command => "! /bin/grep -R -E \"::ERROR::|err.X\" ${::puppet_user_home}/.ACME",

provider => 'shell',

require => Exec["install-${acme_product}"]

}

# need to clean up tmpdir

if $clean_up_tmpdir {

exec { "${tmpdir}-remove":

command => "/bin/rm -fr ${tmpdir}",

provider => 'shell',

require => [ Exec["install-${product}"], Exec["${product}-log-check"], ],

}

}

} else { # uninstall

<snip>

Sample acme::acme_install (5 of 5)

Any errors?

Clean up

Problem is (still) idempotency

● Multiple resources cannot be chained to an unless clause of an exec

● Anchors are a part of this● Maybe 3.4 contains() will help fix this, we

have not had time to evaluate

Sufact (su fact)

● Puppet module● Modest ambitions● Turns out to be really awesome

https://github.com/TheDarren/fact

Sufact (su fact)● fact { "su_support": value => "IDG" }

● Then:○ $su_support == "IDG"

We use this to flag and pin versions of vendor software.

Puppet

● Built each version of each installer as separate module

acme::yyy123

where yyy is a short name of the product and 123 is the “version”Write facts to set these values, then check

class acme::foo490(

$owner = $::puppet_user,

$group = $::puppet_group,

$install_root = "/opt/app/${::puppet_user}",

$source_dir = '/software/install/installers',

$tmpdir = "/tmp/tmpfoo490-${::puppet_user}",

$installationFile = 'foo_4.9.0_linux26gl23_x86_64.zip',

$installer = 'Installer-lnx-x86.bin',

$version = '4.9.0',

# these values are in the silent file

$silent_environmentName = 'FOO',

$feature_acme = hiera('acme_feature_xxx', “blargh”,

$ensure = true,

){

Sample foo490 (1 of 3)

Hiera pattern

Silent values

$installationRoot="${install_root}/somedir"

$product = 'acme490'

anchor { "${product}_installed": }

if $ensure {

# Note: Config files/cron entries, etc. here

if $acme490_installed == undef {

$installer_content = template("acme/${product}/silent.erb")

acme::acme_install { $product:

installationFile => $installationFile,

installationRoot => $installationRoot,

installer => $installer,

product => $product,

tmpdir => $tmpdir,

Sample foo490 (2 of 3)

Ensure pattern

Call our defined type

Anchor pattern

version => $version,

creates_file => "foo${product}",

ensure => true,

group => $group,

owner => $owner,

source_dir => $source_dir,

installer_content => $installer_content,

before => Anchor["${product}_installed"],

}

fact { "${product}_installed":

value => $version,

require => Acme::Acme_install[$product],

}

}

} else { # uninstall

Sample foo490 (3 of 3)

Fact pattern

Anchor pattern

Combining Vendor productsclass acme::foo12 (

$installationRoot = "/opt/app/${::puppet_user}/somedir",

){

anchor { 'acme::foo12::begin': }

anchor { 'acme::foo12::end': }

class { 'acme::foo123':

#parameters

}

Anchor [‘acme::foo12::begin’] ->

Class [‘acme::bar123’] ->

Class [‘acme::baz124’] ->

Anchor [‘acme::foo12::end’

<snip>

Dependency chaining pattern

Testing the Puppet installations

● Similar to Rspec-system/Beaker● Totally home-grown● Written in shell● Integration testing● Linux Container (LXC) Based

Testing the Puppet installations

● Fires up LXC● Builds ssh bridge to LXC● Run “puppet apply”● Runs suite of 'checks'● Teardown LXC● Repeat for ever suite of checks

Testing the Puppet installations

● Work in progress● Plan to leverage this for refactoring and

beautification project● Plan to use Jenkins for reporting

Testing the Puppet installations#!/bin/bash

testname=foo610

if [ -d "/somedir/config" ];then

:

else

exit 1

fi

Challenges● Often used the “dot-uh-oh” versions● Large and many installers● Many different versions● Testing● Puppet development environment● Installation validation● Orchestration issues (queue creation, stop

commands before upgrading files, start & wait, etc.)

Future● Automation of pre-installation (checks)● Automatically add the monitoring/syslog

configs (both are done manually now)● Possibly pull post-processing into puppet● md5sum checking of installed files● Possibly look into app deployments using

puppet (need orchestration in a big way)

Questions

Any questions?