puppetconf 2016: delivering premium quality modules: using beaker and vmpooler for multi-platform...

DeliveringPremiumQualityPuppetModules

RenéLast&DimitriTischenkoPuppetconf2016

Using Beaker and VMpooler for Mul4-Pla7orm Tes4ng

Whoarewe?

Agenda

AboutKPNOurproblemOursoluBon

AboutKPN

Ourproblem

KPNdecidedtomanage10Kserversforvariousclients

withPuppet…

Ourproblem

…mostofthemWindows.

Ourproblem–consequences(1)

NeedtosupportWindows

Ourproblem–consequences(2)

Needtoguaranteestableandpredictableinfrastructurestate

Ourproblem–consequences(3)

Needtoguaranteecodequality

OursoluEon(1)

Adedicatedteamtodevelopmodules

OursoluEon(2)

AnautomatedtesBngfactory

TesEngFactoryRequirements

AutomatedpipelineconfiguraBonPuppet3.xtestsPuppet4.xtestsUnittestsonLinuxandWindowsAcceptancetestsonallplaQorms(currently8)Runtestsinparallel

ModuleCI/CDPipelineOverview

syntaxcheck

Auto

puppet-lint

rspectest

acceptancetest

ReleasetoTEST

Auto Auto Manual

ReleasetoPROD

ManualChangein

git

Periodictrigger

NoEfybymail&Hipchat

Fail Fail Fail Fail

PuppetCIComponents

JenkinsMaster

JenkinsLinuxSlave

JenkinsWindowsSlave

vmpooler

GithubEnterprise

vcenter

beaker

CreateVMs

Linuxpipeline

WindowspipelineMulE-PlaXormAcceptanceTests

Syntax,lint&rspec

Syntax,lint&rspec

SystemsUnderTest

Jenkinsmasterandbuildslaves

rtyler/jenkinspuppetmodule

Ruby

maestrodev/rvmPIKoruru!=rvmL

Jenkinsjobs

Jenkinsjobbuilderstankevich/python

PuppeEzingJenkinsJobs

puppet Yamlfile Jenkinsjobbuilder xml Jenkinsjob

Howtoaddamodule

1.  kpn_jenkins::pipeline::module{'kpn-puppet-forge/puppet-kpn-hosts':2.  ensure=>present,3.  test_platform=>['linux','windows'],4.  pe3_syntax=>false,5.  pe4_syntax=>true,6.  pe4_unit=>true,7.  pe3_unit=>false,8.  acceptance_test=>true,9.  acceptance_platform=>['centos-7-x64-pe4',10.  'windows-2008r2-x64',11.  'windows-2012r2-x64',12.  'windows-2008r2-x64-pe4',13.  'windows-2012r2-x64-pe4',14.  'windows-2016-x64-pe4',15.  ],16.  hipchat_notification=>true,17.  hipchat_room=>'Moduleteam',18.  email=>‘puppetteam@kpn.com',19.  }

Howtoaddamodule

JenkinsMaster

JenkinsLinuxSlave

JenkinsWindowsSlave

vmpooler

GithubEnterprise

vcenterCreateVMs

WindowspipelineMulE-PlaXormAcceptanceTests

Syntax,lint&rspec

Syntax,lint&rspec

SystemsUnderTest

Beaker

beaker

Linuxpipeline

Beaker

AtestharnessfocusedonacceptancetesBngusing(virtual)machines

Beaker

Runstestson“real”servers

Beaker

SupportsvmproviderslikeVagrant,vSphere,docker,AWSorvmpooler

BeakerInstallaEon

#geminstallbeakerbeaker-rspecbeaker-puppet_install_helper

BeakerConfiguraEonspec/acceptance/nodesets/*.yml 1.  HOSTS:2.  centos-7-x64-pe4:3.  roles:4.  -agent5.  platform:el-7-x86_646.  hypervisor:vmpooler7.  template:centos-7-x648.  CONFIG:9.  masterless:true10.  pe_ver:'2016.2.0'11.  pe_promoted_builds_url:'http://repo.kpn.com/pe'12.  pooling_api:'http://vmpooler.kpn.com:4567'13.  ssh:14.  auth_methods:15.  -password16.  user:root17.  password:XXXXXXXX

PrepareaSystemForTesEngspec/spec_helper_acceptance.rb 1.  require'beaker-rspec/spec_helper'2.  require'beaker-rspec/helpers/serverspec'3.  require'beaker/puppet_install_helper'4.  5.  #Installpuppetagentonanode6.  run_puppet_install_helper7.  8.  #CopyourmoduledependenciestotheSUT9.  system('gitclonegit@github.kpn.com/stdlibspec/fixtures/modules/stdlib')10. 11. copy_module_to(hosts,:source=>'spec/fixtures/modules/stdlib',12. :module=>'stdlib')13. 14. #Orifyouhaveinternetaccess15. install_puppet_module_via_pmt_on(host,:module_name=>'stdlib')

ExampleTestspec/acceptance/*_spec.rb 1.  require'spec_helper_acceptance'2.  3.  describe'kpn-centerity_agentmodule'do4.  context'installcenterity_agentsoftware'do5.  pp=<<-EOS6.  class{'centerity_agent':}7.  EOS8.  it'1stapplyshouldrunwithouterrors'do9.  apply_manifest(pp,:catch_failures=>true)10. end11. it'2ndapplyshouldrunwithoutchanges'do12. apply_manifest(pp,:catch_changes=>true)13. end14. end15. end

TesEngforInstalledPackages

1.  context'Testinstalledsoftware'do2.  casefact('osfamily')3.  when/RedHat/4.  describepackage('sshd')do5.  it{shouldbe_installed}6.  end7.  when/windows/8.  describecommand('C:\Programs\Chocolatey\bin\clist-l')do9.  its(:stdout){shouldmatch/sshd/}10. end11. end12. end13. end

Run,beaker,Run!

#exportBEAKER_setfile=~/nodesets/${platform}.yml

#rakebeaker

RulesforWriEngAcceptanceTests(1)

Onlytestyourpuppetcode,

notthesupplier’ssocware

RulesforWriEngAcceptanceTests(2)

Abeakertestshouldbeself-contained

RulesforWriEngAcceptanceTests(3)

Ifyoucanrspecit,don’tbeakerit

ImprovingBeakerDocumentaEon

heps://github.com/RARYates/beaker

JenkinsMaster

JenkinsLinuxSlave

JenkinsWindowsSlave

GithubEnterprise

vcenterCreateVMs

WindowspipelineMulE-PlaXormAcceptanceTests

Syntax,lint&rspec

Syntax,lint&rspec

SystemsUnderTestbeaker

vmpooler

vmpoolerLinuxpipeline

vmpooler

VmpoolerpoolsVMs

Providesconfigurablepoolsofinstantly-available

(running)virtualmachines

vmpooler

NeedsitsownDHCPzone

andproperlyconfiguredDDNS

vmpooler

Systemrunningbeakermustusevmpooler’s

nameserver

vmpooler

vmpoolerhasastrangelogo

vmpoolerinstallaEon

include‘vmpooler’

vmpooler

❝Templateset-upislecasanexercisetothereader.Somehow,eitherviaPXE,embeddedbootstrapscripts,orsomeothermethod--clonesofVMtemplatesneedtobeabletoset

theirhostname,registerthemselvesinyourDNS,andberesolvablebythevmpoolerapplicaBonacercompleBngtheclonetaskandbooBngup.

❞

vmpoolerhostnamingpuzzle

vmname!=hostnameneedtosethostnameautomaBcallytovmnamedon’twanttogiveeachvmvspherecredenBals

so-howtodoit?

Just5linesofcode(2monthslater)#Annotatewithcreationtime,origintemplate,etc.#AddextraconfigoptionsthatcanbequeriedbyvmtoolsconfigSpec=RbVmomi::VIM.VirtualMachineConfigSpec(annotation:JSON.pretty_generate(name:vm['hostname'],created_by:$config[:vsphere]['username'],base_template:vm['template'],creation_timestamp:Time.now.utc),extraConfig:[{key:'guestinfo.hostname',value:vm['hostname']}])

Seangthehostname

Linux:#hostname=$(vmtoolsd--cmd"info-getguestinfo.hostname")Windows:>$hostname=cmd.exe/c'C:\Progra~1\VMware\vmtoolsd.exe--cmd"info-getguestinfo.hostname"'

vmpoolerlessonslearned

• Canusemorelogging(PRsforthcoming)• Poolnameneedstoequaltemplatename• Docsrefertovmtemplates,needtobevms• WillnotinstallifESXhostisnot“green”• CanusebeeerexcepBonhandling

Conclusions

AutomatedpipelineconfiguraBonPuppet3.xtestsPuppet4.xtestsUnittestsonLinuxandWindowsAcceptancetestsonallplaQorms(currently8)Runtestsinparallel

PPP

P P

P

Whatdidwelearn?

+ Regressiontest+ MulBplaQorm

+ Catcherrorsearly

Whatdidwelearn?

FutureImprovements

Speedoftests

FutureImprovements

StabilityofvmpoolerWindowssupport

Andnow…forsomenumbers

ModulesperOperaEngSystem

Windows32

Linux43

LinuxandWindows

16

TestTypes

102 9479

0

20

40

60

80

100

120

Syntax Rspec(unit) Acceptance

#modules

TestRunsFlow8%

Init14%

Syntax21%

Unit18%

Acceptance39%

Top10ModuleswithFailedTests

Vmpooler:AverageTimetoReady

Q&A

WindowsModules

•  kpn-ad_ds•  kpn-dotnet•  kpn-rdp•  kpn-kms•  kpn-chocolatey_client•  kpn-chocolatey_server•  kpn-local_security_policy