juc europe 2015: the famous cows of cambridge: a non-standard use case for jenkins
TRANSCRIPT
The Famous Cows of Cambridge A non-standard use-case for Jenkins
Sarah Woodall, NXP Semiconductors
#jenkinsconf
Footer
What this talk is about
• How (and why) we use Jenkins to test our software automatically using a farm of custom hardware
• The plugins and features of Jenkins that help • Improvements we would love to see • Why we think the Workflow Plugin will be important
2
#jenkinsconf
Setting the scene
#jenkinsconf
Some of the world’s most photographed cows
4
Photo by Alex Brown (originally posted to Flickr as Cows and King's) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
#jenkinsconf
The product that our Jenkins builds and tests
A software development toolchain based on Eclipse and GCC/GDB
• Designed for ease of use with NXP’s ARM-based LPC microcontrollers
• Hosted on Windows, Mac, Linux (separate installers)
#jenkinsconf
LPCXpresso in normal use
6
Host computer
running
LPCXpresso
Debug probe
USB
Development board
Target MCU
Stub
#jenkinsconf
LPCXpresso’s components
Open-source IDE built and tested by the Eclipse Foundation
Public releases three times a year
GNU tools (GCC compiler, GDB debugger, ...) built and tested by ARM
Public releases four times a year
Project wizards; Debug perspective; MCU definitions; support for advanced features such as Trace, etc ...
Written locally: our responsibility to build and test
Debug stubs: let GDB talk to NXP MCUs on target boards via USB
Many target/probe/host platform combinations => many stubs!
Written locally: our responsibility to build and test
Target software: C Libraries; LPC-Open related components; ...
Written locally: our responsibility to build and test
#jenkinsconf
Testing LPCXpresso’s debug stubs
8
#jenkinsconf
Manual testing will not get us very far
9
#jenkinsconf
Debug stubs: the scale of the testing problem
This matrix is only a subset • New targets and debug probes
are constantly being added
• We have to run the same tests over again for each host platform
#jenkinsconf
It’s not enough to run the tests just once
Why test frequently? • It’s much easier to find problems straight away • Developers see what is wrong while their work
is still fresh in their minds
Why test everything, everywhere? • Many of the stubs share common code:
change one – break the others! • Stubs work with the host OS, so they behave
differently on each platform
11
#jenkinsconf
The solution: automate testing with Jenkins
• Jenkins builds the three LPCXpresso installers – (How it does that is a whole talk for another day)
• A set of host machines are set up as Jenkins slaves – Windows 7, Windows 8.1, Ubuntu, Fedora, Mac, …
• TestInstall (a matrix job) – Uses the Copy Artifact plugin to get the right installer – Runs it silently and checks the result
12
#jenkinsconf
Test a wide range of target/probe combinations
• Using a big matrix job, each testing slave can – use every kind of debug probe to communicate
with target boards – execute a set of standard images compiled for
each kind of target hardware
13
#jenkinsconf
The snag: how to power-cycle the test boards?
• In normal use, nobody would expect to have multiple boards and debug probes attached to one computer – They are not designed to work like that!
• Many of them have to be switched off and on again as well as being booted before use
• How can we simulate this from Jenkins?
14
#jenkinsconf
The answer: our test farm has COWs
15
COW = “Collection of Wires”
#jenkinsconf
The test farm (much simplified)
16
Jenkins master
TestWindows TestMac TestUbuntu ... many similar Jenkins slaves
Ermintrude Daisy Poppy Willow
USB
probe +
target
... many similar cows
... and very many moreprobe +
target
probe +
target
probe +
target
probe +
target
probe +
target
probe +
target
probe +
target
probe +
target
USB
#jenkinsconf
Part of the Cambridge herd
17
#jenkinsconf
An expatriate cow
18
Our Jenkins master drives this laptop in California as a (rather slow) slave
#jenkinsconf
How are our cows connected today?
• Cows are physically plugged in to particular test machines
• We run a special job to tell Jenkins whenever a cow has been moved
• All the test jobs begin by copying over the artifact that this job created
19
#jenkinsconf
Summary Display shows test coverage available
The result of the ConfigureTestFarm job shows us where the gaps in our testing are
#jenkinsconf
Test results overview shown in Jenkins
Results table created with the Jenkins Summary Display Plugin
#jenkinsconf
Tests are defined using Robot Framework
• Open source; easily extended using Python • Works on all our supported platforms • Ready-made output display formatted in HTML • Easy to integrate with Jenkins
22
robotframework.org
#jenkinsconf
Links in our results table lead to Robot’s reports
#jenkinsconf
Drilling down further shows more detail
#jenkinsconf
A cow in action
25
#jenkinsconf
A cow driving a development board
26
#jenkinsconf
More automation
Testing other aspects of LPCXpresso
#jenkinsconf
Importing and compiling 2000 templates
28
#jenkinsconf
Uploading, downloading, …
• Jenkins uploads our build artifacts to a cloud server for easy access by colleagues in remote locations
• A shared cloud slave enables communication with a distant group’s Jenkins instance – Their Jenkins uploads zipped packages as artifacts – Our Jenkins fetches them, unpacks them and runs
them through our test farm, reporting results by email
29
#jenkinsconf
The plugins and features of Jenkins that help us most
#jenkinsconf
Our favourite plugins
• Exclusion • Conditional buildstep • EnvInject • Parameterized trigger • Matrix combinations • Promoted builds
• Copy artifact • Node/label parameters • Summary display • Description setter • Build timeout • Green balls
31
#jenkinsconf
Things we do with build parameters
• Check out from a branch of the source
• Name a separate exclusion resource on each slave
• Allow interactive input to Groovy scripts
• Select matrix combinations dynamically
32
#jenkinsconf
Selecting matrix combinations dynamically
33
This is part of the configuration for a job called LaunchTestMatrix
#jenkinsconf
Matrix combinations example
This is generated automatically by a script COMBINATIONS=(PROBE=="LPC-Link"&&(TARGET=="LPC2138"||TARGET=="LPC11U14_201"||TARGET=="LPC1114_301"||TARGET=="LPC1549"))||(PROBE=="RedProbe+"&&(TARGET=="LPC2368"||TARGET=="LPC1768"))||(PROBE=="LPC-Link2"&&(TARGET=="LPC4337"||TARGET=="LPC1768"||TARGET=="LPC4330_SPIFI_1M_64K"||TARGET=="LPC1227_301"||TARGET=="LPC1830_NGX_SPIFI"||TARGET=="LPC4370_SPIFI"||TARGET=="LPC1549"||TARGET=="LPC11U68"||TARGET=="LPC54102J512"||TARGET=="LPC810"))||(PROBE=="RDB-Link"&&(TARGET=="LPC1768"))||(PROBE=="CMSIS-DAP"&&(TARGET=="LPC11U68"||TARGET=="LPC824"||TARGET=="LPC1549"||TARGET=="LPC4370_SPIFI"||TARGET=="LPC54102J512"||TARGET=="LPC1768"))34
#jenkinsconf
Exclusion to manage shared resources
The whole point of the cows is to have only one target board/probe combination switched on at one time. • What if more than one job is running on the slave?
Our solution: use the Jenkins Exclusion Plugin • Define a resource called ${SLAVE}_BOARDS• Claim it before running each test
35
#jenkinsconf
Our Jenkins wish-list
#jenkinsconf
We do love Jenkins, but it would be nice if …
• You could select one configuration in a matrix job when copying artifacts from a permalink
• There was proper support for source-code branches
• The Cross-platform shell plugin worked better
• Matrix jobs were first-class citizens
37
#jenkinsconf
Automated is good, but scripted is better!
#jenkinsconf
Our job structure is quite complicated
Visualization created with the Jenkins Dependency Graph Plugin
39
#jenkinsconf
We think the Workflow plugin will help
• Allows complicated job structures to be expressed concisely as scripts
• Easier to read, to write and to manage
But • We shall need good support for matrix jobs
40
#jenkinsconf
What is special about our use-case for Jenkins
#jenkinsconf
Our world is a matrix world
• Our test farm is a sparse multi-dimensional matrix • The hardware is reconfigurable, so the Jenkins
model of it has to be flexible, too • We need build parameters and combination filters to
manage this complexity • The proper functioning of matrix jobs is essential
42
London, UK 23-24 June