wellcome trust centre for gene regulation & expression college of life sciences, university of...
TRANSCRIPT
![Page 1: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/1.jpg)
Wellcome Trust Centre for Gene Regulation & ExpressionCollege of Life Sciences, University of DundeeDundee, Scotland, UK
Continuous Integration
1
![Page 2: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/2.jpg)
Wellcome Trust Centre for Gene Regulation & ExpressionCollege of Life Sciences, University of DundeeDundee, Scotland, UK
Continuous Integration
2
![Page 3: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/3.jpg)
3
Plan
1. Why OME needs Continuous Integration?
1. OME Deliverables and Releases
2. OME source code management
3. Continuous Integration
2. OME Continuous Integration servers
1. Introduction
2. Travis CI
3. Jenkins CI
3. CI documentation
![Page 4: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/4.jpg)
4
OME releases: Deliverables
o OMERO.server / OMERO.clients
• Windows, Linux, Mac OS X
• Ice (3,3, 3.4, 3.5)
o Tools/plugins
• OMERO.imagej,
• OMERO.matlab,
• OMERO.py
o OMERO Virtual appliance
o Documentation
• API documentation
• Sphinx documentation
o Java ARchives (JAR)
• loci_tools.jar
• ome_tools.jar
• ….
o Bio-Formats Tools
• Command-line tools
• MATLAB toolbox
o Documentation
• API documentation
• Sphinx documentation
![Page 5: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/5.jpg)
5
OME releases
o OME 4.4.x series (June 2012 – Jan 2014)
• 12 releases in 19 months (1 major, 11 minor)
• 3 emergency releases (4.4.1, 4.4.3, 4.4.8)
o Quality Assurance
• Ensure quality standards are being met
• Robustness/stability
• Catching bugs early
o Every code change must be tested and reviewed
![Page 6: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/6.jpg)
6
Source code management
![Page 7: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/7.jpg)
7
Source code: Pull Requests
![Page 8: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/8.jpg)
8
Source code: changes frequency
OMERO Bio-Formats Documentation Scripts
Number of commits 5502 3033 1375 111
Number of mergedPull Requests
812 366 254 23
Number of unique authors
25 26 26 7
Number of merged Pull Requests/day
2 1 1 <1
Number of commits/day
11 6 3 <1
Changes on dev_5_0 branch between 25 Sept 2012 and 20 Jan 2014
![Page 9: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/9.jpg)
9
The problem
http://geekandpoke.typepad.com
![Page 10: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/10.jpg)
10
Continuous integration
o Continuous integration: perform software integration
frequently
o Automate the build
• OMERO suite (server, clients…), Bio-Formats suite
• OME Consortium products
o Automate the tests
1. Run Bio-Formats automated tests against the data repository
2. Run the OMERO.server integration tests of the OMERO.server
o Automate the deployment
• Deploy all deliverables for PR review/ QA release
• Deploy the latest version of the documentation
![Page 11: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/11.jpg)
11
Continuous integration server
Source: http://www.appfoundation.com/ci/
![Page 12: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/12.jpg)
12
Continuous integration server
BENEFITS
o Automation (build, test,
deployment)
o Integration with SCM
o Easy access to daily
deliverables
o Time trigger/change
polling
o Build history
COSTS
o Writing/maintenance of
CI automated scripts
o Team resources
• Server setup
• Server maintenance
• Individual job setup
o Team communication
![Page 13: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/13.jpg)
13
OME CI servers
o http://about.travis-ci.o
rg
o Externally hosted
o Integration with
GitHub
o Support Java, Python,
C+
o Matrix jobs
o Setup in .travis.yml
o Build queues/ 50 min
timeout
o http://jenkins-ci.org
o Hosted by OME team
o Integration via scc tools
o Node-specific languages
o Chained jobs
o Build system (ant,
build.py)
o Control over node
number/throttling/time
![Page 14: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/14.jpg)
Typical CI workflow
![Page 15: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/15.jpg)
15
Travis CI : introduction
Activated for core OME source code repositories
![Page 16: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/16.jpg)
16
Travis CI: GitHub integration
![Page 17: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/17.jpg)
17
Travis CI: OME usage
Source: .travis.yml
Code validation Build Test
OMEROpy flake8 ./build.py build-default ./build.py
-py test
java ./build.py build-default test-compile
Bio-Formats
maven mvncpp cmake + make
cppwrap mvn + cmake + makesphinx make clean html latexpdf
Documentation make clean html latexpdf
Scripts flake8
![Page 19: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/19.jpg)
19
OME Jenkins CI: build view http://ci.openmicroscopy.org/job/JOBNAME
![Page 20: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/20.jpg)
20
Jenkins CI: job naming scheme
o $COMPONENT-$VERSION-$TYPE-$DESCRIPTION,
e.g. OMERO-5.0-merge-integration
o COMPONENT: deliverable name
• OMERO, BIOFORMATS, Consortium or third-party deliverable
o VERSION: version number e.g. 5.0
o TYPE: usually merge, latest, release
o DESCRIPTION:
• Short description (2-3 words max)
• Allows to create personal views
http://ci.openmicroscopy.org/pview/
![Page 21: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/21.jpg)
21
Jenkins CI: views
o Versions
• 4.4, 5.0, 5.1: for OME main products (OMERO, Bio-Formats)
o Components
• OMERO/Bio-Formats
• Consortium
• Third Party (Pytables, ITK, Zlib,…)
• Mgmt
o Experimental
o Failing
• all non-Experimental jobs with failing/unstable status
![Page 22: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/22.jpg)
22
Jenkins CI: job types
o LATEST jobs
built from the HEAD of the development branch, i.e. origin/dev_5_0
o MERGE jobs
built from the HEAD of a development branch with Pull Requests
merged in, i.e. snoopycrimecop/merge/dev_5_0/latest
o RELEASE jobs
associated with a tag, e.g. 5.0.0-rc1
![Page 23: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/23.jpg)
Jenkins CI: chained jobs
o Morning jobs
• merge Pull Requests
• build artifacts
• deploy merge server
• import data
• run integration tests
• build and deploy
staging
documentation
![Page 24: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/24.jpg)
Jenkins CI: deployment servers
o One node associated to each development branch
o Multiple OMERO instances running on different ports
• 4064: merge server for PR review
• 14064: latest server
• 24064: integration server for integration tests
Series Development branch Server name
4.4.x dev_4_4 howe.openmicroscopy.org.uk
5.0.x dev_5_0 gretzky.openmicroscopy.org.uk
5.1.x develop trout.openmicroscopy.org
![Page 25: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/25.jpg)
Continuous integration documentation
![Page 26: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/26.jpg)
Continuous integration documentation
![Page 27: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/27.jpg)
Continuous Integration: future directions
o Lots of progress in the last 6 months
• Cleanup of the job (integration tests)
• Server maintenance and stabilization
• Rationalization/organization
o Future steps
• Refactoring of Consortium projects
• Integration tests in Travis build
• Maintenance/improvements of the CI documentation
• Refactoring of the 5.0 jobs using new 5.1 jobs
![Page 28: Wellcome Trust Centre for Gene Regulation & Expression College of Life Sciences, University of Dundee Dundee, Scotland, UK Continuous Integration 0](https://reader035.vdocument.in/reader035/viewer/2022062716/56649dbd5503460f94aafbfb/html5/thumbnails/28.jpg)
28
CI Agent: Snoopy Crime Cop
o GitHub user
• https://github.com/openmicroscopy/snoopycrimecop
• Member of openmicroscopy and ome organizations
o Perform actions
• Performs local PR merging using GitHub API
• Open PRs for submodule bumps
• Push merge/branches release tags to his forks
o Forks of main repositories
• see https://github.com/snoopycrimecop/openmicroscopy
• used for pushing merge branches/ staging release tags