building eclipse projects on the new cbi build farm generation... · more slaves dedicated master...
TRANSCRIPT
![Page 1: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/1.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Next Generation CI
Building Eclipse projects on the new CBI build farm
Frederic GurrEclipse Foundation
EclipseCon Europe - October 25, 2018
![Page 2: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/2.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
History & Status quo
2
![Page 3: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/3.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
History
3
2011-2012
More slaves
Dedicated master for performance tests
2013
Hudson instance per project
Part of the CBI initiative
Shared instance with slaves for Windows and MacOS
2009
Single Hudson instance
Maintained by the community
2010
1 Master, 2 Slaves
Maintained by Eclipse IT team.
![Page 4: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/4.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
History
4
2017
Mass migration to Jenkins
Migration tool https://github.com/eclipse-cbi/hipp2jipp
Custom scripts
2016
Hudson ecosystem not catching up
Started to evaluate Jenkins
2018
Jenkins instance per project
No more Hudson
![Page 5: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/5.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Today’s infrastructure
5
• 220+ Jenkins instances
• Beefy machines(total ~250 cores, ~1.5TB RAM)
• A few dedicated slaves(e.g. for Windows and Mac)
https://ci.eclipse.org
![Page 6: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/6.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Today’s infrastructure
6
• Managing 10s of Jenkins is “easy”, managing 100s of them is time consuming
• We want to spend our time on areas that add value: • Harden digital signature and packaging services• Ease publication/promotion (Maven central, Nexus, ...)• Better support of static analysis tools (SonarQube, …)• Better integration with GitHub
![Page 7: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/7.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Today’s infrastructure
7
• We also want to offer better performance• Pool our hardware to use it more efficiently• QoS - better isolation and control over resources usage• Be able to add additional resources more easily
• Either cloud-based or on-premise
![Page 8: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/8.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Next generation CI
8
![Page 9: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/9.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
A duet solution
9
( )
![Page 10: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/10.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
10
Operation Center
JIPPS
![Page 11: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/11.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
11
Operation Center
JIPPS
Kubernetes/OpenShift Cluster
Jobs run on temporary, dynamically provisioned agents
![Page 12: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/12.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
12
Operation Center
JIPPS
Kubernetes/OpenShift Cluster
Jobs run on temporary, dynamically provisioned agents
Dedicated agents
![Page 13: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/13.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
13
Operation Center
JIPPS
Kubernetes/OpenShift Cluster
Jobs run on temporary, dynamically provisioned agents
Dedicated agents
Shared agents
![Page 14: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/14.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Next generation status quo
14
Already running on new infra:• Every new provisioned JIPP
• Early adaptors like Xtext
• ~40 Jakarta EE projects
![Page 15: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/15.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CBC and cluster problems
15
• We had some stability issues in the beginning
• Responsive and excellent support by CloudBees
• Cluster is now much more stable
![Page 16: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/16.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Migrating OSS Jenkins instances to CBC
16
• How?• No built-in solution for mass migration of jobs• => Custom solution
• What needs to be moved?• Job configurations• Views• Credentials• Maven settings
![Page 17: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/17.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Roadmap
17
June2018
Setup
OpenShift/CJE running on Foundation infrastructure
August 2018
Early adopters
New JIPPs are created on CJE and a couple of volunteer projects are migrated
Soon
Migration
All JIPPs are migrated to CJE
October2018
JakartaEE
All JakartaEE projects run on CJE
![Page 18: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/18.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
18
FAQ & Best practices
![Page 19: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/19.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
19
• Why does my build take so long and how can I make it faster?
![Page 20: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/20.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
20
• Why does my build take so long and how can I make it faster?
• What are Jenkins pipelines and why/when should they be used?
![Page 21: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/21.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Jenkins Pipelines
21
• Store your job config in your repo
• Config linked to code
• Config history
• Scripted Pipeline vs Declarative Pipeline
• Recommended for docker based builds
![Page 22: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/22.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
22
• Why does my build take so long and how can I make it faster?
• What are Jenkins pipelines and why/when should they be used?
• Can I finally run docker based builds?
![Page 23: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/23.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
23
• Why does my build take so long and how can I make it faster?
• What are Jenkins pipelines and why/when should they be used?
• Can I finally run docker based builds?• What do I need to do for the migration?
![Page 24: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/24.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Best practices
24
• Modular builds and tests => Parallelization
• Play fair and don’t be a resource hog
• Build configuration in Code => Pipelines
• Docker
More: http://wiki.eclipse.org/CI_best_practiceshttps://wiki.eclipse.org/Jenkins#FAQ
![Page 25: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/25.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Questions?
25
![Page 26: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/26.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Takeaways
26
• Less administration => more tools and services
• New infra => better utilization and scaling• Stateless build agents will require
adaptations
![Page 27: Building Eclipse projects on the new CBI build farm Generation... · More slaves Dedicated master for performance tests 2013 Hudson instance per project Part of the CBI initiative](https://reader035.vdocument.in/reader035/viewer/2022081611/5f03b7927e708231d40a6ca4/html5/thumbnails/27.jpg)
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
27
Thank you!
Please evaluate the talks on eclipsecon.org!