how openstack is built
TRANSCRIPT
![Page 2: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/2.jpg)
How Big is OpenStack?
265,850 commits
5,744 developers
54 project teams
585 git repos
2
![Page 3: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/3.jpg)
OpenStack CI System
3
• Developed and managed by the OpenStack Infra team
![Page 4: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/4.jpg)
Ansible+Puppet - two invisible hands.
■ All configuration of CI infra nodes is managed by Puppet manifests ■ The ‘Puppetmaster’ is actually an Ansible control machine■ Puppetmaster holds playbook + hiera data■ Ansible playbooks trigger ‘puppet apply’ on the nodes
4
+
![Page 5: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/5.jpg)
Git - the mighty archivist
■ the most widely used distributed open-source VCS
■ Served by ‘cgit’■ Changes submitted with the help
of ‘git-review’ tool
5
![Page 6: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/6.jpg)
Gerrit – the gatekeeper
■ Originally developed by Shawn Pearce at Google for Android development.
■ Manages git repositories■ Provides code review and
approval rules■ Collaborates with human and non-
human gatekeepers
6
![Page 7: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/7.jpg)
Zuul – the dispatcher
■ Written by James E. Blair■ Defines and enforces the gating
process■ Listens to Gerrit events■ Manages the queue and
dependencies■ Dispatches the verification
process to Jenkins (through Gearman)
7
![Page 8: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/8.jpg)
Zuul – the beautiful dispatcher
■ Written by James E. Blair■ Defines and enforces the gating
process■ Listens to Gerrit events■ Manages the queue and
dependencies■ Dispatches the verification
process to Jenkins (through Gearman)
8
![Page 9: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/9.jpg)
Gearman - the outsourcer■ a framework to farm out work to
other machines or processes
9
■ sends jobs to Jenkins through jenkins gearman plugin (by Khai Do & James E. Blair)
■ manages the job queues
![Page 10: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/10.jpg)
Jenkins - the butler :)
■ Continuous Integration Server■ Executes testing jobs, records executions results■ Jobs performed on slaves■ “Our jenkins servers, at peak load, runs 20,000+ jobs
per day. At that load we require many jenkins slaves (900+)…” (From Jenkins Gearman plugin page)
■ Jobs created and managed by Jenkins Job Builder■ devstack slaves provisioned by Nodepool
10
![Page 11: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/11.jpg)
Jenkins Job Builder■ Configures Jenkins jobs using YAML files stored in git■ Defaults, macros and job configs
- job: name: example-docs node: node-label
triggers: - zuul
builders: - git-prep - docs
publishers: - scp: site: 'scp-server' files: - target: 'dir/ectory' source: 'build/html/foo' keep-hierarchy: true - console-log 11
![Page 12: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/12.jpg)
Nodepool - the provisioner
■ Manages a pool of Devstack images to use in project testing.■ Provisions nodes on HP and Rackspace clouds. (using novaclient)■ Makes sure there are always available nodes for each provider type.■ Listens to events from Jenkins ZeroMQ publisher plugin. (by C.Boylan)■ Tears down old instances when tests are completed.
12
![Page 13: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/13.jpg)
DevStack Gate■ The integration test that verifies that all projects still work together after the
changes■ Provides a full OpenStack installation on a single machine with Devstack■ Installs Devstack and runs Tempest (the OpenStack integration test suite)
13
![Page 14: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/14.jpg)
Logstash
■ All jenkins job logs are indexed and stored by Logstash
■ Logs pushing also handled by Gearman jobs
14
■ Elastic Search helps analyze the test results
![Page 15: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/15.jpg)
Wrap Up
■ Git■ Gerrit■ Zuul■ Gearman■ Jenkins Job Builder■ Jenkins■ Devstack■ Nodepool■ Logstash
15
Only 9 out of 32 projects/tools managed by the Openstack InfraTeam
![Page 16: How Openstack is Built](https://reader031.vdocument.in/reader031/viewer/2022020410/58f1ecbc1a28ab980b8b45db/html5/thumbnails/16.jpg)
Some References:
■ Scaling the OpenStack Test Environment - http://amo-probos.org/post/15■ Zuul: a Pipelining Trunk Gating System - http://amo-probos.org/post/14■ Multiple Jenkins Masters with Jenkins Gearman Plugin - http://
docs.openstack.org/infra/publications/gearman-plugin■ Understanding the OpenStack CI System - www.joinfu.com/2014/01/
understanding-the-openstack-ci-system/■ cgit - https://git.zx2c4.com/cgit/about/
16