![Page 1: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/1.jpg)
DevOps StackReid Holmes https://upload.wikimedia.org/wikipedia/commons/thumb/b/b5/Devops.svg/2000px-Devops.svg.png
Chris Parnin: https://github.com/CSC-DevOps/Course
![Page 2: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/2.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
WaterfallWaterfall modelProblems?
Benefits?
Still in use?
![Page 3: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/3.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
AgileAgile Processes
Iteration: Plan, Requirements, Development, Testing.
Problems?
Benefits?
Where not in use?
![Page 4: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/4.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Continuous DevelopmentContinuous * (Perpetual Development)
We’ll circle back to this….
![Page 5: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/5.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Who does Ops?Operations Responsibility
http://perfcap.blogspot.ca/2012/03/ops-devops-and-noops-at-netflix.html
![Page 6: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/6.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
DevOps Values‣ No silos‣ One team, owning changes‣ Developers are responsible for supporting their
code‣ Leads to a quality-focused culture‣ Carrying a beeper makes you careful
![Page 7: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/7.jpg)
REID HOLMES - CPSC 410: Advanced Software Engineering
Configuration ManagementTracking and control activities that manage baseline and alternative versions of systems. CM enables changes to flow through different versions and releases while supporting traceability and reproducibility.
![Page 8: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/8.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Configuration Management‣ Baseline: snapshot of a system.‣ All changes are submitted as change requests
against the baseline.‣ Gates can be established to ensure changes are
scrutinized before they are applied.‣ CM does not just apply to code:‣ Documentation‣ Environment‣ Libraries
![Page 9: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/9.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
CM at Google‣ Monorepo:‣ 1 billion files‣ 9 million source files‣ 2 billion LOC‣ 35 million commits‣ 45,000 commits / day‣ 86 terabytes in total
HTTPS://WWW.YOUTUBE.COM/WATCH?V=W71BTKUBDQE
![Page 10: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/10.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Branching
HTTPS://WWW.ATLASSIAN.COM/GIT/TUTORIALS/COMPARING-WORKFLOWS/GITFLOW-WORKFLOW
‣ Branches: mechanisms for allowing concurrent changes to be made to the repository.
![Page 11: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/11.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Modern Branching
![Page 12: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/12.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Branching Anti-Patterns‣ Merge paranoia: avoid merging any changes.‣ Merge mania: spend all your time merging.‣ Big bang: try to merge all branches simultaneously.‣ Branch mania: too many branches.‣ Cascading branches: branching but never merging.‣ Dev freeze: halting progress to allow merging.‣ Spaghetti branching: unstructured merging.
HTTP://WWW.HILLSIDE.NET/PLOP/PLOP98/FINAL_SUBMISSIONS/P37.PDF
![Page 13: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/13.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Branching Anti-Patterns
![Page 14: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/14.jpg)
REID HOLMES - CPSC 410: Advanced Software Engineering
Nightly Build
Systems should be built and executed on a nightly basis. Smoke tests are often used to quickly check the build is not functionally broken.
![Page 15: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/15.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Nightly Build‣ Build code and run smoke tests.‣ Popularized my MSFT in the mid ‘90s
‣ Benefits‣ Minimizes integration risk‣ Reduces risk of low quality‣ Enables early defect diagnosis‣ Improves morale
![Page 16: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/16.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Build Automation‣ Automated build steps:‣ 1) Checkout the system.‣ 2) Acquire all dependencies.‣ 3) Build the system.‣ 4) Report build status.
‣ Can be built:‣ On demand‣ Through a schedule ‣ Via a trigger
![Page 17: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/17.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Build Automation
HTTPS://UPLOAD.WIKIMEDIA.ORG/WIKIPEDIA/COMMONS/9/91/BOEING-WHICHATA_B-29_ASSEMBLY_LINE_-_1944.JPG
![Page 18: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/18.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Big Steps1. Checkout the system:‣ Configuration Management (git, hg)
2. Acquire required dependencies:‣ Package Managers (apt, npm, gem)
3. Build the system:‣ Build Managers (ant, mvn, gulp, rake)
4. Run tests:‣ Test Runners (jUnit, karma, rake test)
![Page 19: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/19.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Smoke Tests‣ Preliminary tests to reveal catastrophic failures.‣ Only a few tests will run.‣ Tests should not be flaky.‣ Tests should be cheap to run.‣ Important for systems with huge test suites.
‣ Validates key functionality.‣ If they do not pass, it is not worth bothering with
the full suite; the build is ‘broken’.
![Page 20: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/20.jpg)
REID HOLMES - CPSC 410: Advanced Software Engineering
Continuous Integration
A practice where developers automatically build, test, and analyze each change committed to the source repository.
![Page 21: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/21.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Why CI?‣ Detect and fix problems faster.‣ Measurable software health.‣ Document environmental assumptions.‣ Effort on automation eases release stress / errors.‣ Global failure feedback. Gives developers trust in
their changes.
![Page 22: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/22.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Principles‣ Invest in automated tools.‣ Frequent commits.‣ Only commit compiling code.‣ Fix broken builds immediately.‣ All tests must pass.‣ Run private builds.‣ Pull code only from known-good configurations.
![Page 23: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/23.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Risks (of not doing CI)‣ Little confidence tests will pass after integration.‣ Long integration phases.‣ Inability to generate testable builds.‣ Fear of making large changes (e.g., refactoring).‣ Late defect discovery.‣ Low quality awareness.
![Page 24: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/24.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Automated Testing at Google‣ 10,000 devs‣ 50,000 builds / day‣ Monorepo‣ 20+ changes / minute
‣ 10 million test suites / day‣ 60 million test cases executed per day
HTTP://WWW.INFOQ.COM/PRESENTATIONS/CONTINUOUS-TESTING-BUILD-CLOUD
Practical Issues: Load Growth and Spikes
● Bigger and bigger tests○ Long running tests with lots of dependencies trigger often
● Load spikes○ Most submits are during west-coast US working hours○ Low-level changes can trigger many thousands of tests○ Everyone submits before going to lunch
![Page 25: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/25.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Code Analysis‣ Linting‣ Flags suspicious source code constructs.‣ Heuristic-based.‣ e.g., use before def, div/0, range violations.
‣ Esp. important for interpreted languages.‣ Dependency analysis‣ Checks code dependencies and identifies call
sites that could be affected by a change.‣ Architectural compliance checking‣ Ensure change does not violate model.
![Page 26: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/26.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
![Page 27: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/27.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
CI Challenges‣ Granularity:‣ Every change? Time delta? Batching?
‣ How to deal with failures:‣ Notify one person? Team? Organization?
‣ Tool support:‣ Remote test execution (Jenkins, Travis CI)
‣ Track build failures (Jenkins, Travis CI)
‣ Track per-test failures (Jenkins)
![Page 28: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/28.jpg)
REID HOLMES - CPSC 410: Advanced Software Engineering
Continuous Delivery
A practice that ensures that a software change can be delivered and ready for use by a customer in production-like environments.
![Page 29: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/29.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Continuous Delivery‣ Dogfooding‣ Heavier infrastructure needs:‣ Test infrastructure.‣ Containers helpful:‣ Build & distribute virtual environment (vagrant
can manage VM or containers like Docker)‣ Why Continuous Delivery?‣ Can deploy at _any_ time (e.g., security fixes).‣ Even better feedback / more confidence.
![Page 30: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/30.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Managing Infrastructure‣ Configuring hosts is not easy in practice:‣ Dynamic:‣ Install/manage hosts (cobbler)
‣ Configure host environment (puppet, chef)
‣ Deploy code to hosts (ansible, puppet, chef)‣ Monitoring hosts (puppet, AppDynamics)
‣ Static:‣ Amazon Machine Image (AMI)‣ Similar to above tools but static snapshots.
![Page 31: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/31.jpg)
REID HOLMES - CPSC 410: Advanced Software Engineering
Continuous Deployment
A practice where incremental software changes are automatically tested, vetted, and deployed to production environments.
![Page 32: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/32.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Deployment Automation
HTTPS://DZONE.COM/ARTICLES/DEPLOYMENT-AUTOMATION-VS-BUILD
![Page 33: 410 27 ops-stack - UBC Department of Computer Sciencertholmes/teaching/2015t1/cpsc410/slides/410_27_ops-stack.pdfContinuous Delivery ‣ Dogfooding ‣ Heavier infrastructure needs:](https://reader033.vdocument.in/reader033/viewer/2022042303/5ece609c30baae06585bbe70/html5/thumbnails/33.jpg)
REID HOLMES - CPSC 410: ADVANCED SOFTWARE ENGINEERING
Example PipelineExample Deployment Pipeline