evolving infrastructure
TRANSCRIPT
![Page 2: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/2.jpg)
Workday Introduction• Background / Our Own Cloud
2.0 Architecture• Config Management / Chef Build Pipeline /
Chef Development Tools
3.0 Architecture• Release & Deployment Changes / Image Build Pipeline /
Release Management / Planned vs. Unplanned Changes
![Page 3: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/3.jpg)
Enterprise HCM & Financials software
in the cloud
First releaseNovember 2006
![Page 4: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/4.jpg)
![Page 5: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/5.jpg)
![Page 6: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/6.jpg)
Workday Introduction• Background / Our Own Cloud
2.0 Architecture• Config Management / Chef Build Pipeline /
Chef Development Tools
3.0 Architecture• Release & Deployment Changes / Image Build Pipeline /
Release Management / Planned vs. Unplanned Changes
![Page 7: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/7.jpg)
![Page 8: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/8.jpg)
One of the DevOps mantras is
Infrastructure is Code
![Page 9: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/9.jpg)
If it’s code…
→ you need to test it
→ you need a build & test pipeline
![Page 10: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/10.jpg)
Local Development
CodeReview
C.I.Unit Tests
All triggered by developers
pushing code
Bronze Cookbook Artefacts
Local Development
CodeReview
C.I.Unit Tests
Local Development
CodeReview
C.I.Unit Tests
![Page 11: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/11.jpg)
C.I.System Tests
C.I.System Tests
C.I.System Tests
Failures(back to dev)
System TestsTriggered on
the Hour
Bronze Cookbook Artefacts
Silver Cookbook Artefacts
![Page 12: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/12.jpg)
C.I.Integration Tests
Gold Cookbook Artefacts
Silver Cookbook Artefacts
Failures(back to dev)
Integration TestsTriggered Several
Times a Day
![Page 13: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/13.jpg)
reekChefSpec
RSpec
![Page 14: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/14.jpg)
System &Integration
reekChefSpec RSpec
Unit Testing
![Page 15: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/15.jpg)
Where Do We Run Our ServerSpec Tests?• Lab Hardware?• Vagrant?• AWS?
![Page 16: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/16.jpg)
The Lab?• Hardware Based• Can’t set machine state before the test run• Can’t reset machine state after the test run
![Page 17: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/17.jpg)
Run In Vagrant?• Good for simple cases• Harder for integration testing a few dozens
Chef roles• Prefer a hosted platform with longer running
nodes for some services like artefact repos
![Page 18: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/18.jpg)
![Page 19: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/19.jpg)
Unit System Integration
![Page 20: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/20.jpg)
https://www.chef.io/delivery/
https://downloads.chef.io/chef-dk/
![Page 21: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/21.jpg)
Workday Introduction• Background / Our Own Cloud
2.0 Architecture• Config Management / Chef Build Pipeline /
Chef Development Tools
3.0 Architecture• Release & Deployment Changes / Image Build Pipeline /
Release Management / Planned vs. Unplanned Changes
![Page 22: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/22.jpg)
Platform Services
Appliances
Workday
Linux Servers
Image Based Deployment
Chef / CobblerBased Deployment
![Page 23: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/23.jpg)
■ Cobbler for the OS■ Chef based deployments
of system / infrastructure changes
■ Custom tooling for applications deployments
■ Cobbler / Chef for bare metal
■ Most services moving to image based deployments
■ Custom deployment tools to manage VM lifecycle
2.0 Deployments 3.0 Deployments
![Page 24: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/24.jpg)
1. Where in the build & test pipeline do the Machine Images get created?
2. What technology & processes are used to create them?
![Page 25: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/25.jpg)
• Early in the pipeline
• Application teams → image artefact
• Image artefact → build & test pipeline
![Page 26: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/26.jpg)
• Lots of tools to choose from:
• Diskimage-builder
• VMBuilder
• Box Grinder
• Packer
• Imagefactory
• We use Oz (https://github.com/clalancette/oz)
![Page 27: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/27.jpg)
KickStartFile
OZTemplate
Base Image
OZCentOS Core
Image Manifest
Unit + SystemTests
System RPMs
![Page 28: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/28.jpg)
OZTemplate
Application Image
OZ
Base Image
Image ManifestApplication
RPMs
Image Build Service
![Page 29: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/29.jpg)
Gold Cookbook Artefacts
Gold Application
Artefacts
Promoteto Staging
UNIT SYSTEM INTEGRATION
UNIT INTEGRATION
I N F R A S T R U C T U R E
A P P L I C A T I O N SSYSTEM
Promote to Production
![Page 30: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/30.jpg)
Image Build Service
SYSTEM TESTS
UNIT INTEGRATION
I N F R A S T R U C T U R E
SYSTEM
Promoteto Staging
A P P L I C A T I O N S
Base Image
Base Image
Application RPMs
Application Image
Application Image
![Page 31: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/31.jpg)
The ability to push code to environments easily and quickly - push button deploys
A stable framework for development, testing, deployment and auditing
Deploy applications and operational environments in the same way each time
Source code, tools and all components that make up releaseIdentifiability
Reproducibility
Consistency
Agility
![Page 32: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/32.jpg)
■ Images■ Startup Properties
■ Cookbooks■ Roles■ Data Bags■ Encrypted Data Bags■ RPMs■ Other artefacts (e.g. ruby
& python packages)
Config Management Image Deployment
![Page 33: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/33.jpg)
■ Destroy and recreate rather than change in place
■ All facets of the OS are captured by the image artefact
■ Server state mutated over time as updates are applied
■ Impractical to manage every last detail of the OS with config management
Config Management Image Deployment
![Page 34: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/34.jpg)
• The ideal is…• Push all changes through the image pipeline
for both planned and unplanned changes• If your pipeline is reliable with a quick
turnaround you can use this for all changes• We also like to have a Break The Glass option
Planned Vs Unplanned Changes
![Page 35: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/35.jpg)
• For low impact changes only• Bash script → RPM• Agent on each VM can deploy RPMs• Trigger remotely via secure channel• Testable, easy to roll out, good for auditability
Break The Glass
![Page 36: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/36.jpg)
MonitoringDeployment
(*)
(*) developed in house
![Page 37: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/37.jpg)
Stats & MetricsLogging
![Page 38: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/38.jpg)
“If it hurts,do more of
it”
![Page 39: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/39.jpg)
http://www.beatcleaver.com/portfolio/https://www.flickr.com/photos/cote/ DC Image
Melissa StolbergMichael CotéPaul McAuley
![Page 40: Evolving Infrastructure](https://reader036.vdocument.in/reader036/viewer/2022070512/5899f9f31a28abc5778b5da3/html5/thumbnails/40.jpg)