the anatomy of continuous deployment at scale - 100 deploys a week at envato 2016
TRANSCRIPT
![Page 1: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/1.jpg)
The Anatomy of Continuous Deployment
At Scale
John Viner (Dev Manager)Mario Visic (Lead Developer of Delivery Engineering)
100 Deploys a Week at Envato
![Page 2: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/2.jpg)
Who is Envato?
![Page 3: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/3.jpg)
A two-sided Marketplace for creative assets that has paid out $400 million to our
“Authors” in the last 10 years
7 Sites where digital professionals can buy web templates, video, audio, code, graphics and 3d
models 7 million registered users10 million items for sale
![Page 4: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/4.jpg)
Themeforest.net
![Page 5: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/5.jpg)
Show me the Tech Stack
Ruby on Rails (100K loc, 25 web servers)
MySql (175 tables)
Memcached + RedisRackspace / VMWare
Amazon AWS
![Page 6: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/6.jpg)
Show me the Numbers!
Peak of 25,000 requests p/min through Rails160m Page Views per Month
500,000 Database calls p/min Serving 200TB of data a month
22,000 Unit Tests, 2,000 Integration Tests
![Page 7: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/7.jpg)
How do we deliver change?
![Page 8: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/8.jpg)
By the numbers … for the month of June 2016
~ 40 developers3000 total builds (2.5 builds p/dev/day)
500 Pull Requests Closed400 deployments (18 p/day avg, 0.5 p/dev/day, peak 25)
4.5 hour cycle time (from first commit to deploy - working hours)
![Page 9: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/9.jpg)
Statistics over Time
Size of a Pull Request (median)37 lines changed3 files changed
Cycle Time of a PR (first commit to deploy)average: 10.8 hours (working hours)median: 2.6 hours (working hours)
Deployments8400 deployments in 2 years
![Page 10: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/10.jpg)
Pretty Charts - Builds
![Page 11: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/11.jpg)
Deployments
![Page 12: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/12.jpg)
Pull Requests
![Page 13: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/13.jpg)
PR Cycle Time (First Commit to Deployment)
![Page 14: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/14.jpg)
Building Envato Market
![Page 15: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/15.jpg)
Feature request
• 2 week sprints• Work with product team to break down work• No task larger than two days• Feature toggle can be used to push incomplete features
![Page 16: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/16.jpg)
Feature request
![Page 17: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/17.jpg)
Implementation
• Write the tests (sometimes first)• Write the code• Peer review• Get it into production
![Page 18: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/18.jpg)
Implementation
Tests live alongside code, stored in Github
![Page 19: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/19.jpg)
Implementation
Github Pull Request are the unit of change, 2 thumbs up by reviewers means the code is good to ship to production
![Page 20: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/20.jpg)
Implementation
![Page 21: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/21.jpg)
Implementation
Buildkite is our CI tool, a merge on master automatically runs a build. A successful build automatically triggers a production
deploy
![Page 22: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/22.jpg)
Deployment
• Entirely Automated• Post deploy monitoring• Quick rollback• Multiple time per hour
![Page 23: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/23.jpg)
Deployment
Samson (open source) is our deployment tool, successful master builds triggers a deploy which is logged in Samson.
![Page 24: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/24.jpg)
Deployment
We can run manual deployments from Samson and view all PRs in a given deployment.
![Page 25: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/25.jpg)
Deployment
We can trace back to specific changes in a deploy.
![Page 26: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/26.jpg)
Deployment
… including the amount of lines changed. A typical PR changes only 37 lines of code across 3 files
![Page 27: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/27.jpg)
Deployment
![Page 28: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/28.jpg)
Why do we do this?
![Page 29: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/29.jpg)
Lowers Risk
Size of Change is Small means …Easier to identify any failureEasier to rollback any failure
Reduces human errors due to reliance on automationSecurity patches can be deployed within minutes
![Page 30: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/30.jpg)
And also
Reduces Waste:Fewer handovers from Development to QA to Ops
Increases Product Agility:New features go to customers sooner to get earlier feedback
Increases developer ownership and engagement:More direct connection between your creation and it being used
![Page 31: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/31.jpg)
How to get there
![Page 32: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/32.jpg)
Start with Culture
Foster an Ownership Culture in Developers
http://webuild.envato.com/expectations/collaboration/4-i-take-responsibility-for-syste.html
![Page 33: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/33.jpg)
Investment in automation of unit tests, functional tests
Tests to cover legacy systems
Add Tests whenever un-tested code is touched
Set Targets and Measure Test Coverage and # Tests
![Page 34: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/34.jpg)
Automate Deployments and Enable Feature Toggles
Automate your manual run sheet one line at a time
Start with Push Button
Feature Toggles help deploy features without showing them
![Page 35: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/35.jpg)
Start Small
Use a less risky or complicated system first
![Page 36: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/36.jpg)
Investment in monitoring and rollback
Knowing that the system is working within seconds becomes critical
Developers need to be “on the pager”
Invest in the ability to Roll Back
Optimise for Mean Time to Recover
![Page 37: The Anatomy of Continuous Deployment at Scale - 100 deploys a week at Envato 2016](https://reader036.vdocument.in/reader036/viewer/2022062901/58f2d97b1a28ab68468b45ab/html5/thumbnails/37.jpg)
Thank you!