agile australia conference 2011 - devops live accounts- continuous delivery_st
DESCRIPTION
Presentation at Agile Australia 2011 http://www.agileaustralia.com.au/2011/topics-day-one.html#liveaccounts-devops Nish Mahanty - Software Delivery Manager, MYOB » Lawrence Song - Technical Architect, MYOB Live Accounts is an online accounting application. It was a 10 year-old legacy system with complex architecture and no test or build scripts. The manual deployment was quite complex, involving deploying one Java application and three DotNet applications to three windows servers. Over the past six months, MYOB has progressed incrementally from manual build and deploy processes based on Perforce, to CI and semi-automated deployments (using Perforce, Hudson, Maven,) to fully automated delivery (using Go, Git, Rake). This talk summarises that journey, and explores the technical challenges and lessons-learned. MYOB has measured the increase in number of deployments, decrease in deployment issues, and decrease in deployment time over the six months. Developers were working closely with Ops to understand the pain points and automated the deployment process as much as possible to make their lives easier. This talk explains the business problem and how to begin the incremental, iterative, adaptive journey to Continuous Delivery for a complex legacy system, illustrated with data and technical tips. Attendees will discover: » The value argument for Continuous Delivery » Clear steps on how to integrate DevOps and progress on the automation journey » Insights into a common set of tools, with the opportunity for a technical in-depth pros and cons discussionTRANSCRIPT
LiveAccounts: Continuous Delivery in the Real World
Nish Mahanty
Lawrence Song
“Rapidly deliver value to our customers” CTO
Good news journey Quadrupled our feature throughput Shortened our delivery times by 87% Improved team satisfaction Metrics Team empowerment Tools
Just the facts Ma’am
Legacy code
Manual Build
Unpredictability
Distributed team
Real bug?
Late night deployments
Develop Smoke
Functional
Smoke Regression
Smoke
Delivery Value Stream
3 3 1
Prod Staging QA 1 3 0.3
Compile Assemble Package Restore DB Apply DB Changes Deploy Package
Assemble Package Restore DB Apply DB Changes Deploy Package
Assemble Package Apply DB Changes Deploy Package
Dev
Dude, where’s my new feature?
(and what we don’t)
What we believe in
Code quality
Current Before
Automated functional tests
0
21
56
79
110
132 146
160 173
S O N D J F M A M
# of Scenarios
60 65 65 65
80 83
93 94 94
S O N D J F M A M
% Coverage
Unit test coverage – new code
Code branches
5
2 1
Before Now Next
Deployment Duration
S O N D J F M
130 98
Cycle Time (Features and Defects)
S O N D J F M
30
9.8
Delivery
Duration
87% faster release
11
1
How we did it
One team
Develop Smoke
Functional
Smoke Regression
Smoke 3 3 1
Prod Staging QA 1 3 0.3
Dev
Build a common vision
Common KPIs
Build Trust
Improve Comms
Empower the team to change things
Approach
• Close collaboration
• Incremental
• Data driven
• Automation
• Consensus base
Develop Smoke
Functional
Smoke Regression
Smoke
Development Environment
3 3 1
Prod Staging QA 1 3 0.3
Dev
3 days to set up environment Shared database Shared environment Manual build No continuous integration build
30 min to set up Database change management Development sandbox Automated builds
Evolution, not revolution
Maven Buildr
Hudson Go
Perforce Git
Manual
No CI
Code Quality
• Agreed on coding standards as a team
• Unit Test Code Coverage: Minimum Value 80%
• Cyclomatic Complexity <= 7
• Checkstyle
• JsLint
• Fail a build
Develop Smoke
Functional
Smoke Regression
Smoke
Deployment
3 3 1
Prod Staging QA 1 3 0.3
Dev
3 days to QA environment 20 page checklist 1 day getting prod data to env Similar timeframes for stg/prod Frustrated QAs
30 min to each environment Auto configuration management 1 click deployment 1 click restore prod db
Develop Smoke
Functional
Smoke Regression
Smoke
Testing
3 3 1
Prod Staging QA 1 3 0.3
Dev
4 days (4 QAs) Tedious regression Slow feedback cycle
1 day (2 QAs) Automated functional tests
Behaviour Driven Dev’t by empowered testers
Automated continuous delivery pipeline
Dev Team
Version Control
Build, Test & Assemble
Automated regression test
QA Env & Smoke test
Staging Smoke Test
check in
trigger
Prod & Smoke Test
trigger
feedback
feedback
one click
one click
one click
feedback
feedback
feedback
What did we learn?
• Automation, Automation, Automation
• Continuous feedback/improvement
• Takes longer to automate for Windows
• Put infrastructure configuration into version control
• Run automated smoke tests for each deployment
Change in value stream times
1 hr 7 hrs 1 hr
Develop Smoke
Functional
Smoke Regression
Smoke 3 3 1
Prod Staging QA
1 3 2 hrs
Dev
30 min 30 min 30 min
Delivery
Duration
87% faster release
11
1
How you can do it
How do I sell this to my stakeholders?
Emphasise business benefits
– increased delivery predictability
– shorter time-to-market
– less rework
– lower risk
– lower application total cost of operation
Show them the metrics!
Where do I start?
Team: What’s the major pain point?
Automate manual processes
Measure stuff!
Use experts to accelerate adoption and reduce learning curve
