scaling continuous integration practices to teams with parallel development
DESCRIPTION
Slides from an Urbancode and Accurev joint webinar: http://www.accurev.com/webinar/20120119-Scaling-CI-Parallel-DevelopmentContinuous integration is simple with a single development team. But when software projects grow to multiple teams and dependencies, continuous integration loses effectiveness due to parallel projects, varying release schedules, and differing cadences between teams. As a result, many teams unknowingly lose the benefits of continuous integration, and therefore suffer from a lack of feedback and poor quality. In this webinar, UrbanCode’s Eric Minick and AccuRev’s Chris Lucca will explain how to: - Scale continuous integration builds across multiple development teams working on parallel projects - Share only code that has passed continuous integration from other teams to avoid broken builds and confusion - Automate the configuration of your test environment to handle fluid projects done in parallelTRANSCRIPT
![Page 1: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/1.jpg)
Scaling Continuous Integration Practices to Teams with Parallel
Development
January 19, 2012
![Page 2: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/2.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Introduction to Today’s Speakers
Chris Lucca, Technical Evangelist, AccuRev
Chris' experience dates back to his days as a build / release specialist and software engineer for IBM-Rational. He has also filled the roles of developer, release engineer and SCM consultant for numerous organizations. In addition Chris is a certified scrum master and a frequent speaker at the AccuRev seminar series.
Now Chris brings the experience and knowledge he has acquired to help keep others in the industry up-to-date on the latest developments, and how they can impact them directly.
![Page 3: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/3.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Introduction to Today’s Speakers
Eric Minick, Lead Consultant, UrbanCode
Eric Minick is a lead consultant at UrbanCode where he helps customers get the most out of their build, deploy and release processes. He has 8 years of automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer.
Eric has been at the forefront of continuous integration for 5+ years and has worked on all three generations of UrbanCode’s Anthill. Keep up with Eric'’s latest insights on UrbanCode’s blog.
![Page 4: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/4.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
![Page 5: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/5.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
![Page 6: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/6.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Jan Mar May JulFeb Apr Jun Aug Sep
Specify Design Code IntegrateWrite tests
DocPlan Test
Traditional Development
![Page 7: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/7.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Traditionally Devs Work in Isolation
![Page 8: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/8.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Too Much Isolation Leads to Integration Hell
![Page 9: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/9.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI: Minimizing Integration Pain
• Integrating frequently is less painful than infrequently• But, If we commit frequently blindly, we have a
broken code base
• Implied condition: each integration will not decrease the code quality• This is the key to CI • The rest is implementation details
![Page 10: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/10.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI: The Simple Implementation
• How do we determine code quality? We test it
• How do we test code? We build it and run tests against it
• This is how we get from CI to CI servers and software builds
![Page 11: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/11.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
So Simple but Very Important
• A build & unit test is a rudimentary check• Gives important feedback• The heartbeat of the dev organization
![Page 12: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/12.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI at Scale?
• With too many committers into the same code line, the likelihood something is broken right now is high
• We stop caring about failures
• The feedback is meaningless, and productivity suffers. CI becomes “constantly broken”
![Page 13: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/13.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
![Page 14: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/14.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Reasons People Work In Parallel
It’s often a business reason…not for optimization
• Maintenance Release – Patches
• Customer Specials - Just for one customer
• Development branches – Developer spikes for
experiments that will integrate at an unknown time
• Code Configurations – Different configurations of code
based on environment, such as UAT, QA
• Special Content – Images, binaries
![Page 15: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/15.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Technical Debt Grows
As code lines diverge, integration becomes a bigger eventual problem. … But we don’t always want to merge immediately
• Maintenance Release – Don’t want new features
• Customer Specials – One directional integration
• Development branches – Dev enjoys a stable playground
• Special Content – Reused at different cadence than code
![Page 16: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/16.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Agenda
• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A
![Page 17: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/17.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Finding the Isolation / Integration Balance
![Page 18: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/18.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Integration Is Easy With a Few People
Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4
![Page 19: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/19.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
It’s hard at scale
Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4
![Page 20: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/20.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Little Hiccup
• Race condition: I could update to the latest code before the build/test notifies me it doesn’t compile
• Cost associated with this is huge
• Leads to “interesting solutions”
• Preflight• Build branches• Build Acceleration• Build Avoidance• Componentization
![Page 21: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/21.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Cost of Integration
Per week:• # of check-ins per developer?• # of developers?• % of check-ins that break things?• # of problems encountered per developer?• Length of time per problem encountered?
200 devs, 2 commits each, 1% are bad = 4 breaks/day
If each breakage cuts productivity 50% for 15 minutes, we lose 100 dev hours / day (@$50/hr = $5000/day)
![Page 22: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/22.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI
CI
CI
Team A
Team B
ws
ws
ws
Multi-Stage Continuous Integration
Minutes Hours
int
Daily
ws
ws
ws
![Page 23: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/23.jpg)
• Continuous
Integration
• Continuous Integration
• Automated Tests
• Unit Tests
• Binary Deploy
• Database Deploy
• Web Deploy
• Database Deploy
• Web Deploy
• Binary Deploy
• Customer Sign-
Off
• Database Deploy
• Web Deploy
• Binary Deploy
• IT Sign-Off
• Rollback (if needed)
x60x30x3x1
x1x4x10x100s
x1x2x5x10
0:102:003:005:00
![Page 24: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/24.jpg)
• Team Specific CI
• Continuous
Integration
• Automated Tests
• Unit Tests
• Binary Deploy
• Database Deploy
• Web Deploy
• Database
Deploy
• Web Deploy
• Binary Deploy
• Customer Sign-
Off
• Database Deploy
• Web Deploy
• Binary Deploy
• IT Sign-Off
• Rollback (if needed)
Automated Merges
![Page 25: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/25.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
A Feature Branch Example
Urbancode’s Website Today… Next (not ready for prime time)
We need two build configurations, deploy configs, test configs, and test environments
![Page 26: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/26.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Building and Testing Multiple Code Lines
• Build and test every active code line (continuously)
• Automatically run code scans, unit tests and functional tests
![Page 27: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/27.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
CI for a New Code Line Must Be Cheap
• Build and deployment configuration• Maximize reuse• Potentially create dynamically
• Tests• Core regression suites shared across code lines• Tests for new functionality shared as code is shared
• Test Environments• No excuse for a lack of test environments today• Deploy “what’s in production” as a baseline
![Page 28: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/28.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Deploy Time Integration Is the Same
• In the team’s environment:• Continuously deploy and execute functional tests• Regularly (nightly / weekly) deploy other
components that have passed integration tests
• When new features work well, deliver to the integration test environment
• Releases to Stage & Prod match the full contents of an earlier test environment
![Page 29: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/29.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
In Closing
• Compile is the first test
• Unit test, deploy, automation can come after
• Get Feedback as Fast as possible
• Different teams need different types of builds
• Separate code configurations based on process
• It’s about people, but tooling helps
![Page 30: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/30.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Best Practices For:Agile
Parallel
Global
Compliance
Change ManagementVisual Process
Automation
What Is AccuRev?
Much more than Version Control…AccuRev enables Software Process Automation
VersionControlSCM
![Page 31: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/31.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Urbancode Products
uBuild (AnthillPro)• Continuous integration and delivery
uDeploy• Deployment and release management
uCloud (soon)• Environment provisioning from your private cloud
![Page 32: Scaling Continuous Integration Practices to Teams with Parallel Development](https://reader031.vdocument.in/reader031/viewer/2022013011/554f907cb4c905435d8b5044/html5/thumbnails/32.jpg)
Copyright © 2012 Urbancode, Inc. and AccuRev, Inc.. All Rights Reserved.
Q&A
Call: +1 216 858-9000Email: [email protected]: www.urbancode.comLearn: blogs.urbancode.com
Call: 1-800-383-8170 or +1-781-861-8700Email: [email protected]: www.accurev.comLearn: blog.accurev.com