scaling continuous integration practices to teams with parallel development

Scaling Continuous Integration Practices to Teams with Parallel

Development

January 19, 2012

Introduction to Today’s Speakers

Chris Lucca, Technical Evangelist, AccuRev

Chris' experience dates back to his days as a build / release specialist and software engineer for IBM-Rational. He has also filled the roles of developer, release engineer and SCM consultant for numerous organizations. In addition Chris is a certified scrum master and a frequent speaker at the AccuRev seminar series.

Now Chris brings the experience and knowledge he has acquired to help keep others in the industry up-to-date on the latest developments, and how they can impact them directly.

Introduction to Today’s Speakers

Eric Minick, Lead Consultant, UrbanCode

Eric Minick is a lead consultant at UrbanCode where he helps customers get the most out of their build, deploy and release processes. He has 8 years of automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer.

Eric has been at the forefront of continuous integration for 5+ years and has worked on all three generations of UrbanCode’s Anthill. Keep up with Eric'’s latest insights on UrbanCode’s blog.

Agenda

• Intro to CI• Overview of Parallel Development• Adapting CI to the Parallel Dev Challenge• Q&A

Agenda

Jan Mar May JulFeb Apr Jun Aug Sep

Specify Design Code IntegrateWrite tests

DocPlan Test

Traditional Development

Traditionally Devs Work in Isolation

Too Much Isolation Leads to Integration Hell

CI: Minimizing Integration Pain

• Integrating frequently is less painful than infrequently• But, If we commit frequently blindly, we have a

broken code base

• Implied condition: each integration will not decrease the code quality• This is the key to CI • The rest is implementation details

CI: The Simple Implementation

• How do we determine code quality? We test it

• How do we test code? We build it and run tests against it

• This is how we get from CI to CI servers and software builds

So Simple but Very Important

• A build & unit test is a rudimentary check• Gives important feedback• The heartbeat of the dev organization

CI at Scale?

• With too many committers into the same code line, the likelihood something is broken right now is high

• We stop caring about failures

• The feedback is meaningless, and productivity suffers. CI becomes “constantly broken”

Agenda

Reasons People Work In Parallel

It’s often a business reason…not for optimization

• Maintenance Release – Patches

• Customer Specials - Just for one customer

• Development branches – Developer spikes for

experiments that will integrate at an unknown time

• Code Configurations – Different configurations of code

based on environment, such as UAT, QA

• Special Content – Images, binaries

Technical Debt Grows

As code lines diverge, integration becomes a bigger eventual problem. … But we don’t always want to merge immediately

• Maintenance Release – Don’t want new features

• Customer Specials – One directional integration

• Development branches – Dev enjoys a stable playground

• Special Content – Reused at different cadence than code

Agenda

Finding the Isolation / Integration Balance

Integration Is Easy With a Few People

Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4

It’s hard at scale

Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4

Little Hiccup

• Race condition: I could update to the latest code before the build/test notifies me it doesn’t compile

• Cost associated with this is huge

• Leads to “interesting solutions”

• Preflight• Build branches• Build Acceleration• Build Avoidance• Componentization

Cost of Integration

Per week:• # of check-ins per developer?• # of developers?• % of check-ins that break things?• # of problems encountered per developer?• Length of time per problem encountered?

200 devs, 2 commits each, 1% are bad = 4 breaks/day

If each breakage cuts productivity 50% for 15 minutes, we lose 100 dev hours / day (@$50/hr = $5000/day)

Team A

Team B

Multi-Stage Continuous Integration

Minutes Hours

• Continuous

Integration

• Continuous Integration

• Automated Tests

• Unit Tests

• Binary Deploy

• Database Deploy

• Web Deploy

• Database Deploy

• Web Deploy

• Binary Deploy

• Customer Sign-

• Database Deploy

• Web Deploy

• Binary Deploy

• IT Sign-Off

• Rollback (if needed)

x60x30x3x1

x1x4x10x100s

x1x2x5x10

0:102:003:005:00

• Team Specific CI

• Continuous

Integration

• Automated Tests

• Unit Tests

• Binary Deploy

• Database Deploy

• Web Deploy

• Database

Deploy

• Web Deploy

• Binary Deploy

• Customer Sign-

• Database Deploy

• Web Deploy

• Binary Deploy

• IT Sign-Off

• Rollback (if needed)

Automated Merges

A Feature Branch Example

Urbancode’s Website Today… Next (not ready for prime time)

We need two build configurations, deploy configs, test configs, and test environments

Building and Testing Multiple Code Lines

• Build and test every active code line (continuously)

• Automatically run code scans, unit tests and functional tests

CI for a New Code Line Must Be Cheap

• Build and deployment configuration• Maximize reuse• Potentially create dynamically

• Tests• Core regression suites shared across code lines• Tests for new functionality shared as code is shared

• Test Environments• No excuse for a lack of test environments today• Deploy “what’s in production” as a baseline

Deploy Time Integration Is the Same

• In the team’s environment:• Continuously deploy and execute functional tests• Regularly (nightly / weekly) deploy other

components that have passed integration tests

• When new features work well, deliver to the integration test environment

• Releases to Stage & Prod match the full contents of an earlier test environment

In Closing

• Compile is the first test

• Unit test, deploy, automation can come after

• Get Feedback as Fast as possible

• Different teams need different types of builds

• Separate code configurations based on process

• It’s about people, but tooling helps

Best Practices For:Agile

Parallel

Global

Compliance

Change ManagementVisual Process

Automation

What Is AccuRev?

Much more than Version Control…AccuRev enables Software Process Automation

VersionControlSCM

Urbancode Products

uBuild (AnthillPro)• Continuous integration and delivery

uDeploy• Deployment and release management

uCloud (soon)• Environment provisioning from your private cloud

Call: +1 216 858-9000Email: sales@urbancode.comVisit: www.urbancode.comLearn: blogs.urbancode.com

Call: 1-800-383-8170 or +1-781-861-8700Email: sales@accurev.comVisit: www.accurev.comLearn: blog.accurev.com

scaling continuous integration practices to teams with parallel development

integration hell copyright

isolation copyright

people chapter

scale chapter

dev organization copyright

parallel developmentjanuary

release engineer

integration pain

Technology

scaling dev teams with purpose

eclipsecon 2012: agile xxl - scaling agile for project teams

parallel trajectories: distance learning, oer, and scaling...

teaching an elephant to dance or scaling agile to large...

scaling up word sense disambiguation via parallel texts

scaling multicore databases via constrained parallel...

scaling humans - ops teams and incident management

strong scaling analysis of a parallel, unstructured...

building and scaling technical teams

scaling agile across project teams and departments - … ·...

scaling sql server 2014 parallel insert

why include business context in project readiness? adapt - a...

understanding application scaling nas parallel benchmarks...

point nine saas summit - scaling teams

parallel scaling analysis for explicit solid dynamics in

scaling up machine learninglarge_scale_survey/suml.pdf ·...

scaling human robot teams

overcoming more impediments to agile transformation -...

rooﬂine scaling trajectories: a method for parallel

vrisha: using scaling properties of parallel programs for...