ImplementingZero-Debt Continuous Inspection in an Agile MannerA Case Study

Brian ChaplinOctober 8, 2013

Agenda• What is technical debt?• What is continuous inspection?• A scalable framework to automate technical

debt reporting• How to control technical debt using the

framework• Best practices• Lessons learned from 2 case studies

– Agile lessons learned• What you can do tomorrow

Technical Debt Metaphor

• Ward Cunningham at OOPSLA 92 : “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite...”

• However,– Every minute spent on debt counts as interest

Consequences of Technical Debt

• Product atrophy• Decreased

predictability• Underperformance• Universal frustration• Decreased customer

satisfaction

• Unpredictable tipping point

• Increased time to delivery

• Significant number of defects

• Rising development and support costs

From Essential Scrum, Kenneth S. Rubin

Types of Debt• Naïve

– reckless debt (Fowler 2009), unintentional debt (McConnell 2007), and mess (Martin 2008).

• Unavoidable– Although this debt might be predictable it’s not

preventable because we can’t foresee how the component might evolve

• Strategic– deliberately make a strategic decision to take shortcuts

during product development to achieve an important short-term goal, such as getting a time-sensitive product into the marketplace.

Naïve Technical Debt DefinedCoverage Uncovered conditions

Uncovered linesComplexity Average method complexity

Total class complexityCompliance 140 static code analysis rules

Critical, major, minor weightsDuplication Duplicated linesOrganization Circular dependencies

Design Fan-out, 6

What is Continuous Inspection

• an approach to running automated code analysis as part of a build in order to find code quality problems

Why Continuous Inspection?

• 5. Sustainable development, able to maintain a constant pace

• 9. Continuous attention to technical excellence and good design

• can help reduce the time spent in long, manual code review sessions

Continuous Inspection Tooling

• SonarQube www.sonarqube.org

• CAST www.castsoftware.com

• Microsoft Visual Studio/Team Foundation Server

Automating Continuous Inspection Reporting

• Necessary for large projects–Agile and timely reporting within hours

• Necessary for zero-debt–Architects only have time for the big

issues–Computer handles the smaller quality

defects

Business Context

• Large project, semi-Agile• IT history of past quality initiatives

– OO and pair programming– Automated unit test generation

• Management goal of 80% test coverage• Continuous integration build/deploy

Characteristics of 2 Case StudiesCategory Java

Open SourceC# dot Net

Developers 100 175LOC 2,200,000 1,600,000Commits per day 30 36Classes 14,000 13,000Unit tests, Coverage 41,000 83% 22,000 63%

File changes per day 225 500Technical debt per day 20 50Code quality since 2009 Dec. 2012

Architecture Requirements for Code Quality (CQ) Statistics

• Must come from existing continuous integration process

• Available within 24 hours• Meaningful to developers, leads and

management• Must involve reviewers• Must tie to accountability points

– Leads, reviewers, submitters, project, business function

• Extract, transform and load

• Match the committer with the quality change

Architecture ETL Overview

Architecture Overview• Extract/Transform/Load

(ETL)• Input

– Source code management system (SCM)

– Project management system

– Code quality metrics

• Output– SQL database– Excel pivot table reports– Defect tickets (project

management system)– Email– Ad hoc management

reports– Program scores and

developer contribution

Establish and Maintain the Quality Database

• Guard against corrupted quality data– Test or build failures– Sonar may record lower stats– Rollback a bad build

• Design for mash-ups• Combine code quality with runtime defect

reporting

Maximize Report Flexibility• Enables timely and actionable reporting• Excel pivot table as data-marts• Refreshed Excel reports accessed via Email

hyperlinks or served up by web server

Bake Code Quality into the Build and Sprint

Notificationsin every phase Sprint

Code Quality ChangeTechnical DebtOpen Tickets To Fix DebtDaily

Integration Tests

Code Quality Champions Personalized CQ ReportContinuous

Auto Build Unit Test Deploy

Email Code Quality Warnings 18

Fast Feedback

Technical Debt Information Radiator

1. Fast feedback: identify and notify Immediately

–Inside the IDE–After the commit

2. Track until it’s fixed

1. Notify Immediately• Email each debt violation within hours

– After each continuous inspection build• 3 times a day or after every CI build

• Notify both committer and code reviewer• Daily contribution

– Best contributors summary– Personalized contribution detail to each

committer• Hyperlink to more reports and wiki

– Direct Sonar hyperlink to the degraded file

Fast Feedback

Project ID

Change ID

Prog. Name

UncoveredLines or Branches

Static Code

Analysis Debt

Excess Complexity

Class Function

ticket1 580932 4/18/2013 Class1 9 9

ticket1 580902 4/18/2013 Class3 4 0 14 2.8

ticket1 580902 4/18/2013 Class5 23 0 13 2.6

Submission Warning(s)You are receiving this message because even though the code quality may have been enhanced, the submission(s) below have decreased the code quality (coverage, compliance, and/or uncovered complexity) or they didn't meet the standards for a new class. Please review these submissions with your reviewer and take the appropriate action.

Sample Violation Email

• Metric• Submitter• Lead• Reviewer• Class• Sprint• Date

2. Track Until It’s

Fixed

2. Track Debt Until It’s Fixed

Making Technical Debt Visible

• Estimate the cost of remediation• Assign owner• Report weekly progress by owner• Report trends• Monetize it, various methods

– SQALE (Software Quality Assessment based on Lifecycle Expectations) from Inspearit, Jean-Louis Letouzey

Monetize the DebtEstimated Remediation Costs

Cost Activity12 minutes To unit test uncovered branch or

line6 minutes To fix static code analysis violation8 hours To split a class that is too complex30 minutes To split a method that is too

complex2 hours To eliminate a block of duplicated

code

Current debt

Ratio of bad to good code

Weekly progress

Report Weekly Progress by Owner

Report Weekly Tech Debt Trends

• By project• Or by

owner

• Staff a code quality desk–Knowledge clearinghouse

• Use code reviewers• Stabilize build and project structure• Establish static code analysis rules and

rarely change them

Best Practices

• Developers must be able to clear unfixable debt from the tracking report

• Run the ETL at least daily• Keep the database accurate• Track both contribution and debt• Recognize code quality champions• Use uncovered line/branch count not

percentage

Best Practices, cont’d

Allow for Exclusions, False Positives

• Integrations, class rename, code moves

• Classes exempted from quality–Registries–Test support

• Unfixable debt–Unreachable test

cases–Exceptions to the

metrics standard–Static analysis false

positives• 0.5% are excluded

Case Study Lessons Learned• Communicate the benefits of code quality

– Maintainability– Less runtime defects, explain carefully– Working with a net

• Help new developers with un-testable code– Train and mentor– Consider refactoring to testable code first, then

write the test

Reviewers Guard Against Doing the Wrong Thing

• Meaningless comments• Unit tests just for coverage that assert

nothing• Allow some large case statements

Too Busy to Keep Debt Down?

Monthly debt rate

Managing the Debt

Rate

• Debt rate is about 15% for both projects

• Varies widely by developer

Techniques to Keep the Debt Rate Low

• Immediate feedback• Open defect tickets at

key points• Train• Assist developers one

on one• Encourage re-

factoring to write testable code

• Code a little, test a little

• Keep management aware

• Continually monitor, don’t let it get out of hand

• Recognize the champions

Count 282 8,447Average value NPE Not Function complexity 2.90 1.88 Lcom4 1.98 1.14 Weighted violations 7.52 2.70 Complexity 66 11 Coverage 85% 83% Duplicated lines 1.99 5.61 Comment lines density 28% 32% Statements 144 25 Score 1.53 1.73 Fan-out 9.45 3.16

Probability =exp( a + b*WEIGHTED_VIOLATIONS + c*COMPLEXITY -d*COVERAGE -

e*COMMENT_LINES_DENSITY - f*SCORE)

10/20 top NPEs

predicted

probability

class name CXTY SCORE

100%class1 528 1.13100%class2 523 1.12

99%class3 464 0.8999%class4 523 1.2599%class5 523 1.4598%class6 369 1.0796%class7 345 1.1896%class8 341 1.3395%class9 399 0.9595%class10 385 0.8393%class11 305 1.0592%class12 372 1.2491%class13 279 1.3290%class14 416 1.1288%class15 263 1.0288%class16 328 1.0287%class17 318 1.0986%class18 428 1.1085%class19 316 0.8984%class20 236 1.22

Does bad code cause defects?

Implement the “Boy Scout rule”

• Encourage Code Quality Improvement On Every Commit

• Score the program’s improvement• Score the committer’s improvement• Recognize the contributors• Ensure the numbers are used properly

Agile Lessons Learned

When and How to Create Debt Backlog Items

• Detailed appropriately– By metric, program file,

developer

• Emergent– At release code branch– Move to later Sprints as

necessary

• Estimated– Use Sonar tech debt

estimate

• Prioritize by– Complexity– Unit tests– Static code

Keep the Pressure Off

• Be positive!• Manage debt informally as sprint

issues• Use TDD and other techniques as

productivity aids• Interpret the contribution metrics

correctly

Fast Feedback

• Reviewers work with submitters to fix same day

• Quick emails with links to more information

• Allow developers to correct immediately after receiving an email

Small Batch Size

• Fix the debt as it’s incurred

• Don’t let it build up

• Minimize tech debt inventory1. Static code violations2. Code and branch

coverage3. Complexity

management4. Circular

dependencies5. Fan-out reduction

Individual vs. Group Accountability

• Idle work vs. idle workers–Attack the most significant debt first

• Fix your own debt quickly–You know the code best

• Fix debt in the code you’re working on–Leave it cleaner than you found it

Minimize Ceremony

• Backlog spreadsheet always available• Morning “warm-up”, fix a little every

day• Only open backlog tickets at release

or sprint boundaries

Code Quality Take-away Messages

Quality Motivates and Enhances Productivity

• Tom DeMarco & Timothy Lister, Peopleware: “Quality, …, is a means to higher productivity”

• Defect removal #1 cause of productivity loss

• Programmers are motivated by emphasis on code quality

Implement CQ Continuous Inspection• Establish CQ point of

accountability• Promote CQ• Track technical debt

– Require remediation• Scales to large projects

– Multi-branch– Multi-language

• Target the reports– End of Sprint or

production release branch

– By lead, committer, reviewer

– Commit quality trend

Embed a Code Quality Expert on the Team

• Unit testing techniques– Mocks– Corner cases

• When to override the warnings• When to clear, not fix, debt• Safe re-factoring

How Much Time for a Code Quality Czar?• 12% of a senior dev’r

cq48%

lead35%

devr9% mgr

5%build

er2%

Roles

Effort breakdownRole task Total

Code Quality Czar   

 

monitor technical debt 13%clear unfixable debt 10%mentor/assist 6%scoring 5%monitor dev branches 4%manual db update/fix/check 3%ad hoc reporting 2%email warning help desk 2%standards setting 2%investigate numbers/database 1%

 Lead developers

report review 16%performance appraisal 13%presentations 4%train 1%trend analysis 1%

Code coding 5%  write SQL 4%Manage awareness 2%  inform/escalate 2%  sell 2%Build sonar 2%

13% more code per yearAll with 100% test

coverage

Keep the code base maintainable so velocity continues

You Can Maintain Velocity and have Zero Debt

What Can You Do Tomorrow?• Implement a code quality dashboard

– Sonar– MS Visual Studio or Eclipse CQ plugin

• Require unit testing• Peer review at check-in• Break the CI build if quality decreases

– Coverage drops below threshold– Critical or major static analysis violations

Thank You!

Brian ChaplinBrian.chaplin@gmail.comTwitter: bchaplin1 NoTechnicalDebt.com