when agile is a quality game changer webinar - michael mah, philip lew

When Agile Is a Quality Game Changer:

Predictive Forecasting of Defect Profiles

with QSM Associates’ Michael Mah

#AgileGameChanger

XBOSoft, Inc. All Rights Reserved.

About XBOSoft

Dedicated to Software Quality ImprovementFounded in 2006

We speed products to market with our expert:• Software QA consulting• Software testing

Global team with offices in San Francisco & Beijing“Thorough, accurate and fast”


House Rules

Participants other than the speakers are muted Ask questions in the GoToWebinar control on the

right side of your screen or through Twitter @XBOSoft

Questions may be asked throughout the webinar - we’ll try to answer them at the end

You’ll receive info on recording after the webinar

Webinar Hashtag: #AgileGameChanger


Meet the XBOSoft Team

Philip LewCEO and Founder, XBOSoft•Relevant specialties and passions

o Software quality process, evaluation, measurement and improvement

o Software quality in use / UX design

o Mobile User Experience and usability

o Cycling and travel


Rachel GeibigMarketing Manager, XBOSoft•Emails you about our industry hot topic webinars•Invites you to download our latest trends in software testing white papers

With over 25 years of experience, Michael and his partners have derived productivity patterns for thousands of software projects collected worldwide across engineering and business applications.His current work examines time-pressure dynamics of teams, and its role in project success and failure. In addition to his background in physics and electrical engineering, he is a mediator specializingin dispute resolution for technology projects.

• Director of the Benchmarking Practice at the Cutter Consortium, a Boston-based IT think-tank

• Past editor of IT Metrics Strategies publication• Managing partner at QSM Associates Inc. based in Massachusetts• Teaches, writes, and consults to companies on measuring, estimating and

managing software projects, whether in-house, offshore, waterfall, or agile

He has a degree in electrics engineering from Tufts University. His training on dispute resolution, mediation, and participatory processes is from the Program on Negotiation at Harvard Law School and the Radcliffe Institute for Advanced Study. Michael is also a private pilot and lives in the mountains of Western Massachusetts.

Michael Mah

Rayleigh Curve Defect Rate

0

20

40

60

80

100

120

140

160

Defect Type (All)

Count of Severity*

Create Date Status Mode Status Severity* TR-Version

Product+*

The QSM SLIM Database

QSM maintains the world’s largest benchmarking database of 12,000+ completed software projects collected worldwide. We put industry productivity statistics on the desktop.

The QSM SLIM database contains projects in all industries, waterfall, Agile, offshore/outsourced, in-house, new development, and maintenance.

SLIM tools enable managers to measure and estimate and in-flight forecast Agile and waterfall projects.

Partial List of Clients

British TelecomSAP MicrosoftIntel AT&T/BellSouth Nationwide Motorola Verizon WirelessRoche Diagnostics

Fiserv Corp IBM Global Misys Healthcare JPMorganChase Boeing Bank of New York Mellon Lockheed Martin Progressive Insurance DirecTV

Input to SLIM

Size

Time

Defects

Effort

Trendline Assessment – Defects/Quality

Defects During Test

100 1,000

Effective SLOC (thousands)

10

100

1,000

10,000

Errors (S

ysInt-Del)

Rel 5.0

Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0

Rel 8.0

Rel 5.0

Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0

Rel 8.0

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Far Fewer Defects: 50% - 66% Below Industry

Trendline Assessment – Build Phase Schedule

Main Build Phase Duration vs Size

100 1,000


1

10

100

Time (M

onths)

Rel 5.0

Rel 6.0

Rel 6.5

Rel 7.5Rel 7.0

Rel 8.0Rel 5.0

Rel 6.0

Rel 6.5

Rel 7.5Rel 7.0

Rel 8.0

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Schedules are Half Industry

Industry Average

Current Performance

Delta

Project Cost $3.5 Million $2.2 Million -$1.3M

Schedule 12.6 months 7.8 months -4.8 mos

QA Defects 242 121 -50%

Staffing 35 35 n/a

Follett vs. Industry Average

Short Feedback Loops

Paired programmers

Instantaneous code reviews

Accelerated learning and execution

Face to face communication channel

Time Boxing

Short iterations

Clear and discernible progress

Anticipation of the next important feature

Efficiency

Avoiding Burnout

XP = Sustainable pace

40 Hour Work WeeksPrevent productivity collapse for overworked teams

Transparency

“Transparency is a great floodlight. People who thrive in political maneuvering hate SCRUM…”

- Ken Schwaber

High-Bandwidth Communication

The best teams have “wide-open pipes”

Domain knowledge moves among the team

Information flows rapidly and accurately

Avoiding Waste and Costly Rework

Rework has high costRework takes timeRework creates defectsRework is badRefactoring can be a cover up

Norbert Wiener, 1894-1964

“He noticed that the Feedback principle is a key feature of life forms from the simplest plants to the most complex animals, which change their reactions in response to their environment.”

Keywords: Shannon’s Information Theory, Cybernetics

Norbert Wiener, 1894-1964

Example Outsourced Project – Whiteboard Sketch

Input to SLIM

Size

Time Effort

SLIM Replica – Release 9.5Staffing & Probability Analysis

Avg Staff (people)<Single Goal - MBI 4.433>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21Jan'06

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'07

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

0

20

40

60

80

100

Avg Staff (people)876543210

CDR&DC&T

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

SOLUTION PANEL <Single Goal - MBI 4.433>

DurationEffortCost

Peak StaffMTTD

Start Date

C&T14.5716

4153.274.80.1

8/4/2006

Life Cycle19.6919

5328.574.80.1

3/1/2006

MonthsPM$ (K)peopleDays

PI=21.3 MBI=4.4 Eff SLOC=686,000

CONTROL PANEL <Single Goal - MBI 4.433>

PI 17.0 25.5

21.3

Peak Staff 59.8 89.7

74.8

Eff SLOC (K) 549 823

686

Main Build Phase Schedule vs. Size

1 10 100 1,000 10,000


1

10

100

1,000

DC

&T D

uration (Months)

AB Rel 1.0

AB Rel 9.5AB Rel 9.0

AB Rel 1.0


AB Medical All AB Medical Sy stems Special Project Misy s - All Projects Av g. Line Sty le 1 Sigma Line Sty le

Trendline Assessment – Main Build Schedule

Company – All Projects

Deadline target for Rel 9.0 is aggressive but consistentwithin the band of history. Rel 9.5 and Rel 10scope/dates are off the chart.

Main Build Phase Staff vs Size

1 10 100 1,000 10,000


1

10

100

1,000

DC

&T P

eak Staff (P

eople)

AB Rel 9.5

AB Rel 10


AB Rel 10

AB Rel 9.0


Planned staffing profiles (RED projects)are much more aggressive than history. Tight deadlines and time-to-marketare the drivers.

Trendline Assessment – Main Build Staffing


Trendline Assessment – Defects/QualityDefects During Test vs Size

1 10 100 1,000 10,000


0.001

0.01

0.1

1

10

100

Errors (S

ysInt-Del) (thousands)


Ambitious scope, aggressive staffing, and tight deadlinesresult in (higher/lower) defects. (Fill in answer)[WATCH YOUR WARRANTEES]


Defects Found

6 12 18 24 30 36 42 48 54 60 66 72 78 84 9005/17

'0306/28 08/09 09/20 11/01 12/13 01/24

'0403/06 04/17 05/29 07/10 08/21 10/02 11/13 12/25 02/05

'05

0

50

100

150

200

250

300

350

Defects

109876543101098765

Current Plan Actuals Green Control Bound Yellow Control Bound Project: CEP

Defects vs. Industry Average

Defects Found

6 12 18 24 30 36 42 48 54 60 66 72 78 84 9005/17

'0306/28 08/09 09/20 11/01 12/13 01/24

'0403/06 04/17 05/29 07/10 08/21 10/02 11/13 12/25 02/05

'05

0

100

200

300

400

500

Defects

109876543101098765

Current Plan Actuals Green Control Bound Yellow Control Bound Project: CEP

Scaled Up Defect Curve – Outsourced Project

Nov 19

Forecast to Complete (4 Month Delay)

Industry Average

Offshore Average

Delta



QA Defects 242 677 +280%

Staffing 35 50 +15

* Normalized to same project size

Offshore vs. Industry Average

Industry Average

Current Performance

Delta



QA Defects 242 121 -50%

Staffing 35 35 n/a

Follett vs. Industry Average

The force of concentration, or “clustering” of human creativity and talent… the powerful (economic) gains that come when smart and talented people (co-)locate in close proximity to one another…

Robert Lucas Revisited

Some Practical Advice

The Source of Our Troubles

Take a Little More Time

Build a Little Less

Use Small A-Teams

Give Them the Best Tools

Curve Fitting and Creating Agile TrendsAgile Trends - Iterations/Build Phase

Time-to-Market

10 100 1,000

New + Modified Size (thousands)

1

10

100

Months

Effort

10 100 1,000


1

10

100

1,000

10,000

Person-M

onths

Average Staff

10 100 1,000


1

10

100

1,000

Headcount

Bugs During QA

10 100 1,000


1

10

100

1,000

10,000

Defects

All Sy stems Av g. Line Sty le 1 Sigma Line Sty le

New Agile Benchmark TrendsAgile Trends - Iterations/Build Phase

Time-to-Market

10 100 1,000


1

10

100

Months

Effort

10 100 1,000


1

10

100

1,000

10,000

Person-M

onths

Average Staff

10 100 1,000


1

10

100

1,000

Headcount

Bugs During QA

10 100 1,000


1

10

100

1,000

10,000

Defects

The S

LIM-S

uite

SLIM-Estimate:Size, Schedule, Cost & Quality

Estimating

SLIM-DataManagerSoftware Project Metrics Repository

SLIM-Control:Variance Analysis

& Adaptive

Forecasting

SLIM-Metrics: Industry

Benchmarking & Process

Improvement

SLIM-MasterPlan: Incremental

Development & Project

Aggregation

The SLIM-Suite

Software Lifecycle Management

QSM SLIM Agile Benchmark TrendsAgile Trends - Iterations/Build Phase

Time-to-Market

10 100 1,000


1

10

100

Months

Effort

10 100 1,000


1

10

100

1,000

10,000

Person-M

onths

Average Staff

10 100 1,000


1

10

100

1,000

Headcount

Bugs During QA

10 100 1,000


1

10

100

1,000

10,000

Defects

Contact Michael

Michael MahManaging PartnerQSM Associates, Inc.Pittsfield MA USA

email: [email protected]: www.qsma.comtwitter: @michaelcmahtel: 1 413-499-0988

mailto:[email protected]

http://www.qsma.com/

Post your webinar questions on Twitter @XBOSoft

Registrants will receive an email with information on where to view the recording and slides from today’s webinar.

Join us to keep updated on all our webinars, reports and white papers:

facebook.com/xbosoft

+xbosoft

linkedin.com/company/xbosoft

Check out our blog: http://xbosoft.com/software-quality-blog/

Download our free white papers: http://xbosoft.com/knowledge-center/

Email us with ideas for future webinars or questions regarding our services! [email protected]

Thank you!

Q&A

www.xbosoft.comXBOSoft, Inc. All Rights Reserved.

http://xbosoft.com/software-quality-blog/

http://xbosoft.com/knowledge-center/

http://www.xbosoft.com/