©

Feedback-FocussedProcess Improvement

Neil Thompson14th European Conference on Software Testing, Analysis & Review

4-7 December 2006: Manchester, England, UKTrack Session T6

©

2

Can Process Improvement for Information Systems learn from these?

TOYOTA CELICA GT4

TOYOTA PRIUS

©

3

Contents of presentation1. Traditional process improvement in STAR 2. “New” methods in manufacturing: how Toyota has been so

successful, comparison with Goldratt Theory of Constraints3. How that new paradigm translates into IT/IS:

• Agile methods• Lessons for process improvement in general…

4. Example: extending TPI® (using Strengths, Weaknesses, Opportunities & Threats)

5. Thinking tools & feedback loops6. Tipping points7. Other ways to apply the above (eg TMMSM, TOMTM, DIY)8. Integrating Goldratt-Dettmer Thinking Tools into Systems

Thinking9. Implications for Time-Cost-Quality-Scope

©

4

“Traditional” Process Improvement in Information Systems Review & Testing

Sources: TMMSM - http://www.stsc.hill.af.mil/crosstalk/1996/09/developi.aspTPI® – based on http://www.sogeti.nl/images/TPI_Scoring_Tool_v1_98_tcm6-30254.xlsTOMTM – based on http://www.evolutif.co.uk/tom/tom200.pdf, as interpreted by Reid, S. Test Process Improvement – An Empirical Study (EuroSTAR 2003)

PREDEFINED SUBJECT AREAS

MATURITY LEVELS

Medicalanalogies

SomeflexibilityTest

OrganisationMaturityTM

TestMaturityModelSM

TestProcess

Improvement®

©

5

How Toyota progressed through Quality to Global dominance, & now Innovation• Quality (top reputation):

– has dominated JD Power satisfaction survey for decade– Toyota Production System (TPS): 14 principles across

Philosophy, Problem-Solving, Process and People & Partners

• Global dominance:– market value > GM, Chrysler & Ford combined– on track to become (2006) world’s largest-volume car mfr

• Innovation (fast):– Lexus: invaded the “quality” market and won– Prius: not evolutionary but revolutionary – and launched 2

months early and sold above expectations– Toyota Product Development System (TPDS):

13 (!) principles across Process, People and Tools & Technology

©

6

PROCESS

Quality

Toyota’s TPS & TPDS merged with Balanced Score-Card

9. Leaders10. People

& Teams:

PEOPLE & PARTNERS(Respect, Challenge & Grow)

PROCESS(Eliminate

Waste)

PHILOSOPHY(Long-Term)

USERQuality

FINANCIAL Quality

PRODUCT Quality

1. …even at short-term expense

2. Continuous process flow

to surface problems

3. Pull to avoid over-production

4. Level workload

5. Stop-to-fix: “right first time”

6. Standardised tasks forcontinuous improv & empowerment

7. Visual control to see problem

s

8. Reliable tested technologythat serves Process & People

11a. Partners

11b. Suppliers

12. See for yourself to thoroughly understand

13. Decide slowly (all options) by consesnus

14. Learning org via reflection & improvement

• C

ustom

er-de

fined

value

(to se

parat

e valu

e-add

ed fro

m was

te)

• Front-load product dev to explore thoroughly

alternatives while max design space

• func exp& cross-func int

• Build learning culture

• Tools

PROBLEM - SOLVING(Continuous Learning

& Improvement)

©

7

GOLDRATT: Drum-Buffer-RopeMaximise throughputCritical Chain manag’t

Monitoring buffers

Cause-effect treesConflict resol diagramsIdentify constraint,“elevate” & iterate

The “new” paradigm in manufacturing: value flow, pull not push, problem-solving

• And now these principles have been successfully applied beyond actual manufacturing, into product development

3. Pull to avoid over-production

• Customer-defined value (to separate value-added from waste)

2. Continuous process flow to surface problems

7. Visual control to see problems

12. See for yourself to thoroughly understand

13. Decide slowly (all options) by consensus • Front-load product dev to explore thoroughly alternatives while max design space

TOYOTA: Takt (rhythm), Low-Inventory (“lean”),Just-In-TimeMinimise wasteAndon (stop and fix)Kanban cardsTagging slow moversOne-page metricsChain of 5 “why”s

Plan-Do-Check-Act

4. Level workload

14. Learning org via reflection & improvement

• But what about development of software?...

©

8

• Alistair Cockburn:– Increasing feedback & communication reduces need for

intermediate deliverables– Efficiency is expendable in non-bottleneck activities

• Mary & Tom Poppendieck: – Map value stream to eliminate waste– Critical Chain project management– Decide as late as possible

• David J Anderson:– Throughput of value through stages of spec, dev & test– “Stratagrams” (my term)

• But whether or not we are using agile methods, we can use new paradigm principles to improve processes…

The new paradigm in IS development: agile methods

Sources: Cockburn, A. Agile software development (Addison-Wesley Pearson Education 2002) Poppendieck, M&T. Lean software development (Addison-Wesley 2003) Anderson, David J. Agile management for software engineering (Prentice Hall 2004)

©

9

Extending the new paradigm to “STAR”: By rearranging TPI’s key areas…

…we can begin to see cause-effect trees…

1.Test strategy

2.Lifecycle model

3.Mom of involv’t

4.Estim & plan

5.Test spec techn

6.Static techn’s

7.Metrics

8.Test automation

9.Test env’t

10.Office env’t

11.Commit & motiv

12.Test func & train

13.Scope of meth’y

14.Communication

15.Reporting

16.Defect mgmt

17.Testware mgmt

18.Test proc mgmt

19.Evaluation

20.Low-lev testing

©

10

Cause-effect trees: can start with TPI’s inbuilt dependencies

…eg for getting to at least level A throughout

1.Test strategy 2.Lifecycle model3.Mom of involv’t

4.Estim & plan 5.Test spec techn6.Static techn’s 7.Metrics

8.Test automation9.Test env’t10.Office env’t

11.Commit & motiv 12.Test func & train13.Scope of meth’y 14.Communication 15.Reporting16.Defect mgmt

17.Testware mgmt

18.Test proc mgmt19.Evaluation 20.Low-lev testing

A:Informal techniques

A:Single hi-level test

A:Budget & time

A:Plan, spec, exec

A:Compl test basis

A:Substantiated A:Product for project

B:Test int in proj org B:Progress, activities,prioritised defects

A:DefectsA:Internal

A:Planning & exec’n

B:+Monitoring &adjustment

A:Managed-controlled

A:Testers & Test Mgr

A:Project-specific

B:Formal techniques

A:Internal

(slightly simplified)

©

11

Can add extra “key areas”, lifecycle inputs & outputs, general categories

…eg TPI / TMap’s four “cornerstones”

1.Test strategy 2.Lifecycle model3.Mom of involv’t

4.Estim & plan 5.Test spec techn6.Static techn’s 7.Metrics

8.Test automation9.Test env’t10.Office env’t

11.Commit & motiv 12.Test func & train13.Scope of meth’y 14.Communication 15.Reporting16.Defect mgmt

17.Testware mgmt

18.Test proc mgmt19.Evaluation 20.Low-lev testing

TECHNIQUESin general

LIFECYCLEin general

ORGANISATIONin general

INFRA-STRUCTURE

in general

INPUTS &INFLUENCES

on STAR

OUTPUTSfrom STAR

+ Test data

+ Risk-Based STAR

©

12

Can go beyond the fixed questions: SWOT each subject area

INPUTS & INFLUENCES on STAR 4.Estimating & planning

(small Post-it® notes are good for this)

STRENGTHS

WEAKNESSES THREATS

OPPORTUNITIES STRENGTHS

WEAKNESSES THREATS

OPPORTUNITIES

Not substantiated, just “wedid it as in previous project”

Monitored, and adjustmentsmade if needed

Source: solid borders denote as in TPI; dashed borders denote additional

System requirements areagreed too late

System specs & designs aredefective, just timeboxed

The most experiencedbusiness analysts are leaving,more may follow

Release dates are fixed

Can’t recruit more staff

The squeeze on testing islikely to worsen

Some managers are consideringagile methods

System specs are heavy textdocuments

Business analysts may bemotivated by UML training

Too busy for well-consideredestimating & planning

©

13

Some cause-effect trees may “re-root” to form loops

Not substantiated, just “wedid it as in previous project”

System specs & designs aredefective, just timeboxed

System specs are heavy textdocuments

Too busy for well-consideredestimating & planning

4.Estim & planINPUTS & INFLUENCESon STAR

Text-only format makes defectdetection less likely

Note: solid borders denote as in TPI; dashed borders denote additional;items without borders have been added after considering the SWOT

If they cantimebox,so can we

Also too busy to do Risk-BasedSTAR

+ Risk-Based STAR

Live systems are buggy

OUTPUTS from STAR

Testers spend much timehelping diagnose, then retest

Management demand inquests,even more time “wasted” (oropportunity for causal analysis?)

Work takeslonger than“planned”

…(STAR) LIFECYCLE

in general3.Moment of involvement

… …

Testing starts later than whentest basis is completeCulture of our testers is to

prefer large text documentsto diagrams

Key testers are still involved in“firefighting” previous release

1.Test strategy

5.Test spec techn

6.Static techn’s

19.Evaluation

20.Low-lev testing

VARIOUSADVERSEEFFECTS(+other areas, + knock-ons)

©

14

A second example: Test Design

Don’t know actual test coverage

Don’t know how to define & use test coverage

Not trained in formal test techniques

“Informal” techniques not really defined

Too many failures in Live

Too busy “firefighting”

Write numerous tests, “to be safe”

When execution time runs short, unsure which tests safest to omit

Test specifications difficult to review

Increased likelihood of coverage omissions & overlapsescaping rectification Difficult to select regression tests

Test Design does appear in TPI, TMM & TOM, but does this loop indicate it deserves more prominence?

No time to tabulate coverage from existing scriptsUnable to tabulate coverage for new tests

Omissions Overlaps

Add even more testsTest execution takes long time

Test specs are large & “texty”

Formal techniques difficult even after training

Tests probably includelow-value conditions & cases

Not time to attend courses

Increased likelihood of coverage omissions & overlapsescaping detection

©

15

New paradigm problem-solving: the Goldratt-Dettmer* “Thinking Tools”

Core problem+(other) Root causes

Intermediateeffects

Undesirableeffects

Prerequisites+Conflicts

Requirements +INJECTIONS

Objective

CURRENT REALITY+ Injections

Intermediateeffects

Desired effects

Intermediateobjectives

Obstacles

Objective

Needs+Specificactions

Intermediateeffects

Objective

CURRENTREALITY

........... What to change to .......(2) (3)

CONFLICTRESOLUTION

.... How to change ....

PRE-REQUISITES

TRANS-ITION

FUTUREREALITY

What tochange (1)

* very slightly paraphrased here

(4) (5)

Sources: Dettmer, W. Goldratt’s Theory of Constraints (ASQ 1997) Thompson, N. “Best Practices” & Context-Driven – building a bridge (StarEast 2003)

©

16

Applying the Thinking Tools to information from SWOT analysisSTRENGTHS

WEAKNESSES THREATS

OPPORTUNITIES

The SWOT method can be “nested”, eg aggregate upfrom individual subject areas to whole lifeycle

CURRENT REALITYFUTURE REALITY

PRE-REQUISITES

TRANS-ITION

Anticipating &overcomingobstacles

Action planning

CONFLICTRESOLUTION

System specs are heavy textdocuments

Culture of our testers is toprefer large text documentsto diagrams

SDLC method does notencourage diagrams

Test specs are large & “texty”

Test coverage omissions & overlaps

Can still improve coverageat macro level with

informal techniques (80/20)

Using extracts from both 1st & 2nd examples

Too many failures in Live

Some managers are consideringagile methods

Business analysts may bemotivated by UML training

STRATEGIC: Improve SDLC method

TACTICAL: Address culture byworked examples of diagrams

TACTICAL: Include tables &diagrams in test specifications

(Use Threats to helpidentify obstacles)

(Use Strengths to helpamplify opportunities)

ONGOING: Techniquestraining &

coaching

©

17

A third example: Defect fixing & retesting Source: Ennis, M. Managing the end game of a software project (StarWest 2000)

“understand relationship between metrics”

Then the “risk spider” might be renamed a ‘correlation amoeba’

Code turmoil

Test failure rate

Test completionslowness Defect detection rate

Defect backlog

Let us reverse 2 labels, to make all 5 quantities bad things

Failing testsslow completion

Failing testsneed fixes

Too many bad fixes,knock-on faults

More defectsadd to backlog

Tests fail ifthey expose

defects

Its shape, changingover time, could show uswhether we havea feedback loop,either bad… … or good

Code turmoil

Test failure rate

Test completionslowness Defect detection rate

Defect backlog

Less slowness Fewer fixesneeded

Code fixes are good,fixing faults

Fewer defects,backlog can reduce

Fewer defectsmeans fewerfailing tests

+

-

©

18

Tipping Points• Reversing the vicious loop into a virtuous loop:

– example on previous slide required only one relationship change…

– ie effect of code turmoil on defect detection rate to tip from positive to negative

• Examples of Tipping Points:– removing New York graffiti reduced serious crime rate– Stanford “Prison” Experiment: spiral of vindictiveness

• Achieving Tipping Point involves:– concentrating resources on a few key areas– a way to make a lot out of a little– fresh thinking, sometimes counterintuitive

Note the similarities with Goldratt’s Theory of Constraints, and more generally the Pareto principle (80-20)

Source: Gladwell, M. The tipping point (Abacus 2000)

©

19

Fourth example: Documentation

• Documentation can also become a vicious loop:– documentation tends to get out of date– “do you want us to fix the documentation or

deliver software?”– can then become unclear whether tests

passed / failed: what is it meant to do?– no-one reads the documentation any more, so

defects in it are no longer detected– so it gets even further from reality…

©

20

Documentation and flow of valueLEVELS OF DOCUMENTATION,pushed by specifiers

WORKINGSOFTWARE

Accepted

System-tested

Integrated

Unit / Component-tested

FLOW OF FULLY-WORKING SOFTWARE,

pulled bycustomer demand

Requirements

+ FuncSpec

+ TechnicalDesign

+ Unit / Componentspecifications

+ Test Specifications

©

21

Difficulties in problem-solving: conflict resolution (eg for documentation)

Objectives of documentation

are to help build and maintain a fit-for-purpose

system by knowingand agreeingwhat built andwhat tested

“We needmore

documentation”

“We needless

documentation”

Developed further to Daich, GT. Software documentation superstitions (STAREast 2002)See also Rüping, A. Agile documentation (Wiley 2003)

CO

NFLIC

T

“Signed-off requirementsare counterproductive tosystems meeting realuser needs now”

“Documented test plansare counterproductive tothe best testing”

“People never readany documentation”

Specifications are likeinventory, no end value

“Reviews are powerful atfinding defects early, but it’sdifficult to review just speech”

“If it’s not written,it can’t be signed off”

“Test reports need tobe formal documents”

“They will when theirmemories have fadedor when there’s acontract dispute”

Documentation varies:need to distinguish necessaryfrom unnecessary

Need to distinguish qualityof documentation, notjust quantity

“Will the live system bemaintained by itsits developers?” No!

Our users cannot be on-sitewith the project throughout

“Are test analysts writingtests for others to run?” No!

Can mix exploratory& scriptedtesting

“Sign-off can be byagreed meeting outcomes” “Are there few enough people

to make frequent widespreadmeetings practical?” No!

“What documentation is neededfor contractual reasons? Stilltime to negotiate?” Yes!

Documentation is still needed for maintenanceafter go-live

Agree in a workshop whatdocumentationis needed

Documentationdoesn’t have tobe paper: usewikis etc

Make maximumuse of tables& diagrams

©

22

Resolving the “conflict” between agile and outsourcing

“Outsourcing /offshoring is

the way to go”

“Agile isthe way to go”

CO

NFLIC

T

Objectives of methodology

are to help build and maintain a

system toappropriate

time-cost-scope-quality/risk

balance

“Does the time-cost-scope-quality/riskpyramid always apply?” No!

Inspired by Guckenheimer, S. with Perez, JJ. Software engineering with Microsoft Visual Studio Team System (Addison-Wesley Pearson Education 2006)

“Low cost is the mostimportant factor for us”

“Fast delivery is the mostimportant factor for us”

“How much documentationwill we need to controlproject & products?

“How confident are we inknowing requirements?

“What are theregulatoryrequirements?

“How will we communicateacross inter-company /international boundaries?

etc…

©

23

Can use these principles to focus TPI, TMM, TOM etc…

• TPI: (as in earlier slides)– use “off-question” information directly via SWOT– use relationships between key areas to identify causes-effects,

loops, and hence biggest-payback improvements• TMM:

– choose between staged & continuous– consider moving selected items between levels, up or down

• TOM:– the “low=1” scores are weaknesses (but you may be able to

think of others); similarly “high=5” scores strengths– look for additional symptoms (via SWOT)

• Generally:– look for Tipping Point improvements from among the many

candidates– seek Tipping Point insights into the change management

challenges (Connectors, Mavens & Sellers)

©

24

…or invent your own methodology…

Risk management Quality management

Insurance Assurance

V-model: what testing against W-model: quality management

Risks: list & evaluate

Define & detect errors (UT,IT,ST)Give confidence (AT)

Use Risk-BasedSTAR

Tailor risks & priorities etc to factors

Refine test specifications progressively: Plan based on priorities & constraints Design flexible tests to fit Allow appropriate script format(s) Use synthetic + lifelike data

Allow & assess for coverage changes Document execution & management procedures

Distinguish problems from change requests Prioritise urgency & importance

Distinguish retesting from regression testing

Use handover & acceptance criteria

Define & measure test coverage

Measure progress & problem significance

Be pragmatic over quality targets

Quantify residual risks & confidence

Decide process targets & improve over time

Define & use metrics

Assess where errors originally made

Define & agree roles & responsibilities

Use appropriate skills mix

Use independent system & acceptance testers

Use appropriate techniques & patterns

Plan early, thenrehearse-run,acceptance tests

Use appropriate tools

Improve efficiency of STAR

Effective STAR

Build lessons learned into checklists

Do Reviews& Analysis

Modified after Thompson, N. “Best Practices” & Context-Driven – building a bridge (StarEast 2003)

©

25

… or just do simple lifecycle-focussed process improvement

1.Test strategy 2.Lifecycle model

3.Mom of involv’t4.Estim & plan

5.Test spec techn6.Static techn’s

7.Metrics

9.Test env’t10.Office env’t

11.Commit & motiv12.Test func & train 13.Scope of meth’y

14.Communication15.Reporting

16.Defect mgmt

17.Testware mgmt

18.Test proc mgmt

(a) MANAGE STAR END-TO-END(whole systems…………...………….whole lifecycle architecture as part of QM/QA)

(b) USE RISK-BASED STAR

…eg “7 Habits of Adequately Effective STAR-ers”

(e) USE FORMAL TECHNIQUES

(f) USE METRICSAND ACT ACCORDINGLY

(g) USE TOOLS APPROPRIATELY

(c) USE STRUCTURED REVIEWS

(d) MANAGE COVERAGE OF TESTS

20.Low-lev testing

8.Test automation

(involving roles &checklists)

19.Evaluation

(including…)

POSITIVEFEEDBACK

©

26

Process Improvement is itself a reinforcing feedback loop…

…but first we may need to reverse the negative loop of process stagnation (through its Tipping Point)

“Can’t do structured reviewsbecause not trained inrisk analysis”

“Now we’re trained inrisk analysis, can dostructured reviewseven better”

“Now we’re trained informal techniques, can dotest coverage even better”

“Can’t do test coveragebecause not trained informal techniques”

“No point in theseimprovements becausenot responsibleend-to-end”

“Now we’ve gotend-to-end responsibility,these improvements willbe even more effective”

“No point in writingchecklists becausepeople know they’rejust wishful thinking”

“Now we’ve gotroot cause analysis,we can make ourchecklists even better”

©

27

Systems thinking: more holistic view, eg people issues

Long working hours

Money now:overtime

Money later:promotionprospects

“PizzaHero”Culture

Fatigue

Healthdamage

Abandon healthy shopping,regular exercise &social activities

Higher “efficiency”(per week)

So what? There are more hours

More mistakes,defects, faults,failures in dev & test…

More remedial work needed

Even more work wantedby management

“Zombie”working

“Must need aTime Management Course” Culture

“Mad butExploitable”Culture

Project “delivered”on time

Tolerable degree oflive failures?

“Stretch goals”

“PizzaParasite”Culture

Psychologicaldamage

Overall “adequate”work

Lower effectiveness (per hour)

Both? Give up!

More work absorbed

Guilt,feelings ofinadequacy

Is this a vicious or virtuous feedback loop?

Alternatingillness &hard workLower output (per year)

Too manylive failures?

Expert orreplaceable?

©

28

Systems Thinking notationDuration ofworking hours

Short-termfinancial reward

Long-termfinancial reward

Peer-approvalreward

Management-approvalreward

Efficiency(per hour)

Effectiveness(per week)

REINFORCINGLOOP

BALANCINGLOOP 1:

Quality targets

BALANCINGLOOP 2:

Capacity of staff

A loop is balancing if it contains an odd number of opposing links;else it is reinforcing

Each author seems to vary; this is Neil Thompson’s,

incorporating elements ofGerald Weinberg’s &

Dennis Sherwood’s

Overallacceptabilityof projects

“Coping”mechanisms

for fatigueetc

HealthDegreeof live

failures

©

29

Connections between & beyond feedback loops

• Goldratt’s Theory of Constraints:– sometimes cause-effect trees can form loops

• Systems Thinking:– loops can have “dangles”

• So let’s fit the two together…!

Code turmoil

Test failure rate

Test completionslowness Defect detection rate

Defect backlog

Failing testsslow completion

Failing testsneed fixes

Too many bad fixes,knock-on faults

More defectsadd to backlog

Tests fail ifthey expose

defects

+

System documentationis too “texty”

No dedicatedconfigurationmanagement team

Difficult to select regression tests

CM not properlycoordinated

Mistakes madein CM Don’t know whether

problems are bad fixesor CM mistakes

Everyone toobusy to arrangea CM team

Difficult to diagnose faults

Everyone toobusy to produceconciseoverviews &distil expertise

©

30

What the new paradigm means for the trad. Time-Cost-Quality-Scope pyramid

Inspired by Guckenheimer, S. with Perez, JJ. Software engineering with Microsoft Visual Studio Team System (Addison-Wesley Pearson Education 2006)Old pyramid reprised from Gerrard, P. & Thompson, N. Risk-Based Testing (EuroSTAR 2002)

Time Cost

Quality/Risk

Scope

OLDPARADIGM Quality

Low-cost

Speed Scope

CurrentRealityTrees

FutureRealityTrees

Note: this loop hasan even number ofopposing links, soshould be reinforcing:but could be viciousor virtuous?

Quality

Low-cost

Speed Scope

STARTHERE

STARTHERE

ConflictResolutionDiagram

Quality inbuilt not imposed,less waste (eg rework),

working software is iterated…

Mythical Man Month:small teams can be

surprisingly productive

?

?

NEWPARADIGM

©

31

Summary

• Traditional process improvement in testing may be improved by building in principles which made Toyota (and others) global successes

• This involves thinking beyond just STAR: need to consider whole lifecycle and quality regime

• You can either fine-tune an existing method (eg TPI, TMM, TOM) or build your own method

• The principles are not proprietary and require no special training

• Focussing on feedback loops is an example of the Pareto principle (80-20)

©

32

• Try it for yourself: – Strengths, Weaknesses, Opportunities & Threats on Post-itTM notes– Draw pencil connectors for current & future causes & effects– Move items around, look for loops– What would it take to balance / reverse vicious loops?

• Toyota: (see slide 5)• Agile methods: (see slide 8)• Selected reading on Goldratt:

– Goldratt: mostly “novels” (everyone quotes The Goal) – so try this…– William Dettmer: 1st of 2 books on the Thinking Tools…

• For Systems Thinking:– Gerald Weinberg: IT/IS context… – Peter Senge: business mgmt context… – Dennis Sherwood: many examples of loops…

• Practical application of some new paradigm principles:– Sam Guckenheimer

Way forward & Where to find out more

©

33

Thanks for listening!

• Contact details:

Neil ThompsonThompson information Systems Consulting Ltdwww.TiSCL.com NeilT@TiSCL.com 23 Oast House CrescentFarnhamSurreyGU9 0NPEngland, UK+44 (0)7000 NeilTh (634584)

• Questions?