Lean Kaizen Applied To Software Testing

Thomas Axen

Agenda

Short about ATP - Test Organization

Why Test Kaizen - Blitz and Relationship to our test improvement

areas

Test Kaizen - Blitz seen from the theory to practices

The Test Kaizen - Blitz process in ATP

Test improvement overall view

Summary

Questions

4

5

The purpose of this presentation

To share my experiences with others

We hope that you can use some of this information in your

daily work

We hope that you come up with some good questions for me,

that I can use in my daily work

6

Short about ATP - General

Employees in ATP

- 800 FTE at Hillerød Campus

- 1000 FTE in 5 call centers

Employees in IT

- 210 FTE

- 80 Consultants

Employees in Test

- 27 FTE

- 23 Consultants

Customers– ATP livslang Pension (ATP)

– Arbejdsgivernes Elevrefusion (AER)

– Arbejdsmarkedets Erhvervssygdomssikring (AES)

– Barsel.dk

– DA-Barsel

– FerieKonto (FK)

– Finansieringsbidrag (FIB)

– Lønmodtagernes Dyrtidsfond (LD)

– Lønmodtagernes Garantifond (LG)

– Supplerende arbejdsmarkedspension for førtidspensionister

(SUPP)

– Den Særlige Pensionsopsparing (SP)

– Finansieringsbidrag (FIB)

– Skattenedslag for seniorer (SFS)

– Pension Danmark (PD)

– Objektiv sagsbehandling – New customers …

- Folkepensionen

- Udbetaling og indtægtsregulering af fortidspension

- Boligstøtte

- Barselsdagspenge

- Forskudsvis udbetaling af børnebidrag

7

Short about ATP - Test Organization

The test resources (50 FTE)

- 3 Test chiefs

- 6 Test Managers (RTM, TI and TM)

- 9 Test Coordinators

- 3 Test Specialists (automated test and test data)

- 6 IT-Testers and students workers (ITU and DTU)

- 23 Test consultants

Responsibility areas

- Test in projects and Sustained Engineering

- Test methods and tools (QC, QTP, ATSA, TDM)

Why Test Kaizen - Blitz

We have used Kaizen on other areas with great success

Top management – see it as a way of ”tuning” the organization

A god way of interacting with the stakeholders, to understand their

needs and attitude towards test

An new positive way of identifying improvements areas on test

All the testers can impact/influence on what the improvement

areas are and what we shall prioritize to work on

It verifies if we are working on the right improvement areas

8

Relationship to our test improvements areas

9

InspirationProcess

improvementImplementation

Test organisation

Improvements

Governance

Prioritizing

Execution

Improvement inspiration

Feedback

TMM assessment

Kaizen - Blitz

Guru

Network

Best Practice

Management decision

Strategy

Project

Sustained Engineering

Test Kaizen - Blitz seen from the theory to practices

Lean thinking:

Elimination of waste

Flow (production processes are made continuous)

Three central concepts:

Waste is ’everything the customer won’t pay for’

Flow implies elimination of batches and queues

Production is ’mass production’

10

…but Isn’t All Testing Waste?

The answer is yes in the context of software production:

11

Customers will not pay you for making errors.

Consequently customer will not pay you for finding and

correcting the errors you make.

But we are not looking at test in the context of software

production. We are looking at test in the context of a customer

checking the quality of software received from a vendor.

In this context test is not waste.

How Do We Identify Waste in Testing?

12

ATP chose Kaizen – Blitz from the Lean Toolbox.

Kaizen means ’continuous improvements’ and is “by the book”:

Dealing with a limited and thus manageable part of a production process (comparable to the work of 5 to 6 persons).

Being performed quickly but having ambitious goals for removing waste from the area under consideration.

Involving the 5 to 6 persons, who operate the process and are familiar with it.

Looking for improvements, which the persons involved are themselves in command of.

Being based on facts and measurements as far as possible.

Being repeated at regular intervals.

ATP chose a pragmatic approach to Kaizen

13

Dealing with a limited and thus manageable part of a production process. The

areas chosen were somewhat big, since it included both Domain, Integration,

and User Test.

Being performed quickly but having ambitious goals for removing waste from

the area under consideration. 3 short workshops over 6 weeks instead of 3

intensive days.

Involving exactly the 5 to 6 persons, who operate the process and are familiar

with it.

Looking for improvements, which the persons involved are themselves in

command of. Several useful improvements were “owned by other parties”.

Being based on facts and measurements as far as possible. Neither time nor

money for measurement on a large scale. (Common sense)

Being repeated at regular intervals. One time must be the first time.

Next time

Next time

The Test Kaizen - Blitz process in ATP

1 Step - Define the reason for the investment in kaizen

- Business case with focus on cost/benefit and scope.

- Communication to sponsor ongoing on status and progress

2 Step - ATP Test Kaizen kick-off

- Selecting the tester group that shall work with the kaizen process

- Short introduction to kaizen, and the plan (30 min.)

3 Step - Interviewing stakeholder and collecting facts

- The VoxPop (Voice of the Customer) input.

4 Step - Workshop, Testers as active players

- Present Voxpop and collected facts

- Kaizen scope and definition of waste in testing

- Value Stream (mapping all test activities and waste pr. Test phase)

5 Step - Workshop, RCA and classification

- Root Cause Analysis (Fishbone and selected defects analysis)

6 Step - Workshop, Consultation

- Classification and prioritization the improvement areas

- Finalize the proposal to management (the presentation)

7 Step - Summary (Presentation for managers and a execution plan)

14

3 Step - Interviewing stakeholder and collecting facts

Objective: To challenge the Kaizen team members attitude

towards their work.

15

Examples:

Far to many errors do not

surface before full production.

The Voice of the Customer:

- ”To little testing. To much test

administration. Spend more time testing!”

- ”I have never been on a project with so many test phases...”

- ...

4 Step - Workshop, Kaizen Scope and Definition of Waste

SIPOC - Scope of ATP Test Kaizen (Black outline in Kaizen scope)

Definition of waste in testing:

All activities not directly involved in finding errors or in

proving the absence of errors are considered waste.

All repetition of test etc. are considered waste.

16

ProgrammingComponent

test

Domain

test

Integrations

test

User

testOperation

4 Step - Workshop, Value Stream (mapping all test activities and waste pr. Test phase)

Objective: To get a clear picture of the time and resources spend on

testing and to identify activities where time and resources are

wasted.

What was done?

Detailed mapping of 3 subprocesses: Domain test,

Integration test and User test (3 value stream maps).

Identification of waste pr. subprocess.

Fordeling af kundeværdi og spil - Best case

0

50

100

150

200

250

300

350

400

1 2 3 4 5 6a 6b 6c 7 8a 8b 9b 9c 10 11

Aktivitet

Re

ss

ou

rce

forb

rug

/tim

er

NVA

BNVA

CVA

Example: Integration test

5 Step - Workshop, Root Cause Analysis

Objective: To prepare for elimination of waste by identifying the

underlying reasons.

What was done?

Systematic root-cause-analysis of all activities involving waste.

A. Testing is not finished within time and budget

I. To few people with proper business and domain knowledge in the individual test phases

a. Different (new) people are testing a given piece of software in each of the test phases.

1. To many handovers between phases.

6 Step - Workshop, classification and prioritization the improvement areas

What was done:

Brainstorm among participants on the basis of the Root

Cause Analysis.

Identification of 38 proposals

for elimination of waste.

Prioritisation of proposals:

’Resources vs. Effect’.

Anchoring:

Who is doing what?

Objective: To formulate operational proposals, leading to elimination

of waste in testing.

Low

Consider Don't

Hig

h

Do Consider

Low High

Resources needed for implementation

Eff

ect

on

waste

elim

inatio

n

6 Step - Workshop, Finalize the proposal to management

We identified 38 improvement areas

We grouped the improvement areas into 4 categories

- Go Fix – without any additional investments (9 areas)

- Investment – We need an investment based on a Business case and return of

investment analysis (3 areas)

- In progress – investment already in place project up an running (3 areas)

- Relationship to other areas (4 areas)

Rewriting of all the improvement areas so management can

understand the value of the investments

20

21

7 Step - Summary, “Go Fix” without any additional investments

Priority 1

- Tester should have more focus on progress, instead of roles

- Release management plan, must reflect dependencies so it is easy for the tester to plan

their test

- With critical issues, it should be possible to gather a team across the normal teams

Priority 2

- Post-mortem on found defect in production and projects (Root course analysis)

- Optimize the defect process

- General education plan (Consultants, PM, Business, Testers)

Priority 3

- Test cases must follow the BPT standard

- Improvement of the code documentation on defect and design changes, so test can

focus and minimizing the cost

- Quality assurance on deliverables in a structured way (Focus on static testing)

7 Step - Summary, “Investment” based on BC and ROI

Organization of the test pr. release

- All hours on test are used optimal – focus on planning, competences and the

team work together

Quality assurance of all software deliverables collected systematically

for reporting to management

- Systematic collection of data, for reporting to management so they can use it

for negotiations with vendors/teams on what the given quality should be for

each of them (Project and SE)

Release regressions test coverage

- Well documented release regressions test coverage pr. customer/solution,

so management can decide if we should “go live”

22

7 Step - Summary, “In progress”

Strategy for how to test letters

Test Data Management (Tools and reuse data)

More automated regression test (BPT, QTP)

23

7 Step - Summary, Relationship to other areas

Release management

- Overall plan, deliverables, dependencies, risk and issues.

- Requirements and change management

Deployment to test environments (36 applications using SOA)

- System test environments

- Integration test environments

Tester education with different focus

- Consultants, Business unit area, End users, Project management

Post-mortem on found defects In production and projects/Root cause

analysis

24

25

Test improvement overall view

Department vision/mission & overall goals- Be proactive (defects prevention, Innovation, think ahead)

- Agreements attitude (Clear, on time, expectation, follow-up)

- Well-known ABP (Stable, optimized, functional)

- Ready for changes (New things, flexible, developing and managed)

Test vision & overall goals- From known to respected in ATP

- Test in ATP should increase one level on TMM(i)

- Effective test with use of Best Practices and front-runners on selected areas

- Test process exists; it is always used and improved continually

- Competent/ professional test resources in the IT-organization

- Competent tests in the business-organization

Test leadership- Updating the A4 plans for the whole test area

- Updating detailed plans for each sub test areas

- Ongoing prioritization and follow-up on tests areas across applications

- Focus on improvements to minimize the effort

TMM – improvements (Level 3)- Continually optimizing/updating/support on test artifacts

- Continually optimizing all test processes

- Overview of all test tools and usages

- Extend the defect – root cause analysis

- More focus on non-functional requirement and test techniques

- More tester education to business, end users and consultants

- Focus on test data availability

- Use review for testing

- Metrics on test

Tester feedback loops- The best test case practice in ATP

- Optimize the usages of our test environments

- Optimizing of the review process on important artifacts

Test Kaizen – Blitz Improvements area- Quality assurance on deliverables in a structured way

- New test organization to handle releases

- Release regressions test coverage

- Tester should have more focus on progress, instead of roles

- Release management plan, must reflect dependencies so it is easy for

the tester to plan their test

- With critical issues, it should be possible to gather a team across the

normal teams, to analyse the issue

- Post-mortem on found defects in production and projects

- Optimize the defect process

- General education plan (Consultants, Business, End users, PM)

- Test cases must follow the BPT standard

- Improvement of the code documentation on defect and design

changes, so test can focus and minimizing the cost

- Strategy for how to test letters

- Test Data Management (Tools and reuse data)

- More automated regression test (BPT, QTP)

”Guru sanity check” Recommendations- All tester roles must be defined and communicated in ATP

- Test management meeting each 14 days

- PM gets a Tester education day, and understand each others roles

Governance projects- Extend the amount of automated regression test

- Optimizing of our Test Data Management area

- Stabilising of all reg. test cases for automation

Project managers improvements areas- Improve the test environment setup

- Test statistic (ex. defects in projects / production)

- Target groups on test artefacts (Who should read what)

- Move the test activities to the start of the project in the risk and QA

- Root cause analysis on project defects

Summary

We will definitely have a test Kaizen again, when we are ready

It is a good way of making sure that all stakeholders are being heard

Great way of getting a ”buy in” from all the testers, where the testers have an

influence on the improvements areas to execute on

It improves the image and awareness about Test in ATP

Remove activities that don’t add value

All relevant improvements areas are identified and prioritized

This is a “new angle” on ways to identify improvement areas

It also spins of improvement areas to other parts of the IT organization

Adjust the communication with respect, so the stakeholders can see the value of

each improvements area from their point of view.

26

27

Questions