#DevoxxPL

User Acceptance Testing

Bogusław OsuchInTENSO

Looking for the Holly Grail

http://www.yourskypeschool.com/book_yss_eng/winnie-the-pooh.pdf

Here is Edward Bear, coming downstairs now, bump, bump, bump, on the back of his head, behind Christopher Robin. It is,

as far as he knows, the only way of coming downstairs…

… but sometimes he feels that there really is another way, if only he could

stop bumping for a moment and think of it...

… and then he feels that perhaps there isn’t…

Origin

Project Characteristics

10 people Agile 6 years

400k lines of code5.0 developmentto maintenance

ratio

Expectation to deliver more

Customer Feedback

Quality should be better

That’s not what we expected

We didn’t implement any reasonable features latelyPerhaps we should start to think about new system?

The Challenge

Agile is not working ?!?

Searching for issues

?Is this a project where Agile is not working?

Project met most Agile assumptoins

Agile was implemented a few months ago

Organization was not ready?

We did something wrong?

We felt that something else is wrong…… but what?

Searching for issues

After deep analysis including customer

we found out that the biggest issue are

User Acceptance Tests and that’s an area which has to be reorganized.

UAT status

as is to be

customer spend a lot of time for execution of manual tests

time for manual test should be as short as therefore, but not less

customer resources are limitedless involvement is test preparation and

better management of customer resources

complicated UAT planning and delivery (paralel streams and versions)

provide better visibility and planning

toolset is not supporting testing process enough

provide better toolset

delivering test environmentprovide whole test proces organization

as a service for customer

„Decision about doing something is one thing but we would like to provide a time of delivery later. We need a flexibility in deciding about content of versions and we expect a short time of delivery version for testing. Additionally regression tests are very important.”

Business need

Idea

Conception v1.0

Test Case

Test PlanRequiremen

t

Defect

Project /CR

Test Plans has to be done on time

Traceability is a must

Test Plan is a set o Test Cases

Defect is created during Test Case executionProject/CR is grouping Requirements

Use JIRA as a tool

Experience

Is it working?

Creating and preparing Test Plans takes to much time

Using the same Test Cases is difficult and we don’t have agregated information about all executions of Test Case

We are still missing something

Conception v2.0

Introducing Test Case Template

Creating Test Plans should be really fast

Time of executing Test Case should be as small as possible

Building Test Case Repository

Test Case

Test PlanRequiremen

t

DefectProject /

CR

Test CaseTemplate

Retrospective #1

Time to prepare Tests Repository of Test Case Templates was significant

Average time to create and prepare one Test Plan for execution was 15x faster!

Retrospective #2

Average there was 15 Test Plans 80 Test Cases each giving 1200 Test Cases for version

Time spend on preparing Test Repository returned after 3 versions (2 months)

Conception v2.1

Link Requirement with Test Case Template to get better traceability

Test Case

Test PlanRequiremen

t

DefectProject /

CR

Test CaseTemplate

Retrospective #3

Test Cases in Test Plan can be created only from Test Case Templates On Test Case Template track all Test Cases created from this Template

Test Case Templates created in Repository as a part of the analysis

Retrospective #4

First we created and executed Test Plans within the internal QA

Next we provided these Test Plans to Customer for execution

Better control of predictable defects, greater willingness to unforseen defects

Conception v2.2

Link Requirement and Test Plan with Sprint

Test Case

Test PlanRequiremen

t

DefectProject /

CR

Test CaseTemplate

Sprint

We found ourselves to have fun playing and experimenting with UAT process, automating step by step

Retrospective #5

Which Requirements in Sprint are not covered by Test Cases – Missing Test Cases Report

Which Test Cases are connected with Requirements not present in Sprint – Missing Requirements Report

Summary

Having all processes in one tool is not necessary but right now it’s hard to imagine that it could be done a different way

Summary

Don’t stop searching improvements

Holly Grail – searching is more important than finding

Summary

There is no one single solution perfect for everyone but everyone can find his own concept for setting up UAT, it’s very important to have a tool which allows to experiment

As a software provider we found that UAT is totally different from internal Quality Assurance

Thank you for your attention