ictss 2010 - iterative software testing process for scrum and waterfall projects

INdT

TestLink

QUALITY

MANTIS

VALIDATION

Verification

Bug

SCRUM

Iterative Software Testing Process for Scrum and

Waterfall Projects with Open Source Testing Tools

Experience

Federal University of Amazonas - UFAM

Nokia Technology Institute - INdT

SELENIUM

Eliane Collins & Vicente Lucena

Natal – RN , Nov 2010

22nd IFIP International Conference on Testing Software and Systems – ICTSS 2010

AGILITY

Company Confidential

2. Environments Context

1. Introduction

Summary

3. Software Testing

Process

5. Conclusion

4. Execution Results


3

IntroductionIntroduction


INTRODUCTION

�Different projects with different development process

request Test activities.

� The Software Testing process must be agile for all projects.

� Project 1 used Scrum. It is an agile development process

focusing on teamwork .

�The test activities must follow the scrum and they may be

executed efficiently and fast .

4


INTRODUCTION

�The Project 2 used Waterfall software development process.

�The system test activities must be done for the whole system

in the test phase.

�Software Testing automation is one of the main approaches

that has been applied to decrease testing costs and time.

5

Test PlanningTest Planning Test Case

Specification

Test Case

SpecificationTest Case

Execution

Test Case

Execution Defect ReportDefect Report

Sequential software testing process


6

Environments ContextEnvironments Context


Environments Context

� Nokia Technology Institute – INdT

�It is an independent and nonprofit institute committed to conducting

research and development of technological solutions through

application development, new technologies and concepts.

�The INdT area where the experiment was designed is called Product

Manufacturing Technologies (PMT).

7



� Project 1

�A system to provide an online customer satisfaction survey result to

the development team by e-mail.

�The system registers the projects, leaders, customers, team members

and the customers’ answers.

�The system sends the reminders to customers online to answer the

questions on the survey and forward the results to the development

team.

�The project was developed under the Scrum process.

�Web Platform: MVC Architecture, Ruby and Java languages, Rails

framework and MySQL database.

8



9

� Project 1 - Scenario

�In the schedule there wasn’t any time planned for test activities.

�No formal Software Testing process was done.

�Just one test analyst.

�Short time for test execution.

�No financial resources available for this activity.



10

� Project 2

�A software project to provide a graphic analysis of factory production

tests in real time in order to advise professionals to support

manufacturing on the quality trends for production line failures.

�The project development used the Waterfall process approach.

�Software platform : Client-Server, Java Swing, IDE Eclipse environment

and Oracle database.

�The System has twelve screens, where six of them were used to show

and analyze graphic charts and the other six had the data inputs,

reports and user configuration.



11

� Project 2 - Scenario

�In the schedule there wasn’t any time planned for test activities.

�No formal Software Testing process was done.

�Outdated documentation of the system requirements.

�Functionalities implemented but not tested

�No financial resources available for this activity.

�4 weeks for a tester and a test analyst to do all test activities for the

whole system


12

Software Testing ProcessSoftware Testing Process


Software Testing Process

� Iterative Software Test Process

�Test Inputs: Project Plan, Requirements or Backlog, Development

Environment.

�Test Deliveries: Test Plan, Test Cases Specification, Test Case Execution

Report and Summary Defect Report.

�Functional Test Automation for basic test cases.

�Exploratory Testing.

�Create Test Cases derived from Backlog stories or Requirements.

�The tester was a scrum team member.

�Interaction with the Development Team

13



� Iterative Software Test Process

14

Planning Environment

Config.

Environment

Config.Criation/ Update

Test Cases

Criation/ Update

Test Cases

New System

Version

New System

Version

Report

Generation

Report

Generation

Defect ReportDefect Report

Test

Execution

Test

Execution

Automated

Test Scripts

Automated

Test Scripts

Regression

Tests

End of

Iterations?

End of

Iterations?

NO

Summary

Defect Report

Summary

Defect ReportYES

Start

End



�Open Source Automated Test Tools

�TestLink to automate the managing, planning, test cases specification

and test execution reports.

�Mantis for Defect Tracking tool, friendly user interface to interact with

developers and can be integrated with TestLink.

15



16

TestLink:

Specification:

Reports:



�Web Functional Open Source Automated Test Tools

�Selenium IDE and Selenium Core.

�Aproach rec-and-play for Mozila Firefox and Internet Explorer

browsers.

�Test scripts and Test Suite created in HTML code.

17



18

�Automated Testing Tools:

�For functional test automation a new open source tool was chosen to

test the java swing desktop application: MARATHON.

�Approach rec-and-play recording the user actions creating test scripts

in Jython language and run them later in the tool .


19

Execution and ResultsExecution and Results


Execution– Project 1

� Sprint Test Activities in First week:

�Test Environment Configuration

�Automated test scripts creation. The test scripts were implemented in

the browser Mozilla Firefox.

� These scripts were edited and gathered in a test suite, and reused in

other test cases;

�the test cases were done according to the acceptance criteria

emphasizing test cases creation for exceptions.

�The system analyst should review the test cases to ensure coverage of

all functionalities;

20



� Sprint Test Activities in Second week:

� The test scripts were executed for Firefox and Internet Explorer

browsers. Manual Exploratory tests were also executed by the tester.

�At Mantis and TestLink, the defects were registered, described and

assigned to the developer who was notified by e-mail.

�The Reports were automatically generated by TestLink.

�When the defects had been fixed, the entire test suite was executed by

Selenium doing the regression test.

�It ensures that another part of the system was not affected by code

changes.

�If the acceptance criteria were met, the same activities of the process

will begin again for the next Sprint.

21



22

�Goal: To ensure in a short time the main features stressed.

�Same test process for the scrum, but with some adaptations:

�A meeting with the development team to understand the project and

verify the requirements documentation updates.

�The testers asked for the prioritized list of the project features where

the most important features had higher priority (As a backlog list).

�The test iterations were planned having the most priority features

being tested first.

�Test Activities executed in 4 iterations (4 weeks to test all

applications).



�Test Activities for iteration:

�Plan the test cases for the priority features in the TestLink Tool.

�Automated test scripts creation for test cases in Marathon.

�Execute the automated tests and the exploratory manual tests too.

�The defects found are registered in Testlink and Mantis.

� When the developer fixes the defect, the code changes, so all

automated tests are executed to do the regression tests.

�The Automated reports are generated and sent to the Project leader.

�The development team releases another system version and the

same activities of the process will begin again for the next test

iteration.

23


Results

24

�General Results:

�100% of the functionalities from backlog were covered in Project 1.

�65% of the system defects were from browser incompatibilities in

Project 1.

�77 defects were registered in Mantis, where 30% were interface

defects and 70% were functional defects in Project 2.

�The regression tests showed defects in the functionalities that were

considered done.

�Good evaluation of the projects.


Results

�Waterfall Results Comparison

25

Test Process CoverageTest

CasesRegression cicles Defects

Manual and

Sequential65% 40 0 27

Iterative and

Automated Process100% 75 3 77


26

ConclusionConclusion


Conclusion

� Lessons learned:

�Plan Test Activities and engage the test team from the beginning of

the project.

�Automated test scripts had to be reworked and updated with every

backlog change for each sprint.

�The scrum developer team had to estimate more time to defects

fixing because of the large amount of problems found during the

regression tests.

�Exploratory tests found the usability and performance problems.

�The test script creation has become faster because of the reuse of the

script code.

27


Conclusion

�The test activities broken into iterations made it possible to

stress the most important features, making the system attend

to what was requested by the client.

�Search for new automation tools for other development

platforms for further projects in the company.

�Search for new techniques to improve agile test process as

Test based on Models to reduce the rework.

28


THANK YOU!

29

ictss 2010 - iterative software testing process for scrum and waterfall projects

Documents