INdT
TestLink
QUALITY
MANTIS
VALIDATION
Verification
Bug
SCRUM
Iterative Software Testing Process for Scrum and
Waterfall Projects with Open Source Testing Tools
Experience
Federal University of Amazonas - UFAM
Nokia Technology Institute - INdT
SELENIUM
Eliane Collins & Vicente Lucena
Natal – RN , Nov 2010
22nd IFIP International Conference on Testing Software and Systems – ICTSS 2010
AGILITY
Company Confidential
2. Environments Context
1. Introduction
Summary
3. Software Testing
Process
5. Conclusion
4. Execution Results
Company Confidential
3
IntroductionIntroduction
Company Confidential
INTRODUCTION
�Different projects with different development process
request Test activities.
� The Software Testing process must be agile for all projects.
� Project 1 used Scrum. It is an agile development process
focusing on teamwork .
�The test activities must follow the scrum and they may be
executed efficiently and fast .
4
Company Confidential
INTRODUCTION
�The Project 2 used Waterfall software development process.
�The system test activities must be done for the whole system
in the test phase.
�Software Testing automation is one of the main approaches
that has been applied to decrease testing costs and time.
5
Test PlanningTest Planning Test Case
Specification
Test Case
SpecificationTest Case
Execution
Test Case
Execution Defect ReportDefect Report
Sequential software testing process
Company Confidential
6
Environments ContextEnvironments Context
Company Confidential
Environments Context
� Nokia Technology Institute – INdT
�It is an independent and nonprofit institute committed to conducting
research and development of technological solutions through
application development, new technologies and concepts.
�The INdT area where the experiment was designed is called Product
Manufacturing Technologies (PMT).
7
Company Confidential
Environments Context
� Project 1
�A system to provide an online customer satisfaction survey result to
the development team by e-mail.
�The system registers the projects, leaders, customers, team members
and the customers’ answers.
�The system sends the reminders to customers online to answer the
questions on the survey and forward the results to the development
team.
�The project was developed under the Scrum process.
�Web Platform: MVC Architecture, Ruby and Java languages, Rails
framework and MySQL database.
8
Company Confidential
Environments Context
9
� Project 1 - Scenario
�In the schedule there wasn’t any time planned for test activities.
�No formal Software Testing process was done.
�Just one test analyst.
�Short time for test execution.
�No financial resources available for this activity.
Company Confidential
Environments Context
10
� Project 2
�A software project to provide a graphic analysis of factory production
tests in real time in order to advise professionals to support
manufacturing on the quality trends for production line failures.
�The project development used the Waterfall process approach.
�Software platform : Client-Server, Java Swing, IDE Eclipse environment
and Oracle database.
�The System has twelve screens, where six of them were used to show
and analyze graphic charts and the other six had the data inputs,
reports and user configuration.
Company Confidential
Environments Context
11
� Project 2 - Scenario
�In the schedule there wasn’t any time planned for test activities.
�No formal Software Testing process was done.
�Outdated documentation of the system requirements.
�Functionalities implemented but not tested
�No financial resources available for this activity.
�4 weeks for a tester and a test analyst to do all test activities for the
whole system
Company Confidential
12
Software Testing ProcessSoftware Testing Process
Company Confidential
Software Testing Process
� Iterative Software Test Process
�Test Inputs: Project Plan, Requirements or Backlog, Development
Environment.
�Test Deliveries: Test Plan, Test Cases Specification, Test Case Execution
Report and Summary Defect Report.
�Functional Test Automation for basic test cases.
�Exploratory Testing.
�Create Test Cases derived from Backlog stories or Requirements.
�The tester was a scrum team member.
�Interaction with the Development Team
13
Company Confidential
Software Testing Process
� Iterative Software Test Process
14
Planning Environment
Config.
Environment
Config.Criation/ Update
Test Cases
Criation/ Update
Test Cases
New System
Version
New System
Version
Report
Generation
Report
Generation
Defect ReportDefect Report
Test
Execution
Test
Execution
Automated
Test Scripts
Automated
Test Scripts
Regression
Tests
End of
Iterations?
End of
Iterations?
NO
Summary
Defect Report
Summary
Defect ReportYES
Start
End
Company Confidential
Software Testing Process
�Open Source Automated Test Tools
�TestLink to automate the managing, planning, test cases specification
and test execution reports.
�Mantis for Defect Tracking tool, friendly user interface to interact with
developers and can be integrated with TestLink.
15
Company Confidential
Software Testing Process
16
TestLink:
Specification:
Reports:
Company Confidential
Software Testing Process
�Web Functional Open Source Automated Test Tools
�Selenium IDE and Selenium Core.
�Aproach rec-and-play for Mozila Firefox and Internet Explorer
browsers.
�Test scripts and Test Suite created in HTML code.
17
Company Confidential
Software Testing Process
18
�Automated Testing Tools:
�For functional test automation a new open source tool was chosen to
test the java swing desktop application: MARATHON.
�Approach rec-and-play recording the user actions creating test scripts
in Jython language and run them later in the tool .
Company Confidential
19
Execution and ResultsExecution and Results
Company Confidential
Execution– Project 1
� Sprint Test Activities in First week:
�Test Environment Configuration
�Automated test scripts creation. The test scripts were implemented in
the browser Mozilla Firefox.
� These scripts were edited and gathered in a test suite, and reused in
other test cases;
�the test cases were done according to the acceptance criteria
emphasizing test cases creation for exceptions.
�The system analyst should review the test cases to ensure coverage of
all functionalities;
20
Company Confidential
Execution– Project 1
� Sprint Test Activities in Second week:
� The test scripts were executed for Firefox and Internet Explorer
browsers. Manual Exploratory tests were also executed by the tester.
�At Mantis and TestLink, the defects were registered, described and
assigned to the developer who was notified by e-mail.
�The Reports were automatically generated by TestLink.
�When the defects had been fixed, the entire test suite was executed by
Selenium doing the regression test.
�It ensures that another part of the system was not affected by code
changes.
�If the acceptance criteria were met, the same activities of the process
will begin again for the next Sprint.
21
Company Confidential
Execution– Project 2
22
�Goal: To ensure in a short time the main features stressed.
�Same test process for the scrum, but with some adaptations:
�A meeting with the development team to understand the project and
verify the requirements documentation updates.
�The testers asked for the prioritized list of the project features where
the most important features had higher priority (As a backlog list).
�The test iterations were planned having the most priority features
being tested first.
�Test Activities executed in 4 iterations (4 weeks to test all
applications).
Company Confidential
Execution– Project 2
�Test Activities for iteration:
�Plan the test cases for the priority features in the TestLink Tool.
�Automated test scripts creation for test cases in Marathon.
�Execute the automated tests and the exploratory manual tests too.
�The defects found are registered in Testlink and Mantis.
� When the developer fixes the defect, the code changes, so all
automated tests are executed to do the regression tests.
�The Automated reports are generated and sent to the Project leader.
�The development team releases another system version and the
same activities of the process will begin again for the next test
iteration.
23
Company Confidential
Results
24
�General Results:
�100% of the functionalities from backlog were covered in Project 1.
�65% of the system defects were from browser incompatibilities in
Project 1.
�77 defects were registered in Mantis, where 30% were interface
defects and 70% were functional defects in Project 2.
�The regression tests showed defects in the functionalities that were
considered done.
�Good evaluation of the projects.
Company Confidential
Results
�Waterfall Results Comparison
25
Test Process CoverageTest
CasesRegression cicles Defects
Manual and
Sequential65% 40 0 27
Iterative and
Automated Process100% 75 3 77
Company Confidential
26
ConclusionConclusion
Company Confidential
Conclusion
� Lessons learned:
�Plan Test Activities and engage the test team from the beginning of
the project.
�Automated test scripts had to be reworked and updated with every
backlog change for each sprint.
�The scrum developer team had to estimate more time to defects
fixing because of the large amount of problems found during the
regression tests.
�Exploratory tests found the usability and performance problems.
�The test script creation has become faster because of the reuse of the
script code.
27
Company Confidential
Conclusion
�The test activities broken into iterations made it possible to
stress the most important features, making the system attend
to what was requested by the client.
�Search for new automation tools for other development
platforms for further projects in the company.
�Search for new techniques to improve agile test process as
Test based on Models to reduce the rework.
28
Company Confidential
THANK YOU!
29