1

Agile Practices (Testing) Benchmarking Case Study

byOpCord

2

About Organization - OpCord

General Organizational Information

Captive / Non Captive Non Captive

About Business Products / Services

Major Verticals Testing (Automation), Mobile Apps, Trainings

Number of Employees 30 - 35

Quality/Process Models embraced

Agile Methodologies

Others 1

Others 2

Contact for this Presentation

hariprakash@opcord.com

3

Agile Penetration @ Organization

Agile Related Information

Type of Project(please elaborate)

Manual Testing , Test Automation, Mobile Apps Development

Domains Supply Chain Mgmt, Logistics, Web Applications, Portals

Testing Technologies

Open Source tools, like, AutoIT, Selenium, Sahi, Jmeter, Jbehave, Sikuli, Junit etc and commercial tools, like, QTP, HPQC etc

Number of Projects & % of Projects

3 - 4

Agile Institutionalized since

Inception, last 2.5 yrs

4

Agile Practices in Project

Project Name Desktop Application Test Automation (DATA)

Project Size / Structure

4 test programmers at OpCord and 6 Developers at Client location (India & US)

Project Type Migration to newer technologies

Collaboration Skype was used for voice calls / screen sharing with US/India folks;TeamViewer was also used for screen sharing;Client visited OpCord team for initial 2 weeks to explain the product/domain and set expectationsDaily meeting with client and clarifications

Agile Approach

Scrum; TDD & CI from XP

Agile Practices sprint/iteration, Standup meeting, continuous integration, Automated testing, code refactoring

Agile Metrics % Automated Test Cases

Tools Testing tool, AutoIT; CI tool, Hudson; VM Player, Google Docs, SVN

Why Agile in this project

Customer was very particular on ATDD and Test Automation;OpCord’s rich experience in Agile helped scale faster on agile testing practices

5

Testing Objectives

Comprehensive testing of the application

Test automation using open source tool from end user’s and developer’s perspective

Test Automation required for different platforms, like, Windows XP, Windows Vista (64 bit & 32 Bit), Windows 7 (64 bit & 32 Bit) etc

Measure Performance Parameters

CPU & Memory utilization for each test case

Time taken to execute every activity, every scenario and for the complete test suite

Testing of Installer

6

Agile Testing Strategy – Automation Tool - AutoIT

AutoIT v3.2 (recent version, 2011) is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting.

Uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks in a way not possible or reliable with other languages (e.g. VBScript and SendKeys).

Compatible with all versions of Windows, 2000 / XP / 2003 / 2008. Works with Windows Vista’s and Windows 7 User Account Control (UAC) also

Has abundant built-in commands

Interact with all standard windows controls, like, move, hide, show, resize, activate, close feature etc. Windows can be referenced by title, text on the window, size, position, class and even internal Win32 API handles.

Detailed helpfile and large community-based support forums

Comes with SciTe editor/compiler, which allows AutoComplete AutoIt3 commands, Intellisense (Show a ToolTip), Code folding for easy code viewing, Auto indentation while typing and dynamic help facility!

7

Agile Testing – Test Environment

Test scripts to be made compatible on 5 windows OSs

Client provided VMs (Virtual Machines) for all required windows licenses. It saved number of machines

In total, 4 machines were allocated with min 3 GB RAM. Min 3 GB RAM is needed to run VMs and automated test suite

We faced issues with Vista OS and had to increase RAM more

Later stage of project, more machines were provided because each test cycle use to take 4-5 hours to complete

8

Agile Testing Strategy – Framework -1

At first, test automation is targeted for Windows XP OS

All common functions were identified, like, main menus, delete, etc and scripts were written

Scripts and Data Separation

Single code base was designed to run on all agreed operating systems

Project Folder structure was decided like any other development project

9

Agile Testing Strategy – Framework -2

Coding guidelines, Naming Conventions, Commenting guidelines were agreed upon and evolved over time

Automation was done at 2 levels Scenario based (Developer’s perspective)

Each scenario had 5 to 20 test cases and In total, about 100 major scenarios (work-flow) were identified with the help of client and automated

Each test scenario (or workflow) was automated as separate function

One scenario takes 5 to 7 minutes to complete the testing and provide test results

Each scenario can be run independently so that developer can test particular functionality

Total Flow (End user’s perspective)

Later, All scenarios were integrated to run as a single test suite

Whole testing was made possible by single command

If a scenario fails than it records it as FAIL and continue to run next test scenario

10

Agile Testing Approach – Collaboration

Client shared the base software and visited OpCord to explain the requirements

Team first wrote the test cases and shared with client developers

Developers developed (or migrated) the desktop application using newer technologies and meanwhile testing team did the automation of test cases

Daily meetings were held with client to clarify the requirements as new application had to resolve issues in earlier version

Skype and TeamViewer were used extensively to do reviews and pair-programming when needed

It was must for everyone to always remain available on skype to have seamless discussions

Skype discussions were saved and sent to all team members, if required

Google docs were used to share test cases and other documents

11

Agile Testing Execution

Every week team released automated scenarios / scripts to developers

Developers used these scripts to fine tune the code and check whether functionality is correctly built or not

Defects were reported in Excel (using Google Docs) or basically failed test cases were considered as defects

After stabilizing scripts on Windows XP, Test environment was set for other OSs and scripts were fine tuned for other OS

Single code base was maintained to ensure compantibility

Build process was automated using Hudson and whole test suite was integrated with build process on cloud

Any time, any one can check the health of the product by just clicking a button

12

Limitations

Sometimes, AutoIT is not able to detect diagrams in the applications

Scripts are sensitive to key input, while running of scripts, no other applications or internet should run on the system.

While scripting, TAB orders were made based on the Control Focus on the Present Window. This situation may not occur during UAT.

Time out & Sleep time issues varies with the environment

If testing involves data creation than data should be deleted once script is over to bring application to same old state

Integration is elaborative – Script editing of a single file requires compilation of other dependency files along with the edited file

13

Recommendations & Significant Benefits

Use freewares like Skype, TeamViewer, Google Docs, BugZilla to collaborate with distributed teams

Test suite automation is like developing application hence similar approach needs to be followed

Agile testing and test automation is a must for products however need to keep maintenance aspect in mind

No. of testers reduced 5 times because scripts were compatible with 5 different operating systems

Total regression testing took only 5 hours per OS and can be scheduled in night

Modular approach to test automation suite allows for reusability of scripts

14

Thank You