how to manage your testing automation project ttm methodology

©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice 1

Lior Katz - Testing Automation & performance leader,

Tact testing (Matrix)

Jul 14th 2010

How to Manage

your Testing Automation Project

Testing Tools Management Methodology

2 Footer Goes Here

Who am I ?

4 Footer Goes Here

“Test Automation is probably the most

misunderstood concept in the field of

software testing. The automation tool

implementation in testing is a costly process

and requires careful planning and execution.

Test Automation has to be treated like a full

blown software development project as the

final key deliverable, is in fact a software

code ” –Software test practitioners

5 Footer Goes Here

“In general, the efforts to implement

Automated Tools and realize ROI have been

marginally successful. One of the main

reasons for this failure has been the lack of

planning and inadequate funding needed to

deliver key initial services around the

deployment of tools,

training, and mentoring of resources by the

purchasing parties. Many IT managers still

believe that the tools will and should work by

themselves.” 2009 WORLD QUALITY REPORT

6 Footer Goes Here

Testing Automation projects Testing Automation project life cycle

Should be - Testing Automation projects life cycle

1 Purchase the tool 2 Recruit developers 3 Understand ROI 4 Fail to implement

1 Analyze the systems

Purchase the tools 2 Recruit developers

and managers 3 Understand ROI 4 Follow TTM

methodology

7 Footer Goes Here

Testing Tools Management methodology

8 Footer Goes Here

TTM methodology - motivation

A methodology developed by myself after years

of experience, to understand the market need and

the breaking points of testing automation projects

and load testing projects.

9 Footer Goes Here

Why Do We Fail?

We have the best people and the best tools

but still most projects are failing

management expectations

10 Footer Goes Here

How to succeed with your testing automation project using the TTM

Why Do We Fail

– management support (not enough)

– high expectations for immediate ROI

– Unrealistic expectations from a tool

– Underestimating the time, cost and effort for the initial introduction and implementation

of a tool (e.g. training, external expertise and changes to the testing process).

– Underestimating the effort required to maintain the test scripts and tool updates

– Over-reliance on the tool results

– robust scripting

–Understand the application and the development process

– How to divide our automation project in order to succeed?

A MUST is to decide first regarding Testing automation packages progress

(Sanity, Mini Regression, Regression, divide by modules)

11 Footer Goes Here

TTM process - 4 steps

1. Define the project scope

2. Follow up

3. Testing Automation methodology

4. closer

12 Footer Goes Here

1. Define the Project Scope

– High Level Design (Test Plan) The document contains the Set we are about to automate, the concept of

automation, verification points, parameters, etc…

13 Footer Goes Here

1. Define the Project Scope

– Design (Test cases QC) Test cases developed in QC cover all required area in the application

14 Footer Goes Here

1. Define the Project Scope

– Estimation module A tool can assist to understand how many working days we need for the

project, include all required steps:

• Testing priority

• All Automation steps

• Risk factor

15 Footer Goes Here

1. Define the Project Scope

– Work plan Following the estimation module solution the exact work plan by days

16 Footer Goes Here

2. Follow Up - Test Cases list

17 Footer Goes Here

2. Follow Up - Work plan

18 Footer Goes Here

3. Testing Automation Methodology

– QTP Standard and Guidelines

Documents introduces basic guidelines and code methods for scripting via

QTP. The document describes all acceptable practices to develop in QTP:

19 Footer Goes Here

3. Testing Automation Methodology

– Code inspection

All of the inspectors to reach consensus on a work product and approve it for

use in the project

20 Footer Goes Here

4. Closer activities

– Final Report - A document contains all project steps include:

– Test case list

– Testing automation coverage

– Functions

– Folder where script

– How to execute the SET

– Result example

21 Footer Goes Here

4. Closer activities

– Code Freeze

A process to save all developed code under one place. (With a limited access)

22 Footer Goes Here

Testing Automation Stages

– Sanity SET

Set of ~50 QTP scripts, verify that new build (version) ready for massive testing process

– Quick ROI Developments

Packages can retrieve the investment immediately (e.g. API, data driven, data inflation)

– Regression by modules - Do Not to develop all regression tests in one

SET, “Before moving forward with the development stage, we need to

analyze our system and divide it by modules, for each module to

develop it’s own automation SET”

– Data Inflation

• Using our GUI interface can help with massive data changes

23 Footer Goes Here 23

TTM methodology - ROI example

– Each execution was reduced by ~20 working days

– No. of execution per version - ~7

– Total = a reduction of 140 working days per version

– One working day = ~225$

Saved for the organization ~31500$ per version

24 Footer Goes Here

Thank you all!!!