how to maximize roi from your test automation strategy and budget

Maximize ROI from Automation Budget

12-12-12

Sreekanth Singaraju

Vice President of QA & Testing Services

Allianceglobalservices.com

2 © Alliance Global Services 2012

Automation


Promise of Test Automation

Accelerated Testing Cycle

Decrease time-to-market for new applications and products – especially for regression testing

Increased Coverage

Enable more extensive and detailed testing

Repeatable and Auditable

Implement formal testing processes that provide evidence of execution and are easy to execute

Accuracy in Execution and Reporting

Not prone to human judgment or errors

Enhanced Productivity

Execute repeatedly and often; Future test cycles will be more productive

Use of correctly implemented automated frameworks aligned with organization objectives, deliver substantial benefits to the organization.


Reality of Most Test Automation

Implementation More expensive than planned

Takes longer

Fewer manual testing resources replaced

ROI is not met

Steady State Costs more to maintain than planned

Never seems to run successfully

Development loses buy-in

Out of sync with the application

Many automation projects unfortunately fail before they deliver projected ROI or substantially under-perform on the objectives.


Impediments to Achieving Best in Class Automation

Objectives Non-existent or Unrealistic automation objectives

Focused around technical goals rather than business goals

Automation Strategy

Highest automation ROI areas not identified

Subscale initiatives or insufficient commitment

Start-Up Costs

Tool licenses and framework creation costs

Up front Labor cost for implementing test cases

Expertise & Assets

Manual test teams lack development expertise

Knowledge of test case lifecycle mgmt. missing

Measuring Effectiveness

Lack of a metrics program to measure alignment with overall objectives

Evaluate effectiveness of testing programs and fine-tune strategy


FIVE PROVEN STRATEGIES FOR A

SUCCESSFUL TEST AUTOMATION


Objectives


Automation Objectives

Testing objectives need to be clearly defined and derived from business objectives.

These then serve as a guidepost for managing Automation programs.

• Execute a rapid Test Assessment for anyone planning to build or

refactor an automated regression test suite to identify business

objectives and current state of capabilities of the test organization

• Automation objectives are inherently easy to quantify, since a

baseline for costs, timelines and resources exists from comparable

manual testing

• Current state of capabilities and assets provides an understanding of

resources needed to implement automation

• Sets accurate expectations to key stakeholders

• Provides benchmarks for evaluating the progress of the automation

initiative


Automation Business Objectives – Example

A high level business objective was provided to decrease cost of testing while

providing same or better level of quality control.

• Utilized Alliance’s Assessment

kit to perform a rapid

assessment in matter of

weeks

• Mapped business objectives

to specific Testing objectives

• Defined automation initiatives

that need to be implemented

to achieve Testing objectives

• Baselined existing metrics on

the key objectives to be used

for measuring ROI and course

corrections


Automation Business Objectives – Example

A high level business objective was provided to decrease cost of testing while

providing same or better level of quality control.

Business Impact Area Current State

Quality

Documented Regression Test scenarios Less than 50

Regression testing for Mini Releases No Exhaustive Regression testing for Mini Releases

UAT Test Validations process Business and IT teams validate tests and is very expensive

Test data planning Test data setup after UAT start date extends the testing cycle time

Speed / Efficiency

Reduced # of Testing Cycles of Major Releases

4 Cycles

Duration of Test execution of Major Releases

25 calendar days

Internal business team effort reduction 221.75 days of effort (60% of IT (10) and 60% Biz (4) over 25 days)

External team effort reduction 13.25 days of effort (25% of 4 resources over 25 days)


Automation Strategy - What To Automate?


Automation Strategy - What To Automate?

Automation Candidates

High Business Value

Workflows based on large number

of test data combinations

Frequently used functionality

Long useful lifespan

End-to-End scenarios across

complex sub-systems and

integration points

Tedious and/or resource

intensive to conduct test

manually

What to Automate is the single most important planning step as it defines directly the achieved business value of the automation program


Prioritization

•Obtain the candidates and assign them scores on Business Value, Expected Use of Scenario and Ease of Automation

•Interface with business and development team to obtain inputs

Code

•Cyclomatic complexity of modules and dependencies

•Interactions with internal external systems

Data & Realtime Tracing

•Volume of Data generated

•Importance of data

•End-user usage needs

What To Automate (cont’d.)

Analyse the applications source code, database and real-time tracing to highlight candidates that should be then prioritized by business value, longevity and ease of automation


Upfront Costs


High Upfront Costs

Tools Explore multiple tools strategy if feasible to decrease the cost of testing

Resources Explore jumpstart programs to get valuable expertise into your teams rapidly

Framework & Build-out Attempt the smallest but highest business

value modules first – outsource if necessary

Infrastructure Utilize cloud or virtual models for Test Environment and Automation Infrastructure


Expertise - Frameworks In order for automated tests to be reusable, repeatable and maintainable, test development standards need to be defined and followed. These can be achieved by defining the framework and guidelines document has to be taken as the base and follow the development architecture finalized. There are some basic features that a framework is expected to have which include:

Application Independent

Easy to Use

Easy to Maintain

Is Scalable

Execute on Demand


Automation Framework

• Automation dependent on an efficient framework and governance

• Utilize quality measurement and reporting to evaluate effectiveness

• Execute and automate all aspects of testing from Framework to TestLab

• Reusable test data and test environment management processes

RESOURCES Experience Expertise

Flexible Scalable

ASSETS Accelerators

Frameworks Templates

TDM & STRATEGY

E2E TEST ENVIRONMENT

METRICS DASHBOARD

AUTOMATED TESTING FRAMEWORK

Successful automation frameworks focus on all aspects of automation and plan for resources and reusable assets


Titanium xAFT – Enterprise Test Automation Framework

Titanium xAFT (eXcel Automated Functional Tester) is

a robust enterprise test automation framework – Single

Script can be executed against Multiple execution

engines and browsers

Supports HP

QTP, Selenium,

TWIN execution

engines

Flexible &

Maintainable

framework

Based on Open

and proven

technologies

Titanium xAFT is a Open-Source Test Automation Framework to develop Automation

rapidly across the enterprise while creating highly maintainable suites.

30% productivity

improvements

over Selenium

automation

Collaborative

script

development

by BAs and

Testers in

Excel & Open

Office

Cross-browser Mobile - Web WebServices

http://blog.kalistick.com/Kalistick/kalistick-new-integrations/attachment/hp-qtp-logo/



Automation initiatives need to have metrics, reporting and analysis as an integral

part of the initiative to ensure success.

• Metrics should capture both business level and execution level values

to ensure efficiency is accurately captured and reported

• Metrics should not be left for the end of the project to implement

• Metrics measurements make implementation level course corrections

and expectations management easier

• Business Objectives identified should be directly tied to the metrics

program and measured throughout the initiative

• Assumptions and approximations can be made at the beginning of the

initiative


Business Metrics

Value Add Before Titanium Testing

Services After Titanium Testing

Services

Quality

Documented Regression Test cases

Less than 50 1900+ implemented in test management tool

Enabled Mini Support releases and Regression testing for Hot fixes

No Exhaustive Regression testing for Mini Releases

Enabled 2 Mini releases and 4 Hot fixes into release calendar

UAT Test Validations Business and IT teams validate tests

Testing team validates tests while Biz, IT approve results

Test data planning Test data setup after UAT start date

Test data setup and approval prior to UAT start date

Speed / Efficiency

Reduced # of Testing Cycles of Major Releases

4 Cycles 2.5 Cycles

Duration of Test execution of Major Releases

25 calendar days 13 calendar days

Internal business team effort reduction

221.75 days of effort (60% of IT (10) and 60% Biz (4) over 25 days)

74.5 days of effort (40% of IT (10) and 40% of Biz (4) over 13 days)

External team effort reduction 13.25 days of effort (25% of 4 resources over 25 days)

0 days of effort (Regression Test bed also supports Releases)


Execution Metrics Dashboard

• Should provide risk-based actionable information

with drill down capabilities

• Aggregates information from all aspects of testing

• Metrics should include

– Testing Confidence

– Testing Design and Execution Velocity

– Design Effectiveness

– Automation Coverage

– Automation Backlog

– Defect Identification Efficiency

– Cost of Quality


Thank You Sreekanth Singaraju

Title – VP of QA & Testing Services

Email - [email protected]

Website - www.allianceglobalservices.com

how to maximize roi from your test automation strategy and budget

Documents

alliance global services

asuccessful test automation

automation strategy

automation programs

automation initiativesthat

automation budget12

future test cycles

exhaustive regression