Automation

Testing

ROI

Presented by:

Yaniv Cherkasky

yaniv@e4d.co.il

054-7722568

Agenda

• First Session – Automation – Why? When? When Not?

– Test Automation Frameworks Models

– Guidelines for first approaching Automation

• Second Session – Automation ROI

– Improving ROI

– Case Study

Automation Testing Definition

Test automation is the use of software to control the execution of tests, Commonly, test automation involves automating a manual process already in place that uses a formalized testing process. (Wikipedia)

When?

• Repeatable Test Scenarios

• Application Logic Level

• Unit Test the code

• Checking the Performance

• Data/Event Driven Tests

• Multiple configurations

When Not?

• Test Cases that are complicated for coding (e.g. Image compression/Video quality etc.)

• Test Cases that we run only a limited number of times

• Premature parts in the Application (Constant Changes)

Why not Automate Everything?

• Low ROI (Return On Investment)

• Many times the Test Case is too complicated for a machine

• Most of the time the Look and Feel needs to be done by Human Tester

Why do it? • Length of the testing phase is reduced

• Higher efficiency (Save Time)

• Improve Productivity (Increase bugs coverage)

• Can reduce testing manpower (save money)

TTM

Resources Quality

Test Automation Types

Load and Performance testing

Installation and configuration testing

Boundary testing

Unit Testing

Compliance testing

Smoke and unit testing

Localization testing

Regression testing

Test Automation Framework common Models

• Data Driven Automation Framework Model

– Mass storage of data

– Limited to the automation tool in use

– Easy to implement, hard to maintain

• Keyword Driven Automation Framework Model

– Data tables using a keyword vocabulary

– Hard to implement, easy to maintain

Test Automation Framework common Models

• Test-driven development (TDD)

—Focus on the task at hand

—TDD leads to more modularized, flexible, and extensible code

—Create unit tests before writing the code

Automation Entry criteria

Frequent builds/patches/iterations

Test cases that need verification

with a high precision

Test case functionality with high complexity

Automation Exit criteria Successful Execution

Script has been reviewed for

coding standards

Review completed for

variables, functions, test

steps etc.

Report Repository

Version controlled

Guidelines for first approaching Automation

1. Perform full analysis breakdown of system features & modules relationship

2. Perform full analysis for manual test cases

3. Mark the test cases which are repeatable which has long execute time

4. Do a research of the environment (technology used, SW/HW limitations, platforms etc.)

5. Make a check list for the required testing types (Functionality, GUI, Performance etc.)

6. Pickup the appropriate test automation tool (MTM )

Guidelines for first approaching Automation

7. Pick the suitable framework structure

8. Conduct a breakdown of all framework methods

9. Mark the common use of framework methods in all Test Cases – these methods will become your first building blocks!

10. Calculate ROI

11. Prepare a presentation with Automation Framework structure & ROI calculation

12. Pick an example

13. Run a POC

Tips & Tricks • Make sure your are considering future maintenance in your

planning – maintenance cost money!

• Don’t start with demanding resources, start with 1 resource for POC

• Use also authentic factors for the ROI (items related to the unique product/environment etc.)

• Automation is a development group, manage it as one!

• When you approach your boss, in any given time, make sure you are translating all to budget terms :-)

• Monitor Automation ROI on a yearly basis comparison

Questions?

Automation ROI It is important to consider the following factors when evaluating the ROI of automation:

• The cost of executing the automated tests. Consider the infrastructure required

• The cost of maintaining the automation as the product evolves release by release.

• The cost of any additional tasks necessitated by the automation

• ROI will vary depending on your business environment and the application under test.

Importance of ROI

• ROI is as important if not more important than technical automation topics

• Test Automation is an investment

Return On Investment

TTM

Resources Quality

ROI Challenge

UNREALISTIC EXPECTATIONS

• 100% Automation

• Immediate Payback

• No Automation Analysis

ROI Dilemma

• Continued automation justification

• Difficult to meet unrealistic expectations

True test automation benefits:

Increased Cost Savings

Increased Efficiency

Increased Quality

Calculating ROI - Factors Automation Testing Factors

– Tool Costs

• Software • License

– Execution Machines

– # of Tests to Automate/Effort Estimation – Automated Development/Debugging Time (average per test) – Automated Test Execution Time (average per test) – Automated Test Analysis Time – Automated Test Maintenance Time

Fix

ed

Variable

Calculating ROI – Factors

Return on Investment

Automation ROI

Number of hourly employees? 500

Average hourly rate of employee? 16.00

Number of minutes to review/total each time card (APA 7)? 3

Hourly rate of person reviewing time card? 16.00

Percentage of inaccurate time cards (APA 1-8)? 2%

Number of standard hours an employee worked in a pay period? 40.00

Daily lost time in minutes for an employee (Calculation, Rouding) (APA 49)? 20

Card Audit Savings

Total savings in minutes 1000

Total minutes saved weekly 5000

Total hours saved weekly 83.3

Savings in dollars weekly $1,333.33

Monthly savings $5,773.33

Yearly Savings $69,333.33

Savings in Lost Time

Daily lost productivity in hours 0.33

Average wages overpaid daily $5.28

Average wages overpaid weekly $26.40

Total wages recaptured weekly $13,200.00

Total wages recaptured monthly $57,156.00

Total wages recaptured yearly $686,400.00

Savings from Eliminating Errors

Total weekly payroll expense $320,000.00

Weekly savings $6,400.00

Monthly savings $27,712.00

Yearly Savings $332,800.00

Total Savings Summary

Weekly audit savings $1,333.33

Weekly error savings $6,400.00

Weekly wages recaptured $13,200.00

Weekly payroll savings $20,933.33

Monthly payroll savings $90,641.33

Yearly Payroll Savings $1,088,533.33

Calculating ROI for Automation Projects is an on-going task…

Let:

NPV(ATCF) = Net Present Value of After Tax Cash Flow generated by the

investment

NPV(IC) = Net Present Value of Required Investment

Profitability Index = 100*NPV(ATCF) NPV(IC)

− NPV(IC)

An outline of the equations underlying this expression is given in the Appendix.

The elapsed time to positive cash flow, i.e. the time at which the

discounted cumulative cash flow crosses into the positive region is also important.

This is sometimes called the discounted payback period.

Improving ROI

Result Improvement

Improving ROI

Result Improvement

Automation ROI

Case Study On-Line Shop

Order Pages

Automation ROI – Case Study • Order pages test includes verifying the on-line buying of

products

• Effort Estimation for manual testing takes about 2 weeks for 1

person

• The frequency of running the test is quite common since full

regression is needed each time a change is made in results

from a new marketing campaign or even a new discount

• Manual test does not cover all buying aspects (boundary

cases) – but only the positive testing required by the

requirements

• Currently, the test duration harms the project dynamic

proceeding

Automation ROI – Case Study cost$ per month

Manual Testing (today) Automatic testing

Effort in days $ cost Effort in days $ cost

Sickness 1 days 113.63 0 days 0

Holiday 1.5 days 170.44 0 days 0

Test Development time 7 days 795.41 19.5 days 2215.785

Test Execution time 14 days 1590.82 1.5 days 170.44

Bugs analysis time 5 days 568.15 1 days 113.63

Report creation 3 days 340.89 1 days 113.63

Test maintence time 1 days 113.63 2 days 227.26

Total: 3692.97 Total: 2840.745

Legend:

Month Total $

Tester Hourly Rate 113.63 22 days 2500

Automation Fixed costs

Testing tool 3000

Automation ROI – Case Study Conclusions

• Currently on “Order Pages” tests we save $ 852.225 per

month (for 1 run) it’s $ 10226.7 per year by using

automatic tests (numbers include lost business,

refunds and testing the complex scenario with all it’s

factors manually)

• Currently in 1 year there are about 5 campaigns of to be

tested…doing it with manual tests will take 120 days. If

using automation for it, it will take 7.5 days. It means:

112.5 working days that are saved!!!

Automation ROI – Case Study Conclusions

• Automation tests can run 10 times more in 1 month then

using manual tests which means: more regression =

more stable product

• Automation 1 test run duration = 7 peoples work

• The cost for the long run is actually decreases!

• Most of the automation costs are fixed costs (one-time

expenses) and are for the short run

Questions?

Thank You!!!