Economic Perspectives in Test Automation:

Balancing Automated and Manual Testing with Opportunity Cost

Paper By – Rudolf Ramler and Klaus Wolfmaier

Presented By – Pallavi Gulati

Topics

Why Automated Testing? Simplistic Cost Model for deciding

between Automated vs. Manual Testing

Flaws in the Simplistic Cost Model New Model based on Opportunity Cost Benefits offered by the new model Future Plans

Why the need to Automate Tests?

Testing accounts for at least 50% of total project cost

Fast-paced iterative development Shrinking schedules and budgets

Test Automation proposed to reduce testing costs

Advantages - Automated Tests

Reduces overall cost of RUNNING tests Though some initial cost required for writing

automated tests Increases the number of tests run Increases frequency at which tests are

run Incremental cost of test running is less

Major concern in deciding whether to fix a particular bug or not

Disadvantages - Automated Tests

Can not automate every test case – timing-related, appearance, user-friendly, some system & integration test cases, involving different hardware setups

May not catch new bugs, targeted towards regression bugs

Higher latency as compared to manual testing which can start immediately

When to go for automation?

How do managers generally decide when to automate?

Decision is based on factors like: Efforts required to run manual test Frequency of testing Cost of third-party / in-house developed

tool Efforts required to write automated tests

Simplistic Cost Model

Vm = Estimated Cost of designing manual tests

Am = Overall cost of manual testing for some # of times

Va = Estimated Cost of designing automated tests (may also include cost of test tool)

Aa = Overall cost of automated testing for some # of times

Flaws - Simplistic Cost Model

Manual & Automated Testing are not comparable Exploring new functionality vs regression testing

Only costs are analyzed not benefits Manual testing starts early in the process but some

latency in automated testing All test cases & executions are considered equal Project Context (budget, resources) not considered Misses additional cost factors like training on the

testing tool, new process adaptation

Why automated tests fail to provide expected benefits?

Due to flaws in estimation using the Simplistic Cost Model and other factors like:

Failure to understand that automated tests are targeted for regression bugs

Gross underestimation of the effort required to develop automated tests

Practically no allocation of efforts for maintaining automated tests

When to automate? vs. What to automate?

It does not help to automate every test

Need to decide trade-off between automated and manual testing

Alternative - Opportunity Cost Model

Opportunity Cost Model

Consider manual & automated test cost Decide how many manual test cases

should be automated Pick those which will give maximum

benefit / risk mitigation / coverage Project budget & deadlines considered

Looks something like...

Opportunity Cost Model Va = 1 hour to

automate 1 test 0 min to run 1

automated test

0 min to come up with 1 manual test

Dm = 15min to execute 1 manual test

Restriction 1

Restrictions 2 and 3

Target Function

Ra = Risk Mitigation for automated testing

Rm = Risk Mitigation for manual testing

This allows us to include the benefit obtained by automating a test

Scenarios for Analysis

Scenarios for Analysis

Model can be extended to include Additional Influencing Factors

Tests that rely solely on automation e.g. 100 concurrent transactions

Productivity changes over time Causes an outward curved graph

Growing Test Effort in iterative development Improves risk mitigation of automated test

Maintenance cost Early / Late return on investment

Developing automated test early using data driven technique

Defect detection capability random input invoking crash

Advantages– Opportunity Cost Model

Draws attention to balancing automated and manual testing

Benefits as well as costs Risk mitigation capability of tests Budget and deadlines More realistic comparison between

automated and manual testing

Future Plans

Reduce complexity & make it more practical to use

Evaluate preciseness and completeness

Identify & add more influencing factors

Analyze need to add more restrictions

Summary

Advantages of Automated Test Simplistic Cost Model Flaws Proposed Opportunity Cost Model Benefits offered Future Plans for expanding the

model

Wake up !!!