Test Automation

What is test automation

Test automation is an activity that aims to test the most frequently used and least frequently

changed parts of an application automatically. This can be done by test automation tools by the help

of programming languages supported. Therefore, related test cases are written as a piece of code.

Thought test automation activity is criticized under testing activity, it is a software development

activity. Every change in the application should be reflected to the test automation scripts so, even

the cleverest test scripts are produced, and maintenance activity comes out as problem of test

automation since the application is a live object.

Test automation has an important role in software quality assurance. Because the needs of

the sector and trend towards to test automation is getting high, there are a large number of test

automation tools exist in the market, but the most famous tools can be listed as follows and more

can be found in the list below :

• QTP: Supported by HP, originally develop by Mercury and imported by HP. It has

many features, easily record and replay, fast and easily development, short learning

curve and powerful support by HP, however for only Windows and costly license

price.

• IBM Rational Robot: Developed by IBM, short learning curve for testing team and

gives options for more sophisticated scripts for test automation engineer, supports

most of the UI platforms, however for only Windows and costly license price.

• Selenium: Open source UI test automation framework, best of the selenium is that

you can write test cases with your favorite language for webpage testing. Although

Selenium is a Firefox plug-in, it can be used for any web browsers. There are lots of

blogs, websites and forum to gather help about new features.

Important points for test automation

Test automation is a live object this means that every change made for the item in the scope

of test automation should affect the test automation scripts. Keeping updated the scripts is crucial.

Therefore, test automation should be started after the test cases are mature and should be apply to

suitable test cases for test automation. This can reduce reworks and unnecessary scripting efforts.

Automation scripts should not cause a fail in the system since the aim of the test is to find

defects in the software in such a case finding the root of a failure is not feasible. Also in such

condition which the automation scripts find a bug or error, it should recover itself then reports the

issue and never stops until the full scripts runs. This can be explain by “transition handling” which

means that test automation is composed of many separate test cases and these test cases run in a

desired/fixed order one by one. Every test case starts then performs its job and then ends then the

following test case does its job but the starting and the end points for all the test cases are the same.

When a failure is catch the test case doesn’t do its job but goes to the end point which is the starting

point of the following test case. This can be illustrated by flower leaf model as shown below:

Another import point is that spending effort for the automation is beneficial or not. The idea

behind the automation is to perform many things on terminals without using human efforts instead

the human can perform test analysts or other test activities. However maintenance effort should be

eliminated because as we said automation is a live object. There should take a statistical study on the

aim of the automation and the gains after a successful automation is applied to the project. This is

named as automation’s return of investment (ROI), a ration of benefits to cost. A mathematical

formula can be written as and more about the calculation of ROI can be found here

http://www.dijohn-ic.com/test_automation_roi.pdf:

ROI should be greater than 100% otherwise test automation is not better than manual test

execution. ROI is calculated for only one execution of the test set but the idea on the automation is

reusing it for every iteration so if you have a frequently release candidate, you have to test your

application before release this means that you should run the automation more than one. If you run

the automation for 3 times effective ROI can be written as:

Graph shows the ROI with execution times:

What are the test automation methodologies

Since an aim to automate everything for every condition is a utopia, focusing on an

applicable automation target should be more realistic. It can be described by SMART which stands

for test automation should be Specific, Measurable, Achievable, Realistic, and Timely. For example,

40% of regression test cases with previously defined 10 users on 3 popular web browsers for an e-

commerce firm could be applicable test automation strategy. After you have decided a strategy, you

can apply a test methodology to practice your test automation. Test methodologies can be described

as follows:

• Record and Replay:

o Records every steps to create a test case

o Easily replay the existing test case

o It doesn’t have loop or if condition or other complicated features

• Functional Test Automation:

o Every test case is automated by a different test scripts

o Test scripts include test data

o No import/export data option

• Data-Driven Test Automation:

o In addition to functional test automation

o Test scripts can run by different data

o Data can be stored/read from an external storage/file

o Only data-sets are updated not the scripts

• Keyword-Driven Test Automation:

o Every case named by a keyword

o The important keywords are sequentially run as a separate test cases

o New test cases can be easily created not need a programming knowledge

o Data and scripts are separately located

o Error handling and Transition handling are difficult

o Creating complex test cases is difficult job

• Hybrid Test Automation:

o Combination of keyword-driven and data-driven automation techniques

o Exception handling and transition handling mechanism can be used

o Desired data can be used for different test cases

o Data and scripts are separately located

Example for keyword-driven testing, important keywords are sequentially run as separate test case

sets. Sample keywords:

open the the webpage named as open;

login the user information named as login;

check the product;

add to basket;

checkout the basket;

logout from the webpage;

close the browser;

Belonging to the keywords above some test case sets may be written as “open + login + logout”,

“open + login + check + logout” and so on.

Mesut Güneş – 22.02.2013

gunesmes@yahoo.com