Software Testing:Software Testing:Tools for Software Test Tools for Software Test

AutomationAutomationRobert V. BinderRobert V. Binder

20002000

EEL6887: Software Engineering Life-Cycle ControlEEL6887: Software Engineering Life-Cycle ControlLoren StroupLoren Stroup

March 22, 2006March 22, 2006

Reference SourcesReference Sources Textbook: Testing Object-Oriented Systems: Models, Patterns, and Textbook: Testing Object-Oriented Systems: Models, Patterns, and

Tools, by Robert V. Binder, Addison Wesley, 2000Tools, by Robert V. Binder, Addison Wesley, 2000 Website: Are Automated Test Tools For Real?, by Alan R. Earls, Website: Are Automated Test Tools For Real?, by Alan R. Earls,

May 1, 2004, May 1, 2004, http://www.adtmag.com/article.aspx?id=9307http://www.adtmag.com/article.aspx?id=9307 Website: A Sampling of Automated Testing Tools, by Lana Gates, Website: A Sampling of Automated Testing Tools, by Lana Gates,

May 1, 2004, May 1, 2004, http://www.adtmag.com/article.aspx?id=9308http://www.adtmag.com/article.aspx?id=9308

OverviewOverview Test Automation DefinedTest Automation Defined Advantages to Automated TestingAdvantages to Automated Testing Limitations and Caveats to Automated TestingLimitations and Caveats to Automated Testing Test Harness DesignTest Harness Design

Test Case/Suite MethodTest Case/Suite Method Test Case/Suite ClassTest Case/Suite Class

Automated Testing and the CMMAutomated Testing and the CMM Views on Automated Test ToolsViews on Automated Test Tools Sampling of Automated Test ToolsSampling of Automated Test Tools SummarySummary ConclusionConclusion

Test AutomationTest Automation Definition:Definition:

Test automationTest automation is software that automates any aspect of testing of an is software that automates any aspect of testing of an application systemapplication system

Test automation includes capabilities to generate test inputs and expected Test automation includes capabilities to generate test inputs and expected results, it has no manual intervention and it can evaluate pass/no passresults, it has no manual intervention and it can evaluate pass/no pass

Automated testing depends on testing goals, budget, software process, kind Automated testing depends on testing goals, budget, software process, kind of application under development, and particulars of the development and of application under development, and particulars of the development and target environmenttarget environment

As project nears completion, two undesirable options are faced:As project nears completion, two undesirable options are faced: Rerun fewer baseline test cases and focus on the tests necessary to validate the Rerun fewer baseline test cases and focus on the tests necessary to validate the

final incrementsfinal increments Add more people to enter and judge the increasing number of baseline test Add more people to enter and judge the increasing number of baseline test

casescases Both options involve risk: The first option has risk concerning error detection, the Both options involve risk: The first option has risk concerning error detection, the

tester is hoping that software fixes have not introduced errors in other test cases. tester is hoping that software fixes have not introduced errors in other test cases. The second option has risk concerning adding more staff on a budget that The second option has risk concerning adding more staff on a budget that cannot change and adding test cases that increases the testing schedule.cannot change and adding test cases that increases the testing schedule.

Test Automation (cont)Test Automation (cont) Six main steps to automated testing (post design/development of Six main steps to automated testing (post design/development of

test driver):test driver): Exercise the Implementation Under Test (IUT) with the automated test Exercise the Implementation Under Test (IUT) with the automated test

suitesuite Repair faults revealed by failuresRepair faults revealed by failures Rerun the test suite on the revised IUTRerun the test suite on the revised IUT Evaluate test suite coverageEvaluate test suite coverage Enhance the test suite to achieve the coverage goalEnhance the test suite to achieve the coverage goal Rerun these suites to support regression testing or other forms of Rerun these suites to support regression testing or other forms of

testing that require the ability to compare baseline resultstesting that require the ability to compare baseline results Last step is the focal point of automated testing, it delivers quick Last step is the focal point of automated testing, it delivers quick

results on how stable the baseline isresults on how stable the baseline is

Advantages to Automated TestingAdvantages to Automated Testing

Permits quick and efficient verification of bug fixesPermits quick and efficient verification of bug fixes Speeds up debugging and reduces “bad fixes” by allowing the Speeds up debugging and reduces “bad fixes” by allowing the

developer to rerun a test case to reproduce the problem and then developer to rerun a test case to reproduce the problem and then verify that the bug fix worksverify that the bug fix works

Allows consistent capture and analysis of test resultsAllows consistent capture and analysis of test results Tester productivity is improved by allowing the time spent entering Tester productivity is improved by allowing the time spent entering

and reentering tests to be used on designing tests that achieve and reentering tests to be used on designing tests that achieve greater coveragegreater coverage

Manual tests are always error-prone while automated tests won’t Manual tests are always error-prone while automated tests won’t type a wrong input or have other distractionstype a wrong input or have other distractions

Automated comparison is a repeatable and efficient way to evaluate Automated comparison is a repeatable and efficient way to evaluate a large quantity of output, testers can become fatigued after a large quantity of output, testers can become fatigued after scanning pages of outputscanning pages of output

Limitations and CaveatsLimitations and Caveats Automating a test is typically more expensive than just running the Automating a test is typically more expensive than just running the

application, if no need to repeat the tests is evident, then automation application, if no need to repeat the tests is evident, then automation is probably not justifiedis probably not justified

When testing applications with significant user interaction, the tester When testing applications with significant user interaction, the tester can use this knowledge to try features and combinations that might can use this knowledge to try features and combinations that might be hard to identify when designing a GUI scriptbe hard to identify when designing a GUI script

The cost of maintenance may be quite high if the requirements and The cost of maintenance may be quite high if the requirements and implementation of the IUT change frequently. If the test automation implementation of the IUT change frequently. If the test automation is not maintained, it will become unusable and the expected return is not maintained, it will become unusable and the expected return will not be realizedwill not be realized

Test Harness DesignTest Harness Design Definition:Definition:

A A test harnesstest harness is a system that supports effective and repeatable is a system that supports effective and repeatable automated testingautomated testing

The test harness is the system that starts the IUT, send test The test harness is the system that starts the IUT, send test messages to it, and then evaluates test pass/no pass resultsmessages to it, and then evaluates test pass/no pass results

A test harness provides some of the following capabilitiesA test harness provides some of the following capabilities Test case/test suite implementationTest case/test suite implementation Interface to the IUT by which test messages may be sentInterface to the IUT by which test messages may be sent Test execution controlTest execution control Automated comparatorAutomated comparator Result loggingResult logging Regression test support – selection of a test suite subset and creation of Regression test support – selection of a test suite subset and creation of

the necessary test configurationthe necessary test configuration

Test Harness Design (cont)Test Harness Design (cont) Test Harness design patterns and architectureTest Harness design patterns and architecture

Test Cases – How should test message be implemented?Test Cases – How should test message be implemented? Test Control – How should the servers of the IUT and its environment Test Control – How should the servers of the IUT and its environment

be controlled and observed?be controlled and observed? Test drivers – How should test code be organized so that it is modular, Test drivers – How should test code be organized so that it is modular,

has a meaningful correspondence to the IUT, and achieves reuse where has a meaningful correspondence to the IUT, and achieves reuse where possible? How can drivers support controllability and observability?possible? How can drivers support controllability and observability?

Test Framework – How should test cases, test suites, drivers, and a Test Framework – How should test cases, test suites, drivers, and a tester interface be organized for an entire application system?tester interface be organized for an entire application system?

Test Harness ArchitectureTest Harness Architecture

Test Case/Test Suite MethodTest Case/Test Suite Method IntentIntent

Implement an API scope test case as a method of a driver classImplement an API scope test case as a method of a driver class ApplicabilityApplicability

Useful at class and cluster scope for unit and integration testing when Useful at class and cluster scope for unit and integration testing when developing a driver to exercise a class, a class cluster, or other developing a driver to exercise a class, a class cluster, or other component that will be tested using its APIcomponent that will be tested using its API

AdvantagesAdvantages Implementation is simple and easy to codeImplementation is simple and easy to code May be used with any kind of API driver class, can be inherited and May be used with any kind of API driver class, can be inherited and

overriddenoverridden DisadvantagesDisadvantages

Pattern required programmer skills to design, develop, debug, and Pattern required programmer skills to design, develop, debug, and maintainmaintain

Test code is susceptible to the same problems as application code Test code is susceptible to the same problems as application code (bugs, low maintainability, low portability)(bugs, low maintainability, low portability)

Test Case/Test Suite Method Test Case/Test Suite Method Pattern StructurePattern Structure

Test Case/Test Suite ClassTest Case/Test Suite Class IntentIntent

Implement test cases or test suites as objects of class TestCaseImplement test cases or test suites as objects of class TestCase ApplicabilityApplicability

Useful at class and cluster scopes for unit and integration testing – developing a Useful at class and cluster scopes for unit and integration testing – developing a driver to exercise a class, class cluster, or other component that will be tested driver to exercise a class, class cluster, or other component that will be tested using its APIusing its API

AdvantagesAdvantages Traceability and related information are easily represented and may be Traceability and related information are easily represented and may be

automatically updated as tests are run and rerunautomatically updated as tests are run and rerun The full power of the implementation language can be used to automate all The full power of the implementation language can be used to automate all

aspects of test executionaspects of test execution DisadvantagesDisadvantages

Pattern required programmer skills to design, develop, debug, and maintainPattern required programmer skills to design, develop, debug, and maintain Test code is susceptible to the same problems as application code (bugs, low Test code is susceptible to the same problems as application code (bugs, low

maintainability, low portability)maintainability, low portability)

Test Case/Test Suite Class Pattern Test Case/Test Suite Class Pattern StructureStructure

Automated Test and the CMMAutomated Test and the CMM Deciding when automated testing is appropriate involves reviewing Deciding when automated testing is appropriate involves reviewing

past metrics on various programs, thus, a CMMI level of at least 3 is past metrics on various programs, thus, a CMMI level of at least 3 is idealideal

Automated testing can be used at any level of the CMM, however, Automated testing can be used at any level of the CMM, however, recording and following process standards is evident at level 3 and recording and following process standards is evident at level 3 and higher, which makes automated testing easier to implementhigher, which makes automated testing easier to implement

At CMM levels 4 and 5, automated testing can be firmly established At CMM levels 4 and 5, automated testing can be firmly established in the company’s processes and standards and the organization can in the company’s processes and standards and the organization can predict trends in product quality and how automated testing will predict trends in product quality and how automated testing will achieve this or if it is cost effectiveachieve this or if it is cost effective

Automated testing may involve training staff members and CMM Automated testing may involve training staff members and CMM levels of 3 and higher have training processes in place and have levels of 3 and higher have training processes in place and have been used many times on other projectsbeen used many times on other projects

Other Views on Automated TestingOther Views on Automated Testing

Downside (Kanglin Li)Downside (Kanglin Li) Vendors tools are impressive during demonstrations of basic things, Vendors tools are impressive during demonstrations of basic things,

however, when the tester needs to test software in depth, more however, when the tester needs to test software in depth, more functionality needs to be added to generated test scripts which can be functionality needs to be added to generated test scripts which can be complex and time consumingcomplex and time consuming

Automated test tools can be expensive and will only be a good Automated test tools can be expensive and will only be a good investment if the automated test scripts are reused frequentlyinvestment if the automated test scripts are reused frequently

Considerations (Theresa Lanowitz)Considerations (Theresa Lanowitz) When considering to adopt automated tools, ensure there are skilled When considering to adopt automated tools, ensure there are skilled

professionals in place and a good processprofessionals in place and a good process Automated tools are only useful when proper training is given to bring Automated tools are only useful when proper training is given to bring

staffing up to speed on the toolsstaffing up to speed on the tools Documented, repeatable development and test process should be Documented, repeatable development and test process should be

already established (i.e. CMMI level 3 and higher is valuable)already established (i.e. CMMI level 3 and higher is valuable) Test automation can significantly reduce risk only if processes and Test automation can significantly reduce risk only if processes and

standards are followed version after versionstandards are followed version after version

Sampling of Automated Test ToolsSampling of Automated Test Tools

QACenter Performance Edition 5.0 – Compuware Corp.QACenter Performance Edition 5.0 – Compuware Corp. Used mainly for load and performance testingUsed mainly for load and performance testing http://www.compuware.comhttp://www.compuware.com

QuickTest Professional – Mercury Interactive Corp.QuickTest Professional – Mercury Interactive Corp. Good for functional and automated regression testGood for functional and automated regression test Uses the concept of Keyword-driven testing to radically simplify test Uses the concept of Keyword-driven testing to radically simplify test

creation and maintenance creation and maintenance http://www.mercuryinteractive.comhttp://www.mercuryinteractive.com

WebLoad Analyzer – RadView Software Inc.WebLoad Analyzer – RadView Software Inc. Good for Web application performance testingGood for Web application performance testing http://www.radview.comhttp://www.radview.com

Sampling of Automated Test Tools Sampling of Automated Test Tools (cont)(cont)

Unified TestPro (UTP) 7.0 – SDT TechnologiesUnified TestPro (UTP) 7.0 – SDT Technologies Good for functional testing; Test Transformation Suite – test Good for functional testing; Test Transformation Suite – test

methodology, tools, best practices, servicesmethodology, tools, best practices, services http://sdtcorp.comhttp://sdtcorp.com

SilkPerformer Component Test Edition – Segue Software Inc.SilkPerformer Component Test Edition – Segue Software Inc. Good for remote component testingGood for remote component testing SilkPerformer Lite is good for Web application testingSilkPerformer Lite is good for Web application testing http://www.segue.comhttp://www.segue.com

SummarySummary Automated testing depends on testing goals, budget, software Automated testing depends on testing goals, budget, software

process, kind of application under development, and particulars of process, kind of application under development, and particulars of the development and target environmentthe development and target environment

Automated Testing has many advantages and disadvantages and Automated Testing has many advantages and disadvantages and must be used wisely and when it is deemed appropriatemust be used wisely and when it is deemed appropriate

A A test harnesstest harness is a system that supports effective and repeatable is a system that supports effective and repeatable automated testingautomated testing

Two kinds of test harness design are Test Case/Test Suite Method Two kinds of test harness design are Test Case/Test Suite Method and Test Case/Test Suite Classand Test Case/Test Suite Class

There are many automated test tools available through many There are many automated test tools available through many companies and vendorscompanies and vendors

ConclusionConclusion It’s up to an organization to decide whether automated testing is It’s up to an organization to decide whether automated testing is

applicable for themapplicable for them A CMMI level of 3 or higher will have the proper training and A CMMI level of 3 or higher will have the proper training and

documentation on how to handle automated testingdocumentation on how to handle automated testing Automated test tools can be expensive and will only be a good Automated test tools can be expensive and will only be a good

investment if the automated test scripts are reused frequentlyinvestment if the automated test scripts are reused frequently Test harness design is a viable option for helping to automate Test harness design is a viable option for helping to automate

testing at an organizationtesting at an organization Automated testing is very powerful for regression testing efforts Automated testing is very powerful for regression testing efforts

when complete systems need a full run through of test cases to when complete systems need a full run through of test cases to ensure no other failuresensure no other failures

Automated testing can greatly reduce risk if the processes and Automated testing can greatly reduce risk if the processes and documentation at an organization are followed consistentlydocumentation at an organization are followed consistently