Testing Tools

Presentation for Senior Mgmt TeamAugust 4, 2003

Agenda• Types of Testing Tools• Demo of Robot Recording and CTA (web-based)• Demo of WA Regressions (virtual terminal-based)• Demo of CA Import Tests• Demo of Execute Action• Business Case for Automation• Other Uses of Automation• Additional Tool Needs• Summary and Questions

Types of Test Tools

CompuwareIBM

(Rational) Mercury SegueHighJumpSoftware

UIRegression

QARun- Robot

- Data DrivenRobot Macros

WinRunner SilkTest

Stress &Performance

QALoad- Robot Virtual

Users- TeamTest- TestStudio

LoadRunner SilkPerformer V

Test Mgmt /Manual Test

QADirector Test Manager TestDirector SilkPlan Pro

DefectTracking

Track Record ClearQuest TestDirector SilkRadar - Defect Advantage- Track for Windows

CoverageTools

PurifyPlus

HomegrownTools

- 2425 Emulator- Automate a Platform

Regression using Architect- XL2XML Utility

Misc. Testing Tools -Worth Mentioning

• Stress/Performance– Empirix (Web Based Application Performance)

• Defect Tracking– TeamTrack by TeamShare

• Coverage Tools– C-Cover by Bullseye– TrueCoverage by NuMega– TCAT C/C++ for Windows by TestWorks– AQTime’s Coverage Profiler by Codework– CoverMeter by Synopsys – LiveCoverage by Paterson Technology – G-Cover by Green Hills Software– VectorCAST by Vector Software– Test Coverage by Semantic Designs– CTC++ by Testwell– Object Coverage by ObjectSoftware– . . .

Demo• Robot/CTA

– Record/Playback• Notepad demo

– CTA Regressions• Item and Container Pages

• Three levels of a UI regression script– Level 1 - Record/Playback Scripts– Level 2 - Portable Scripts

• Use internal functions to make scripts portable– OS independent– database independent– directory location– Platform version– Timing independent (speed of machine)

• Error handling (script continues even when the unexpected happens)• Log condition of environment• Database dump/compare

– Level 3 - Data driven Scripts• Single script that is adaptable by configuring a database

Demo

• WA– Receiving– Demonstrate how an actual defect was found

using this script– Using Virtual Terminal screen verification points– Also using internally developed Robot functions to

verify database state

Demo

• CA Import TestingXL2XML

XMLFile

AdvantageLink for XML

XMLFile

XMLFile

AAD

Import Folder

WebWiseCollaboration

Advantage

Excel Spreadsheet

Baseline

Import to CA

Rational Robot

View Test Data

Table Dumps

Demo

• Platform Testing– Copy the latest version of the test application and scripts to the test

environment– Open Advantage Architect and the test application then compile and

activate– Start the Advantage Workflow Engine– Open the Application Status Console and System Status Console– Execute the test– Current test exercises the runtime functionality of “Execute Action”

parameter types

Business Case for Automation

• Benefits of automation– Increase speed and accuracy– Reusability– Repeatability– Broader test coverage– Discover defects early

Cost of Defect Resolution

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

Planning

Design

Coding

Testin

g

Implem

entat

ion

Producti

on

Defects

When to invest in Automation

• Multiple Hardware and Software configurations

• Multiple Builds• Product Longevity• Product Sales Vision• Repetitious or tedious testing

Case Study

Item Manual Tester Manual Cost Automated Tester Automated Cost

Cost $50.00 per hour $60.00 per hourTime to author 1 test case 1 hour per or 7 per day 1 hour per or 7 per dayTime to author 100 test cases 11 days $4,400 11 days $5,280Time to automate 0 days 0 15 days (30% + manual dev time) $7,200Time to execute and analyze EACH test case and results 3 per hour or 24 per day 20 per hour Time to execute and analyze ALL test cases and results 4 days $1,600 5 hours $300Total run through the code release for 1 release $6,000 $12,780

Item Manual Tester Manual Cost Automated Tester Automated Cost

Time to test for each subsequent release 4 days $1,600

Rework on scripts at 5% of original development time or 6 hours $360

First Release Labor Cost

Subsequent Releases Labor Cost

Case Study Continued...

Item M anual Tester M anual Cost Automated Tester Automated Cost

First Quarter plus three times subsequent quarter cost 6000 + (3 * 1600) $10,800

Rework on scripts at 5% of original development time times 4 or 3 days $13,860

Cost of testing on all pertinent combinations (Sql Server & Oracle on Win 2003)

All tests run on 2 configurations for each code change. $21,600

No additional cost for addition configuration $13,860

Item M anual Tester M anual Cost Automated Tester Automated Cost

Subsequent Release Cost * 4 $6,400 $1,440Cost of testing on all pertinent combinations (Sql Server & Oracle on Win 2003)

All tests run on 2 configurations for each code change. $12,800

No additional cost for addition configuration $1,440

First Year Labor Cost Summary

Subsequent Yearly Labor Cost

Yearly Savings Summary

Total Costs Per Tester 2003 2004 2005

Cost per license $5,765 $0 $0Cost for support $1,153 $1,153 $1,153Labor Cost $13,860 $1,440 $1,440Total Cost $20,778 $2,593 $2,593

Labor Cost $21,600 $12,800 $12,800

Total Cost Savings $822 $10,207 $10,207

Automated Costs

Manual Costs

Automated Cost Yearly Savings

Current Use of Automation at HighJump

• Tools we own/use– Two licenses of Robot

• Current Projects– CTA, WA– Advantage Platform– CA– WebWise regressions for Microsoft Gold Certification

• Planned Projects– “Reference” application for testing platform services– Automated nightly builds with smoke test

• Subset of product regressions used as build verification– Continue automating all WA processes– EA

• Testing all possible filters for correct runtime behavior• Regression test of UI for defining all filter types

– Installation regression (across all products?)

Additional Tool Needs

• UI Regression– Additional Robot license just purchased– Purchase first WinRunner license

• Performance/Stress Test– Research and recommendation

• Coverage– Research and recommendation

Summary & Questions

• Product Development committed to realizing the benefits of test tools at HighJump

• Research will continue to identify and select tools for performance testing and coverage

• Some tools may have practical application in professional services team as well