GUI TestingGUI Testing

By Norbert HachéBy Norbert Haché

ContentsContents

What is GUI testingWhat is GUI testing Elements of GUI testingElements of GUI testing Old Approach (TRUMP Project)Old Approach (TRUMP Project) ScriptingScripting Capture / ReplayCapture / Replay Full Test IntegrationFull Test Integration Evaluation of CAPBAKEvaluation of CAPBAK DemoDemo

What is GUI TestingWhat is GUI Testing

Graphical User Interface (GUI) Graphical User Interface (GUI) TestingTesting

Methods used to identify and Methods used to identify and

conduct GUI tests, including the conduct GUI tests, including the use of automated tools. use of automated tools.

Source: (www.systemhouse.mci.com)Source: (www.systemhouse.mci.com)

Elements of GUI TestingElements of GUI Testing

A processA process A GUI Test PlanA GUI Test Plan A set of supporting toolsA set of supporting tools

source : www.csst-technologies.com/guioutln.htmlsource : www.csst-technologies.com/guioutln.html

Old Approach Example Old Approach Example (TRUMP)(TRUMP)

Was Done by manually stepping Was Done by manually stepping through thousands of pages of test through thousands of pages of test procedures.procedures.

Labour intensive, highly error Labour intensive, highly error prone.prone.

Needed to be redone each time Needed to be redone each time regression testing was required.regression testing was required.

Very expensive.Very expensive.

ScriptingScripting

Another Programming Language.Another Programming Language. Needs to be subjected to some Needs to be subjected to some

form of formal verification.form of formal verification. Eliminates human error during Eliminates human error during

execution of the test.execution of the test. Can be used (sometimes with Can be used (sometimes with

modifications) for regression modifications) for regression testing.testing.

Scripting cont’dScripting cont’d

Script Replay tool

Capture/Replay ToolsCapture/Replay Tools

A capture replay tool is a set of A capture replay tool is a set of software programs that capture user software programs that capture user inputs and stores it into a format (a inputs and stores it into a format (a script) suitable to be used at a later script) suitable to be used at a later time to replay the user inputs.time to replay the user inputs.

Note: Throughout this presentation I use Note: Throughout this presentation I use capture/replay and capture/playback to mean capture/replay and capture/playback to mean the same.the same.

Available Capture replay Available Capture replay toolstools

QA Partner [Segue Software, Inc)QA Partner [Segue Software, Inc) Xrunner & WinRunner (Mercury Interactive)Xrunner & WinRunner (Mercury Interactive) QC/Replay (CenterLine Sftware)QC/Replay (CenterLine Sftware) Evaluator (Eastern Systems)Evaluator (Eastern Systems) CAPBAK (Software Research, Inc)CAPBAK (Software Research, Inc) Vermont High Test Plus (Vermont Creative Vermont High Test Plus (Vermont Creative

Software)Software) WITT (IBM)WITT (IBM) ATS/X-Tester (Automated Testing Solutions Inc.)ATS/X-Tester (Automated Testing Solutions Inc.)

source : www.stlabs.com/marick/faqs/t-gui.htmsource : www.stlabs.com/marick/faqs/t-gui.htm

Capture Replay toolsCapture Replay tools

Capture/Replay

Script

Capture Replay ModelsCapture Replay Models

OS

Device Driver Device DriverDevice Driver

Capture/Playback tool Application

Application Events Self Events

Capture Replay Models Capture Replay Models (cont’d)(cont’d)

OS

Device Driver Device DriverDevice Driver

Application

Application Events Self Events

Capture/Playback tool

Capture Replay Models Capture Replay Models [cont’d][cont’d]

OS

Device Driver Device DriverDevice Driver

Capture/Playback tool

Application

Application Events Self Events

Full Test IntegrationFull Test Integration

Major drawback in Capture/Playback Major drawback in Capture/Playback tool is that when the GUI changes, tool is that when the GUI changes, input sequences previously recorded input sequences previously recorded may no longer be valid.may no longer be valid.

A test system which makes the A test system which makes the maintenance of Capture/Playback maintenance of Capture/Playback generated test scripts easy and fast is generated test scripts easy and fast is a must for such a tool to be of any a must for such a tool to be of any use.use.

Full Test Integration Full Test Integration (cont’d)(cont’d)

A capture/playback tool that support the A capture/playback tool that support the following capabilities could be used in a more following capabilities could be used in a more capable and fully integrated test development capable and fully integrated test development environment:environment:• record scripts of user/system interactionsrecord scripts of user/system interactions• user access to scripts for user access to scripts for

editing/maintenanceediting/maintenance• user ability to insert validation commands in user ability to insert validation commands in

the scriptthe script• allows replay of the recorded script.allows replay of the recorded script.

Full Test Integration Full Test Integration (cont’d)(cont’d)

A fully integrated GUI test development A fully integrated GUI test development environment would also require the following environment would also require the following additional characteristics:additional characteristics:• Script editing using higher level abstractions Script editing using higher level abstractions

such as icons etc.such as icons etc.• High level view of what functionality is High level view of what functionality is

being tested.being tested.• The ability to generate many variations of a The ability to generate many variations of a

recorded script without having to manually recorded script without having to manually edit the script itself.edit the script itself.

Full Test Integration Full Test Integration (cont’d)(cont’d)

A product called TDE under A product called TDE under development by Siemens is to provide development by Siemens is to provide exactly this kind of functionality exactly this kind of functionality (currently only at the prototype level)(currently only at the prototype level)

Source: ISSTA 98 Proceedings of the ACM SigSoft Source: ISSTA 98 Proceedings of the ACM SigSoft International Symposium on Software Testing and International Symposium on Software Testing and Analysis.Analysis.

Full Test Integration Full Test Integration (cont’d)(cont’d)

TDE capabilities:TDE capabilities:• uses higher level scenario language instead of uses higher level scenario language instead of

scripting. This allows graphical editing of the test scripting. This allows graphical editing of the test sequence and easy creation of variations.sequence and easy creation of variations.

• Has a test designer, which through user Has a test designer, which through user interactions with the system, builds an internal interactions with the system, builds an internal model of the system’s GUI to produce a high level model of the system’s GUI to produce a high level test design representing many executable scripts.test design representing many executable scripts.

• Test design library.Test design library.• Test generator engine to convert high level Test generator engine to convert high level

scenario into tests scripts.scenario into tests scripts.

TDE

Test Scripts

TestScript

GUIInfo

GUIReplay

record

Test DesignLibrary

Test DesignerTest

DesignTest

GenerationEngine

Tester

Specify Variations

Test Scenario ScreenTest Scenario Screen

TDE SUT RepresentationTDE SUT Representation

Full Test Integration Full Test Integration (cont’d)(cont’d)

When the GUI changes, instead of editing the When the GUI changes, instead of editing the hundreds of generated test scripts, the editing hundreds of generated test scripts, the editing is done at the scenario level where it is much is done at the scenario level where it is much easier and faster. This is followed by the easier and faster. This is followed by the automatic regenaration of the test scripts from automatic regenaration of the test scripts from the scenario.the scenario.

TDE can detect and analyse the differences TDE can detect and analyse the differences between a new GUI and its previous version. It between a new GUI and its previous version. It then makes assumptions about the changes then makes assumptions about the changes that can be subsequently overriden by the that can be subsequently overriden by the tester prior to script regeneration.tester prior to script regeneration.

Full Test Integration Full Test Integration (cont’d)(cont’d)

Using the prototype it was shown that in Using the prototype it was shown that in 30 minutes, a tester was able to create a 30 minutes, a tester was able to create a single scenario that produced 2500 test single scenario that produced 2500 test cases which exercised every significant cases which exercised every significant combination of input values and action combination of input values and action choices available for the particular choices available for the particular application.application.

Source: ISSTA 98 Proceedings of the ACM SigSoft Source: ISSTA 98 Proceedings of the ACM SigSoft International Symposium on Software Testing and AnalysisInternational Symposium on Software Testing and Analysis

Evaluation of CAPBAKEvaluation of CAPBAK

CAPBAK is a capture/replay tool CAPBAK is a capture/replay tool developed by Software Research developed by Software Research Inc of San Fransisco.Inc of San Fransisco.

Versions are available for Windows Versions are available for Windows 95, NT as well as for X-Windows.95, NT as well as for X-Windows.

My evaluation for the Windows 95 My evaluation for the Windows 95 version.version.

CAPBAK User InterfaceCAPBAK User Interface

CAPBAKCAPBAK

Very simple and intuitive to use.Very simple and intuitive to use. Documentation provided was dated.Documentation provided was dated. Had several bugs that need to be Had several bugs that need to be

fixed to make the product more fixed to make the product more robust.robust.

Provides many synchronization Provides many synchronization mechanisms, not all of which I could mechanisms, not all of which I could get to work properly.get to work properly.

CAPBAKCAPBAK

Provides OCR to save window text Provides OCR to save window text for baselining window image.for baselining window image.

Provides automatic detection of Provides automatic detection of changes to the window at the changes to the window at the bitmap and text level.bitmap and text level.

Provides an object mode to record Provides an object mode to record at the widget level. (I could not get at the widget level. (I could not get this to work properly).this to work properly).

CAPBAKCAPBAK

For more information, you may For more information, you may read the hand out on my read the hand out on my evaluation of CAPBAK.evaluation of CAPBAK.

DemoDemo

I have developed a basic MS Windows I have developed a basic MS Windows application that experiments with the basic application that experiments with the basic principles of user entry capture.principles of user entry capture.

It is very basic and does not attempt to filter It is very basic and does not attempt to filter out any particular events. I did it to show how out any particular events. I did it to show how easy it is to do such thing using the first model easy it is to do such thing using the first model showed earlier in the presentation which I am showed earlier in the presentation which I am showing here again.showing here again.

OS

Device Driver Device DriverDevice Driver

Capture/Playback tool Application

Application Events

USER32.EXE

Applicationbeing

captured

CaptureMain Window

Application

CaptureDLL

Console output

SummarySummary

GUI Testing using capture/replay GUI Testing using capture/replay tools is a useful technology if it can tools is a useful technology if it can be used within a test system that be used within a test system that allows efficient and high level allows efficient and high level maintenance capabilities of the maintenance capabilities of the test design.test design.