presentation paper bio return to main menu t4 return to ...€¦ · – winrunner: testscript...

P R E S E N T A T I O N

International Conference On

Software Testing, Analysis & ReviewNOV 8-12, 1999 • BARCELONA, SPAIN

Presentation

Bio

Return to Main Menu

Presentation

Bio

Return to Main Menu T4

Thursday, Nov 11, 1999

Software Test Improvement

in Small Organisations:

Project VISTA

Rene (I.G.) Kamphuis

Paper

11-11-1999 11

VISualisation Testing Automation

René Kamphuis

ECN-EE,

P.O. Box 1,

1755 ZG Petten,

The Netherlands

[email protected]

Eurostar-99

11 november 1999

Barcelona

11-11-1999 2

Contents:

• Context

• VISTA

• Follow-up after the end of the project

11-11-1999 3

ECN (Energieonderzoek Centrum Nederland)

• Netherlands Energy Research Foundation

• Mission: contribute to a clean, renewable and reliable energy supply for aviable world

• ECN employs more than 800 staff.

• Contracts are obtained from the government and from national and foreignorganisations and industries.

ECN: Research areas

• Fuels, Conversion and the Environment

• Nuclear Research & Nuclear Facilities

• Policy Studies

• Solar and Wind Energy

• Energy Efficiency

11-11-1999 4

ECN: Energy Efficiency

• Programme Energy Efficiency

• Fabrication Technology Services

• Engineering Services:

– Mechanical Engineering

– Process Engineering

– Process Automation

– Software Engineering

11-11-1999 5

Working areas EE/PA and SE :In ECN-projects or external assignments

• GIS-based decision support systems

• Model based capacity prediction software

• C/S-applications

• Editors/viewers for geometric models

• Graphical front-/backends for numerical simulation programs

• Database applications

• Logistic optimization packages

• Advanced simulation and visualisation (CSE)

• AI-techniques (ANN, GA,KBS)

11-11-1999 6

Context

• Tools should be embedded in existing procedures and initiatives (CMM:CM(2), PD(2), PM(2), SPE(3) and PR(3))

• Linkable to existing GUI and Visualisation tools (Visual Studio, WindowsNT, Unix, X-11)

• Starting point is level 2-/ISO-9001 organisation.

• Metrics to evaluate pre- and post-situation difficult to get

11-11-1999 7

Process improvement issues with respect to testing

• Tracking of defects and management of repair not structured

• Testing, especially in multiplatform environments, takes too much (30-50 %)effort (> NPlatforms)

• Predictability of defect removal too low; cascading effect

• Testing is delivery driven; not structural part of defined life-cycle activities

• No separation of black-box/white box testing

• Lack of integration with other key process areas

• Testing assigned to final stages when budget is exhausted

11-11-1999 8

Expected benefits of structured testing

• Validation/verification as early as possible

• Tool supported and managed defect tracking

• Reduced cycle time

• Effective in multi-platform porting

• Process control by metrics

• Increase visibility of software development process

• Earlier defect detection

• Automation of the testing process

11-11-1999 9

Workpackages (ESSI-PIE)

• 1-11-1997: Start

• I: 1-3-1998: Evaluation, hands-on training, positioning in softwaredevelopment cycle and planning for base-line project.

• II: 1-5-1998: Implementation: learning, tailoring to developmentenvironment and procedures. Create testsuites.

• II. 1-9-1998: Weave in delivery schedule of baseline project. Collect metrics.

• III: 1-11-1998: Analysis and evaluation

• 1-3-1999: End

11-11-1999 10

WP I: Tool selection criteria

• Programmable; steep learning curve for software engineers

• Maintainable scripts (change resistant)

• Enabling Black-box approach early in development

• Adaptable in technically/scientific development environment (e.g. supportdiff in file-compare)

• Support multiple operating system environments

• Supporting test-structuring and management

• Configurability

• Should not change behaviour of the application (debugger effect)

• Low-impact on the test-organisation

11-11-1999 11

Tool-components

• Test management

– Structuring and concatenating of scripts in sets

– Automatically execute tests

– Defect tracking

• Session recording

– Context sensitive vs. analog

– Scripting (C-like, VB-like,…)

– Reuse for different test-cases

11-11-1999 12

WP I: Results

• Tool selected: Mercury Testsuite

– TestDirector: management tool

– WinRunner: testscript designer

• Hands-on training course (8 engineers; two days)

• Awareness of software developers increased

• Problems:

– Tools automate task of an emerging discipline of testers -> broad rangeof terminology and skills to be mastered

– Finding of and aligning to a suitable base-line project

– Collection of metrics

– Multi-platform usage

11-11-1999 13

WP I: Results:Usage of blackbox tools in test structuring

Usability Alternatives

Module test - InspectionWhitebox

Integration test -/+ Frequent builds;Inspections

System test + Frequent builds

Porting test ++

Delivery testing ++

11-11-1999 14

WP I: ExperiencesDefining and planning WP II-base line

• Designing test-ware and making a good test-plan takes time and requires alot of domain knowledge and detail.

• Design aspects play an equally important part as for software. Testing needsarchitecture, design and implementation phases

• Alignment with existing development scheme needs changes from thatscheme and a change of development practices.

• Making a comprehensive set of test-cases leads to scrutinizing the functionalspecifications

11-11-1999 15

WP II : Base-line project Characteristics

• LECTOR-4:

– Model based capacity prediction system

– 8 m/y in 2.5 years; critical to user organization

– Client(GUI;front-/backend;VB/VC++))

– Server (mathematical model in Fortran-77; Unix)

– 32-bit development environment; 16-bit delivery; traditional userorganisation

– integrated set of graphical component editors

– in transition stage to adaptive/perfective maintenance phase and wide usein organisation

11-11-1999 16

WP-II: Steps

• Define 0-situation

• Tailoring of test-tools.

• Implementation of test-application; creation of test-suites

• Coupling to development/maintenance of new versions of the product.

• Apply to version delivery

11-11-1999 17

WP-II: Baseline start :Defects per component editor as reported

Bug statistics LECTOR4

0

5

10

15

20

25

30

35

40

Aggregatie All Kwgeg Opdrachten Scenario Server Tabeldef

Module

Nr

of

bu

gs

Aug97

Dec97

Feb98

Mar98

11-11-1999 18

WP-II Start: User detected defect classificationBug statistics LECTOR4

0

5

10

15

20

25

30

35

Ease ofuse

Codingerror

No error Omissionspecs

Inaccurate Sitedependent

Postponedimpl.

Bug Type

Nr

of

bu

gs Aug97

Dec97

Feb98

Mar98

11-11-1999 19

WP-II Start: Time track defect-repair

Time track bug fixing

0

50

100

150

200

250

Jun-97 Aug-97 Sep-97 Nov-97 Jan-98 Feb-98 Apr-98

Data

Nr

of

bu

gs Found

Solved

Pending

11-11-1999 20

WP-II Start: Internal time-track bug-fixing

Time track bug fixing

0

20

40

60

80

100

120

140

Apr-97 Jun-97 Aug-97 Sep-97 Nov-97 Jan-98 Feb-98 Apr-98

Data

Nr

of

bu

gs Found

Solved

Pending

11-11-1999 21

WP-II Start: Conclusion

• New delivery of some components gives cascading effect in defect rate ofothers due to integrated architecture

• Coding defects are detected after delivery

• User drives priority in defect removal (not design or internal validation)

• Interpretation of defects partly ambiguous

• Engineering approach needed

11-11-1999 22

WP-II: Activities

• Test cases (based on use cases) defined for all components

• Test scripts built including bill-of-materials

• Test scripts aggregated to test-sets

• Integration with development and product delivery started

• Test-ware consultancy aided initial steps

• Test-configuration management en test-management documents added to thesystem development handbook

• Software development trajectory adapted to allow testing in an earlier stage

05/10/99 23

Adapted process model

FD TD ImplementTest andRework

FOTO

Test-planImplement

Test

Re-work

11-11-1999 24

Test management

• Defect registration

• Automatic and planned tests

• Version.Release.Build control=Functional.Adaptive/Corrective.Internalworking version

Test automation

• Defect-tracking and reporting with TestDirector

• Testscripting and automated execution with WinRunner

• Planned execution of testing several software configuration items:WinRunner/TestDirector

11-11-1999 25

Base-line/problems

• 16/32 bits platforms for Windows

• Planning maintenance activities; repeat procedures for minor repairs ?

• Separation of roles in teams in view of organisation size

• Domain knowledge

Base-line/benefits

• Delivery with increased accordance to specs

• Management of defect repair

• Signalling/early detection

• Test-effort reduction/shift to higher levels of abstraction

• Improvement of internal testing

11-11-1999 26

WP-II: Experiences

• Drastic decrease in number of defects after delivery; delivery centred testing

• ‘Minor’ errors detected before delivery; probable prevention of cascading

• Tools are software themselves with interoperation problems

• Defect tracking very useful

• Defect management and structured problem reporting pays off

• Technical-scientific software is inherently complex; complete test-coverageis impossible

Within EE/SE,PA• Sustained formalisation, disciplined approach

• Managed

• Separated roles tester/programmer

• For certain application types

11-11-1999 27

Application potential

Type Process TD WR C/S + GUI ++ ++ ++

“C”/Fortran +GUI

++ + +

Database-appl’s

+ + +

WWW-appl’s + 0 - Graphics en

Vis. + 0 --

Dacq. + RealTime

+ ++ + (intermittent)

Fortran codes + 0 -

11-11-1999 28

WP-III: Evaluation; Improvement issues

• Tracking of defects and management of repair ; ++

• Testing, especially in multiplatform environments, takes too much (30-50 %)effort (> NPlatforms); -

• Predictability of defect removal too low; avoid cascading effect; +

• Testing is delivery driven; should be a structural part of defined life-cycleactivities; +

• No separation of black-box/white box testing; +

• Lack of integration with other key process areas; +

• Testing assigned to final stages when budget is exhausted; +

11-11-1999 29

WP-III; Found benefits

• Validation/verification as early as possible; +

• Tool supported and managed defect tracking; +

• Reduced cycle time; +/-

• Effective in multi-platform porting ; --

• Process control by metrics; +

• Increase visibility of software development process; +

• Earlier defect detection; +

• Automation of the testing process; +/-

11-11-1999 30

VISTA: Lessons learnt

• Structured testing starts in the design phase

• Other processes in the organisation have to be disciplined and structured aswell

• Automated test-tools in a small organization require a productchampion/guru

• Automation tools are state-of-the-art now; first experiences very encouraging

• Involvement and awareness of software engineers during introduction taketime

• Not every software software engineer is a good tester

• Involve end-user

• Don’t underestimate trajectory to gain sufficient skills

11-11-1999 31

Follow up

• GIMP: measurement and logistics system for pharmaceutical industry

• Test-management and structuring very important to comply to standards

• Thorough pre-delivery testing

of 9Project VISTA Software process improvement in small organisations

Software Test Improvement in Small Organisations:Project VISTA

I.G. KamphuisECN, Netherlands Energy Research Foundation

Unit Energy Efficiency,Group Software Engineering,

P.O. Box 1,1755 ZG Petten,The Netherlands.

Tel 31-224-564544,Fax. [email protected]

Abstract

The objective of the VISTA-project was to introduce the use of automated testing tools andmanagement of the testing process in the software development process at ECN. In a baselineproject, metrics and experiences, were collected with newly designed procedures and tools. Theresults are described especially with emphasis on key-aspects and lessons learned for a smallrelatively small software producing organisation in a technical R & D setting.

Introduction

ECN is a technological research centre working on a renewable, clean and reliable energy supply.ECN is grouped into business units, which each have gained a position in the market on a certainkind of energy generation technology. The software-engineering group in ECN has an applicationdriven, project directed approach for making applications. Due to the complexity of possible userinteraction, software testing presently typically takes 30-50% of the total cost of visualisationsoftware development. Applications, available on multiple operating system platforms require moretesting effort than single-platform applications. To improve both aspects, an ESSI/PIE-funded projectiVISTA project was started.

Objectives

The objectives of the project were:• Structure and define the testing efforts to increase predictability and save on cost• Automate the test management and testing process.• Use the tools to facilitate multi-platform testing.• Gain experience with test maturity concepts as a small organisation.

Environment

The software engineering group (18 persons), due to ECN’s R&D environment, is dedicated to alarge diversity of operating system platforms and programming languages. The group uses structured


design methods, supported by CASE-tools. It serves all test installations at ECN with extensivelynetworked software in an R&D setting.

The complete working portfolio is diverse with respect to size and to application area. Some projectsare done in one-person assignments. In major projects, teams of up to four people are engaged.Recently, positive responses were obtained when a PSP-program was introduced.

In ECN, tracking and managing software defects is not standardised. Cascading of effects of defectremoval impairs predictability. Up to delivery, continuously compromises are made to reachdeadlines. Testing and correction activities are sometimes postponed until a next set of functionalityis built or parallel to developing new versions.

The base line project ,aimed to gain experience, concerned a large client-server system for modelcalculations with an extensive user interface. It presented a largest common denominator of tools andexpertise’s in the group.

Expected outcomes

As a result a shorter time-to-market in our original project proposal by merely automating tests.Furthermore, awareness and test structuring are equally important. Other improvement issues arerelevant. As soon as something, is available to integrate it should be built and tested to make defectsvisible as early as possible. An important attribute of a module and very generally speaking a designis testability.

The tool should automate a number of repetitive tasks now done by highly qualified personnel. Itshould support seamless porting and testing in multi-platform environments.

Control possibilities of the process by metrics in terms of defect management procedures should beavailable. Some unequivocal project independent statistical parameters should be made available tohelp managing and tracking the project. Structuring and embedding testing in the life-cycle modelshould precede automated testing.

A subdivision of usage potential of the new techniques versus application type in the ECN/EEportfolio was our fourth aim.

The fifth aim was the get insight in the technical and people management aspects of introducing thenew techniques and tools.

Preparation-phase/tool selection

A significant number of test-tools are available on the market. Each test-tool has its strengths andweaknesses but also its familiarity with the ECN software engineering operating domain. Some toolsare database transaction, stress test or load-test oriented. Non-tool selection criteria were:

• Suitability for white-box testing;• Ample possibilities for stress testing;• Possibility of SQL-transaction or load testing. Above attributes are mostly found in non-technical/business IT-environments. Our criteria were:• Programmable: software engineers also have the role of tester (besides roles as application

consultant, functional and technical designer, programmer and maintenance programmer). Asoftware engineer covers a varying number of tasks in a project.

• Maintainable scripts and modular test-scripts. Test-sets should be built by combining modulartest-scripts. Blackbox testing implies that differences in functionality lead to differences in


behaviour and to changes in test-scripts. Useful language attributes and standard functionspresent in programming languages to facilitate maintenance should be available.

• Try to define products as early as possible. Test-tools should be present from the integrationstage onwards. Component-based development should be supported.

• Support non-UI tasks. In technical scientific environments apart from user interface action andvisualisation a large number of files may be generated, that defines a test case. Certain Unix-primitives to handle files should be supported and automatically tested.

• Support multiple operating systems. Enable automatic delivery, test-set execution and testingfrom one common source for several versions of operating systems.

• Tools should consist of a user interface capture-and-playback tool, but also of a defectmanagement tool, which is equally important.

The main asset for us would be a tool supporting test-script generation, test automation and test-structuring. From the tools available on the market, three main candidates were selected. In the end,the Mercury test-suite (Winrunner(WR)/TestDirector(TD)) came as most appropriate. Mercuryscored best for programmability (‘‘C’’-like), possibilities to define teststructuring and GUI-objectcontext-sensitivity. The tool does not record sessions analogously but knows about UI-semantics. Thetool supports a GUI-Map, which allows developers to correlate objects from the programmingdomain with the testing domain. The concept increases the maintainability of test-scripts.Furthermore, a strong point of the tool-set is the availability of TestDirector, a test structuring,reporting and maintenance tool.

Problems encountered with phase I are the following:• Tools automate the task of an emerging discipline of testers. Test design and testing is becoming

a separate discipline in software development organizations. In a small team of developers it isvirtually impossible to define separate testteams in view of the size of a typical application andnumber of persons involved in software development.

• A baseline project should be comparable in timeframe (effort to spend, calendar time) with theVISTA-project working package II.

• Metrics: enough figures exist from the current situation, but all connected to one generalisedclass of ‘‘bugs’’. Timing and management information on defect removal falls short.

To deploy structuring of the testing process, apart from the usage of tools, a variant of the well-known V-model (verification/validation) was described as a chapter of our handbook for systemdevelopment. In this model project activities are viewed in a testing framework with specific rolesand responsibilities. Inspections take the role of non-automated white-box tests. Reviews play asimilar role with respect to functional and technical design documents. In module-tests it is best toleave responsibility to individual software engineers to deliver modules that adhere to a certaininterface specification. The acceptance test is the validation against the requirements. In automatingwith the selected test-tool the following test-phases can be improved. The same, to a lesser extent,holds for the integration test. Usability of a black-box testing tool is largest from the systemintegration test stage onwards.

Part of our CMM-related activities were coupled to yield a chapter “Test Management Standards” forour system handbook. CMM-activities PR and SPE in the context of testing were described in thisdocument together with the place of automated testing in these KPA’s. The automated tools aregeared to the later stages of the internal software construction trajectory and the end-user acceptancetesting.

Lector, the baseline project adopted, is a software system, forecasting the participation and mobilityof students in the Dutch educational system. The tool allows model calculations for several userdefined aggregation levels in various scenarios. The Netherlands Ministry of Education, Culture andScience in Zoetermeer, The Netherlands uses the tool for capacity assessment for various school


types. The software was encountering its fourth major release. In this release, the user community onthe Ministry is increased from 3 to 15 persons. Lector consists of Graphical User Interface PC-clientand a model-calculating server on a Digital Unix workstation. The architecture mimics the nuclearviewer architecture proposed originally to serve as the base-line project. The server softwareprogrammes for such computations are very large and use alphanumeric input. Visualisation softwaresuch as the client-software allows the user to graphically represent and manipulate scenarios andaggregations of input and output streams in an intuitive manner. Furthermore the results can bevisualised in concise and informative, but end-user friendly, manner. The visualisation software isimplemented on Microsoft Windows (3.11 and NT) platforms. The server runs under Unix. The Lector-software is improved continuously. The client software must follow this development,which means extension in terms of user population, usability and functionality. Multi-platformtesting is important as in the user environment 16-bit Windows 3.11 is used and in the developmentstage 32-bit Windows NT 4.0 is used as an operating system. This gives us the additional opportunityto gain experience with porting tests. The increase in user community and the more formal way ofinteracting with the customer, was one of the factors that urged us to further structure our testingtrajectory. Based upon historic information, a database was filled with existing defect information of the base-line project, prior to the tool introduction. Important classifications in this defect database were thetype of defect, the time between detection, diagnosis and fix, and the stage in the softwareconstruction process. Cascading error effects were seen in products previously working well If we tryto classify defects, coding errors are the main source of errors. Improvement of user convenience andinterpretation errors (and not defects in functionality) is the main following defecttype. Furthermore,it appears, that user detected bugs were ‘‘fixed’’ very quickly as compared to the time to repair forinternally found defects. Internally diagnosed errors were prioritised lower than end-user detectederrors. Furthermore, some regression of errors could be seen. The preliminary results suggest a veryproductive usage in our software development trajectory for structuring and automated tools:management of defects can be done much more stringently and regression errors can be detected inour own development environment and not in the end-user operational environment.

Selecting and defining reproducible test cases requires some rethinking of parts of the design andimplementation so far. The TestDirector-tool urges the test engineer to rethink, structure andaggregate test examples and test cases according to a number of test-sets. Furthermore, teststructuring activities are pulled forward to an early phase in the development. The tool automatesinternal tracking of defects. All conclusions lead us to focus on internal process improvement steps.

Implementation of tools and procedures

Test cases, covering a large part of the Lector software, were designed and assembled in differentcombinations as separate test-sets. The testing environments of the VISTA test-tools and Lector-4component editors in their development environment were coupled. For Lector, a number of pre-existing test-sets were incorporated in the suite. The test cases were programmed and debugged withthe Mercury-tool WinRunner. Combination of the test-scripts and customising them yielded test-setsintroduced in the TestDirector tool. The test-sets now are available for automated execution andguide the pre-delivery stage.

The test-tool needed some tailoring. It has to be made flexible with respect to several testingenvironments by using configuration parameters. Test script-coding standards and best practices havebeen defined. According to the test-plan, screens have to be linked to the GUIMap, a very usefulmapping scheme between the GUI and the test-environment. Seamless coupling to the development-trajectory is of extreme importance. The GUIMap, a kind of repository of UI-components, facilitatesbuilding test-scripts. In respect to the broad functionality of the tools, it was considered useful toassign a guru/product champion in the process. As a last stage of phase II, the test-sets were mademandatory for new maintenance releases of the Lector-4 software to end-users.


A test-plan and definite formal test cases were defined. These were ‘‘programmed’’ into the test-automation environment. To prevent interdependency and cascading of errors in a multi-componentsystem ,a kind of bill-of-materials test, had also to be defined. This answers the question if allcomponents are in the same condition present as in a previous delivery. In the future, the possibilityin the Mercury tool-set to check compliance of a GUI to Windows style-guides is a valuable option. Making test-scripts required much more effort than just recording user interface actions. Definingtest-files, database and composition of test-sets as structured “programs” requires more effort.Control structure usage and primitives in test-script-languages require careful consideration.Integration in our development environment has been achieved. A test-tool is software itself andshowed some errors; therefore a maintenance contract and help-desk facilities of the tool are anessential pre-requisite for continuity. Errors encountered pertained to the complex technicalarchitecture of the application. This architecture also hindered making a comprehensiveimplementation all testing cases. No primitives within the test-tools were found, that enabled a clientapplication to get “job-status” information from a server process. Furthermore, using logicalvariables to discriminate between different testing environments needed some local development andfine-tuning. Therefore, only a part of the total system could be instrumented and tested with theautomated tools.

Practical difficulties in using the tools existed in network connectivity between our DEC-Unix- andNT-filesystems exiisted. Problems with permissions, naming conventions and protections in ourenvironment caused extra work to program into the test-scripts. Intermittent change of the operatingplatform between different Windows-standards for the development and the user environmentscaused extra problems. The GUIMap functionality of the tools relieved the tasks in some respect.With respect to the organisation, a clear distinction had to be made between the developers and thetest-team.

Regarding the first positive results with the LECTOR-project, for another project the testplan wasimplemented using the tools. Using the database in TestDirector, user diagnosed errors wereregistered accompanied by status information and information about planning defect removal. Inretrospect, the phase II baseline project was challenging and ambitious in terms of improving theprevious experiences with customer deliveries, culture of the development team, design andprogramming attitude of the test-team, built-up in the project, and technical complexity.

On the other hand, maximisation of the learning effect for the organisation within the tight timereserved for VISTA activities was achieved. Activities in phase II were closed with the first majormaintenance delivery of the Lector-software. To augment our experience, apart from the baselineproject, during the end of phase II a second project was chosen to gain further quantitative resultswith the new methods. This system did not have an as complex technical architecture as LECTOR.The ease of implementation of the tests was very much larger and with the experiences gained acomplete set of tests could be generated. For this project, the design was made in OMT using theSelect OMT-tool. The object-oriented analysis in terms of use cases enabled a straightforwardtranslation into test cases.

Technical Evaluation

Technical difficulties exist for heterogenous hardware architectures. Linking and enablingcommunication from a Windows-NT/95/3.11 client process to a UNIX server present somedifficulties on the application level. On the testing tool level, these difficulties were prohibitive forimplementation of the complete test-plan. Furthermore, 32-bit development 16 bit-delivery platformsautomated test-tools are hard to cope with. Introducing two more tools on a development platformapart from compiler and workbench tools increases version compliance difficulties. This gave someproblems with setting up and replaying reliable tests. The GUIMap feature in WinRunner partlyrelieved this issue, but some PC-packages (Quattro, Access) do not adhere to the standards of MFCset by Microsoft and the GUIMap becomes of no use.


Only the WinRunner tool covered the internal pre-delivery testing process. Introduction of structuredtesting has changed some configuration management, product development and software productengineering procedures, supported by the tools, already existing. Both the external defect registrationand the internal testing process made use of the TestDirector tool. Using the tools for UNIXplatforms brings a five-fold increase in spending and is not within reach considering the assignmentportfolio at this time. This presents a serious drawback for the given toolset, as ECN/EE has aportfolio of visualisation applications portable on the Windows and the Unix platforms.

Design methods supporting structured testing are now emerging. For instance, in many objectoriented design methodologies, Use Cases are of prime importance for elicitation of userrequirements. Use Cases also define test cases and test sets in a natural way. In this respect, recentcollaboration between our tool vendors, Mercury, and our CASE-tool vendor, Select, is of particularinterest.

Apart from a project plan, for larger software development projects a detailed test-plan will benecessary. Early introduction of the tests specified from the system integration stage will allow us todetect and correct defects in an earlier stage and to rethink parts of the design in terms of test-ware togenerate.

Some technical difficulties with the deliverables in the Lector-maintenance became manifest earlierin the development process than before due to the structured testing effort. Problems would haveappeared without the tools at the customer site. Scripts aimed at checking other aspects of theapplication due the regression nature of the test-sets yielded unexpected, but very valuable, results.

At last, it should be realised that separate roles and technical environments of test and developmentteams must be set up and be kept in existence in the maintenance phase of projects.

Business evaluation

For every software development activity, throughout the process, using structured test proceduresenhances quality. A direct business advantage of automated testing can only be partly quantified atthe moment, as the scope of our maintenance efforts in the Lector-project is only beginning toemerge. In respect of EE’s projects portfolio the use potential is collected per application type. Thelargest potential can be found for large projects with distinct roles in the user organisation and thedevelopment organisation for testers. Visibility of the efforts in testing to the client organisation thenis of utmost importance. For WWW-applications, standards in GUI’s have not yet settled andbrowser wars makes automated testing a tricky business. For real-time data-acquisition and controlsystems the system behaviour changes by introducing automated tools. This can lead to testtoolgenerated intermittent errors. For the GUI-part there are possibilities. For advanced graphicsapplications with a research character, there are no significant gains to be expected. On the contrary,the dynamics of state-of-the-art visualisation cannot be fitted in the WR/TD architecture.

In our base-line project the total amount of testing time has not decreased as much as was expected.Roughly the effort is the same. However, the activities are shifting towards designing and structuringtest cases and to awareness in earlier phases of the testing process of the added value of testing. Ourcustomer achieves definite turn-offs in the number of problem reports. The amount of error reportsfrom testtool guided deliveries are an order of magnitude smaller than before introduction of thetools.

Organisational evaluation

In the organisation, management commitment to software improvement efforts is variable. Usingtesting tools and collecting information from the user with regard to tests early in the specification


trajectory will enhance end-user involvement and satisfaction and in more management commitmentin the beginning stages of a project.

More specifically, in an organisation with the product portfolio as ECN/EE introducing the newlyacquainted expertise in testing and automation of the testing process must be a project managementdecision at the beginning of the project. It leads to a distinct place for testing in the software lifecycle and to the assignment of distinct roles to development and test-teams in the project.

For smaller projects, assigning different teams is not effective as the spreading of problem domainknowledge over a larger number of people not outweighs the gains achieved by independent test-teams. In small projects extending the number of domain expert’s causes extra time to get acquaintedwith the functional specifications.

A procedure for defect handling has been established. Phases include registration, analysis,verification, resolution, planning and repair. Generally, a more disciplined and rigorous look onsoftware construction was developed in the project. Tools were not found to be more important thanthis disciplined approach. In view of the size of the projects and the size of the development groupthe critical mass and benefits for a dedicated software testing group is reached in larger projects byassigning a one-person test-team.

Cultural aspects

In the ECN organisation application domain (Energy System related) knowledge is needed apart fromsoftware engineering knowledge. This factor, coupled to the focus of our clients on working energyapplications, might complicate our improvement effort. As a measure to consolidate the pathpresented, a testing guru with detailed knowledge of the testing tools is assigned. Furthermore, atesting plan wizard has been assigned. His role is to assess test-plans from the experience in test-design, obtained in the VISTA-project.

Testing tools, as we use them in the VISTA project, are mostly in use in large software packageconstruction firms, where a separate department of testers uses them. These financial-economicalsoftware settings have a far more disciplined approach to the software making process as is the casein the R&D setting of ECN/EE. In this sector, structured and automated testing is common. Due toour organisational focus, it is not to be envisaged that such a separate discipline will emerge in ourgroup.

A disciplined approach is of utmost importance to success of software testing techniques. In R&Dsettings the manner of working, is more iterative and explorative with a less disciplined way ofworking. This is even more applicable for our internal customers. The internal approach is productsfocused and not processes focused. A product has to gain its targeted results. Affinity with testsstructuring and that this leads to longer-term guarantees for maintenance and reliability has to beeducated to our clients. There is a way to go here still.

In the organisation the tools now have found their way. Recently, a project with very intense productquality requirements has been done, in which the testtools have been given an even more intensiverole in the software construction process right from the beginning.

The error-registration and defect-reporting procedures are considered to add to an necessaryformality to the defect repair process. The value is seen and the benefits are realised.

Apart from the toolage expertise, additional expertise has been built up about test structuring anddesigning testware. Due to our broad scope of projects and variance in duration and the fast pace ofdevelopments in the IT-industry, the number of software development packages to master for asoftware engineer is large. The functionality of test management and test automation tools is ample.It takes several weeks to master and tailor the tools and underlying concepts to a programmer’s


particular working situation. Not for every software engineer the ability to use the wholefunctionality of the test-tools can be expected. In this respect, the training course only elicitedawareness; no real settled hands-on experience.

Conclusions

• Tools are not the most important point in test automation. As for all automated systems successcritically depends upon a structured approach of the testing process and the translation intoworkable procedures.

• Making a good testplan is a considerable effort. The effort is comparable with making a technicaldesign. Especially adjusting the level of detail of the testcases requires expert skills andjudgement. It takes time for an organisation to gain and implement the desired expertise.

• As for software, test-scripts require maintenance. Test-teams have to be assigned a role inmaintenance as well.

• Test-tools offer a broad range of functionality. Careful selection of a tool most appropriate to theorganisation’s development process and subsequent selection of what components of a toolset touse needs careful consideration.

• Automated test-tools in a small organisation require a product champion/guru and a structuredprocess of testing.

• Involvement and awareness of software engineers during introduction take time. Skills have to bebuilt up. Separate development and test-teams have to be established in the project. With the EE-gamma of projects not all projects can benefit from the new procedures and tools.

• Implementation of tools in an organisation leads to technical problems and inter-platformidiosyncrasies as with the installation of other software tools. Compromises have to be regardedand workarounds have to be designed to establish network connectivity and allow flexibleconfiguration.

• Test design and making tests can be started from the requirements phase. The internal testingefforts are deemed to be underestimated by programmers.

• Software engineers are excited about test-tools. A steep learning curve for test-tools is nowpossible. Further projects have profited from the experiences and tool-functionality now. A time-series evaluation tool for evaluation and classification of measurements on various signalsdetermined from wind-turbines is a second candidate for usage. Structuring is more importantthan tool usage, but tools can ignite interest for software developers in test structuring. Business point of view

• The qualitative results suggest a short payback period even in small organisations for a certainrange of application types. Faster deliveries and, through early test structuring, a stronger andmore flexible position to the end-user and customer in an R&D organisation can be achieved.Structured testing procedures as introduced in our organisation add to total product quality.Extra investment is necessary in the beginning of projects; rewards can be expected near deliveryand in maintenance. In our case the number of error reports after delivery showed a dramaticdecrease.


• In a new project to be performed for a pharmaceutical firm the tools were used right from thebeginning of the implementation trajectory.

i VISTA: Visualisation Testing Automation, ESSI Project number 24513. Final Report April 7th, 1999. ECN,Petten, The Netherlands.

Rene (I.G.) Kamphius

René (I.G.) Kamphuis was project leader of the EU-ESSI VISTA-project,aimed at introducing structured testing and testing tools in a technicalscientific environment. He has a long experience in software developmentand managing software projects at ECN (the Netherlands EnergyResearch Foundation). He led the implementation of parts of CMM inthe software development group of ECN. Large projects, he was involvedin, included data-acquisition and control systems for wind-turbines,visualisation and monitoring software for energy generation plants anddecision networks for accident management. He holds a Ph.D. fromGroningen University in chemical physics. His current research interestsinclude software design in general and process optimisation of small andlarge scale energy generating systems by using small distributedinformation systems.

presentation paper bio return to main menu t4 return to ...€¦ · – winrunner: testscript...

Documents