experience with keyword automation

8/14/2019 Experience With Keyword Automation

1/23


2/23

A Vision on Automation

Have a good method integrated method for testing and automation testers in the driver seat

maximum flexibility, best technology minimal technical efforts maximum maintainability

Have a good tool or toolset that supports that method specifically designed to provide a framework for testing and automation does not stand in the way of the engineers encourages testers to follow the method supports maximum manageability of

products

process

Succeed in the three "success factors for automation" test design automation solution

organization


3/23

Keywords, Action Based Testing

number name quantitynew product P-9009 Sledge Hammer 5

number quantity

add quantity P-9009 20

add quantity P-9009 3add quantity P-9009 6

number quantity

check uantit P-9009 34

add quantity

check quantity

new product

function action_AddQuantity. . . .

end

function action_CheckQuantity. . . .

end

function action_NewProduct. . . .

end


4/23

Keywords, Action Based Testing

Test Development Plan

Test Cases

TestObjectives

Test Module 1

Test Cases

TestObjectives

Test Module 2

Test Cases

TestObjectives

Test Module N

Actions

. . .

ACTION AUTOMATION

Break down

Automate


5/23

History of Action Based Testing

Started in 1994 in The Netherlands, as "actionwords"

Presented to an international audience in 1996

Used in hundreds of projects in over 10 countries

Many industries, platforms, project sizes and

project circumstances

Successful, but there are pitfalls


6/23

Example of a Test Module, "high level"

test objectivetest objective


7/23

Example of a Test Module, "low level"

TEST MODULE Product Maintenance

VERSION 1.0

. . .

TEST CASE Test Case 03 Use an invalid number

test requirement add 003 A number must be divisable by 9

source control destination

click main new new product

window control value

enter new product number P9001

source control

click new product process

text

check message Invalid product number.

test objective


8/23

The Three Holy Grails of Test Design

The right approach per module

The proper level of test specification

The right breakdown into test modules


9/23

Case: Stock Exchange

Transition from floor based to screen based trade

Created on basis of an existing standard package result: very little specifications

Consisting of four major, different, systems that need to work in real-time

Failures and bugs are not an option: core of the financial system of the country

100K USD per second parties not necessarily following rules

In-depth knowledge limited to four people nicknamed "The Four Daltons", after characters

in a European cartoon about the wild west

none of the four Daltons was in testing, testing was in a vacuum

Three months to go... test development (and scripted automation) had failed test department was in conflict with development department internal and external auditors had raised the alarm

Crown Prince would come to put the new system into use


10/23

Case: Stock Exchange

Test set: make it comprehensive make it in-depth and aggressive make it easy to assess and approve

Organization: get the right people involved (testing, automation, etc) use scarce resources efficiently (in particular the four Daltons) work with stake holders to let the process be transparent

Technical: use of the keyword method ("action words") use "test objectives" so auditors can see quickly what you're testing use great test design, don't mix apples and oranges

"Sign off lubrication": auditors signed off on the tests, not the test results "the test is complete", not "the system works well"

Results: deadline was met one day before the automated tests were the only ones used for acceptance no functional errors found afterwards


11/23

Case: A Financial Exchange

Fixed-time fixed-price development

Large scale test, wide user involvement

First large scale project with actions

Very high pressure on test execution too early system delivery to meet the fixed-time deadline political dynamics underestimated, in particular by the test manager testing team not necessarily independent from the vendor (us...) most key functional knowledge invested in one test designer (and he left

just before execution started)

Good numbers complete in 6 weeks, as opposed to 2 months planned

still overshot total timeline with 3 weeks, guess who got the blame... found several hundred bugs

missed 12 bugs, only one of them should have been found


12/23

Case: International Financial Project

One of the largest projects with action words

Over 10000 windows, meant for use in 85countries

Long development cycle (4 years, 400 pp)

Maintenance very hard

Testing main bottle-neck

Much investment in automation techniques were

needed to become successful


13/23

Case: A Financial Project

Large project, with many consultants embraced the approach

however, they were from our competitor

One of the first with action words

However, were confident that they could do thetest development without help

Result: too many hard to implement action words almost the end of the action words method

one memo saved the day


14/23

Case: A Pension Fund

Relatively small organization

In one year: Totally new system 2k1 compliance

Complex business logic, knowledge with users Around 20% of entire organization involved

Test automation only option

Significant part of solution: soap operas

Extra problem: many consultants inexperienced: project much larger than anticipated


15/23

Case: American Bank

Project for a new teller system

Large, state of the art

Many system releases, many adjustments

Need for very high level of automation

Over 1 million test lines, in over 600 test modules

Initially little attention paid to "holy grails" UI and functional tests in the same modules virtually un-maintainable, came close to killing the project

Emergency re-organization of the test modules

after system changes the tests would run again within a day


16/23

Example of Going High Level

section Get cost values

key

key navigate F7

key navigate 5

window

wait for controls loaded view account info

text

check breadcrumb # view account info breadcrumb

window field

check focus view account info cancel

start cost cost in cost out net cost

capture cost values >> starting cost >> total cost in >> total cost out >> total net cost


17/23



18/23


19/23



20/23

Case: Oil Exploration

Wide variety of sophisticated systems, in particularfor oil exploration

Complex 3D graphics, which can be manipulatedin many ways virtually impossible to test with functional cases (the bottleneck is

not technology, but specific enough test design)

Solution: "4D Testing" capture the actual graphical command stream

play back to a human tester for approval, using "questions" createdas part of the test case design

on repeated tests: if the same output is seen again with the same

question, don't ask the human tester anymore


21/23

Case: The French Director

Mid size company

Struggling under high pressure

Testing of their main product, standard financialsoftware

Control and priority main issue

Unfamiliar business culture

Main instrument: module break down


22/23

Strategic Issues for Testing

Time to market

Quality to market

Control

Motivation (testing must be fun)

Long term stability and re-usability


23/23

Some Lessons I Learnt

Keyword are not magic but can help well for large scale automation I'm not aware of other alternatives Important to do it right, if you can't, don't do it at all

Test design is the most important success factor have well-differentiated tests with each a well-defined scope stay in the scope (for example: avoid checks "on the fly")

Involvement of users is essential, but make sure you learn yourself about the business as well don't allow monopolies on business or technical knowledge

Cooperation with developers is equally important they know where to find good bugs they can tell you the truth, early

they can help in automation

Mind the politics testing is virtually always political generally the tester gets the bullet

specific for automation and keywords: manage expectations understand and handle the politics (see the slides of my "Testing in the Cold" class

for some ideas)

experience with keyword automation

Documents