delivering test automation success
TRANSCRIPT
-
8/14/2019 Delivering Test Automation Success
1/136
Managed TestAutomation
March 31 April 2, 2009
Hans Buwa lda , CTO, Log iGear Corpora tion
"Managing Mr. Playback . . ."
-
8/14/2019 Delivering Test Automation Success
2/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Introduction
Who is here? industries?
involvement in testing?"Mr. Playback"
wants to know...
-
8/14/2019 Delivering Test Automation Success
3/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Implement effective and maintainable automated tests
Managing testing and test automation
Main focus: concepts, techniques and experiences To a lesser extend: in-depth treatments, full coverage of possible issues
We will use the "Action Based Testing" method as aguideline today, as an example of an approach: Representative of keyword-based approaches Incorporates most of the key ideas and concepts
Home match for your teacher
Not everything will be equally interesting, or accessible, toeverybody
Objectives
-
8/14/2019 Delivering Test Automation Success
4/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Outline of this Tutorial
TESTAUTOMATION
TEST
DEVELOPMENT
ORGANIZATION&
MANAGEMENT
INTRODUCTIONAND
OVERVIEW
1
2 3
4
-
8/14/2019 Delivering Test Automation Success
5/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Introduction and Overview
-
8/14/2019 Delivering Test Automation Success
6/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
specification development
Testing Under Pressure
DEADLINE
testing
-
8/14/2019 Delivering Test Automation Success
7/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
specification development testing
Develop tests in time:
Test design Auditing, acceptance Preparations
Automation
Testing Under Pressure
DEADLINE
-
8/14/2019 Delivering Test Automation Success
8/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The Challenges for a Test Process
Testing should be effective
Testing should be efficient
Testing should be under control
Testing should be fun
-
8/14/2019 Delivering Test Automation Success
9/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Organize it Well
Test Development
Test Automation
Technology/Infrastructure
ProductionMarketing/
Sales
SystemDevelopment
End UserDepartments
Quality Assurance
Management
After Sales/Help Desk
Customers
Vendors
Government
Agencies
Publicity
EXTERNAL INTERNAL
-
8/14/2019 Delivering Test Automation Success
10/136
-
8/14/2019 Delivering Test Automation Success
11/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Why test?
People expect us to do
Somebody wants us to
Increases certainty and control Showing absence of problems
Finds faults, saving time, money, damage Showing presence of problems
-
8/14/2019 Delivering Test Automation Success
12/136
-
8/14/2019 Delivering Test Automation Success
13/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Why Automate Tests?
It is more fun
Can save time and money potentially improving time-to-market
Reduces involvement of valuable specialists "cans" application business and application knowledge
Consolidates a structured way of working when established as integral part of system development process
Speeds up development shorter turnaround between changes
more security
-
8/14/2019 Delivering Test Automation Success
14/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Why not Automate?
Can rule out the human elements promotes "mechanical" testing might not find "unexpected" problems
More sensitive to good practices pitfalls are plentiful
Creates more software to manage
Needs/uses technical expertise in the test team
Tends to dominate the testing process at the cost of good test development
-
8/14/2019 Delivering Test Automation Success
15/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Different Points of View
1. Does the system comply with the requirementsthe positive angle
2. Are there any problems we should know about
the negative angle
3. Will the system work in practicethe pragmatic angle
-
8/14/2019 Delivering Test Automation Success
16/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
What are Typical Problems
No clear direction what, when, how, who, ...
Focus on tools and technology
Test design is not well thought through main driver for success of failure
Automation lacks architecture, is not transparent and hardto manage
Testing is underestimated or avoided testing is difficult and expensive management is often in denial it looks unattractive to spend money on testing
Test automation is underestimated test automation can be very difficult "use the head first, before using the hands"
-
8/14/2019 Delivering Test Automation Success
17/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The 5% Challenge for Test Automation
No more than 5% of all test cases should be
executed manually
No more than 5% of all efforts around
testing should involve automating the tests
-
8/14/2019 Delivering Test Automation Success
18/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Approaches to Automate Test Execution
Record & Playback
Scripting
Keywords
see also: "Software Test Automation", Mark Fewster and Dorothy Graham
-
8/14/2019 Delivering Test Automation Success
19/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
1. Record and Playback
select window "Logon"
enter text "username", "administrator"
enter text "password", "testonly"
push button "Ok"select window "Main"
push button "New Customer"
expect window "Customer Information"
select field "First Name"type "Paul"
select field "Last Name"
type "Jones"
select field "Address"type "54321 Space Drive"
.
.
.
-
8/14/2019 Delivering Test Automation Success
20/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Can help well to generate early versions ofindividual steps in a larger automated script
Also useful to get to know a test tool, or the UI ofthe system under test
In its pure form maintainability of generated testscripts can become a major liability
Puts the testers in the automation role "loose a good tester, gain a poor programmer"
1. Record and Playback
-
8/14/2019 Delivering Test Automation Success
21/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
2. Scripting
Test CaseDesign
Test CaseAutomation
Test CaseExecution
TEST DESIGNER
AUTOMATION ENGINEER
MR. PLAYBACK
-
8/14/2019 Delivering Test Automation Success
22/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
2. Scripting
. . .Function EnterCustomer ( FirstName, LastName, Address )
Click("New Customer");ExpectWindow("New Customer");EnterField("First Name", FirstName);
EnterField("Last Name", LastName);EnterField("Address", Address);. . .Click("OK");
End Function
. . .
Function MainLogon();EnterCustomer("Mary", "Jones", "123 Palm Drive");EnterCustomer("Paul", "Franklin", "321 Regent Street");. . .LogOff();
End Function. . .
-
8/14/2019 Delivering Test Automation Success
23/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
2. Scripting
Structured process test cases are designed first and then automated automation is managed as a programming task
done by a specialized automation engineer
Engineering approach emphasis on structure and re-use similar to regular programming projects common tasks (like "logon") are factored out data can be made external to the programming
High quality, limited coverage approach is good (see above), but is essentially a waterfall approach efforts are substantial, burdening engineering capacity hard to keep manageable, in particular in large projects sensitive to changes in the behavior of the application under test
-
8/14/2019 Delivering Test Automation Success
24/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
3. Keywords, Action Based Testing
number name quantitynew product P-9009 Sledge Hammer 5
number quantity
add quantity P-9009 20
add quantity P-9009 3add quantity P-9009 6
number quantity
check uantit P-9009 34
add quantity
check quantity
new product
function action_AddQuantity. . . .
end
function action_CheckQuantity. . . .
end
function action_NewProduct. . . .
end
-
8/14/2019 Delivering Test Automation Success
25/136Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
3. Keywords, Action Based Testing
Describe test cases as a series of actions withkeywords and arguments, in a sheet like format
Externalize those descriptions out of the scripting
Let the testers (not the automation engineer)
define the keywords, as by-product of the testdesign
Organize the test cases into "test modules", eachwith a clear and differentiated scope
Focus the automation efforts on the actions, not
on the test cases
-
8/14/2019 Delivering Test Automation Success
26/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
3. Keywords, Action Based Testing
Test Development Plan
Test Cases
TestObjectives
Test Module 1
Test Cases
TestObjectives
Test Module 2
Test Cases
TestObjectives
Test Module N
Actions
. . .
ACTION AUTOMATION
Break down
Automate
-
8/14/2019 Delivering Test Automation Success
27/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Example of a Test Module, "high level"
test objectivetest objective
-
8/14/2019 Delivering Test Automation Success
28/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Example of a Test Module, "low level"
TEST MODULE Product Maintenance
VERSION 1.0
. . .
TEST CASE Test Case 03 Use an invalid number
test requirement add 003 A number must be divisable by 9
source control destinationclick main new new product
window control value
enter new product number P9001
source control
click new product process
text
check message Invalid product number.
test objective
-
8/14/2019 Delivering Test Automation Success
29/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Enter a user id that is greater than 10 characters,enter proper information for all other fields,and click on the "Continue" button
There should be an error message statingthat "User Id must be less than 10characters".
Enter a User Id with special character's), enterproper information for all other fieldsand click on the "Continue" button
An error message should be displayedindicating that "User Id cannot containsome special characters".
Enter the information, with a password of 4
characters and click on the "Continue" button
Check for an error message saying:
"Password must contain at least 5characters".
Comparing FormatsMost values are implicit. The tester has tofigure them out during execution.
classic format
user id message
check registration dialog aaaaabbbbbc User Id must be less than 10 characters
user id message
check registration dialog rsoudre User Id cannot contain some special characters
password message
check registration dialog test Password must contain at least 5 characters
keywords
Execution instructions arerepeated in multiple test cases
-
8/14/2019 Delivering Test Automation Success
30/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Common Terminology
"Keywords" usually base level, interacting with the user interface
related to control classes (like textboxes, lists and menu's)
examples: "click", "enter", "select"
"Business Templates" typically high level, making automation scripts re-usable
examples: "login", "enter transaction"
"Actions" can be at multiple levels (interface, business oriented, test logic, ...) defined as part of the test design (not per se to re-use existing
automation)
a method like Action Based Testing allows earlier defined actions to
be used to define higher level action words
-
8/14/2019 Delivering Test Automation Success
31/136
-
8/14/2019 Delivering Test Automation Success
32/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Risks of the Keyword Methods
Sometimes hailed as panacea, complications can beunderestimated often treated as a technical "trick"
The method needs understanding and experience to besuccessful pitfalls are many, and can have a negative effect on the outcome
Lack of structure can lead to lack in manageability in particular in the test design
Automation might take more time and costs than expected
Impact of changes might be higher than expected
Results can be disappointing and boring
-
8/14/2019 Delivering Test Automation Success
33/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Combining Approaches . . .
Use keywords for the automation-readydescription of test cases
Use scripting to set up structured automation forthe actions
Use record and playback to record actions
-
8/14/2019 Delivering Test Automation Success
34/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Some "Paradigm Shifts" . . .
From testing to: test development
From process oriented to: product oriented
From being part of a waterfall to: agile testdevelopment
From automation-centric to: test-centric
From selective automation to: integral automation
From requirement-based to: scenario-based
-
8/14/2019 Delivering Test Automation Success
35/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Test Development
TESTAUTOMATION
TESTDEVELOPMENT
ORGANIZATION& MANAGEMENT
-
8/14/2019 Delivering Test Automation Success
36/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Recent Software Error
Six months before Global Surveyor fellsilent, engineers sent up incorrectsoftware commands that disabled its
solar panels . . .
source: www.cnn.com/2007/TECH/space/04/13/mars.surveyor.ap/index.html
-
8/14/2019 Delivering Test Automation Success
37/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
...TO-3.51 The exit date must be after the entry date...
State your Objectives . . .
// Test Case: Billy Goodfellow// Test Objective: TO-3.51Function EnterBilly
Press "New Employee"
Enter Field "Name", "Goodfellow"Enter Field "First Name", "Billy"Enter Field "Entry Date", "2002-10-02"Enter Field "Exit Date", "2002-10-01". . .
Press "Process"Find Text "Exit date must be after entry date"if Found then
Result passedelse
Result failed, "No error message"
. . .End Function
-
8/14/2019 Delivering Test Automation Success
38/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
State your Objectives . . .
...TO-3.51 The exit date must be after the entry date...
test objective TO-3.51
name entry date exit date
enter employment Bill Goodfellow 2002-10-02 2002-10-01
check error message The exit date must be after the entry date.
-
8/14/2019 Delivering Test Automation Success
39/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Traceability of Results
Requirements
TestObjectives
Test
Cases
TestResults
-
8/14/2019 Delivering Test Automation Success
40/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Possible Project Life Cycle
TestDevelopment
Plan
Automation:- design- research
- development- testing- support
Focus on:- actions
- interface handling- technology- never on tests
Test ModuleDevelopment
Test Module
Acceptance
Plan of Approach
Execution andFollow Up
Management
Stake Holders
Business Users
IT Department
Development
TechSpecialists
intake
-
8/14/2019 Delivering Test Automation Success
41/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Examples of Time Lines
System under test becomes available in phases
Test Development Plan
Test Development
Automation
Execution and follow up
Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Test Development Plan
Test Development
Automation
Execution and follow up
Sep Oct Nov Dec Jan Feb Mar Apr
Jan Feb Mar
System under test becomes end of the cycle
G
-
8/14/2019 Delivering Test Automation Success
42/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The Three Holy Grails of Test Design
The right approach per module
The proper level of test specification
The right breakdown into test modules
G il 1 B kd Di id d C
-
8/14/2019 Delivering Test Automation Success
43/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 1: Breakdown, Divide and Conquer
Mission, Requirements, System Design, Use Cases,Priorities, Planning, Budget, Stakeholders, ...
Identify test modules priority and planning what, when, who, how
Development
Execution
Maintenance
Test Module 1
Analysis
Development
Execution
Maintenance
Test Module 2
Analysis
Development
Execution
Maintenance
Test Module N
Analysis
. . .
G il 1 E l b kd
-
8/14/2019 Delivering Test Automation Success
44/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 1: Example breakdown
UI oriented tests does function key F4 work does listbox xyz the right values is the tab order correct
Do individual functions work like transactions in a financial system
Alternate paths in use cases
like cancel a transaction
End-to-end tests
Simulating business processes
Tests with specific automation needs like multi station tests
Tests of non-UI functions
High ambition level tests (aggressive tests)
Q ti
-
8/14/2019 Delivering Test Automation Success
45/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Question
Can you give a set ofbreakdown criteria fortests?
???
-
8/14/2019 Delivering Test Automation Success
46/136
G il 1 Wh t i d b kd ?
-
8/14/2019 Delivering Test Automation Success
47/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 1: What is a good breakdown?
Well differentiated and clear in scope
Balanced in size and amount
Reflects the level of tests
Modules are stand-alone, not dependent on each
other
Fitting the priorities and planning of the project
E l T t D l t Pl
-
8/14/2019 Delivering Test Automation Success
48/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Example Test Development Plan
R i P ith St k H ld
-
8/14/2019 Delivering Test Automation Success
49/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Review Process with Stake Holders
Test Team sends draft
Module to Stake Holder
Stake Holder reviews:- coverage- correctness
Stake Holder returnsnotes:- additions- corrections
Test Team receives andprocesses notes
changes needed? Stake Holder returnsnotice of approval
Test Team marks theModule as "Final"
END
no
yes
START
G il 2 A h T t M d l
-
8/14/2019 Delivering Test Automation Success
50/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 2: Approach per Test Module
Organize the test design process around the test modules
Plan the test module: when to develop: is enough specification available
when to execute: make sure the functionality at action level is well-testedand working already
Process: analysis of requirements
formulation test objectives creating test cases
Identify stakeholders and their involvement: users, subject matter experts
developers auditors
Choose testing techniques if applicable: boundary analysis, decision tables, transition diagrams, soap opera testing,
...
G il 2 E th b ll S
-
8/14/2019 Delivering Test Automation Success
51/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 2: Eye on the ball, Scope
Always know the scope of the test module
The scope should be unambiguous
The scope determines many things:
what the test objectives are which test cases to expect
what level of actions to use
what the checks are about and which events should generate a
warning or error (if a lower functionality is wrong)
Grail 2 E amples of Testing Techniq es
-
8/14/2019 Delivering Test Automation Success
52/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 2: Examples of Testing Techniques
Equivalence class partitioning any age between 18 and 65
Boundary condition analysis try 17, 18, 19 and 64, 65, 66
Error guessing try Ccile Schfer and test the sorting of the name list
Exploratory
"Exploratory testing is simultaneous learning, test design, and test execution",James Bach, www.satisfice.com
Model-based see Harry Robinson: www.model-based-testing.org see Better Software Magazine (STQE Magazine), March 2003
Error seeding deliberately injecting faults in a test version of the system, to see if the tests catch
them handle with care, don't let the bugs get into the production version
Grail 2: More Testing Techniques
-
8/14/2019 Delivering Test Automation Success
53/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 2: More Testing Techniques
State transitions
Decision tables
Soap Opera Testing
Recommended:
"Why Software Fails", James Whittaker and Alan Jorgensen, and
"Testing Computer Software", Cem Kaner, Jack Falk and Hung Nguyen
State Transition Diagram
-
8/14/2019 Delivering Test Automation Success
54/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
State Transition Diagram
first check
second check
self destructdoors open
initial
signal
correct
correct
incorrect
incorrect
signaltampering
time-out
Question: what test cases would you create to cover these situations?
doors closed
alert state
State Transition Table
-
8/14/2019 Delivering Test Automation Success
55/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
State Transition Table
NR STATE EVENT OUTPUT NEXT
1 doors closed signal beep first check2 doors closed tampering hi-lo alert state
3 first check correct code green doors open
4 first check incorrect code hi-lo second check
5 doors open close signal beep doors closed6 second check correct code green doors open
7 second check incorrect code hi tone self destruct
8 self destruct anything boom oblivion
9 alert state time out hi tone self destruct
10 alert state signal beep first check
Car Safety System, Model 007, Luxury Version*
*certain insurance restrictions might apply, please check with your agent
Example of a Decision Table
-
8/14/2019 Delivering Test Automation Success
56/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Example of a Decision Table
dialog data correct n y y y y
large order n n y y
online customer n y n y
show error and resume x
give discount 20% x x
give discount 3% x
notify marketing x x
send to priority execution x
display price info x x x x
TEST CASE 1
TEST CASE 2
TEST CASE 3
TEST CASE 4
TEST CASE 5
-
8/14/2019 Delivering Test Automation Success
57/136
No Y2K Problems in Auckland Airport??
-
8/14/2019 Delivering Test Automation Success
58/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
No Y2K Problems in Auckland Airport??
-
8/14/2019 Delivering Test Automation Success
59/136
-
8/14/2019 Delivering Test Automation Success
60/136
The Mechanical Approach for Test Development
-
8/14/2019 Delivering Test Automation Success
61/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The Mechanical Approach for Test Development
Start with a (preferably long) list of requirements
Make a test case for every requirement
Use a standardized test technique to translate the
requirements into test cases
Hire (many) people to perform the tests manually
.
-
8/14/2019 Delivering Test Automation Success
62/136
Soap Operas
-
8/14/2019 Delivering Test Automation Success
63/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Soap Operas
Ashley hears about Jack's deposit when he thought he had
to go. Victoria lectures her father about what's wrong with him
and Nikki but Victor advises her that it's none of her businessOlivia learns Dru has no regrets about leaving and takes great
satisfaction in having Lily as her companion. Dru then asks Olivia
why she is raking Malcolm over the coals. Stopping by Gina's,Nikki spots Brad and sits with him, admitting she doesn't want to
be alone tonight. Victor stops by Mack's party at the Crimson
Lights. Ashley takes a home pregnancy test. Worried about Billy,Raul makes call and J.T. claims he doesn't know where Billy is.
Raul rushes over and finds Billy out cold in the snow. Raul worries
when he can't find a pulse. . . .
Some Other Examples
-
8/14/2019 Delivering Test Automation Success
64/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Some Other Examples
Pension Fund
World Wide Transaction System for an International Bank
William starts as a metal worker for Industrial Enthropy
Incorporated in 1955. During his career he becomes ill, works
part time, marries, divorces, marries again, gets 3 children, one
of which dies, then his wife dies and he marries again and gets 2
more children.
A fish trade company in Japan makes a payment to a vendor on
Iceland. It should have been a payment in Icelandic Kronur, but
it was done in Yen instead. The error is discovered after 9 days
and the payment is revised and corrected, however, the interest
calculation (value dating)
Properties of Soap Operas
-
8/14/2019 Delivering Test Automation Success
65/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Properties of Soap Operas
Using a recurring theme, with episodes
About real life
But condensed
And more extreme
Soap Operas for Testing
-
8/14/2019 Delivering Test Automation Success
66/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Soap Operas for Testing
Define a scope of the test to develop
Identify with the business environment
Identify which elements would make things difficult
Draft test cases (typical some dozen lines)
Write them down in modules
Soap Count the Goodies
-
8/14/2019 Delivering Test Automation Success
67/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Soap Count the Goodies
In this episode, the Goodies have won $750,000 in the
lottery. They decide to buy that big house on the corner of
the street that theyve always dreamed about. Father, BingGoody, goes to the banking office to make the necessary
arrangements. Of course, the bankruptcy that Bing had two
years ago complicates the approval, but Jim, the guy at the
bank, is very helpful and even sells Bing on the idea of a
beautiful vacation property in Mexico. Bing has a brilliantidea. Why not ask neighbor Jones if he is interested in co-
owning the vacation property? Like everything else with Mr.
Jones, his credit is perfect.
The Story Continues
-
8/14/2019 Delivering Test Automation Success
68/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The Story Continues
One week later: After a small party with some new
friends, only about $720,000 of the prize money is left,
so the mortgage arrangements need to be changed. Jim,Mr. Jones, and Bing decide to meet the next morning to
make the necessary adjustments. Since it was a great
party, they are not at their best when they meet, so many
mistakes are made. They correct some immediately,
others they miss and will need to fix later. They find thatan additional second mortgage on Mr. Joness home is
needed in order to still qualify for the vacation home.
What Will Happen Next?
-
8/14/2019 Delivering Test Automation Success
69/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
What Will Happen Next?
See it all . . .
. . . in the next test module ! !
What This Soap Can Test
-
8/14/2019 Delivering Test Automation Success
70/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
What This Soap Can Test
Entering a customer with a big family
Mortgage arrangements including a first and a second house
Property abroad
Both a primary residence and a vacation property
Weighing of the income of a second owner
Co-ownership on the second house only Establishing a second mortgage to finance a vacation property
Bankruptcy two years ago
Modification of the down payment
Correcting errors upon entry
Corrections after the application has been processed
Killer Soaps let me wash
-
8/14/2019 Delivering Test Automation Success
71/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Killer Soaps
Aggressive tests, aimed at finding hidden
problems
Run when everything else has passed
Ask experienced users and subject matter experts
for ideas
let me washthis thing...
Test Analysis / Test Creation
-
8/14/2019 Delivering Test Automation Success
72/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Test Analysis / Test Creation
Test Analysis
- what do we want
- what do we need
Test Creation
- confrontation with reality
- put it to the testseparate
relate
and
TESTOBJECTIVES
TESTCASES
Questions to Answer with a Test Collection
-
8/14/2019 Delivering Test Automation Success
73/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Questions to Answer with a Test Collection
mechanical soapssoaps +
objectives
1Does the system complywith the requirements *** * ***
2Are their any problems(defects and/or failures)we should know about
* *** ***
3Will the system work inpractice ** ** ***
Grail 3: Specification Level, choosing actions
-
8/14/2019 Delivering Test Automation Success
74/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Grail 3: Specification Level, choosing actions
Scope of the test determines the specification level
As high level as appropriate, as little arguments as
possible Use default values for non-relevant arguments
Clear names (usually verb + object works well)
Manage the Actions
Document the Actions
By-product of the test design
-
8/14/2019 Delivering Test Automation Success
75/136
Designing Actions/Specification Level
-
8/14/2019 Delivering Test Automation Success
76/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Designing Actions/Specification Level
Scope of the test determines the specification level
As high level as appropriate, as little arguments as
possible Use default values for non-relevant arguments
Clear names (usually verb + object works well)
Manage the Actions
register them, keep them organized
Document the Actions
intention, arguments, assumptions, conditions, implementation notes, ...
Levels of Actions
-
8/14/2019 Delivering Test Automation Success
77/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Levels of Actions
High level actions to capture functions in abusiness process hide unneeded details
always hunt down details usually implemented with lower level actions
Low level actions to access the system interface either built-in in a action based tool like TestArchitect
or tailor created as part of an interpretation harness, usingscripting and/or record&playback
Example of Going High Level
-
8/14/2019 Delivering Test Automation Success
78/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
a p e o Go g g e e
Example of Going High Level
-
8/14/2019 Delivering Test Automation Success
79/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
p g g
section Get cost valueskey
key navigate F7
key navigate 5
window
wait for controls loaded view account infotext
check breadcrumb # view account info breadcrumb
window field
check focus view account info cancel
start cost cost in cost out net cost
capture cost values >> starting cost >> total cost in >> total cost out >> total net cost
Example of Going High Level
-
8/14/2019 Delivering Test Automation Success
80/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
p g g
Example of Going High Level
-
8/14/2019 Delivering Test Automation Success
81/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
p g g
section Capture initial number
key
key navigate F7
key navigate 3
page tab
locate page tab Scan Criteria
window
wait for controls loaded search
text
check breadcrumb # search breadcrumb
window control value
select search scan direction Backward
window control value
enter value search business date match # bus date
source control
click search go
window
wait for controls loaded search results
store ascapture sequence num >> seq num
Example of Going High Level
-
8/14/2019 Delivering Test Automation Success
82/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
p g g
-
8/14/2019 Delivering Test Automation Success
83/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Test Automation
TESTAUTOMATION
TEST
DEVELOPMENT
ORGANIZATION& MANAGEMENT
Components of Automation
-
8/14/2019 Delivering Test Automation Success
84/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
p
Tests and test data what needs to happen to prove or disprove the functionality under
test best to keep this separate from the technical part of the automation
Programming/scripting technical specification of the individual steps of the automation
Interface information what is where on screens, web pages, etc shield volatile physical details
UI and non-UI automation functions the actual (usually tool provided) commands to work with the
interfaces of the system under test
Automation: Implementing Actions
-
8/14/2019 Delivering Test Automation Success
85/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
p g
In a keyword driven approach the automationfocuses on the actions: as opposed to programming test cases
in general this will lead to fewer and shorter scripts etc in theautomation
In the Action Based Testing method actions are
implemented in three ways: built-in in the tool, like click, enter, check etc
defined in an action definition, using other (lower level) actions similar to functions in a programming language
programmed in a programming or scripting language
-
8/14/2019 Delivering Test Automation Success
86/136
Automation Overview
-
8/14/2019 Delivering Test Automation Success
87/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Actions
Test Modules
ActionDefinitions
Low Level
Actions
InterfaceDefinitions
(UI, SQL, OLE, ...)
Test Architect
Built-in
Additional
Scripting (C#, VB, ...)
90% +
split up in lower level actions provided defaults values for
arguments
shield (volatile) technicaldetails of an interface
provide "logical" names forcontrols and their classes
test objectives test cases
Doing the Count Down . . .
-
8/14/2019 Delivering Test Automation Success
88/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
g
6000
250
Test Cases Actions ProgrammedActions
3000
22
200
20
In a good application of the keywords approach a large increase in test cases (like doubling the amount)should result in a modest increase in actions, and a minor increase, if any, in programmed actions.
Programming
-
8/14/2019 Delivering Test Automation Success
89/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
g g
Often called "scripting", but much like regular programming
Traditional playback tools mostly have built-in scripting
languages often Basic look-a-likes, WinRunner's TSL is a C look-a-like
More and more tools support existing programminglanguages like Visual Basic
Regular languages can in some cases do the job too in particular for non-gui's for example scripting languages like Python and Ruby but also Java, C++, VB, C#, etc
Functionality of an Action Interpreter
-
8/14/2019 Delivering Test Automation Success
90/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Function Interpret
While not end of file doRead next line
Split the line into arguments
Look up the action in the "action list"
Execute the action
Print the results of this line in the reportEnd of the while loop
End
ReportInterpreter
Atomic and Generic
-
8/14/2019 Delivering Test Automation Success
91/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
In ABT key recommendations for programmedactions are to make them: atomic, only one small task per action
generic, maximum re-use
Avoid programming test cases or high level
actions split them up in well-defined low level actions
Automation: example test lines
-
8/14/2019 Delivering Test Automation Success
92/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
System Under Test Test Lines
Automation, step 1: make "Action Functions"
-
8/14/2019 Delivering Test Automation Success
93/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
identify theUI element
perform theoperation
identify theUI element
perform thecheck
Automation, step 2: "Divert" to the action functions
-
8/14/2019 Delivering Test Automation Success
94/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
call this function for each line in the test module
Interface Information and Interface Handling
-
8/14/2019 Delivering Test Automation Success
95/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Handling interfaces is the essential challenge automation works "outside in"
interfaces are not necessarily designed with test tools in mind
Common interfaces GUI
Web
API's, with methods and/or functions message protocols, on basis of TCP/IP or SS7
embedded software
databases with SQL
files (like in batch runs)
Identifying Elements of an Interface
-
8/14/2019 Delivering Test Automation Success
96/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
All elements have properties can be used for identification and verification
Some elements can be easily identified like buttons, checkboxes, frames
Other elements have only volatile and/orambiguous properties like text boxes, list boxes, combo boxes
can differ across runs, or across versions/builds
Text labels tend to be "invisible" only bitmap
Interface Definitions
-
8/14/2019 Delivering Test Automation Success
97/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Quickly define how interfaces must be accessed
Commonly used for GUI's, however applicable toother interfaces as well
Typically kept in a table or spreadsheet
separate from the test modules
Components of Interface Definitions
-
8/14/2019 Delivering Test Automation Success
98/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Interface entity window
web page
text screen 3270 page (typically based on the HLLAPI standard)
class member
api function
message database table
Interface element
button text box
link
field
argument ....
Example Interface Definition
-
8/14/2019 Delivering Test Automation Success
99/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
window control valueenter add dialog number 1234567
enter add dialog name saw
enter add dialog description Cut through things
enter add dialog quantity 500
source controlclick add dialog ok
INTERFACE ENTITY add dialog
interface entity setting title Add A New Item
ta name ta c ass a e
interface element number textbox Item Number
interface element name textbox Name
interface element description textbox Descriptioninterface element quantity textbox Quantity On Hand
ta name ta class caption
interface element ok button Process
interface element cancel button Cancel
Some Tips to Get Stable Automation
-
8/14/2019 Delivering Test Automation Success
100/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Make the system under test automation-friendly developers are not always inclined to do that, but it pays off ask to add specific property values to the GUI interface controls for automated
identification like accessible name in .Net and Java, or id in Web controls
Use "active" timing wait based on conditions in the system under test, avoid fixed sleeps
Test your automation develop separate tests for the actions themselves
Use automation to identify differences between versions of the systemunder test
test in a loop if all control can still be found
Keep an eye on the test design unsuccessful automation virtually always can be traced back to poor test design
"Accessible Name" property
-
8/14/2019 Delivering Test Automation Success
101/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Testing the automation . . .
-
8/14/2019 Delivering Test Automation Success
102/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The automation is a service provider to the testers: provide working actions
provide accurate up to date interface definitions
In each project consider having the following
folders:
Automation Tests
Action Tests
Interface Tests
Cross Platform Testing
-
8/14/2019 Delivering Test Automation Success
103/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
No longer absolute dominance of Windows and IE
Diversity in: operating systems
Windows (XP, Vista, Windows 7), Linux (variety of distributions), Mac widget toolkits
.Net, InfraGistics, QT, Java, STK+, Flex, ...
browsers Internet Explorer, FireFox, Safari, Chrome,... within browsers: Ajax, JavaScript, Java,
VB Script, Flash, ...
non-traditional platforms Windows Mobile, Android, Symbian, RIM, ... Netbooks!
non-client platforms
Cloud computing, SOA, ...
This trend will continue: even if Windows and/or IE keep dominating the alternatives can't be ignored much
longer, diversity is the name of the game
Cross Platform Automation
-
8/14/2019 Delivering Test Automation Success
104/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The key ingredient for success is design test design is even more the decisive factor
automation needs to be layered and flexible
tooling needs to plug in into the overall model
UI LEVEL TESTS
HIGH LEVEL ACTIONS
LOW LEVEL ACTIONS
GENERIC CONTROLS AND OPERATIONS
PLATFORM 1 PLATFORM n. . .
HIGH LEVEL TESTS API LEVEL TESTS
Non-UI Testing
-
8/14/2019 Delivering Test Automation Success
105/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Examples application programming interfaces (APIs)
embedded software
protocols files, batches
databases
command line interfaces (CLIs)
multi-media
Impact is mainly on the automation test design should in most cases be transparent towards the
specific interfaces
Multiple System Access
-
8/14/2019 Delivering Test Automation Success
106/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
System (part)Under Test
Automation Scheme
APIaccess
protocolaccess
UIaccess
databaseaccess
Test Modules, driving eitherone or multiple interfaces
Device Testing
-
8/14/2019 Delivering Test Automation Success
107/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Software Under Test
MiniAgent
ABTAutomation
InterfaceInfo
Testing HostDevice
Testing a Service Architecture or Protocol
-
8/14/2019 Delivering Test Automation Success
108/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Automation
InterfaceDefinitions
stash(optional)
CLOUD,SOA, ...
SEND
RECEIVE
Testing the Cloud, testing the SOA
-
8/14/2019 Delivering Test Automation Success
109/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
A lot of business as usual for testing
Test everything always puts an extra need on high coverage automation
not necessarily time to "maintain" not-working automation
Where is what, and is it safe is data where you think it is, and what you think it is are protocols and safety measures working
Various perspectives unit level, service level, client level, user level testing functions, testing interfaces, testing protocols
Expect the unexpected, "jungle testing"
We still have to learn
"Jungle Testing"
-
8/14/2019 Delivering Test Automation Success
110/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Expect the unexpected prone to unexpected requests prone to unexpected situations in data prone to attacks
how does a generic design respond to a specific unexpected event?
Difference in thinking coding bug: implementation is different from what was intended/specified jungle bug: system does not respond well to an unexpected situation
To prepare study the matter (common hack attacks, ...) make a risk analysis
make time to discuss it (analysis, brainstorm) involve people who can know use "exploratory testing" (see James Bach's work on this) use an agile approach for test development applies equally to testing, requirements, design
testing should focus on the specific attacks
From the Headlines
-
8/14/2019 Delivering Test Automation Success
111/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Bugs can be data driven
They can happen in real time the earlier you see them the better
There will be material and immaterial damage Your tests won't be perfect
sometimes unexpected simply means unexpected
learn, generalize, prevent the bad bug, and its family, for the future 20/20 hindsight is an acceptable technique...
Handling Batch Processes
-
8/14/2019 Delivering Test Automation Success
112/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Transfer Jones Williams 100Check Balance Jones -100Check Balance Williams 100 Jones Williams 100
Batch Run
Jones -100
Williams 100
Test Module
"Delayed Checks"
Batch Input File
Batch Output File
Check Balance Jones -100Check Balance Williams 100
1
1
2
2
3
Multi-media: The "Play List" Approach
-
8/14/2019 Delivering Test Automation Success
113/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The testers focus on a play list items are played/shown while the machine tells what it is the tester
should see/hear
the tester acknowledges/falsifies the system stores checksums of those items that were ok
efficient and relatively easy to store and handle
Checksums can be kept for files, bitmaps etc
Next time only the changed items go on the playlist again when the checksum is different from the previous run/version
in most systems this will only be a limited amount of items
Performance Testing
-
8/14/2019 Delivering Test Automation Success
114/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
The topic is complex, many components andphenomena
To formulate tests in actions can be surprisinglystraightforward
Often performance testing isn't testing, but moreclose to research
The three controls you can/should address:hardware (equipment, infrastructure, etc)software (programs, database models, settings, etc)demands (1 second can be 10 times more than 2 seconds)
See also: "Web Load Test Planning", Alberto Savoia, StickyMinds.com
Data Driven Testing
-
8/14/2019 Delivering Test Automation Success
115/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Separate test logic from the data
Have the data, including expected outcomes, in a
separate file or table can also be generated during test execution, for example extracted
from a database
Use placeholders in the test case
Works for both scripted and keyword drivenapproaches
Data Driven Testing, Example
-
8/14/2019 Delivering Test Automation Success
116/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
-
8/14/2019 Delivering Test Automation Success
117/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Topics Test Management
TESTAUTOMATION
TESTDEVELOPMEN
T
ORGANIZATION& MANAGEMENT
Organization
-
8/14/2019 Delivering Test Automation Success
118/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Much of the success is also gained or lost in how you organize theprocess who does test design who does automation
what to outsource, what to keep in-house
Write a plan of approach for the test development and automation scope, assumptions, risks, planning methods, best practices
tools, technologies, architecture stake holders, including roles and processes for input and approvals team . . .
Assemble the right team testers, lead testers automation engineer(s) managers, ambassadors, ...
Test design is a skill . . .Automation is a skill . . .
Management is a skill . . .
-
8/14/2019 Delivering Test Automation Success
119/136
Typical Time Allocation
-
8/14/2019 Delivering Test Automation Success
120/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
TEST DEVELOPMENT
AUTOMATION
time
efforts
Embedding QA
-
8/14/2019 Delivering Test Automation Success
121/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Clear division of tasks and responsibilities: testers (who will now become test developers)
users, power users, subject-matter specialists
test automation engineers (do they work for you?) developers
other stakeholders
Decision making and communication who's call is it to pass a system for release what happens with bugs
who does which testing
what whistles to blow/not blow, when
Flexibility testing is a "2nd order dependency" activity
firmness on the integrity of provided information (but be practical)
Question
-
8/14/2019 Delivering Test Automation Success
122/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
What are good skills for atester?
???
Skills for the Team
-
8/14/2019 Delivering Test Automation Success
123/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Typical skills needed testing (real testers)
subject matter
beware of "guru's", don't allow singleindividuals to monopolize key knowledge
technology (often underestimated and then weak spot)
engineering practices (like configuration management)
soft skills (communication, managerial, diplomacy)
Acquiring the skills
training, reading, practicing, coaching interest, curiosity
hiring, borrowing
overall picture counts, not everyone needs to know everything
The Tester as a Detective
-
8/14/2019 Delivering Test Automation Success
124/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Looking for the bad guys (bugs)
Specific personality traits: authority, respected
inconspicuous, not threatening
able to look past excuses and stories
mindful of details
Mr. Authority
Mr. Inconspicuous Mr. No Nonsense Mr. Details
Tester Needs Hunting Instincts
-
8/14/2019 Delivering Test Automation Success
125/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Off-shoring
-
8/14/2019 Delivering Test Automation Success
126/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Can a keyword drivenapproach be done off-
shore? If yes, what would it take?
If no, why not?
???
Off-shoring Keyword Automation
-
8/14/2019 Delivering Test Automation Success
127/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Potential can get a lot more tests at a lower cost
potential of double cost benefit: 1. off-shore, 2. automation risk of double backlash: 1. bad design, 2. bad automation
flexibility, particularly for "bursty" projects ramp up quickly, scale back quickly
Pitfalls little "ownership" with the system under test and the subject matter under
test a keyword method is a subtle approach (see the rest of this tutorial...). Thismakes outsourcing challenging: mind the "Automation Cliff" . . .
very hard to go back when things turn out disappointing
What to do keep local ownership provide clear direction, constant attention and coaching supervise, supervise, supervise ask to create example test modules first and review them carefully
Organization and Project Level
-
8/14/2019 Delivering Test Automation Success
128/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
ACTIVITIES AT PROJECT LEVEL
ACTIVITIES AT ORGANISATION LEVEL
ACTIVITIES AT PROJECT LEVEL
ACTIVITIES AT PROJECT LEVEL
Beyond the projects: Test Governance
-
8/14/2019 Delivering Test Automation Success
129/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
TEST COMPETENCE MANAGEMENT(improvement focus)
initiation, know how, support, ...
experiences, products, people, ...
pilot projects
training
coaching
resource mediation
test quality assurance
test environments
testware management
vendor contacts
TEST BUSINESS POLICIES AND DIRECTIONS(significance, position, organization)
managem
ent&contr
ol
TestMaintenance
TestExecution
TestDevelopment
TestDesign
Test Strategy& Planning
TEST PROJECT MANAGEMENT(solution focus)
Test Policy
-
8/14/2019 Delivering Test Automation Success
130/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Corporate point of view on testing (or software quality)
Not a common topic in many organisations
However can be useful to have
Policy Model
-
8/14/2019 Delivering Test Automation Success
131/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Business Environment
Organisation Environment(Including Policies)
IT OrganisationAnd Policies
Organisation andPolicies of Testing
How to Get It
-
8/14/2019 Delivering Test Automation Success
132/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Think about it yourself
Identify the stake holders
Use existing policy documents (if any)
Discuss it: Within the testing department Externally (IT, business, stakeholders)
If possible start low profile and end high profile
Policies should be recognized and understood
Consider the use of an incremental document
Typical Issues (1)
-
8/14/2019 Delivering Test Automation Success
133/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
What is the significance of testing and what may itcost
Connection to critical success factors of thecompany
Do we have problems
When should testing be done in the life cycle
Who should be involved (test development,assessment, reporting)
Who is responsible
Typical Issues (2)
-
8/14/2019 Delivering Test Automation Success
134/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
How to deal with testing expertise
Centralized / decentralized
Methods and tools
Migration to a desired situation
Automation or not
Examples of policy statements
-
8/14/2019 Delivering Test Automation Success
135/136
Managed Test Automation | 2009 by LogiGear Corporation. All rights reserved. Made available under the EPL v1.0
Testing and test automation are regarded as businesscritical
Testware (developed tests and automation) is to be treatedas business asset
User friendliness is a major priority in testing
Test Automation is used as much as possible
QA is responsible for timely and accurate information on
system quality, independent of other stakeholders
System Development is responsible for system quality
Releasing a system is a business responsibility
-
8/14/2019 Delivering Test Automation Success
136/136