the 5 percent rules of test automation
TRANSCRIPT
-
8/14/2019 The 5 Percent Rules of Test Automation
1/38
The 5% Cha llengesof Test AutomationHans Buwa lda , CTO, Log iGear Corpora tion
March 31 April 2, 2009
-
8/14/2019 The 5 Percent Rules of Test Automation
2/38
2009 LogiGear Corporation. All Rights Reserved
Scope of this Workshop
Raising the bar for automation
the 5% challenges and their rationale
Show techniques, concepts, ideas toachieve the higher barhow could you achieve the challenges
How feasible is all of this in practice . . .
-
8/14/2019 The 5 Percent Rules of Test Automation
3/38
2009 LogiGear Corporation. All Rights Reserved
specification development
Testing Under Pressure
DEADLINE
testing
-
8/14/2019 The 5 Percent Rules of Test Automation
4/38
2009 LogiGear Corporation. All Rights Reserved
specification development testing
Develop tests in time:
Test design Auditing, acceptance Preparations
Automation
Testing Under Pressure
DEADLINE
-
8/14/2019 The 5 Percent Rules of Test Automation
5/38
2009 LogiGear Corporation. All Rights Reserved
The 5% Challenges
No more than 5% of all test cases shouldbe executed manually
No more than 5% of all efforts around
testing should involve automating the tests
-
8/14/2019 The 5 Percent Rules of Test Automation
6/38 2009 LogiGear Corporation. All Rights Reserved
Why a High Automation Degree ??
The best way to prepare for efficiency in the crunch zone
good manual test cases can help too, but marginally
Buy time to do more "exploratory testing" valuable addition to testing with test cases, since it is likely to find other
and unexpected bugs
Credible pay-off for the cost of introduction of automation
initial costs are: tooling, learning curve, adaptation of existing tests Automation is better positioned to identify bonus bugs
on average 15% of fixes cause new bugs
many of these bugs are hard to find without integral testing
often a result of violating overall architectures the bugs occur because data is left in inconsistent state
Automated test cases have a better chance of being kept up to dateif they form the majority of the testware
-
8/14/2019 The 5 Percent Rules of Test Automation
7/38 2009 LogiGear Corporation. All Rights Reserved
Why < 5% Automation Efforts ??
Automation should not dominate testing
it is not a goal in itself
may never be a bottleneck
Testers should be able to focus on testing
better tests (higher ambition level)
communication with stake holders
High automation efforts can aggravate the crunchzone, instead of relieving it
invitation to Murphys law
automation
should deliver,
not dominate
-
8/14/2019 The 5 Percent Rules of Test Automation
8/38 2009 LogiGear Corporation. All Rights Reserved
Vision on Automation
Have a good method
integrated method for testing and automation
testers in the driver seat maximum flexibility, best technology
minimal technical efforts
maximum maintainability
Have the right tool specifically designed for keyword based testing
flexible enough to allow for the right mix of high and low level testing
allow the testers to quickly build keyword tests, without hassle
is not overly complicated in the automation Succeed in the three "success factors for automation"
test design
automation solution
organization
-
8/14/2019 The 5 Percent Rules of Test Automation
9/38 2009 LogiGear Corporation. All Rights Reserved
Base Requirements
Test development and automation must be fully separated
testers are not / should not be programmers
the automation people must implement the action words, and test them
Test cases must have a clear and differentiated scope
and not deviate from it
The tests must be written in the right level of abstraction
for example higher business level, lower UI level, or both
the tooling must provide this flexibility
The method must be simple, not add to the complexity of theautomation
easy straightforward interpretation of tests, with minimal programming
-
8/14/2019 The 5 Percent Rules of Test Automation
10/38 2009 LogiGear Corporation. All Rights Reserved
Approaches to Automation
1. Record and Playback
2. Scripting / programming
3. Keywords
(simplified, see also: "Software Test Automation", Mark Fewster and Dorothy Graham)
it is my believe that an approach based onkeywords is a prerequisite to achieve a highlevel of stable automation
-
8/14/2019 The 5 Percent Rules of Test Automation
11/38 2009 LogiGear Corporation. All Rights Reserved
Keywords, Action Based Testing
number name quantitynew product P-9009 Sledge Hammer 5
number quantity
add quantity P-9009 20
add quantity P-9009 3add quantity P-9009 6
number quantity
check uantit P-9009 34
add quantity
check quantity
new product
function action_AddQuantity. . . .
end
function action_CheckQuantity. . . .
end
function action_NewProduct. . . .
end
-
8/14/2019 The 5 Percent Rules of Test Automation
12/38 2009 LogiGear Corporation. All Rights Reserved
Keywords, Action Based Testing
Describe test cases as a series of actions withkeywords and arguments, in a sheet like format
Externalize those descriptions out of thescripting
Let the testers (not the automation engineer)define the keywords, as by-product of the testdesign
Organize the test cases into "test modules",each with a clear and differentiated scope
Focus the automation efforts on the actions, not
on the test cases
-
8/14/2019 The 5 Percent Rules of Test Automation
13/38 2009 LogiGear Corporation. All Rights Reserved
Example of a Test Module
test objectivetest objective
-
8/14/2019 The 5 Percent Rules of Test Automation
14/38
2009 LogiGear Corporation. All Rights Reserved
Example of a Test Module, "low level"
TEST MODULE Product Maintenance
VERSION 1.0
. . .
TEST CASE Test Case 03 Use an invalid number
test requirement add 003 A number must be divisable by 9
source control destination
click main new new product
window control value
enter new product number P9001
source control
click new product process
text
check message Invalid product number.
test objective
-
8/14/2019 The 5 Percent Rules of Test Automation
15/38
2009 LogiGear Corporation. All Rights Reserved
Achieving the Challenges...
1. Test design
2. Automation architecture
3. Organization
Three major success factors:
-
8/14/2019 The 5 Percent Rules of Test Automation
16/38
2009 LogiGear Corporation. All Rights Reserved
Success Factor 1:
Test Design
-
8/14/2019 The 5 Percent Rules of Test Automation
17/38
2009 LogiGear Corporation. All Rights Reserved
Test Design
Effective test breakdown (into test modules)
make sure every test module has a clear focuskeep different kinds and levels of tests separate
Right level of actions
as high level if possible, hiding as many details asmuch as possible
but not if the details are relevant for the test
It is my believe that test design, not automationor the tool, is the deciding factor in achieving
the 5% challenges
-
8/14/2019 The 5 Percent Rules of Test Automation
18/38
2009 LogiGear Corporation. All Rights Reserved
Breakdown: Divide and Conquer
Mission, Requirements, System Design, Use Cases,Priorities, Planning, Budget, Stakeholders, ...
Identify test modules priority and planning what, when, who, how
Development
Execution
Maintenance
Test Module 1
Analysis
Development
Execution
Maintenance
Test Module 2
Analysis
Development
Execution
Maintenance
Test Module N
Analysis
. . .
-
8/14/2019 The 5 Percent Rules of Test Automation
19/38
2009 LogiGear Corporation. All Rights Reserved
Splitting into Test Modules
UI oriented tests does function key F4 work
does listbox xyz the right values is the tab order correct
Do individual functions work like transactions in a financial system
Alternate paths in use cases like cancel a transaction
End-to-end tests
Simulating business processes Tests with specific automation needs
like multi station tests
Tests of non-UI functions
High ambition level tests (aggressive tests)
-
8/14/2019 The 5 Percent Rules of Test Automation
20/38
2009 LogiGear Corporation. All Rights Reserved
Levels of Actions
High level actions to hide details
to hide unneeded details always hunt down details
implemented in terms of lower level actions
Low level actions to access the
automation technologyeither built-in in a tool like TestArchitector create yourself in a harness
this is the only real programming you would do
-
8/14/2019 The 5 Percent Rules of Test Automation
21/38
2009 LogiGear Corporation. All Rights Reserved
Example of Going High Level
-
8/14/2019 The 5 Percent Rules of Test Automation
22/38
2009 LogiGear Corporation. All Rights Reserved
Success Factor 2:
Automation Architecture
-
8/14/2019 The 5 Percent Rules of Test Automation
23/38
2009 LogiGear Corporation. All Rights Reserved
Automation: Implementing Actions
In a keyword driven approach the automation focuses onthe actions: as opposed to programming test cases in general this will lead to fewer and shorter scripts etc in the
automation
In the Action Based Testing method actions areimplemented in three ways: built-in in the tool, like click, enter, check etc defined in an action definition, using other (lower level) actions
similar to functions in a programming language
programmed in a programming or scripting language
-
8/14/2019 The 5 Percent Rules of Test Automation
24/38
2009 LogiGear Corporation. All Rights Reserved
"Action Definition"
-
8/14/2019 The 5 Percent Rules of Test Automation
25/38
2009 LogiGear Corporation. All Rights Reserved
Automation Architecture
Pay attention to where and how to implement actions using the action definitions (or something similar) for straightforward
actions use programming for
complex actions (typically involving loops) wrap specific technical code to access the interface
Select the right tools and technologies
some of the considerations are subtle technical choices strongly influence the ease of the automation make sure the technology is organized as durable/re-usable as possible Go for a mix of technologies (UI, APIs, SQL, TCP/IP, ...)
Manage the handling of the interface (UI, API, ...) make sure all dialogs, pages, controls etc get meaningful logicalnames (mapped to the actual items in the UI etc)
let the developers help you if necessary by providing easy to identifyproperties of UI elements (for example the id in Web apps, and
AccessibleName in .Net)
-
8/14/2019 The 5 Percent Rules of Test Automation
26/38
2009 LogiGear Corporation. All Rights Reserved
Tips for Stable Automation
Make the system under test automation-friendly developers are not always inclined to do that, but it pays off
ask to add specific property values to the GUI interface controls forautomated identification like accessible name in .Net and Java, or id in Web controls
Use "active" timing
wait based on conditions in the system under test, avoid fixed sleeps Test your automation
develop separate tests for the actions themselves
Use automation to identify differences between versions of the
system under test test in a loop if all control can still be found
Keep an eye on the test design unsuccessful automation virtually always can be traced back to poor
test design
-
8/14/2019 The 5 Percent Rules of Test Automation
27/38
2009 LogiGear Corporation. All Rights Reserved
Success Factor 3:
Organization
-
8/14/2019 The 5 Percent Rules of Test Automation
28/38
2009 LogiGear Corporation. All Rights Reserved
Organization
Much of the 5% success is also gained or lost in how you organizethe process
who does test design who does automation what to outsource, what to keep in-house
Write a plan of approach:
scope, assumptions, risks, planning methods, best practices tools, technologies, architecture stake holders, including roles and processes for input and approvals
team . . .
Assemble the right team testers, lead testers
automation engineer(s) managers, ambassadors, ...
see also:QA All-Stars: Building
Your Dream Team, HansBuwalda, Better
Software September
2006
-
8/14/2019 The 5 Percent Rules of Test Automation
29/38
2009 LogiGear Corporation. All Rights Reserved
Separation of Automation
TestDevelopment
Plan
Automation:
- design- research- development- testing- support
Focus on:- actions- interface handling- technology- never on tests
Test ModuleDevelopment
Test ModuleAcceptance
Plan of Approach
Execution and
Follow Up
intake
-
8/14/2019 The 5 Percent Rules of Test Automation
30/38
2009 LogiGear Corporation. All Rights Reserved
Are the 5% rules Achievable?
To achieve to high automation coverage tends to befairly straightforward
once the actions work all tests are automated dont settle for too little
be ware of "creative" complex solutions in test design andautomation, they tend to obscure readability and maintainability
The automation efforts need to be well managed, butthen can be kept low as well need of automation specialization and intelligence in the team
a method like ABT can provide a proper framework, but is only astart
The key to success is: think it through dont hesitate to revisit your design and automation
make sure the tests are easy to run
-
8/14/2019 The 5 Percent Rules of Test Automation
31/38
2009 LogiGear Corporation. All Rights Reserved
Typical Time Allocation
TEST DEVELOPMENT
AUTOMATION
time
efforts
-
8/14/2019 The 5 Percent Rules of Test Automation
32/38
2009 LogiGear Corporation. All Rights Reserved
What Does It Take More
The method, tooling and success-factors
mentioned earlier in this presentation Focus on your objectives, and monitor:
make sure stuff gets automatedmake sure the method is applied consistently
make sure the test designs are good
go aggressively after "can't do" claims
Give clear direction
-
8/14/2019 The 5 Percent Rules of Test Automation
33/38
2009 LogiGear Corporation. All Rights Reserved
Case Study
Large IT provider
New version of one of their major web-
sites
Test scope was user acceptance test
(functional acceptance)
the users were the business owners
Development was off-shore
-
8/14/2019 The 5 Percent Rules of Test Automation
34/38
2009 LogiGear Corporation. All Rights Reserved
Case Study
Test development was done separate from
automationtime-line for test development: May Oct
time-line for automation (roughly): Jan Feb
All tests were reviewed and approved by
the business owners
acceptance was finished by the end of thetest development cycle
-
8/14/2019 The 5 Percent Rules of Test Automation
35/38
2009 LogiGear Corporation. All Rights Reserved
Test Development PlanNr Module Business Owner Date to BO1 Portal Navigation, Audience Robyn Peterson 05 / 232 Portal Navigation, Search Ted Jones 05 / 273 Membership, registration Steve Shao 06 / 03
4 Portal Navigation, Category Ted Jones 06 / 085 Portal Navigation, Topic and Expert Ted Jones 06 / 13
6 Access Control Mike Soderfeldt 06 / 177 Portal Navigation, Task Ted Jones 06 / 228 Contact DSPP Ted Jones 06 / 279 Portal search Mike Soderfeldt 07 / 0110 Membership, review and update Steve Shao 07 / 05
11 Program contact assignment Alan Lai 07 / 1112 Company, registration Steve Shao 07 / 14
13 Catalog, view and query Robyn Peterson 07 / 1914 Site map Ted Jones 07 / 2515 Membership, affiliation Steve Shao 07 / 2816 Learn about DSPP Ted Jones 08 / 0117 Products and services Steve Shao, Robyn Peterson 08 / 08
18 What's new Ted Jones 08 / 11
19 Company, life cycle Steve Shao, Alan Lai 08 / 1720 Specialized programs Ted Jones, Steve Shao 08 / 2221 Customer surveys Ted Jones 08 / 2922 Software downloads Mike Soderfeldt 09 / 01
23 Newsletters Ted Jones 09 / 0624 Internationalization and localization Ted Jones 09 / 1325 Membership, life cycles Steve Shao 09 / 19
26 Collaboration, forums Ted Jones 09 / 2327 Collaboration, blogs Mike Soderfeldt 09 / 2828 Collaboration, mailing lists Ted Jones 10 / 03
-
8/14/2019 The 5 Percent Rules of Test Automation
36/38
2009 LogiGear Corporation. All Rights Reserved
Review Process with Stake Holders
Test Team sends draftModule to Stake Holder
Stake Holder reviews:- coverage- correctness
Stake Holder returnsnotes:- additions- corrections
Test Team receives andprocesses notes
changes needed?Stake Holder returnsnotice of approval
Test Team marks theModule as "Final"
END
no
yes
START
-
8/14/2019 The 5 Percent Rules of Test Automation
37/38
2009 LogiGear Corporation. All Rights Reserved
Case Study, Results
All tests were developed and reviewed on schedule many notes and questions during test development phase
The automation was 100% of the tests all actions were automated, thus automating all test modules
The test development took an estimated 18 personmonths one on-shore resource, two off-shore resources
The automation took between one and two months focused on actions most time was spent in handling changes in the interface (layout
of pages etc)
-
8/14/2019 The 5 Percent Rules of Test Automation
38/38
Summary
Good to set aggressive targets for the automation: it costs money and efforts it should not get in the way of the testers making good tests
Elements for success: test design automation architecture
organization
Focus on the method first the tool is only a minor element use a method like Action Based Testing that is based on keywords and
addresses topics like test design
The 5% targets can be achieved: automation coverage can commonly achieve, can even reach 100% to limit efforts (to 5% of all efforts) requires serious planning and
thinking ahead. It also does not include a learning curve for the method