agile test automation-gm4b-abridgedxunitpatterns.com/~gerard/nontechtestautomation.pdf ·...
Post on 12-Jun-2020
2 Views
Preview:
TRANSCRIPT
1
1 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Agile Test Automation StrategyFor the Non-technical
Gerard Meszaros
Testing2010@gerardm.com
2 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
I.T.
EmbeddedTelecom
My Background
Gerard MeszarosTesting2010@gerardm.com
•Software developer
•Development manager
•Project Manager
•Software architect
•OOA/OOD Mentor
•Requirements (Use Case) Mentor
•XP/TDD Mentor
•Agile PM Mentor
•Test Automation Consultant
•Lean/Agile Coach/Consultant
2
3 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Learning evolves; paper doesn’t
• Michael Bolton
– (the tester, not the singer!)
• Slides may be different from what you have printed off. Sorry!
– Latest version at:
– http://agile2010.xunitpatterns.com
– http://Testing2010.gerardm.com
4 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Know Thy Audience• Who’s ... Already involved in an agile project?
• ... Just starting with Agile?
• ... a Product Owner / Manager?
• ... a Project Manager?
• ... a Tester or Test Manager?
3
5 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Know Thy Audience (2)• Are you ... automating to find bugs?
• ... automating to prevent bugs?
• ... providing acceptance tests before the software is built?
• ... convinced you need automation?
• Do you have ... an automated func’l test suite?
• ... “complete” automated test coverage?
• ... “fragile tests”?
6 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Common Questions
Why?
How Management?
How Technical?
• Why do I need to be concerned with Test Automation?
• Why do I need to be more concerned with Agile (than Waterfall?)
• Can all testing be automated?
• How does test automation fit within an overall test strategy?
• Why do I need to tell the development team I expect automated unit tests?
• What kinds of test automation do I need?
• Who needs be involved in preparing Functional tests?
• How does automated testing support agile principles and values?
• How do agile principles & values support automated testing?
• What is the Fragile Test problem and how do we avoid it?
• What does it take to be successful with agile test automation?
• What are the common approaches to test automation?
• What Tools should we be using?
• How do we choose which tool to use?
• What about legacy code? How do we test it?
4
7 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Agenda
• Why Do We Need Test Automation?
– What’s Wrong with the status quo?
– How much is it costing?
• Automation and / or Traditional Testing?
– Changing the Role of Tests
– Automated vs. Manual Tests
– How is agile approach different?
• How to Automate Effectively?
– How can we avoid Fragile Tests?
– What are Common Approaches to Test Automation?
– What is the Role of Unit Testing?
• Agile Values & Test Automation
• Conclusion
8 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: Why Do I need to be concerned with Test Automation?
• To keep software soft. To get the product you expected.
• To make your expectations about the product clear.
• Iterative & Incremental Development has large impact on the amount of testing to be done. Automation makes it manageable.
5
9 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Product Owner Goal
• Goal: Maximize business value received
Features Money
Product Owner
Concept
Minim
ize
Maximize
Quality is Assumed; Not Managed
10 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Why Quality Often Sucks
• Iron Triangle of Software Engineering:
You can fix any three; the fourth is the outcome
Resources
Time
Functionality
• What about Quality?
Quality
6
11 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
In Agile, we “Pin” quality
using automated tests
Why Quality Often Sucks
• Iron Triangle of Software Engineering:
You can fix any three; the fourth is the outcome
Functionality
Quality
• What about Quality?
Resources
Time
12 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
CHAOS 2004 – Resolution of Projects
7
13 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
CHAOS 2004 – Project Success Trend
Copyright © 2004 – The Standish Group International, Inc.
Challenged
Succeeded
Failed
14 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
CHAOS 2004 – Average % Cost Overrun
8
15 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Is This Good Enough?
Who thinks this is good enough?
Who would let this doctor operate on you?
16 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
History of Medical Sterility• 1844 Ignaz Semmelweis noted midwife
wards had 1/3 the infection rate of doctors wards
• Deduced due to better cleanliness habits. Proposed rigorous hand washing before surgery.
• Actively resisted by doctors despite much better results.
• 1865 Joseph Lister requires hand-washing in his wards. Adopted after much controversy.
• 1920 Doctors still wear blood-spattered aprons in operating room.
• Today, sterile technique recognized as mandatory. Still requires reinforcement:
9
17 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Would you ...
... ask your doctor to reduce the cost of the operation ...
... by skipping the sterile technique ?
Test Automation is like hand washing:Improved results but at a cost.
18 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Minimizing Cost of Product
Total cost includes:
• developing the software
• verifying the newly built functionality
• verifying old functionality still works
• fixing any bugs found
• Verifying noting was broken by fixes
Agile Test Automation can reduce the cost of all of these activities.
10
19 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Overall Strategy
• Avoid Defects by Providing Acceptance Tests Up Front
• Reduce Cost of Fixing Bugs by Finding Defects Earlier
• Find Defects Earlier by Testing Earlier
• Test Earlier by Developing Incrementally
• Pin the functionality using Automated Tests
• Reduce the Cost of ReTesting by Automating Acceptance Tests
• Reduce Cost of Fixing Bugs thru Automated Unit Testing
• Make Test Automation Cost-Effective thru Design for Testability
20 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Preventing Defects
• Type NF bugs: New Func. is wrong
• Type RB bugs: New bugs in old func. (Regression Bugs)
Concept
Version 1.0
EvolvedConcept
Initial
NF
Version 1.xVersion 1.0
RB
NF
11
21 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Test&Fix Cycles
DevelopmentVerification & Acceptance
Requirements
B6! Missed!
Requirements
Are you Playing Battleship™?
22 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What is the cost of NF Bugs?
• Late discovery of bugs results in Delay
1. Missed Sales due to missing Market Windows
2. Delayed Revenue
3. Damaged Reputation
4. Increased development & testing costs
Calculating 1-3 depends heavily on your business
#4 can be calculated based on the delay
12
23 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
S/W Development Value Stream
What is this costing us?
• In delayed delivery?
• In extra effort?
Write Code
Insert Defect
Deploy to QA
Test Code
Log Defect
Defect List
Write Code (inserting defects)Fix -----
24 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Wasted Time – Delayed Delivery
• Imagine it takes 2 test cycles to fix enough bugs to release– 2 weeks to fix bugs
– 1 week waiting to deploy
– 1 week retesting
• Delivery is delayed by 8 weeks!– Benefit: 20% cost reduction on $10m/year
– Cost = 8/52 * .2*10m = .15*2m = $307,700
Defect List
Write Code
Insert Defect
Deploy to QA
Test Code
Log Defect
Write Code (inserting defects)
Avg: 2 weeks
Avg: 1 week delay
Avg: 1 week
Fix -----
13
25 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Wasted Effort – Per Bug
• Imagine we have 100 KLOCS of code delivered with a defect rate of 10 bugs per KLOC.
– 1000 bugs; 12 hours each to find, fix and retest
=12,000 hours of wasted effort
=1500 person days
= 7.5 person years @ 100K / person-year = $7.5m
Defect List
Write Code
Insert Defect
Deploy to QA
Test Code
Log Defect
10 per KLOC
Write Code (inserting defects)Fix -----
1 hours per bug8 hours per bug
3 hours per bug
26 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Wasted Effort – Test Cycles
• Imagine we have 100 KLOCS of code delivered with a defect rate of 10 bugs per KLOC.
– 30 person team
– Test&fix duration of 50 work days (5 cycles @ 10 days)
=1500 person days
= 7.5 person years @ 100K / person-year = $7.5m
Defect List
Write Code
Insert Defect
Deploy to QA
Test Code
Log Defect
10 per KLOC
Write Code (inserting defects)Fix -----
1 hours per bug8 hours per bug
3 hours per bug
14
27 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Exercise: How Much are NF Bugs Costing You?How much delay?
1. Current number of test&fix cycles: _____
2. Duration of typical test&fix cycle: _____
3. Current test&fix delay (1 x 2) : _____
4. Reduced number of test&fix cycles: _____
5. Saved test&fix cycles (1-4): _____
6. Total reduction in delay (2 x 4): _____
Saved cost of delivery
7. Cost of team per day: $ _____
8. Actual cost of delay (5 x 7): $ _____
Cost of delay:
– Lost Revenue/Benefit due to market window _____
– Delayed R/B due to delayed delivery _____
28 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What is the cost of Regression Bugs?
• Cost of finding the bugs (regression testing)
• Cost of fixing the bugs
• Cost of fixing goes up the longer the time between introducing the bug and finding it
• Ways to reduce this interval:
– NF Bugs: Incremental Acceptance Testing
– Regression Bugs: Automation helps
15
29 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Exercise: Cost of Regression1. How many times per year do you need to do
regression testing? _____
2. How much elapsed time? _____
3. How much effort is involved? _____
4. Total delay (1 x 2): days _____
5. Total cost (1 x 3): person-days _____
6. Suppose you reduced the iteration length (say, 2 weeks?) What would it cost now?
(n times 5) person-days _____
7. Could you include it in your daily build? ____
30 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Agenda
• Why Do We Need Test Automation?
– What’s Wrong with the status quo?
– How much is it costing?
• Automation and / or Traditional Testing?
– Changing the Role of Tests
– Automated vs. Manual Tests
– How is agile approach different?
• How to Automate Effectively?
– How can we avoid Fragile Tests?
– What are Common Approaches to Test Automation?
– What is the Role of Unit Testing?
• Agile Values & Test Automation
16
31 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How does test automation fit within an overall test strategy?
• Acceptance Tests are used to “Pin” the functionality we want to preserve.
• Automation is used to reduce the repetitive work involved in verifying the functionality of each successive build of software.
It does not replace:
• Exploratory testing in which testers try using the software and look for unexpected problems.
• Usability testing, in which real users are asked to complete typical task
Use machines for what they are good at so people can focus on what they are good at!
32 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Traditional Role of Testing
Thanks to Brian Marrick and Mary Poppendieck
Shigeo ShingoCo-inventor of
Toyota Production System
Inspection to find
defects is Waste
Inspection to preventdefects is essential
Acceptance Tests
Regression Tests
Usability Tests
Exploratory Tests
Unit Tests
Component Tests
Property Tests(Response Time,
Security, Scalability)
ReportCard
Critique Product
FunctionalityUsabilityScalabilityResponseAvailability
BCABC
Business
Facing
Technology
Facing
GGM37
Slide 32
GGM37 Add "Pin" to Regression Tests and Unit TestsGerard Meszaros, 07/08/2010
17
33 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Changing the Role of Testing
Acceptance Tests
Regression Tests
Usability Tests
Exploratory Tests
Unit Tests
Component Tests
Property Tests(Response Time,
Security, Scalability)
Business
Facing
Technology
Facing
Prevent anticipatable defects from happening
Find non-anticipatable Defects, ASAP!
Thanks to Brian Marrick and Mary Poppendieck
Critique Product ReportCard
FunctionalityUsabilityScalabilityResponseAvailability
BCABC
Define ProductRequirements
SoftwareDesign
34 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Prevention: - Building the Right Product
What the customer thought they wanted
What the customer actually asked for
What the customer realized they actually needed
What development thought the customer asked for
What development actually built
What testing thought the customer asked for
What testing actually tested for
18
35 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Building the Right Product
• How do we eliminate the waste caused by building the wrong product?
– Missed requirements?
– Misunderstood requirements?
– Unneeded functionality?
36 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Building the Right Product
• How do we eliminate the waste caused by building the wrong product?
– Missed requirements?
– Misunderstood requirements?
19
37 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Example-Driven Development
• A.K.A.
– Acceptance Test Driven Development
– StoryTest-Driven Development
• Concrete examples flesh out requirements
• Testers flush out missed scenarios
• Examples/Tests are executed as
product is built
38 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Life Cycle of an Example / Test
User Goal
Feature
StoryTitle
Story Narrative
Story Scenarios
Story Example
Executable Example
Satisfied Example
New!
FormalizationAutomation
ProductDevelopment
Make ConcreteAdd Data
DefineAcceptanceCriteria
20
39 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Knowing the Score Earlier, Always
• Development needs feedback
– How are we doing?
– What works, what doesn’t
• The sooner development gets feedback, the less effort gets wasted:
– Building more software based on wrong assumptions
– Trying to remember how the software works
» Because it’s been weeks or months since I wrote it
– Debugging the software
» Because I can’t remember what/when I changed it to work that way
GGM34
40 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Reducing the Cost to Fix Defects
• Why?
• We can remember where we put the newly inserted defect because
1. We know what code we were working on
2. The design of the code is still fresh in our minds
• We may have to change less code
– Because we wrote less code based on the defect
Cost to understand and fix a defect goes
up with the time it takes to discover it.
Slide 39
GGM34 Merge with previous slide?Gerard Meszaros, 06/08/2010
21
41 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
41
Continuous Acceptance Testing!
• Defines what “Done Looks Like”
– Several to many tests per User Story / Feature
Write StoryTest
Write StoryTest Build Code Test Code Test Story
Build Code Test Code Test Story
• Tests executed as soon developer says “It’s Ready”
– End-of-iteration: OK
– Mid-iteration : Better
42 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Readiness
Assessment
42
Continuous Readiness Assessment!
• Defines what “Done Looks Like”
– Several to many tests per User Story / Feature
Write StoryTest
Write StoryTest Build Code Test Code Test Story
Build Code Test Code
• Executed by developers during development
– To make sure all cases are implemented
– To make sure it works before showing to business
Test Story
• Tests executed as soon developer says “It’s Ready”
– End-of-iteration: OK
– Mid-iteration : Better
Manual Tests: OKAutomated: Better
Reduces # of testCycles;
Saves your time!
22
43 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What Do You Need to Do?
• Break work (functionality) down into small chunks that can be developed and tested incrementally
– Feature or Product Backlog
• Provided detailed examples of how it should work for each chunk
– Acceptance Tests
• Plan to verify each chunk as soon as it is finished
– May require more resources earlier in project
• Make it clear that Development is expected to run all the tests before delivery for testing
Ideally, automatable
Encourages automation
44 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How is the agile approach to test automation different?
• We automate the tests for a different reason
– Defect Prevention vs Detection
– To communicate requirements
– To “Pin” the functionality once it’s built
• We automate the tests a different way
– We don’t rely on GUI-based automation
– Many different kinds of tests
– Using tools that support collaboration & communication
» in addition to confirmation
• We plan the automation based on ROI
– Goal isn’t: 100% automation
– Goal is: To maximize benefit while minimizing cost
23
45 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Success Factors
Business
Not Willing or Able
Willing & Able
Te
ch
nic
al
Not Willing or Able
Willing & Able
New!
Failure to Automate
Automate the Wrong Tests or Too Late
Success Possible
Won’t Even Try
46 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Maximizing Test Automation ROI
• Need to Treat Automation as an Investment
• Need to Prioritize / Triage Which Tests to Automate
• At least 3 Approaches to Choose From:
– Traditional QA-Based Automation
– Collaborative Critical Path Automation
– Collaborative Selective Automation
New!
24
47 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
QA Automation After Dev Complete
A.K.A. Traditional Approach to Automation
Summary:
– Done by QA Department (i.e. Testers)
– After Product is Built
– Typically done using (C)OTS Record & Playback tools
Issues:
• Too Late for Defect Prevention
– Tests aren’t available to development team
• Too Late to Ensure Easy Automation
– System not Designed for Testability
• Tools Create Fragile Tests
– We’ll Discuss this in Detail Later
New!
48 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Collaborative Automation on Critical Path
A.K.A. Dogmatic (A)TDD Approach
Summary:
–Goal: 100 % automation
–Automate Tests Before Building Functionality
» Test automation task in each Story
Issues:
• Some Tests are MUCH Harder to Automate
• May Increase Costs and Delay Benefits of Functionality
• May Cause EDD to be Abandoned
New!
25
49 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Collaborative Automation based on ROI
A.K.A. Pragmatic Approach
Summary:
– Goal: Just Enough Automation
– Apply Agile Priniciples to Implementation of Automation
Issues:
– Won’t Have Complete Test Coverage
– Can Lead to Automation Being Dropped in Favour of More Functionality
– Requires a Disciplined Product Owner, or,
– A Fixed Budget for the Automation
New!
50 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Life Cycle of an Example / Test
User Goal
Feature
StoryTitle
Story Narrative
Story Scenarios
Story Example
Satisfied Example
New!
Executable Example
FormalizationAutomation
ProductDevelopment
Make ConcreteAdd Data
DefineAcceptanceCriteria
26
51 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What Do You Need to Do?
• Ask Test Automaters for Estimates of Effort
– For each different kind of test
• Ask Testers for Estimates of Impact of Not Having Automation
– For each different kind of test
– Effort to Run Tests Repeatedly
– Risk / Impact if Tests Weren’t Run Often Enough
• Prioritize Automation Stories Based on ROI
– Highest Impact to Effort Ratio
• Ensure that High ROI Automation gets Done
– Don’t Let New Features Crowd Out Essential Automation
New!
52 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Short ExerciseGGM19
Slide 52
GGM19 Devise an exercise.
Maybe a group exercise:Identify possible issues with doing ATDD.Discuss how you could surmount them.Gerard Meszaros, 03/08/2010
27
53 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Agenda
• Why Do We Need Test Automation?
– What’s Wrong with the status quo?
– How much is it costing?
• Automation and / or Traditional Testing?
– Changing the Role of Tests
– Automated vs. Manual Tests
– How is agile approach different?
• How to Automate Effectively?
– How can we avoid Fragile Tests?
– What are Common Approaches to Test Automation?
– What is the Role of Unit Testing?
• Agile Values & Test Automation
54 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Typical Test Strategy #1
• Large numbers of (possibly automated)
“customer” tests
• Very few if any automated component or unit tests
SystemTests
UnitTests
Manual or Record & Playback
Typical when testing (& automation) is “QA’s job”
QA Automation
ComponentTests
TypicallyManual
TypicallyManual
28
55 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Typical Test Strategy #2
• Large numbers of automated unit tests
• Some component tests
– Business Rules
– Technical Components
• Much fewer automated “customer” tests
Unit Tests Hand-Coded
All must be runnable by developers
Manual or automated
SystemTests
Agile (A)TDD
Component TestsHand-Written
56 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Anatomy of an Automated Test
Our System
Other System
Other System
InterfaceBusinessLogic Database
Container Services
Other System
Test
Test Scenario Name-----------------------Preconditions----------------------1. Do Something2. Check Something-----------------------Clean Up
1&2 May be repeated
Fixture Setup
FixtureTeardown
Fixture(state)
Fixture(adapter)
Given...
When ...Then ....
29
57 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: What makes automated tests fragile?
What, when changed, may break our tests accidentally:
– Behavior Sensitivity
» Business logic
– Interface Sensitivity
» User or system
– Data Sensitivity
» Database contents
– Context Sensitivity
» Other system state
Our System
Other System
Other System
InterfaceBusinessLogic Database
Container Services
Other System
Test
58 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Our System
Other System
Other System
InterfaceBusinessLogic Database
Container Services
Other System
Interface Sensitivity• Tests must interact with
the SUT through some interface
• Any changes to interface may cause tests to fail.– User Interfaces:
» Renamed/deleted windows or messages
» New/renamed/deleted fields
» New/renamed/deleted data values in lists
– Machine-Machine Interfaces:» Renamed/deleted functions in API
» Renamed/deleted messages
» New/changed/deleted function parameters or message fields
Window
Window
Field
Button
Title
Caption
Link
Data Grid
E.g.: Move tax field to new popup window
30
59 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Interface Sensitivity Impact *
• Symptoms:
– Unrecognized interaction
» e.g. window or message
– New unrecognized field
– Wrong data
• Impact:
– Many of these will be fatal.
– Non-fatals ones need manual inspection
– Every test that uses modified interface may fail
» Large maintenance impact
“*” means “skipped during presentation”
60 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How to avoid Interface Sensitivity?
• Express the tests using Intention-Based language
– Describe steps in business domain terms (Customers, Accounts), not UI terms (Screens, Fields, Buttons)
• Bypass the UI to an Intention-Based API
– Don’t test business logic via the UI
– Hand-script tests against Intention-Based API
– Do Built-in R&PB recording at Intention-Based API
• Wrap User Interface with an Adapter
– Record, Refactor (Extract Adapter), Playback
» Adapter converts Intention-Based API to User Interface
– Hand-script tests against the Intention-Based Adapter
31
61 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Keeping Tests Simple: Testing via API
• API’s need to be designed in
– Design for Testability
CoreBusinessLogic
Test Invoice Generation-New Customer
-----------------------------Logged in as ClerkItem1, Item2 exist-----------------------------1. CreateCustomer “Acme” 2. CreateAccoun NewCust3. AddPurchase Item14. AddPurchase Item25. GenerateInvoice NewAcct6. ….
UserInterface
What we want to write:SUT
�API
Intention-based Keywords
62 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
When There’s No API Available
• Large gap between:
– what we want to write & What can be executed
– Many tests to adjust when UI changes � High Maintenance Cost
CoreBusinessLogic
Test Invoice Generation-New Customer
-----------------------------Logged in as ClerkItem1, Item2 exist-----------------------------1. CreateCustomer “Acme” 2. CreateAccoun NewCust3. AddPurchase Item14. AddPurchase Item25. GenerateInvoice NewAcct6. ….
UserInterface
SUT
Test Invoice Generation-New Customer
-----------------------------Goto Login creenEnter “Clerk” in UserName fieldEnter “Pw123Secret” in Password fieldEnter …..-----------------------------Goto Cust ScreenClick “New Customer”Enter “Acme” in Name fieldEnter “123 Main St.” in Addr fieldEnter …..GotoScreen( “Account” )Find customer “Acme”Click “Add Account”Enter “Credit” in Type fieldEnter …..
Without a test API we have to write:
IntentionObscuringCode
CodeDuplication
� No API
32
63 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial�Adapter
Keeping Tests Simple: Testing via Adapters
• Adapters can be tacked on– Single place to adjust when UI changes– But may be complex and error prone
CoreBusinessLogic
Test Invoice Generation-New Customer
-----------------------------Logged in as ClerkItem1, Item2 exist-----------------------------1. CreateCustomer “Acme” 2. CreateAccoun NewCust3. AddPurchase Item14. AddPurchase Item25. GenerateInvoice NewAcct6. ….
UserInterface
� No API
What we want to write:
Code inAdapter SUT
Intention-based Keywords
64 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Our System
Other System
Other System
InterfaceBusinessLogic Database
Container Services
Other System
Behavior Sensitivity• Tests must verify the
behavior of the system.– Behavior also involved in test
set up & tear down
• Any changes to business logic may cause tests to fail.– New/renamed/deleted states
– New/changed/removed business rules
– Changes to business algorithms
– Additional data requirements
Object
Object
StateIdentity
AlgorithmRule
E.g.: Change from GST+PST to HST
33
65 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Behavior Sensitivity Impact *
• Symptoms:
– Tests block due to invalid inputs
– More/fewer items found
• Impact:
– Tests that depend on this behavior to setup the test fixture may also fail unexpectedly (fragile test)
66 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Avoiding Behavior Sensitivity• Limit the usage of SUT behaviors not specific to the
current test
– Minimize test fixture set up using other parts of SUT
» Use API’s or backdoors (e.g. database) to setup SUT
– Or, build that logic into reusable test assets (e.g. Keywords) that can be used from many tests but only maintained in one place (like an adapter)
• Minimize duplication of coverage between Test Scripts
– In part 2 of tests
– Use Data-driven tests instead
Four Parts to Every Test:1. Put SUT into known
state2. Exercise the SUT3. Compare the response
and end state with expected
4. Clean up
Here
Here
Behavior Sensitivity Strikes
Here
And sometimes
34
67 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Our System
Other System
Other System
InterfaceBusinessLogic Database
Container Services
Other System
Data Sensitivity• All tests depend on
“test data” which are:– Preconditions of test
– Often stored in databases
– May be in other systems
• Changing the contents of the database may cause tests to fail.– Added/changed/deleted
records
Table
RowRowRow
E.g.: Change customer’s billing terms
68 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Data Sensitivity Impact *
• Symptoms:
– Too many records found
– Too few records found
– Unique key violation
• Impact:
– Any test that depends on database contents may fail
– Tests that add to database may fail
35
69 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Erratic Tests – Interacting Tests
If many tests use same objects, tests can affect each other’s results.
–Test 2 failure may leave Object X in state that causes Test n to fail.
Test 1 read:5
TestRunner 1
Test n
read:99
Symptoms:
–Tests that work by themselves fail when run in a suite.
–Cascading errors caused by a single
bug failing a single test.
» Bug need not affect other tests directly but leaves fixture in wrong state for subsequent tests to succeed.
Fail! (expects 99)
Object X
a: 5
Test 2 update(99)
Object X
a: 99
Fail!
Object X
a: 5
70 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Erratic Tests – Unrepeatable Tests
If many test runs use same objects, test runs can affect each other’s results.
– Test 2 update may leave Object X in state that causes Test 1 to fail on next run.
TestRunner 1
Symptoms:– First run after opening the TestRunner or re-
initializing Shared Fixture behaves differently» Succeed, Fail, Fail, Fail
» Fail, Succeed, Succeed, Succeed
– Resetting the fixture may “reset” things to square 1 (restarting the cycle)
» Closing and reopening the test runner for in-memory fixture
» Reinitializing the database
Fail! (expects 5)
Object X
a: 5
Test 1 read:5
Test n
Test 2 update(99)
Object X
a: 99Test 1
36
71 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Erratic Tests – Test Run War
• If many test runners use the same objects (from Global Fixture), random results can occur.
– Interleaving of tests from parallel runners makes determining cause very difficult
Test 1
Test 2
Test n
Object Xupdate
read Test 1
Test 2
Test n
Object Xupdate
read
TestRunner 1 TestRunner 2
Fail!
Fail!
Fail!
72 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How do we avoid Data Sensitivity (or “Fragile Fixture”)?
• Avoid having tests share data
– With other tests in test suite
– Across test runs
» E.g. rebuild the environment between runs
– Across test runners
» E.g. separate environment per runner
• Each test should:
– Create any/all data it plans to modify
– Clean up after itself
• Test should not:
– Modify any data shared with other tests
» E.g. Reference data such as postal codes
37
73 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Our System
Other System
Other System
InterfaceBusinessLogic Database
Container Services
Other System
Context Sensitivity
• Tests may depend on another systems’ state
– Stored outside the application being tested
• Changing the state of the context may cause tests to fail.
– State of the container
» e.g. time/date
– State of related systems
» Availability, data contents
Security System
User: XPermissions: none
Container Services
TimeDate
Customer X
E.g.: Run test in a shorter/longer month
74 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Context Sensitivity Impact *
• Symptoms:
– Same system “build” behaves differently during different test runs
• Impact:
– Unrepeatable tests
– Nondeterministic tests
38
75 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How to avoid Context Sensitivity?
• Design SUT so that external dependencies can be simulated:
• Time/date under test control
• Security requests redirected to a test stub
• Access to other databases redirected to controlled instances
Requires Design for Testability (DfT) by Development
Collaboration!
Our System
Other System
Other System
InterfaceBusinessLogic Database
Container Services
Other SystemSecurity System
User: XPermissions: none
Container Services
TimeDate
Customer XTest Stub
Test Stub
76 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: What are the common approaches to test automation?
Test Preparation
Test Language
Test Interface
Test Data
Recorded Code Raw UI Global, Static
Hand-written Keyword Adapter Per Run
Data API Per Test
Our System
InterfaceBusinessLogic Database
Container Services
Test Scenario Name-----------------------
----------------------
-----------------------Clean Up
API
AdapterVia
Raw UI
How Set U
p?
Fixture(state)
Preconditions
1. Do Something2. Check Something
GGM26
Slide 76
GGM26 Animate the columns if possibleGerard Meszaros, 05/08/2010
39
77 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
executiondefinition
• User executes tests manually; tool records as tests
• Tool replays tests later without user intervention
Fixture
Test Result Repository
Test Script Repository
TestRecorder
SUT
TestScript 1
TestScript 2
TestScript n
ExpectedOutputs
Inputs
Inputs
Outputs
InputsTestRunner
Script nResult
TestResults
ExpectedOutputs
Recorded Tests
The tests are are code/data interpreted by the test runner.
2 Variations:1. External COTS 2. Built-in Custom
78 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
(C)OTS Record&Playback
Test Preparation
Test Language
Test Interface
Test Data
Recorded Code Raw UI # Global, Static
Hand-written Keyword* Adapter Per Run
Data API Per Test
Notes:* Keywords, if used, tend to be very low level:
•GotoWindowNamed: name•SelectFieldNamed: name•EnterText: text•(Not the same as true Keyword-Driven testing)
# Most COTS Tools operate at UI or HTTP interface; many open-source tools do so as well
40
79 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Sample Recorded Test@@ GoToPageMaintainTaxonomy()
Browser("Inf").Page("Inf").WebButton("Login").Click
Browser("Inf").Page("Inf_2").Check CheckPoint("Inf_2")
Browser("Inf").Page("Inf_2").Link("TAXONOMY LINKING").Click
Browser("Inf").Page("Inf_3").Check CheckPoint("Inf_3")
Browser("Inf").Page("Inf_3").Link("MAINTAIN TAXONOMY").Click
Browser("Inf").Page("Inf_4").Check CheckPoint("Inf_4")
@@ AddTerm("A","Top Level", "Top Level Definition")
Browser("Inf").Page("Inf_4").Link("Add").Click
wait 4
Browser("Inf_2").Page("Inf").Check CheckPoint("Inf_5")
Browser("Inf_2").Page("Inf").WebEdit("childCodeSuffix").Set "A"
Browser("Inf_2").Page("Inf").WebEdit("taxonomyDto.descript").Set "Top Level"
Browser("Inf_2").Page("Inf").WebEdit("taxonomyDto.definiti").Set "Top Level Definition"
Browser("Inf_2").Page("Inf").WebButton("Save").Click
wait 4
Browser("Inf").Page("Inf_5").Check CheckPoint("Inf_5_2")
@@ SelectTerm("[A]-Top Level")
Browser("Inf").Page("Inf_5").WebList("selectedTaxonomyCode").Select "[A]-Top Level“
@@ AddTerm("B","Second Top Level", "Second Top Level Definition")
Browser("Inf").Page("Inf_5").Link("Add").Click
Manually Added
Manually Added
Manually Added
Manually Added
Manually Added
80 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Record, Refactor, Playback
• Use Test Recording as a way to capture tests
• Remove duplication by replacing with calls to domain-specific Test Utility Methods
– using Extract Method refactorings
• Make Test Utility Methods reusable
– Replace Hard-Coded Literal Values with variables/parameters
• Effectively turns recorded tests into programmed or keyword-driven test scripts
– But, still through UI Adapter & original tool choice
Most appropriate with legacy systemsEspecially with many interfaces
41
81 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Recorded Tests Can Work Well V
Satisfied Customer Says:
“T Not only did this save 10's of man-years of testing effort, but it even uncovered before unknown bugs in the legacy system which we considered to be the gold standard. T”
V in the right circumstances:
– We retrofitted the capability to record and playback right into the legacy application.
– We came up with strategies to mitigate each of the four sensitivities.
– See:
82 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Built-In Record&Playback
• User executes tests manually; SUT records as tests
• Tool replays tests later without user intervention
The tests are data
interpreted by the test runner.
SUT
Test Script Repository
TestScript 1
TestScript 2
TestScript n
execution
TestRunner
InputsExpectedOutputs
TestRecorder
Inputs
ExpectedOutputs
GGM25
Slide 82
GGM25 Get Stickman / Actor symbolGerard Meszaros, 05/08/2010
42
83 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Built-in Record&Playback
Test Preparation
Test Language
Test Interface
Test Data
Recorded Code Raw UI Global, Static
Hand-written Keyword Adapter Per Run
API Per Test
84 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Sample Built-in R&PB Test Recording
2. Supply Create
43
85 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Prev. Rec. User Input
Actual choicesplus
test results
Previouslyrecordedchoices
Raw XML for “Designation” Field *
<field name="designation" type="selection"><used-value>DIRECTIONAL</used-value><expected>
<value>DIRECTIONAL</value><value>WORK</value><value>PSGR</value><value>PLOW</value><value>PLOW WORK</value><value>ENG</value>
</expected><actual>
<value status="ok">DIRECTIONAL</value><value status="ok">WORK</value><value status="ok">ENG</value><value status="ok">PSGR</value><value status="surplus">MIXED</value><value status="ok">PLOW</value><value status="ok">PLOW WORK</value>
</actual></field>
Record
ingPlayb
ack
86 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
• Test automater writes test code to exercise the software
• Runs test code whenever testing required
Hand-Coded Tests
executiondefinition
The tests are a program that tests the program
A.K.A. Scripted,Programmed
TestRunner
Test Result Repository
Test Script Repository
TestScript 1
TestScript 2
TestScript n
Fixture
S/W Devt.Tool
SUT
ExpectedOutputsInputs
Inputs
Outputs
Script nResult
TestResults
Inputs
ExpectedOutputs
TestScript n
44
87 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Hand-Coded Tests
Test Preparation
Test Language
Test Interface
Test Data
Recorded Code Raw UI Global, Static
Hand-written Keyword Adapter Per Run
Data API Per Test
• Hand-written code requires software development skills and test automation skills.
• Developers need training on writing clear tests!
88 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Sample Hand-Coded Test public void testAddItemQuantity_severalQuantity () {
// Fixture set up:final int QUANTITY = 5 ;Product product = createAnonymousProduct();Invoice invoice = createAnonymousInvoice();
// Exercise SUTinvoice.addItemQuantity(product, QUANTITY);
// VerifyLineItem expectedLineItem = newLineItem( invoice,
product, QUANTITY, product.getPrice()*QUANTITY );
assertExactlyOneLineItem( invoice, expectedLineItem );}
45
89 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Sample Hand-Coded Test - Poorpublic void testAddItemQuantity_severalQuantity () {
// Setup Fixturefinal int QUANTITY = 5;Address billingAddress = new Address("1222 1st St SW", "Calgary",
"Alberta", "T2N 2V2", "Canada");Address shippingAddress = new Address("1333 1st St SW", "Calgary",
"Alberta", "T2N 2V2", "Canada");Customer customer = new Customer(99, "John", "Doe", new
BigDecimal("30"), billingAddress, shippingAddress);Product product = new Product(88, "SomeWidget", new
BigDecimal("19.99"));Invoice invoice = new Invoice(customer);// Exercise SUTinvoice.addItemQuantity(product, QUANTITY);// Verify OutcomeList lineItems = invoice.getLineItems();if (lineItems.size() == 1) {
LineItem actualLineItem = (LineItem)lineItems.get(0);assertEquals(invoice, actualLineItem.getInvoice());assertEquals(product, actualLineItem.getProduct());assertEquals(quantity, actualLineItem.getQuantity()); assertEquals(new BigDecimal("30"),
actualLineItem.getPercentDiscount());assertEquals(new BigDecimal("19.99"),
actualLineItem.getUnitPrice());assertEquals(new BigDecimal(“69.96"),
actualLineItem.getExtendedPrice());} else {
assertTrue(“Invoice should have exactly one line item“, false);}
Developersneed
training on writing
clear tests!
90 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
executiondefinition
• The tests are expressed in domain-specific vocabulary.
• The tests are read & executed by a test interpreter written by techies.
Keyword-Driven Tests
Prepared like Hand-Coded Tests but with a much more limited vocabulary.
46
91 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Keyword-Driven Tests
Test Preparation
Test Language
Test Interface
Test Data
Recorded Code Raw UI * Global, Static
Hand-written Keyword Adapter Per Run
Data API Per Test
Notes:• While the Keyword Interpreter may go against the Raw
UI, it is better to delegate to an adapter if no API is available.
92 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Test Invoice Generation-New Customer
-----------------------------Logged in as ClerkItem1, Item2 exist-----------------------------1. CreateCustomer “Acme” 2. CreateAccount NewCust3. AddPurchase Item14. AddPurchase Item25. GenerateInvoice NewAcct6. ….
SUT
Sample Keyword-Driven Test
ComponentUnder Test
ResultsDocument
Result of step
Result of step
Fit Library
Fit Test Runner
Keyword Interpreter
Keyword Interpreter
47
93 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
executiondefinition
• The tests are expressed as tabular data by users.
• The tests are read & executed by a test interpreter written by techies.
Data-Driven Tests
Runs the same test script many times; once per set of data.
94 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Data-Driven Test
Test Preparation
Test Language
Test Interface
Test Data #
Recorded * Code Raw UI Global, Static
Hand-written Keyword Adapter Per Run
Data API Per Test
Notes:* The underlying script may be either hand-written or
recorded and parameterized. But the input values and expected outputs are almost always prepared by hand.
# The inputs/outputs are per test (per row) but there may be global or per-run data used as reference data by the underlying script.
48
95 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Sample Data-Driven Test in FIT
• Same script is run for each row of table
• Avoids duplication of test script.
• Compact summary of input values & results
• Sometimes called “Business Unit Test” or “Business Rule Test”
PayrolFixtures.WeeklyCompensation
Standard Hours
Holiday Hours
Hourly Wage
Pay( )
40 0 10 $400
40 0 20 $800
41 0 20 $830
40 1 20 $840 expected
$800 actual
41 1 20 $870 expected
$830 actual
�---Inputs---� �Outputs�
SUT
ComponentUnder Test
ResultsDocument
Marked upTable
Fit Library
Fit Test Runner
Table Interpreter
96 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How to choose test automation tool?
1. Determine test automation goals
– Finding bugs?
– Preventing bugs?
– Frequent/Rapid Regression Testing?
2. Choose the test automation approach(es)
– Hand-written or Recorded?
– Code, Data-driven, or Keyword-driven?
3. Choose tool(s) that supports approach(es)
– E.g. xUnit for Hand-Coded Unit Tests
– E.g. RobotFramework for Hand-Written Acceptance Tests
– E.g. Fit for Hand-written Business Rule Tests
49
97 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How to choose which test prep. strategy?
Fixture
Test Result Repository
Test Script Repository
TestRecorder
SUT
TestScript 1
TestScript 2
TestScript n
ExpectedOutputs
Inputs
Inputs
Outputs
InputsTestRunner
Script nResult
Test
Results
Expected
Outputs
Test Result Repository
Fixture
Test Script Repository
DocumentEditor
SUT
TestScript 1
TestScript 2
TestScript n
ExpectedOutputs
Inputs
Inputs
Outputs
TestRunner
TestInterpretter
Script nResult
TestResults
InputsExpectedOutputs
?
GGM15
98 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Which Test Preparation Strategy?
Recorded Tests:
• Fast to create– “No coding skills Required”
• Don’t support example-driven development– Only useful after the fact
• Unmaintainable– Re-record, or
– Coding skills required
• Done by Testers– Fairly non-technical
• Unintelligible to Business– Because it’s too detailed
– Doesn’t capture business intent
Hand-Coded Tests:
• Harder to create– Coding skills required
• Supports example-driven development – Can be done ahead of time
• Requires S/W Skills– Design-for- Testability
– Test s/w engineering
• Done by Techies– Development or Technical
testers
• Unintelligible to Business– Because it’s in code
GGM24
Slide 97
GGM15 Fix this up - make it coherentGerard Meszaros, 30/07/2010
Slide 98
GGM24 Need to add Data-Driven and Keyword-Diven Tests
Make this more comprehensiveGerard Meszaros, 06/08/2010
50
99 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Why Build R&PB into your Application?
• Gives you complete control over what to pay attention to and what to ignore!– E.g. Ignore the time/date fields
• Tests will run much faster bypassing the User Interface
• May be less expensive than commercial tools, if:– Large numbers of users, or
– Low cost to build » I.e., simple design or easy-to-use
framework
Why Not:
• Can be a significant investment of time and money
– Expect it to cost twice what you first estimate
• Can require a fair bit of expertise
100 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Why Build R&PB into your Application? *
Can make it much easier for your business people to use:• Domain-specific terminology
– Business concepts, not technical
– E.g. Select Direction South
vs. Select value “south” from dialog box with title=“Direction”
• Testing rules built in
– No need to fiddle with generic sensitivity settings in each recorded test
– E.g., Always ignoring time fields
• Simplified test setup
– Knows what starting state consists of and how to cause it to be
– E.g., Managing Database Snapshots
51
101 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Why Use a (C)OTS R&PB Tool?
• Available Right Away
– Free trial downloads
– A lot of test automation expertise is built in
• May Be Lower Risk
– If you can address the 4 sensitivities
Why Not?
• Expensive to buy
– Per-user licensing cost can make it very expensive to deploy
– May have to limit who uses it
• May force you to work at too low a level
– E.g., screen widgets
– May cause you to loose focus on the big picture
• Giving up control of your destiny
– Bound to a tools vendor and their vision of the future• Open Source may address cost
• But not other issues.
102 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Why Hand-Code Tests
• Most flexibility
• Tests will be automated by developers
• Technical product
– E.g. For developers
Why Not?
• Requires programming skills
• Business people won’t understand tests/examples
• More effort
• Less collaboration
52
103 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Why Use Keyword-Driven Tests
• Tests/examples can be automated and reviewed by business experts
• Encourages collaboration between business, testers & developers
• Less effort per test script
Why Not?
• Less flexible
• Requires collaboration!
104 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Why Use Data-Driven Tests
• Can test large numbers of data combinations easily
• Avoids duplicated test script code
• Less effort per test condition
Why Not?
• Less flexible; need separate test format for each business rule
• App. is not rules intensive
• Requires collaboration!
53
105 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What You Need to Do
• Ensure Development and Testing understand that you expect them to:
– Build sustainable test automation
– Work together to ensure system is easily tested
• Ask for regular updates on how test automation will be built
– Ensure they have addressed all 4 sensitivities
– You will need to sustain it long after Dev’t & Test have moved on to other projects!
• Provide knowledgeable resources to specify the business scenarios to be tested
– Maybe even build the test cases
106 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Exercise
• If you have done test automation in the past:
– Which approaches have you used?
– What problems did you run into?
– How did these relate to the Four Sensitivities?
• Which of these approaches look most promising for your current / next project?
– Recorded COTS
– Recorded Built-in
– Keyword-Driven
– Data-Driven
– Hand-Coded
54
107 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: Who needs be involved in preparing Functional tests?
• Business subject matter experts are needed to specify the tests/examples to be automated.
• Testers should also be involved as they often think of scenarios that business people don’t think of (what could possibly go wrong?)
• Test automation engineers or software developers are required to build the automated scripts (Hand-scripted or Record-Refactor-Playback) or the test script interpreters (Keyword-Driven).
A Collaborative Approach Works Best!
108 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
ApplicationCollaborative Test Automation
What:
• Define test agreeing on document format
• Implement keyword interpreters
• Run tests using Test Runner
• Fit marks up Tabular Data with test results
ComponentUnder Test
TestsDocument
TabularData
TabularData
ResultsDocument
Marked upData
Marked upData
Fit Library
Keyword Interpreter
Keyword Interpreter
Fit Test Runner
Who:
Joint
Dev’t or TE
Anyone
Automatic
55
109 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Collaborative Test AutomationPayrolFixtures.WeeklyCompensation
Standard Hours Holiday Hours Hourly Wage Pay( )
40 0 10 $400
40 0 20 $800
41 0 20 $830
40 1 20 $840
41 1 20 $870
110 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Collaborative Test AutomationPayrolFixtures.WeeklyCompensation
Standard Hours Holiday Hours Hourly Wage Pay( )
40 0 10 $400
40 0 20 $800
41 0 20 $830
40 1 20 $840 expected
$800 actual
41 1 20 $870 expected
$830 actual
56
111 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Sample Fit Column Fixture
public class WeeklyCompensation ColumnFixture {
public int StandardHours;
public int HolidayHours;
public Currency Wage;
public Currency Pay( ) {
WeeklyTimesheet timesheet = new WeeklyTimesheet( StandardHours, HolidayHours);
return timesheet.CalculatePay( Wage );
}
}
112 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Requires access to software internals
Sample Fit Column Fixture
public class WeeklyCompensation ColumnFixture {
public int StandardHours;
public int HolidayHours;
public Currency Wage;
public Currency Pay( ) {
WeeklyTimesheet timesheet = new WeeklyTimesheet( StandardHours, HolidayHours);
return timesheet.CalculatePay( Wage );
}
}
57
113 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Sample Multi-Table FIT Test• Load data into table using Column Fixture:
• Exercise system using Action Fixture
• Verify database contents using Row Fixture
PayrolFixtures.EmployeeTableLoad
Number Name Hours Add()
10001 Gerard Meszaros 45 OK
10002 John Smith 40 OK
10003 Jane Doe 420 OK expectedToo Big actual
PayrolFixtures.EmployeeCompensationThisWeek
Number Name Hours Pay
10001 Gerard Meszaros 45 $950
10002 John Smith 42 expected40 actual
$860 expected$800 actual
10003 surplus Jane Doe 20 $800
PayrolFixtures.RunPayrollActionFixture
RunJob Payroll
114 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What You Need to Do
• Dedicate good (or better) business resources to specifying the requirements as examples/tests
– Requires good knowledge and communication skills.
– Don’t fob off the people you can spare
• Ensure you include positive & negative cases
– May require experienced tester to bring this perspective
• Ensure Development team is involved
– To understand testability requirements of application.
GGM17
Slide 114
GGM17 Fill inGerard Meszaros, 30/07/2010
58
115 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: Why We Need Unit/Component Tests?
Acceptance Tests
Regression Tests
Usability Tests
Exploratory Tests
Unit Tests
Component Tests
Property Tests(Response Time,
Security, Scalability)
Business
Facing
Technology
Facing
Prevent anticipatable defects from happening
Find non-anticipatable Defects, ASAP!
Thanks to Brian Marrick and Mary Poppendieck
Critique Product
ReportCard
FunctionalityUsabilityScalabilityResponseAvailability
BCABC
Define ProductRequirements
SoftwareDesign
116 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What are Unit/Component Tests?
• Lets us test code we cannot hit in func. tests
– Avoids nasty lurking bugs that are only found by users
• They run much faster than functional tests
– Makes retesting bugs cheaper
• Helps developers find their bugs much sooner
– Makes fixing them cheaper
Application
Customer (Functional) Testing
ComponentTesting
Component
Class
Unit Testing
59
117 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Insert Defect
DetermineFix
Identify/ Debug
Run Unit Tests
Preventing Coding Defects
Write Req’ts
Review Req’ts
Update Req’ts
Sign Off on Req’ts
Test Application
Deploy to QA
Deploy System
DesignSoftwareDetection!
Write Code
Unit Test Code
Rework!Rework!
118 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Preventing Coding Defects
Write Req’ts
Review Req’ts
Update Req’ts
Sign Off on Req’ts
Write Unit Tests
Run Unit Tests
Test Application
Deploy to QA
Deploy System
DesignSoftware
Write Code
Prevention!
DetermineFix
Identify/ Debug
60
119 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Prevention: - Coding Defects
Write Req’ts
Review Req’ts
Update Req’ts
Sign Off on Req’ts
Write Unit Tests
Run Unit Tests
Test Application
Deploy to QA
Deploy System
DesignSoftware
Write Code
Test-Driven Development
Prevents bugs crawling back in
Prevent defects in new code
120 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
What Do You Need to Do?
Make it clear to development team that:
• They are to do proper unit testing.
• All estimates must include unit test cost
• Ideally, automated tests and test harnesses
Like reminding the Doctor that Sterility is not optional
61
121 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: Why do I need to tell the development team I expect automated unit tests?
• They may not feel empowered to impose this overhead on development because they don’t recognize the business value.
• They may have to been told in the past by Product Owners (or development managers) “not to waste their time” doing automated testing.
122 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: What Tools should we be using?
• Won’t talk about specific tools per se but:
• Tools should make the job as easy as possible for the people doing the work:
• The people preparing the tests or examples
• The people running the tests or examples
• The people supporting the tests or examples
•
62
123 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How does automated testing support agile principles and values?
• AT provides rapid feedback for developers.
• AT reduces the waste involved in repeatedly retesting the same functionality.
• AT improves quality by preventing defects from being introduced (by providing concrete examples, and by avoiding regression.).
• AT enables cost-effective regression testing of each increment of functionality.
124 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Q: How do agile principles & values support automated testing?
• Collaboration between test automaters and developers ensures the system is built to support testability.
• The preparation of automated functional tests/examples forces collaboration between subject matter experts (business-oriented people) and technical people (test automaters and developers.)
• Testers are motivated to help developers do better unit/component testing.
63
125 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Test Automation Pyramid
• Large numbers of very small unit tests
• Smaller number of functional tests for major components
• Even fewer tests for the entire application & workflow
Unit Tests
ComponentTests
SystemTests
Various automation
tools(or manual)
Keyword or xUnit
xUnit
126 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Application not Designed for Testability
Causes of Fragile Tests:
• Interface Sensitivity
• Behavior Sensitivity
• Data Sensitivity
• Context Sensitivity
MonolithicApplication
Test
UI Logic
Business Logic
Technical Logic
Data Access Logic
ControlPoint
Observation Point
See: http://agileregressiontestpaper.gerardmeszaros.com
64
127 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Application with Database
MonolithicApplication
Test
Observation Point – Indirect Output
Control Point– Indirect Input
ControlPoint
Observation Point
UI Logic
Business Logic
Technical Logic
Data Access Logic
128 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
ControlPoint
Observation Point
Application Designed for Testability
UI LayerBusinessLogic Test
ControlPoint
Observation Point
Business Logic Layer
Technical Layer
Data Access Layer
TechnicalLayer Test
ControlPoint
Observation Point
Data AccessLayer Test
ControlPoint
Observation Point
UI Test
Observation Point – Indirect Output
Control Point– Indirect Input
65
129 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Collaboration Required!
DevelopmentIndependent Verification
Requirements
B6! Missed!
Requirements
130 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Collaboration Required!
Requirements
B6! Missed!
Requirements
DevelopmentIndependent Verification
66
131 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Collaboration Required!
Requirements Requirements
This Software is Ready for Testing
Let’s Test it Together
This test is tedious but essential.
Let’s automate
it.
GGM11
132 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Strategy Summary
• Avoid Defects by Providing Acceptance Tests Up Front
• Reduce Cost of Fixing Bugs by Finding Defects Earlier
• Find Defects Earlier by Testing Earlier
• Test Earlier by Developing Incrementally
• Pin the Functionality using Automated Tests
• Reduce the Cost of ReTesting by Automating Acceptance Tests
• Prioritize Automation by ROI = (Value / Cost)
• Improve Internal Quality and Reduce Cost of Fixing Bugs thru Automated Unit Testing
• Make Test Automation Cost-Effective thru Design for Testability
Slide 131
GGM11 Maybe omit this slide?Gerard Meszaros, 30/09/2008
67
133 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
Thank You!
Gerard MeszarosTesting2010@gerardm.com
http://www.xunitpatterns.com
http://www.gerardmeszaros.com
Slides: http:// Agile2010.xunitpatterns.com
Call me when you:• Want to transition to Agile or Lean
• Want to do Agile or Lean better
• Want to remove waste from your process
• Want to teach developers how to test
• Need help with test automation strategy
• Want to improve usability of your applications
Jolt Productivity Award winner - Technical Books
Coming Soon:
134 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial
References
• For Success, Build Record/Playback into Your Application - StarEast 2008 Class
– http://agile2010.xunitpatterns.com/
top related