exploratorytesting 1 introduction - testing education · introduction ©2002 amland consulting 1-1...
TRANSCRIPT
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-1
Introduction OutlineThese slides are distributed under the Creative
Commons License. In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one. For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-2
Introduction Outline
Introduction
Test Management and Techniques
ET Planning, Exec. and Documentation
ET Styles
ET Management
• Introduction to testing.• What is Exploratory
Testing?• Where to use it?• When to use it?• Introduction to Risk
5.
4.
3.
2.
1.
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-3
1. Introduction
1.1 Introduction to testing – thinking like a tester1.2 Introduction to Exploratory Testing
1.3 Introduction to Risk and Risk-Based Testing
1.
2.
3.
4.
5.
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-4
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Different testing approaches
1. Skeptical approaches2. Analytical approaches3. Information-driven approaches4. Time-honored but less effective
approaches5. Experiential and intuitive approaches6. And…?
Ross Collard (2002)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-5
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Skeptical approaches
“In God We Trust, Everything Else We Test!”“The barbarians (software engineers) are at the gate”“Let’s use a scatter gun to test with, and see what bugs we hit”
Ross Collard (2002)
Different testing approaches (1)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-6
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Analytical approachesLet’s analyze the functional specs. to understand the system’s expected behavior“Let’s develop a model of the system, and then use this conceptual model as a basis for testing”Let’s derive the test cases by analyzing the description logic, process flows, equivalence classes, changes of state, or input combinations, etc.”
Ross Collard (2002)
Different testing approaches (2)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-7
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Information-driven approaches“Let’s focus the depth and intensity of the testing in the high risk areas, based on the perceived threats and vulnerabilities of the system”“Let’s follow this bug list or check list in our testing”“Let’s go ask the software engineers what to test, because they know how the system works”“Let’s look under the hood and read the code”“Let’s follow the clients’ direction, because they have the final sign-off authority”
Ross Collard (2002)
Different testing approaches (3)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-8
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Time-honored but less effective approaches
“Let’s follow the book”“We always do it this way”“You shouldn’t change that features because it will screw up our testing.” The tail wags the dog.“Testing is easy, or at least a lot easier than software design, programming and maintenance.”“Anyone can do testing”“Errors just happen. They are caused by bad luck.”
Ross Collard (2002)
Different testing approaches (4)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-9
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementExperiential and intuitive approaches
“Let’s think blue-sky, speculate and follow our intuition.”“We have good hunches about where the bugs are lurking.”“Let’s jump in an explore the system’s behavior hands-on, so we can decide how to test it.”“Let’s find the important bugs fast, and worry about the test paperwork later.”
Ross Collard (2002)
Different testing approaches (5)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-10
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Exploratory Testing– Let’s explore, design the tests and test the
system concurrently (James Bach)– Let’s learn about the system, test it and
reports bugs as we go (Cem Kaner)– Let’s structure and document our creative
testing so we know where we have been– Let’s apply everything we have learned about
testing as we learn about the system, let’s do ”thinking-while-testing”!
Different testing approaches…
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-11
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management What’s Special about a Tester’s Brain?
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-12
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementEpistemology – the Study of Knowledge
Epistemology is the study of how we knowwhat we know. The philosophy of science
belongs to Epistemology.
All good testers practice Epistemology.
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-13
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Basic Skills of EpistemologyAbility to pose useful questions.Ability to observe what’s going on.Ability to describe what you perceive.Ability to think critically about what you know.Ability to recognize and manage bias.Ability to form and test conjectures.Ability to keep thinking despite already knowing.Ability to analyze someone else’s thinking.
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-14
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
From Rapid Software Testing, copyright © 1996-2002 James Bach
Tunnel-Vision is Our Great Occupational Hazard
Problems you
can find with your
biases…
invisible
problems
invisible
problems
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-15
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management A Tester’s AttitudeCautious
Jump to conjectures, not conclusions.Practice admitting “I don’t know.”Have someone check your work.
CuriousWhat would happen if…?How does that work?Why did that happen?
CriticalProceed by conjecture and refutation.Actively seek counter-evidence.
Good testersare hard to fool.
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-16
1.
2.
3.
4.
5. To improve our ”judgement and skills”:
Heuristics:
A heuristic is a fallible method for finding the solution to a problem. It's essentially a plausible guess, or a mechanism that helps generate plausible guesses.
James Bach, [email protected]
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-17
1.
2.
3.
4.
5. Heuristics…continued:
"Avoid driving while intoxicated, because there is an elevated danger of an accident" is a heuristic.
"Never drive when intoxicated" is, by contrast, a rule.
James Bach, [email protected]
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-18
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Testing is done in Context1. The value of any practice depends on its context. 2. There are good practices in context, but there
are no best practices. 3. People, working together, are the most important
part of any project's context. 4. Projects unfold over time in ways that are often not
predictable. 5. The product is a solution. If the problem isn't solved, the
product doesn't work. 6. Good software testing is a challenging intellectual process. 7. Only through judgment and skill, exercised
cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
http://www.context-driven-testing.com/
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-19
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Exercise: The Triangle Program
Specification:This program takes three numbers as input.The numbers represent the dimensions of a triangle.When you click on the check button, the program tells you what kind of triangle the sides represent:
scalene (no side equal to any other)isosceles (two sides are equal)equilateral (all sides are equal)
Please test this program.
From Black Box Software Testing, copyright © 1996 – 2002 Cem Kaner
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-20
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementExample tests: The Triangle Program
Example of tests or groups of tests:Test case for a valid equilateral triangleAt least three test cases that represent valid isosceles triangles (all permutations, e.g. 3,3,4; 3,4,3; 4,3,3)Test case in which one side has a zero value
See Meyer’s Answer in his book
Myers 1979, page 3.
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-21
1. Introduction
1.1 Introduction to testing – thinking like a tester1.2 Introduction to Exploratory Testing
1.3 Introduction to Risk and Risk-Based Testing
1.
2.
3.
4.
5.
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-22
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management What is Exploratory Testing?"Exploratory testing involves simultaneously learning,
planning, running tests, and reporting / troubleshooting results."
Dr. Cem Kaner (2001)
"Exploratory testing is an interactive process of concurrent product exploration, test design and test execution.”
” To the extent that the next test we do is influenced by the result of the last test we did, we are doing
exploratory testing.”
James Bach, Satisfice (2001)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-23
Said about eXtreme ProgrammingAgile software development is not conventional software development done more quickly or done on tippie-toe. Agile software development is software done differently.
Ron Jeffries, (e-mail on agile-testing list, April 24, 2002)
•proven (no single technique is new)•application oriented•planned and disciplined•controllable and reliable•risk minimizing•Two sides of extreme programming:
• for the developer: freedom, flexibility, fun• for the manager: controllability, reliability, high quality
Martin Lippert (University of Hamburg), ICSTEST 2002
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-24
The eXtreme Programmingand Exploratory Testing Analogy:
Agile software testing is not conventional (scripted) software testing done more quickly or done on tippie-toe.
Exploratory Testing:•proven (no single technique is new)•application oriented•planned and disciplined•controllable and reliable•risk minimizing•Two sides of Exploratory Testing:
• for the tester: freedom, flexibility, fun• for the manager: controllability, reliability, high quality
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-25
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management ET vs. Scripted Testing
Jarle Våga (2002)
FullyScriptedTesting
Ad-hocTesting
AutomatedTests
BugHunting
ExploratoryTesting
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-26
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management What is Scripted Testing?
Small (but realistic) example:
How to script and test this login? (Functional tests only – not security!)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-27
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Sample test scripts (4 of “many”):
Sample test script 1:Launch the Login screenEnter User-id: “xyz”Enter Password: “zyx”Press <Enter>Expected result: login ok
Sample test script 2:Launch the Login screenEnter User-id: “xyz”Enter Password: “zyx”Click the “Login” buttonExpected result: login ok
Sample test script 3:Launch the Login screenEnter User-id: “”Enter Password: “zyx”Press <Enter>Expected: login rejected
Sample test script 4:Launch the Login screenEnter User-id: “”Enter Password: “zyx”Click the “Login” buttonExpected: login rejected
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-28
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementSample Generic Scripts (2 of “many”)
Sample generic test script 1:
Launch the Login screenEnter valid User-idEnter valid PasswordPress <Enter> or click buttonExpected result: login ok
Sample generic test script 2:
Launch the Login screenEnter invalid User-idEnter valid PasswordPress <Enter> or click buttonExpected result: login rejected
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-29
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Sample test “Pattern” script (checklist)
Input fields:Valid dataInvalid dataLength > maxLength = max +1Length = maxLength = max –1Combinations of above…
Actions:KeyboardButtons…
Operations:Add, Modify, Inquiry, Delete
What to test for each…
…
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-30
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementWhen to use Exploratory Testing? (1)
A common goal of exploration is to probe for weak areas of the program.
Test team’s resource consumption per week:25% of the group’s time developing new tests50% executing old tests (including bug regression)25% on exploratory testing
Cem Kaner (2001a)
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-31
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementWhen to use Exploratory Testing? (2)
When there is little or no specifications and / or requirementsWhen you have little or no domain knowledgeWhen you don’t have time to specify, script and test
Uncertainty and Time Pressure!
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-32
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementWhen to use Exploratory Testing? (3)
Exploratory Testing is extremely useful when faced with software that is
UntestedUnknown orUnstable
The tester must create a map of the application as he goes on testing it.
Harry Robinson, http://www.testingcraft.com/exploratory-robinson.html
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-33
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET ManagementWhen to use Exploratory Testing? (4)
Take a more scripted approach when:There are little uncertainty about how to testNew tests are relatively unimportantThe need for efficiency and reliability in executing tests is worth the effort of scriptingWe are prepared to pay the cost of documenting and maintaining tests
From Black Box Software Testing, copyright © 1996 – 2002 Cem Kaner
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-34
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
The Fallacy of Repeated Tests:Clearing Mines
minesFrom Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-35
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Totally Repeatable TestsWon’t Clear the Minefield
mines fixesFrom Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-36
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Variable Tests areTherefore More Effective
mines fixesFrom Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-37
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Sample Product Test Cycle1. Receive the product.
Formal buildsInformal buildsSave old builds.
2. Clean your system.Completely uninstall earlier builds.
3. Verify testability.Smoke testingSuspend test cycle if the product is untestable.
4. Determine what is new or changed.Change log
5. Determine what has been fixed.Bug tracking system
6. Test fixes.Many fixes fail!Also test nearby functionality.
7. Test new or changed areas.Exploratory testing.
8. Perform regression testing.Not performed for an incremental cycle.Automated vs. manualImportant tests first!
9. Report results.CoverageObservationsBug status (new, existing, reopened, closed)Assessment of qualityAssessment of testability
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-38
1. Introduction
1.1 Introduction to testing – thinking like a tester1.2 Introduction to Exploratory Testing
1.3 Introduction to Risk and Risk-Based Testing
1.
2.
3.
4.
5.
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-40
1.
2.
3.
4.
5. Typical Questions for Testers
How much testing is enough?When should we stop testing?When is the product good enough for release?How good is our testing?
è Managing RISK!
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-41
1.
2.
3.
4.
5. Product Quality and test coverage
Quality Risk Coverage
(potential faults)
Customer use Coverage
100%
100%
Worst
Poor Perfect!
Good
Rex Black 1999
ET Workshop v. 1.20 -Introduction ©2002 Amland Consulting 1-42
Introduction Summary
Introduction
Test Management and Techniques
ET Planning, Exec. and Documentation
ET Styles
ET Management
• Introduction to Testing.• What is Exploratory
Testing?• Where to use it?• When to use it?• Introduction to Risk
1.
2.
3.
4.
5.