Download - Testing 29/Apr/2009
© 2008 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice
Testing 29/Apr/2009Petr Panuš[email protected] Manager & Offshore Team Manager, SOA Center HP Software R&D
Agenda1. Why is SW testing necessary2. Testing principles3. Testing design techniques
Why is SW testing necessary• SW does not always work as we wish• Defects in SW can cause harm
−Not all defects in SW result in a failure• Testing reduces the probability of
undiscovered defects remaining in the software
• Testing gives us the confidence about the tested SW
Testing Principles
Testing Principles• ISTQB defines 7 Testing Principles
1. Testing shows presence of defects2. Exhaustive testing is impossible3. Early testing4. Defect clustering5. Pesticide paradox6. Testing is context dependent7. Absence-of-errors fallacy
Testing shows presence of defects• Tests show the presence not the absence
of defects.• Testing can’t prove that SW is defect free• Testing reduces the probability of
undiscovered defects remaining in the software
• However, the fact that no defects were found, is not a proof of correctness.
Exhaustive testing is impossible• It is not cost-effective
−Defects are not of an equal risk• We need to prioritize our tests
−Use risk analysis and priorities to focus testing efforts
−Use appropriate testing techniques to meet the risk, giving us confidence in what testing has been performed
• Exhaustive testing is equivalent to Halting problem−There is no algorithm that can for an arbitrary
program and its input validates if the program works (with the given input) fine or not
Early testing• In SW development, what can be tested
−Requirement specification−Use-case Analysis document−Technical Analysis document−Design document−Functional Implementation−Performance−Usability
• What can be also tested−Test Description−Documentation
Price of potential defect fix increases
Example
Example – Early testing• Defect in H310: JBoss 4.3.0 not supported
on Win 2008 (although Hermes 310 PRD requires this combination to be supported)−7 other duplicates reported−Involving 4 people from QA−9 people from DEV
Defect clustering• Defects get clustered since different
reasons−A component might be more complex than
others−A component might be developed by a less
experienced developer−A component might be developed by a less
careful developer−A component might have a poorer specification−A component might need more refactoring
(introducing more other defects)• Another explanation:
http://parlezuml.com/blog/?postid=242
Pesticide paradox• Old tests will eventually stop finding new
defects−Defects create immunity to these tests
• To find defects, new tests need to be introduced−Or old tests refactored
• Conclusion: Regression tests do not find majority of new defects
Testing is context dependent• Different kinds of tests are run in different
periods−Defect testing
• To discover faults or defects in the software where its behavior is incorrect or not in conformance with its specification;
• A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.
−Validation testing• To demonstrate to the developer and the system
customer that the software meets its requirements;• A successful test shows that the system operates as
intended.−We start with defect testing and perform
validation testing later in the SW development process.
Testing is context dependent II.• We can also test differently because of
−The type of industry (safety-critical, business, nuclear)
−Number of customers (impact the SW makes)• One customer patch• New version of SW for potentially many customers
Absence-of-errors fallacy
Absence-of-errors fallacy• We test a system to see if it meets the
documented requirements• We find and fix defects to demonstrate that
the system meets these specifications• Finding and fixing defects does not help if
the system is unusable and does not fulfill the user’s needs and expectations
Testing Techniques
Testing design techniques• Black-box techniques
−Equivalence partitioning−Boundary value analysis−Decision tables−State transitions
• White-box techniques−Statement testing and coverage−Decision testing and coverage−Other structural-based coverage
No details about the product implementation are known
Tester knows how the tested requirement is implemented
Equivalence partitioning• Tax Calculator based on annual income• Input – integral number
• Partitions of equivalenceInvalid part Valid part
#1Valid part #2 Valid part #3
… -2, -1 0, 1, … 19999 20000, … 50000
50001, …
Income Tax< $20000 No tax$20000 - $50000 15%> $50000 20%
Equivalence partitioning II• Split input into partitions
−Valid partitions−Invalid partitions
• Have one test-case for each partition−Even invalid ones
• One more example: Enter number of a month
Invalid partition 1
Valid partition Invalid partition 2
… -2, -1, 0 1, 2, … 12 13, 14, …
Boundary value analysis• Similar as Equivalence partitioning but
−Test boundary values• N-BVA (BVA = 0-BVA)
−Test N boundary values −BV, BV+1, BV-1, … BV+N, BV-N
Decision tables
Decision tables - Example• Example – HP SOA Systinet Licensing
Framework• HP SOA Systinet consists of
−Core, Reporting, Lifecycle, etc.−Policy Manager (optional)−Contract Manager (optional)
• HP SOA Systinet License can limit−Number of users−Number of days
• Default license included in the installer −Policy & Contract Manager included−Limited to 60 days, unlimited to number of users
Decision tables• Example – Testing license application
Conditions
Actions
Decision tables• Example – Testing license application
Conditions
Default PM on CM on # days limited # users limited
Actions
Try PM
Try CM
Login after expirationLogin more users than allowed
Decision tables• Example – Testing license application
Conditions
Default Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N N N N N N N N N N N N N NPM on Y Y Y Y Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y N N N N N N N NCM on Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N# days limited Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N# users limited
Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N
Actions
Try PM
Try CM
Login after expirationLogin more users than allowed
Decision tables• Example – Testing license application
Conditions
Default Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N N N N N N N N N N N N N NPM on Y Y Y Y Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y N N N N N N N NCM on Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N# days limited Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N# users limited
Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N
Actions
Try PM X
Try CM X
Login after expiration
X
Login more users than allowed
Default license
Decision tables• Example – Testing license application
Conditions
Default Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N N N N N N N N N N N N N NPM on Y Y Y Y Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y N N N N N N N NCM on Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N# days limited Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N# users limited
Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N
Actions
Try PM X
Try CM X
Login after expiration
X
Login more users than allowed
Default license
Decision tables• Example – Testing license application
Conditions
Default Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N N N N N N N N N N N N N NPM on Y Y Y Y Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y N N N N N N N NCM on Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N# days limited Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N# users limited
Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N
Actions
Try PM X X
Try CM X X
Login after expiration
X X
Login more users than allowed
X Default license Visibilit
y Edition
Decision tables• Example – Testing license application
Conditions
Default Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N N N N N N N N N N N N N NPM on Y Y Y Y Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y N N N N N N N NCM on Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N# days limited Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N# users limited
Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N
Actions
Try PM X X X
Try CM X X X
Login after expiration
X X
Login more users than allowed
X Default license
UnlimitedStandardEdition
Visibility
Edition
Decision tables• Example – Testing license application
Conditions
Default Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N N N N N N N N N N N N N NPM on Y Y Y Y Y Y Y Y N N N N N N N N Y Y Y Y Y Y Y Y N N N N N N N NCM on Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N Y Y Y Y N N N N# days limited Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N Y Y N N# users limited
Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N Y N
Actions
Try PM X X X X
Try CM X X X X
Login after expiration
X X X
Login more users than allowed
X X Default license
UnlimitedStandardEdition
LimitedStandardEdition
Visibility
Edition
Decision tables II• Catches all possible combinations• Helps to analyze the situation and to decide
−Which use-cases must be tested−Which use-cases does not make sense−Requires knowledge of the business
environment• Helps to prioritize the use-cases
State transitions• Example – Testing ‘Contract Request
Lifecycle’Create Request
Accept Request
Reject Request Revoke
Request
Delete Request
State transitions II• Transition diagram shows valid transitions
−Does not show invalid (that should be also tested)
• Good for testing use-cases that are possible to be described by transitions between states
• The scenarios may contain test-cases for each−State−Transition−Event that triggers state change (transition)−Action that may result from those transitions
White-box techniques
Statement testing and coverage
Decision testing and coverage
Other structure-based techniques• Condition Coverage
−All single boolean conditions in a single statement must be evaluated to true and false
−Full condition coverage does not imply full decision one!
• Condition/Decision Coverage −Hybrid metric composed by the union of
condition coverage and decision coverage.
Other structure-based techniques II• Multiple Condition Coverage
−All possible combinations of all boolean conditions in a single statement must be evaluated to true and false
• Path Coverage−Whether each of the possible paths in each
function have been followed. A path is a unique sequence of branches from the function entry to the exit.
• Function, Call, LCSAJ, Loop, Race, etc. coverages
• FMI, see http://www.bullseye.com/coverage.html
Example• RTCA published DO-178B that requires
minimal coverage for aeronautics SW systems based on their criticality:
Effect of System Failure Example Required Code Coverage
Catastrophic Crash100% condition/decision coverage 100% statement coverage
Hazardous Passenger fatality100% decision coverage100% statement coverage
Major Passenger injury 100% statement coverage
Minor Flight plan change No code coverage requirement
No effectEntertainment system failure No code coverage requirement
Summary
Summary• Testing Principles
−Testing shows presence of defects−Exhaustive testing is impossible−Early testing−Defect clustering−Pesticide paradox−Testing is context dependent−Absence-of-errors fallacy
• Testing Techniques−Black-box techniques−White-box techniques
Q&A