system test methods testteme the test challenge bottom up testing strategy integration test system...
TRANSCRIPT
System Test Methods
TEST TEME
The Test Challenge
Bottom Up Testing Strategy
Integration Test
System Test
Types of Testing
Unit Test = Code-based Testing
Integration Test = Data-based Testing
System Test = Requirements-based Testing
Testing Interdependencies
Software Testing Strategies
Hierarchical Software Systems
BIG-BANG Strategy
TOP-DOWN Strategy
BOTTOM-UP Strategy
INSIDE-OUT Strategy
OUTSIDE-IN Strategy
BRANCHWISE Strategy
Comparison of the Test Strategies
Program Test Methods
Control Flow Testing (White-Box)
Data Flow Testing (Grey-Box)
Functional Testing (Black-Box)
Testing Heuristics
Roll of Testing in the Life Cycle
1
2
3
6
7
5
4
8
9
10
13
14
15
18
19
17
16
20
21
22
11 23
12 24
• Budget-, Time- and Personnel Constraints
• Great Diversity of SW-Environments
• Applications are becoming increasingly large and complex
• SW-Developers are neither trained nor motivated to test
• Testers are willing but incapable
• Lack of a testing culture
The Testing Challenge
TEST TEME-1
„How do you eat an elephant ... bit by bit“African Proverb
• Class Test– Test of each individual Class as a standalone Unit
• Integration Test– Test of several related classes and components
• System Test– Test of subsystems and the entire system
• Acceptance Test– Test by the User in a live environment
Bottom Up Testing Strategy
TEST TEME-2
„Integration testing exercises interfaces among units within the specified scope to demonstrate that the units are collectively interoperable“
Binder, 1999
• Classes• Class Hierarchies• Packages of related Classes = Components• Subsystems• Client/Server = Frontend/Backend • Entire Application
Integration Test
TEST TEME-3
• The Whole is more than the Sum of the Parts– „Individual verification of components cannot garantee
a correctly functioning system“ - Binder, 1999
• Demonstrating the required Functionality– „If you do not have written objectives for your product,
i.e a specification, or if your objectives are unmeasurable, then you cannot perform system testing“ - Myer, 1976
• Determining when to release a product ?!– Probability of remaining Errors..
System Test
TEST TEME-4
ClassesunderTest
Class Flattener
Test Driver
Tes
t S
tubs
Tes
t M
eth
ods
Integration Testing
Specified Arguments
Message Generator
Input Messages
Componentsunder Test
Return Messages
Message Validator
Unit Testing
ExpectedReturn Values
Replay Capture
Web Pages
Web Client
Web Server
Data
System Testing
DataGenerator
DataValidator
Retrieval Updates
Types of Testing
TEST TEME-5
A Test is always a test against Something
TEST TEME-6
Concept
System
Test Cases Test ResultsTest Cycle
newSystem
Test Cases Test ResultsTest Cycle
currentSystem
Verification = Comparison
In order to verify a system, the requirements must be independent of the code. The task is to test the system against the requirements by extracting test cases from the requirements specification. Verification implies demonstrating that the system fulfills it‘s requirements.
Once a system exists, every new version can be tested against the previous one. In this case the test cases can be extracted from the existing system and enhanced by test cases taken from the change requests. The new version is verified against the old one plus the new requirements.
Initial Testing Regession Testing
Code based Testing
TEST TEME-7
CodeTest inputs Test outcomes
Specification Expected outcomes
Interfaces/Databases
Predicate Paths
SpecifiedPaths
Actual Paths
compare Paths
Corresponds to a Unit Test (White-Box)
Unit Test = Code-based Testing
TEST TEME-8
Block
Block
Block
Block
Block
Block
IF THEN ELSE
DO WHILECASEBlock
IF THEN
41 5
7 8
6
9
10
2
3
= Testpoint
WHITE-BOX-Method
(Modultest)
Test Object:
Data based Testing
TEST TEME-9
CodeTest inputs Test outcomes
Specification Expected outcomes
compare Data
Representative Values
Interfaces/Files/Tables
Specified Results
Interfaces/Databases
Corresponds to an Integration Test (Grey-Box)
Client
Server
Data-server
Interface
Interface
Inte
rfac
e S
imul
atio
n
Inte
rfac
e V
alid
atio
n
GREY-BOX Method
(Program Test)
Input Data
Output Data---------
Integration Test = Data-based Testing
TEST TEME-10
Specification based Testing
TEST TEME-11
CodeTest inputs Test outcomes
Specification Expected outcomes
compare
Files/Tables/Panels
Specified Results
ExpectedValues
Interfaces/Databases
Corresponds to a System Test (Black-Box)
Outputs
SystemTest = Requirements-based Testing
TEST TEME-12
Test Object
Web Pages
Masks
Files
Databases
Parameters(Arguments)
Messages
Web Pages
Masks
Files
Databases
Reports(Results)
Messages
SYSTEM INPUTS SYSTEM OUTPUTS
Application Function
BLACK-BOX Method
(System Test)
Integration Testing Strategies
TEST TEME-13
BIG-BANG
TOP-DOWN
BOTTOM-UP
INSIDE-OUT
OUTSIDE-IN
BRANCH-WISE
Layered Software Systems
TEST TEME-14
Framework
GUI-Components
ErrorhandlingModule
PrintModule
XMLOutputModule
XMLInput
Module
SQLAcess
Module
OrderArt-3
Module
OrderArt-4
Module
OrderArt-1
Module
OrderArt-2
Module
Order Entry Processing System
Sample of a three level Program
ServiceLevel
LogicLevel
PresentationLevel
BIG-BANG Strategy
TEST TEME-15
GUIComponents
ErrorhandlingModule
OrderArt-3
Module
OrderArt-4
Module
OrderArt-1
Module
OrderArt-2
Module
PrintModule
XMLOutputModule
XMLInput
Module SQLAccessModule
TOP-DOWN Strategy
TEST TEME-16
Framework
GUIComponents
Stub StubStub
Stub StubStub Stub
BOTTOM-UP Strategy
TEST TEME-17
Framework
GUIComponents
ErrorhandlingModule
PrintModule
XMLOutputModule
XMLInput
Module
SQLAccessModule
Driver Driver DriverDriver Driver
Test-driven Development
INSIDE-OUT Strategy
TEST TEME-18
Driver
StubStub
OrderArt-3
Module
OrderArt-4
Module
OrderArt-1
Module
OrderArt-2
Module
OUTSIDE-IN Strategy
TEST TEME-19
Framework
GUIComponents
ErrorHandlingModule
PrintModule
XMLOutputModule
XMLInput
Module
SQLAccessModule
Driver DriverDriver Driver
Stub Stub Stub Stub
BRANCHWISE Strategy
TEST TEME-20
Framework
GUI-Components
StubPrint
ModuleStub
XMLInput
Module
SQLAccessModule
Stub StubOrderArt-1
ModuleStub
(Thread Testing)
One Transaction at a timeIs tested through to end
Straight thru testing
Comparison of Test Strategies
TEST TEME-21
Point of View TOP-DOWN BOTTOM-UP INSIDE-OUT OUTSIDE-IN BRANCH-WISE
Division of Labor poor good poor good poor
Controllability poor good poor medium medium
Test Aides required Stubs Drivers Stub&Drivers Stub&Drivers Stubs
Test Case Definition difficult easy medium easy medium
Test Visibility poor good medium good poor
Duration of Test medium long long short short
According to G. Myers: Software Reliability Principles and Practices
Principle SoftwareTest Approaches
TEST TEME-22
Control Flow Testing
Statement Coverage Branch Coverage Path Coverage
Data Flow Testing
Repreäsentative value coverage Boundary value coverage
Equivalence Class Coverage Data Relation Coverage Cause & Effect Analysis
State Coverage
Functional Testing
Result coverage = all desired results Rule coverage = all relevant rules Input oriented = all possible inputs
Control-Flow Testing (White-Box)
TEST TEME-23
Entry
Exit
(1) (2) (3) (5) (6)
(7) (8)
(9)
(10)
(4)
(11)
(12)
( ) = Control- branch
Paths =
1, 2, 3, 4, 6, 111, 2, 3, 5, 6, 111, 2, 7, 8, 10, 111, 2, 7, 9, 10, 111, 12,
Data-Flow Testing (Grey-box)
TEST TEME-24
ProgramInputs/Arguments Outputs/Results
Predicates
Representative ValuesBoundary ValuesEquivalence Classes
Discrete ValuesValue DomainsFunctional ResultsRelations
True/FalseCause/Effect
Functional Testing (Black-Box)
TEST TEME-25
Software SystemSystem SpecificationArgumentsRulesResults
InputsPathsOutputs
Testing Heuristics
TEST TEME-26
The specification of expected results is an essential part of the test.
Programmers are not in a position to test their own programs.
The results of each test case must be verified.
All exception conditions should be tested and confirmed.
It is not enough to confirm that a program does what it should do, but also to confirm that it does not do what it should not do.
Test cases must be repeatable.
Never assume there are no errors, software is always erroneoius.
Errors tend to cluster. If an error is found, other errors are likely to be found around it.
A good test case is a case with a high probability of discovering an error. The goal of testing is to find errors.
A test with no defined end criteria is no test.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.