cen 5070 – software v&v what is software testing © 2001-8, dr. e.l. jones
TRANSCRIPT
CEN 5070 – Software V&V
What is Software Testing
© 2001-8, Dr. E.L. Jones
9/2003 Testing Concepts 2
Purpose
This module presents the basic concepts of software testing. We present several aspects of software testing, and a give a generic lifecycle.
"… arbitrarily selected test set ... results in inefficient testing, leaving
some functions untested while performing redundant testing of others."Darlene Mackay, Quality Consultants Unlimited
9/2003 Testing Concepts 3
Agenda
• What is Testing
• Testing Lifecycle
• A Framework for Testing
• A Complete Example
9/2003 Testing Concepts 4
WHAT IS TESTING?
• Executing software to determine whether it meets its specification
• Executing a software element to discover defects or anomalies
• Executing software to evaluate its behavior in different environments
• Inspecting software element code to discover defects or anomalies.
9/2003 Testing Concepts 5
TESTING IS HARD
• Exhaustive testing is impossible
• Software is complex
• Specifying requirements is hard
• Testing is a fallible, human process
• Consequences of failure not always understood in advance
9/2003 Testing Concepts 6
WORKING SMART
• Test small pieces
• Use proven techniques
• Use tools to advantage
• Start early
• Cost of errors grows exponentially with time they go undetected
9/2003 Testing Concepts 7
Agenda
• What is Testing
• Testing Lifecycle
• A Framework for Testing
• A Complete Example
9/2003 Testing Concepts 8
A TESTING LIFECYCLE
Analysis
Design
Implementation
Execution
Evaluation
Specification
Test Strategy/Plan
Test Script, Data, Driver
Defect DataProblem Reports
Test Results
Test Cases
9/2003 Testing Concepts 9
TESTING vs DEVELOPMENT
• Work broken into stages
• Require similar skills
• Require management oversight
• Produce artifacts used to assess quality of the work
• Test artifacts are overhead, not part of delivered system.
9/2003 Testing Concepts 10
TEST ARTIFACTS
• Strategy/Plan (must know techniques)
• Test cases – central artifacts
• Test machinery to apply test cases
• Data showing results
• Reports that interpret test results
• List of problems/defects to be fixed.
9/2003 Testing Concepts 11
TESTING IS HIERARCHICAL
• Unit testing – smallest components
• Aggregate testing – assemblies of units
• Integration testing – combinations of assemblies
• System testing – system built from assemblies
9/2003 Testing Concepts 12
TESTING OBJECTIVES
• To demonstrate conformance
• To find defects
• To measure behavior in specific environments• Load / stress testing
• Performance testing
• To measure trustworthiness/reliability
9/2003 Testing Concepts 13
Agenda
• What is Testing
• Testing Lifecycle
• A Framework for Testing
• A Complete Example
9/2003 Testing Concepts 14
REQUIRED ELEMENTS OF A TESTING METHODOLOGY?
• Specification for the software
• Process for designing test cases
• Repeatable process for designing, running and evaluating tests
• Accountable artifacts of testing activities
• Economical use of human, time and computing resources
9/2003 Testing Concepts 15
Agenda
• What is Testing
• Testing Lifecycle
• A Framework for Testing
• A Complete Example
9/2003 Testing Concepts 16
BLACK-BOX TESTING
• Testing based on the specification rather than the implementation.
• Specification defines the expected response(s) to stimuli
SoftwareunderTest
Stimuli Response(s)
9/2003 Testing Concepts 17
BLACK-BOX TECHNIQUES
• Functional testing -- tests the behavior of the software.
• Boundary testing -- tests behavior at the lower/upper bounds of input values
• Random testing -- tests using randomly generated stimuli
• Intuitive (ad hoc) testing -- error guessing
9/2003 Testing Concepts 18
FUNCTIONAL TEST DESIGN METHODOLOGY
• Specification
• Identify behaviors
• Develop test cases
• Write test script
9/2003 Testing Concepts 19
EXAMPLE A(1) Specification
• Compute pay for an hourly employee, given the number of hours worked and the hourly pay rate. Compute overtime at 1.5 times hourly rate for hours in excess of 40.
SoftwareunderTest
Hours Pay
Rate
9/2003 Testing Concepts 20
EXAMPLE A(2) Identify Behaviors
• Case 1: No overtime (Hours <= 40)• Expect Pay = Hours * Rate
• Case 2: Overtime (Hours > 40)• Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)
9/2003 Testing Concepts 21
EXAMPLE A(3) Create Test Cases
• Case 1: No overtime (Hours <= 40)• Use Rate = 10, Hours = 30
• Expect Pay = Hours * Rate = 300
• Case 2: Overtime (Hours > 40)• Use Rate = 10, Hours = 50
• Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)
= 550
9/2003 Testing Concepts 22
EXAMPLE A(4) Write Test Script
StepStimuli Expected Response
Hours Rate Pay =
1
2
30
50 10
10 300
550
9/2003 Testing Concepts 23
A MORE COMPLEX EXAMPLE (B)
• Increased number of behaviors
• Use of decision table to document
behaviors
• Test case generation from decision table
9/2003 Testing Concepts 24
EXAMPLE B(1) Specification
• Compute pay for employee, given the number of hours worked and the hourly pay rate. For hourly employees (rate < 30), compute overtime at 1.5 times hourly rate for hours in excess of 40. Salaried employees (rate >= 30) are paid for exactly 40 hours.
9/2003 Testing Concepts 25
EXAMPLE B(2) Identify Behaviors
• Case 1: Hourly AND No overtime • (Rate < 30) & (Hours <= 40)• Expect Pay = Hours * Rate
• Case 2: Hourly AND Overtime • (Rate < 30) & (Hours > 40)• Expect Pay = 40*Rate+1.5*Rate*(Hours - 40)
• Case 3: Salaried (Rate >= 30) • Expect Pay = 40 * Rate
9/2003 Testing Concepts 26
DECISION TABLE
Condition
c1: Rate < 30 | Y Y N N
c2: Hours <= 40 | Y N Y N
Action
a1: Pay = Straight time | X
a2: Pay = Overtime | X
a3: Pay = Professional | X X
Columns defineBehaviors
9/2003 Testing Concepts 27
EXAMPLE B(3) Create Test Cases
• One test case per column of decision table
• Case 1: Hourly, No Overtime
• Case 2: Hourly, Overtime
• Case 3: Salaried, No Extra Hours
• Case 4: Salaried, Extra Hours
• Order the test cases by column
9/2003 Testing Concepts 28
EXAMPLE B(4) Write Test Script
StepStimuli Expected Response
Hours Rate Pay =
1
2
30
50 10
10 300
550
3 30 40 1600
4 50 40 1600
9/2003 Testing Concepts 29
RULES -- DECISION TABLES
Condition
c1: Rate < 30 | Y Y N N
c2: Hours <= 40 | Y N Y N
Action
a1: Pay = Straight time | X
a2: Pay = Overtime | X
a3: Pay = Professional | X X Use X to select action(s)
Elementaryconditions
Use 'Y', 'N', '-' or space
9/2003 Testing Concepts 30
NEXT TIME
• Test Case Generation Strategies
• Documenting the Test
• Specification Techniques