Introduction ToSoftware Testing
Brian Nielsen [email protected] of Embedded Software Systems
Aalborg University, Denmark
C SS10101110110101011011010101110111
CISS
Ariane 5 crash• An overrun exception stopped both replicated
position calculation computers (SRI’s) at time H0+36.7. Fail-stop was a wrong design decision
• Overrun caused by higher velocity in Ariane 5 compared to Ariane 4
• Overrun occured in a module which was irrelevant after take-off
• Simulations were not run with the new velocity (would have caught the failure)
CISS
Introducing, Detecting andRepairing Errors Liggesmeyer 98
CISS
What is testing?
CISS
TestingTesting:• to check the quality (functionality, reliability,
performance, …) of an (software) object
-by performing experiments-in a controlled way
• In avg. 10-20 errors per 1000 LOC• 30-50 % of development time and cost in embedded software
CISS
Testing Objectives• To identify as many errors as possible• To bring the software to an acceptable level of
quality• To determine risk of release• Perform testing efficiently, effectively and within
budget and time constraints• Compile a record of software errors for future
error prevention and correction (learning organization).
CISS
A Self-Assessment Test [Myers]
• “A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.”
• Write a set of test cases to test this program
CISS
Test cases for:
A Self-Assessment Test [Myers]
1. valid scalene triangle ?2. valid equilateral triangle ?3. valid isosceles triangle ?4. 3 permutations of previous ?5. side = 0 ?6. negative side ?7. one side is sum of others ?8. 3 permutations of previous ?
9. one side larger than sum ofothers ?
10. 3 permutations of previous ?11. all sides = 0 ?12. non-integer input ?13. wrong number of values ?14. for each test case: is
expected output specified ?15. check behaviour after
output was produced ?
What is a Test?
Software under Test
Test Data Output
Test Cases
Correct result?
Oracle
CISS
Testing vs. Model-Checking• examines physical
system• as finished as possible• in an environment close
to its real usage• no proof• very shallow test
sample• ad-hoc, error prone,
laborious ?!!
• examines an abstract model
• proof of correctness• cheap and early design
verification• only selected, user
specified properties, or generic
• state space explosion• requires formal models
Complementary Techniques!! Apply Both!!
CISS
Types of Testing
unit
integration
system
performancerobustness
functional behaviour
white box black box
Level
Accessibility
Aspect
usability
reliability
CISS
Quality-Characteristics (ISO-9126)• Functionality
• Suitability, accuracy, security, compliance, interoperability
• Reliability• maturity, fault tolerance, recoverability
• Usability• understandability, learnability, operability
• Efficiency• time behaviour, resource utilization
• Maintainability• Analysability, changeability, stability, testability
• Portability• Adaptability, intallability, conformance, replaceability
⇒ functional testing
⇒ reliability testing
⇒ usability testing
⇒ performance testing
⇒maintainability testing ??
⇒ portability testing ?
CISS
Whitebox Exampleint invoice (int x, int y) {int d1, d2, s;if (x<=30) d2=100;else d2=90;s=5*x + 10 *y;if (s<=200) d1=100;else if (s<=1000) d1 = 95;
else d1 = 80;return (s*d1*d2/10000);
}
ifd2=100 d2=90
ifd1=100
if
d1=95
d1=80
return
x>30
s>200
s>1000
Test Data Expected Output
X=5 Y=5 75
X=31 Y=10 229.5
X=30 Y=100 977.5
Test Cases
CISS
Black box testing
SUT
requirements
input
events
output
y
x
domain testing
CISS
Unit test
test driver
Unit under test
test stub
A
FE
B C D
G
Component Hierarchy
test stub
•C-function•Abstract Data Type•Object•Process / thread
•Function call•Method invocation•Message passing•Shared variables
Stub: a code fragment that simulates a missing unit
•Simulate•Pre-programmed return values•Simplified impl.
Driver: a code fragment that invokes the U-ut Simulate
•Supplies input data•Invokes Unit•Collects and compares results•~xUnit testing frame work
CISS
Unit-testing• Goal: Ensure that module functions correctly
according to module spec• Module interfaces are tested for proper
information flow.• Local data are examined to ensure that integrity is
maintained.• Boundary conditions are tested.• Basis path testing should be used.• All return values• All error handling paths should be tested.
• Drivers and/or stubs need to be developed to test incomplete software.
• Mostly white-box techniques.
CISS
Integration Testing• Goal: Systematic construction of program
structure and testing for interfacing errors• Loss of data/information across interfaces• Side-effects (shared data-structures)• Accumulation of errors (imprecision)
• Incremental• Consider required cost and thoroughness• Mostly blackbox, supplemented with white-
box tests to ensure coverage
CISS
TestA
TestA,B,C,DE,F,G
TestB
TestC
TestD
TestE
TestF
A
FE
B C D
G
Big-bang integration
Integration Order
1. Test all components in isolation2. Integrate all in one step
Component Hierarchy
- Stubs and drivers needed!- Difficult to localize cause of errors- Difficult to distinguish interface faults
TestG
CISS
TestE
TestA,B,C,DE,F,G
TestF
TestG
TestB,E,F
TestC
TestD,G
A
FE
B C D
G
Buttom-Up integration
Integration Order
1. Test lower levels first (utility routines)2. Implement driver (easy) to invoke module3. Use tested modules as stubs
Component Hierarchy
+No stubs needed!-Mundane low-level utilities tested first, but most
important top level modules tested late • Critical errors found late• Main design problems are at higher-
levels• Timing determined by high-level
modules
CISS
TestA,B,C,DE,F,G
TestA
TestA,B,C,D
A
FE
B C D
G
Top-Down integration
Integration Order
1. Test higher levels first (controlling modules)2. Implement stubs to deliver responses3. Use tested modules as drivers
Component Hierarchy
+No drivers needed!-Many stubs may be needeed-Stubs may be problematic to write
Must provide values as expected by testerMay require frequent alteration
CISS
TestE
TestA,B,C,DE,F,G
TestF
TestG
TestB,E,F
TestD,G
A
FE
B C D
G
Sandwich integration
Integration Order
1. Combination of B-up and T-Down2. Divide system into upper,
middle (target), and lower levels3. B-up in low-levels4. T-Down in upper levels
Component Hierarchy
+ combines B-up and T-Dn+ Integration possible early-Individual components not tested thoroughly
before integration, especially middle level
TestA
CISS
Testability = how easy can a program be tested
• operability - it operates cleanly• observability
• the results are easy to see• distinct output is generated for each input• incorrect output is easily identified
• controllability• processing can be controlled• tests can be automated & reproduced
• decomposability - software modules can be tested independently
• simplicity - no complex architecture and logic• stability - few changes are requested during testing• understandability - program is easy to understand
CISS
Verification & Validation• Verification :
Software correctly implements a specific function
“building the product right” (Boehm81)
• concerns development process
• intermediate products
• e.g. checking the testability of the specification
• Validation :
Tracable to user requirements
“building the right product” (Boehm81)
• whether final product meets requirements and needs
• e.g. by reviewing the specification
CISS
System Test (Pfleger)• Unit & integration: code implements design correctly• System testing: system does what customers expect
• Constructed software tested against established requirements during requirements analysis
• Validation criteria satisfied?• Are all user-visible functional, behavioral, performance
requirements satisfied?
Functiontest
Non-functionaltest
Acceptancetest
Installation/Production
test
Systemfunctional
requirements
RequirementsFor other
Quality aspects
•Performance•Stress / load•Reliability,•recoverability•Security •Installability,•…
Customer RequirementsSpecification
UserEnvironment
SystemIn
useIntegratedmodules
functioningsystem
accepted,validatedsystem
verifiedsystem
CISS
System test
• 2*CRTG (4 channels) 2 * 200 k€
CISS
Test Equipment
• Complete Type Approval Test System (3 M€)
CISS
Acceptance Test (Pfleger)• By customers,
• User’s understanding of requirements• Customer writes, conducts, and evaluates
acceptance tests• Functional Acceptance Test (FAT)• Production Acceptance Test (PAT)• Techniques
• Benchmark tests• Pilot tests ~ experimental, limited edition
– Alpha test: in-house, controlled environment, developer supervised
– Beta test: “live” test at customer sites, unsupervised• Parallel test: run concurrently with old system
CISS
Who Should Test?
• Developer• Understands the system• But, will test gently• And, is driven by
deadlines
• Independent tester• Must learn system• But, will attempt to break it• And, is driven by “quality”
CISS
Risk Based Test Strategy• Make best possible use of resources by
identifying and prioritizing quality aspects and subsystems• Higher risk ⇒ more testing• No risk ⇒ no testing
• Risk = chance of failure × damage
Use frequencyChance of error being present
ComplexityNew tools/techniquesInexperienced developers
Cost of repairLoss of market shareLegal claim
CISS
Testing• Dynamic testing is the process of executing a
program or system with the intent of finding error (Glenford Meyers’ definition)
• Static testing is any activity that aims at finding defects by inspecting, reviewing, walking through, and analyzing any static component of the software (code, documents, and models)
• Debugging is an ad hoc activity performed by individual developers to find and remove bugs from a program.
• Testing is a planned activity
CISS
Testing• Quality assurance activities by running code –
examines functionality in the form used by customers
• Software testing is a formal process carried out by a specialized testing team in which a software unit, several integrated software units or an entire software package are examined by running programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases.
CISS
Lifecycles & testing
CISSimplementation
code
detailed design
requirements
specification
acceptancetest
systemtest
moduletest
unit-test
V - Model
integrationtest
architecturespec
acceptance test spec
system test spec
integration test spec
module test spec
unit test spec
CISS
• Testing as phase of development process vs.
• Testing as a process itself• with its own phases
• in parallel with development process
Life Cycle : Testing as a Process
plan
ning
& co
ntro
l
prep
arat
ion
test
gene
ratio
n
test
exec
utio
n
com
plet
ion
quality report
implementationspecificationrequirements
T
L
OI
Techniques
Life-cyclefor testing
OrganizationInfrastructureand tools
CISS
Challenges of Testing• New embedded systems
• more functionality• increasingly advanced• faster time-to-market• higher quality
• Testing • more to be tested• more complicated• in less time• more thorough
• skilled developers and testers• advanced testing tools and techniques• well organized• using solid development method
CISS
• No realistic reliability models for software
• Moving implementation deadlines...... but fixed delivery deadlines
systemdelivery
startproject
code delivery =start test execution
Challenges of Testing
CISS
Challenges of Testing• Infinity of testing:
• too many possible input combinations -- infinite breadth• too many possible input sequences -- infinite depth• too many invalid and unexpected inputs
• Exhaustive testing never possible:• when to stop testing ?• how to invent effective and efficient test cases with high
probability of detecting errors ?
• Optimization problem of testing yield and invested effort• usually stop when time is over ......
CISS
Challenges of Testing• What is an effective method to measure coverage ?
• Regression testing: • very important• very boring and expensive• must be automated
• Bad specification or no specification at all• what do you test ?• test oracle problem
• Many operating environments and contexts
CISS
Challenges of Testing• Lack of appropriate tools
• Diversified fields• Experts dispersed, but also doesn’t talk
across application domain.• Tools are specialized, sells in low volume• Tools are expensive, • Tools are immature• No money available for test tools
CISS
Software Quality
CISS
Software Qualities
MaintainerGood Documentation
Readable CodeGood Design
Customer
Low CostPortability Increased
productivity
ReliabilityCorrectnessEfficiency
User
FunctionalityEase of useEase of learning
CISS
Costs of Poor Quality• Increased time to find and fix problems• Increased cost to distribute
modifications• Increased customer support• Product liability• Failure in the market place
CISS
Software Quality Assurance• Quality Control
• Process definition and standards • Formal technical review• Test planning and review
• Quality Assurance• Analysis and reporting• Measurement
CISS
Assuring that a software system meets a user’s needs
Verification vs. Validation:
• Verification: • “are we building the product right?”• The software should conform to its specification
• Validation:• “are we building the right product?”• The software should do what the user really
requires
CISS
static analysisstatic analysis
reviewingreviewinginspectioninspection
walk-throughwalk-throughdebuggingdebugging
certificationcertification
CMMCMM
quality controlquality controlrequirement managementrequirement managementsoftware processsoftware process
There is More than Testing
CISS
Summary
CISS
Some Testing Principles• Testing starts during the requirements phase• The programmer shall not be the tester• A test case specifies the test inputs
and the expected outputs• Test cases shall also cover invalid and unexpected inputs• Test cases shall test that the program does what it should do
and that it does not do what it should not do• Test cases shall be recorded for reuse• A test is successful when it detects an error
( but the project manager thinks differently ! )• No risk, no test
CISS
Lessons• Testing starts when project starts• Have a development approach
• How to develop (waterfall, cyclic, etc) : roles and activities• Clear expectations between testers and developers• Clarity about phases and tasks• Easier planning
• Test execution takes 3 steps• Test, fix, re-test
• Let the programmers find bugs• 80 % of bugs can be found by programmers (but they don’t)
• Every morning discuss the issues found• Testers create test report at end of the day• Next morning discuss which issues have to be solved• Programmer and tester fix issue together
CISS
Non-Structured testing• Start too late• Don’t organize and manage your test process• No test plan (V&V plan)• Unclear responsibilities• Conflicting interests• No fixed or stable specifications• Don’t learn
• from a defect• from a test• from statistics
• Use test tools in a chaotic environment• No configuration management and change control• No standard test procedures• Don’t educate• Do not provide career options
CISS
END