software reliability and testing - summer semester 2013 ... · software reliability and testing...
TRANSCRIPT
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
TestingSoftware Reliability and Testing
Summer semester 2013/2014Fundamentals of Testing
Barbara Russo
27 Feb 2014
1 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Table of Contents
1 Testing and reliability
2 Testing
2 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Testing and reliability
3 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Testing
According to IEEE definition:Software testing is the process of analyzing a software itemto detect the differences between existing and requiredconditions (that is, bugs) and to evaluate the features of thesoftware itemIn practice:Testing is the process of executing a program with the intentof finding an error
4 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Reliability
According to IEEE definition:Reliability is the ability of a system or component to performits required functions under stated conditions for a specifiedperiod of time
5 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Testing
Exhaustive testing is not feasible:if a failure is detected then the software is a failure software,but this does not imply that if no failure has been detectedthe software is correct
6 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Testing
Measuring and evaluating a software process may lead to theonly notion of “good” process we can define:a “good” process is correlated with the probability that nofailures occur in a given interval of time → Softwarereliability
7 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Testing
• Cost of software failures surpasses the cost of testing• Expenses for debugging and recall of product• Loss of customers? faith in company
• Effective test building• Uncovering as many defects as possible with minimum
amount of time and effort
• Is all of this feasible?
8 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Basic Questions
• When do testing start? When is it complete?
• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?
• How can we assess the readiness of a product forrelease?
• How can we control the quality of a product to release?
• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?
9 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Basic Questions
• When do testing start? When is it complete?
• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?
• How can we assess the readiness of a product forrelease?
• How can we control the quality of a product to release?
• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?
9 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Basic Questions
• When do testing start? When is it complete?
• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?
• How can we assess the readiness of a product forrelease?
• How can we control the quality of a product to release?
• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?
9 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Basic Questions
• When do testing start? When is it complete?
• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?
• How can we assess the readiness of a product forrelease?
• How can we control the quality of a product to release?
• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?
9 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
Basic Questions
• When do testing start? When is it complete?
• What particular techniques should be applied duringsoftware development to get acceptable quality atacceptable cost?
• How can we assess the readiness of a product forrelease?
• How can we control the quality of a product to release?
• How can the development process itself be improvedover the course of current and future projects toimprove products and make verification morecost-effective?
9 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
When do testing start? When is it complete?
• Testing activities are spread all over the developmentprocess
• There are several techniques at different stages of theprocess
• Different techniques involve different stakeholders
10 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
When do testing start? When is it complete?
Figure : Main analysis and testing activities through the softwarelife cycle. Source: Pezze and Young (reference book)
11 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
What techniques should be applied duringsoftware development?
• Feasibility study to understand and select appropriatetechniques
• Verification to lead development according torequirements
• Validation to check product against users’ expectiations
12 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
What techniques should be applied duringsoftware development?
• Techniques depend on quality, cost, scheduling,resources
• Not “one technique”• Effectiveness for different classes of fault to capture.• Applicability at different stages of the project (early
stage artefacts might not be tested with automatedtools)
• Differences in purpose. Techniques can have differentgoals (to understand the coverage or to detect faults)
• Trade-off cost and assurance, e..g, focus on key fewproperties
13 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
What techniques should be applied duringsoftware development?
• Feasibility study includes• Tentative architectural design to modularise work and
identify properties that can be verified in differentsubsystems
• Draft plan breaking projects into incremental deliveriespreliminary decisions about test and analysis techniques
14 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
How can we assess the readiness of a product forrelease?
Dependability properties:
• Availability
• Mean Time Between Failures
• Reliability
15 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
How can we assess the readiness of a product forrelease?
Examples of dependability measures:
• Availability: daily down time
• Mean Time Between Failures (MTBF): days betweentwo consecutive failures
• Reliability: number of functionalities that correctlyoperate
16 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
How can we assess the readiness of a product forrelease?
How:
• alpha testing: performed by users in a controlledenvironment. Capture operational profiles decide by theorganisation.
• beta testing: performed by users in a their ownenvironment. Capture different operational profiles
17 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
How can we control the quality of successivereleases?
• Major revisions also called “point releases.” The fullquality process is repeated including beta testing andregression testing.
• Small revisions also called “patch releases.” Revisionrelated to incorporate solution to fault fixing. Fasttesting often automated is foreseen. Subset ofregression testing.
• in Open Source Software we might have “milestones,”“release candidates,” and “major releases”
18 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
How can we control the quality of successivereleases?
19 / 19
Software Reliabilityand Testing
Barbara Russo
Outline
Testing andreliability
Testing
How can the development process itself beimproved over the course of current and futureprojects?
• Data collection across projects for modelling andprediction, e.g., faults and their severity or priority
• Personal Software Process (Humphrey)
20 / 19