seng 521 software reliability & testing
DESCRIPTION
SENG 521 Software Reliability & Testing. Best Practice SRE (Part 12c). Department of Electrical & Computer Engineering, University of Calgary B.H. Far ( [email protected] ) http://www.enel.ucalgary.ca/~far/Lectures/SENG521/12c/. Incremental Implementation. - PowerPoint PPT PresentationTRANSCRIPT
SENG521 (Fall 2002) [email protected] 1
SENG 521SENG 521Software Reliability & Software Reliability & TestingTesting
Best Practice SRE
(Part 12c)Department of Electrical & Computer Engineering, University of Calgary
B.H. Far ( [email protected] )http://www.enel.ucalgary.ca/~far/Lectures/SENG521/12c/
SENG521 (Fall 2002) [email protected] 2
Incremental ImplementationIncremental Implementation Most projects
implement the SRE activities incrementally.
A typical implementation sequence:
SENG521 (Fall 2002) [email protected] 3
Feasibility PhaseFeasibility Phase Activity 1:Activity 1: Define and classify failures
Define failure from customer’s perspective Group identified failures into a group of severity classes from
customer’s perspective Usually 3-4 classes are sufficient
Activity 2:Activity 2: Identify customer reliability needs What is the level of reliability that the customer needs? Who are the rival companies and what are rival products and what is
their reliability?
Activity 3:Activity 3: Determine operational profile Based on the tasks performed and the environmental factors
SENG521 (Fall 2002) [email protected] 4
Feasibility Phase: BenefitsFeasibility Phase: Benefits Activity 1 and 2:Activity 1 and 2: Define and classify failures,
identify customer reliability needs Benefits:Benefits: Release software at a time that meets customer
reliability needs but is as early and inexpensive as possible
Activity 3:Activity 3: Determine operational profiles Benefits:Benefits: Speed up time to market by saving test time,
reduce test cost, have a quantitative measure for reliability
SENG521 (Fall 2002) [email protected] 5
Requirements PhaseRequirements Phase Activity 4:Activity 4: Conduct trade-off studies
Reliability and functionality Reliability, cost and delivery date
Activity 5:Activity 5: Set reliability objectives based on Explicit requirement statements from a request for
proposal or standard document Customer satisfaction with a previous release or similar
product Capabilities of competition Trade-offs with performance, delivery date and cost Warranty, technology capabilities
SENG521 (Fall 2002) [email protected] 6
Requirements Phase: Requirements Phase: BenefitsBenefits
Activity 4:Activity 4: Conduct trade-off studies Benefits: Benefits: Increase market share by providing a software
product that matches better to customer needs
Activity 5:Activity 5: Set reliability objectives Benefits:Benefits: Release software at a time that meets customer
reliability needs but is as early and inexpensive as possible
SENG521 (Fall 2002) [email protected] 7
Design PhaseDesign Phase Activity 6:Activity 6: Allocate reliability among acquired software,
components, hardware and other systems Determine which systems and components are involved and how
they affect the overall system reliability
Activity 7:Activity 7: Engineer to meet reliability objectives Plan using fault tolerance, fault removal and fault avoidance
Activity 8:Activity 8: Focus resources based on operational profile Operational profile guides the designer to focus on features that are
supposed to be more critical
SENG521 (Fall 2002) [email protected] 8
Design Phase : BenefitsDesign Phase : Benefits Activity 6:Activity 6: Allocate reliability among acquired software,
components, hardware and other systems Benefits:Benefits: Reduce development time and cost by striking better
balance among components Activity 7:Activity 7: Engineer to meet reliability objectives
Benefits:Benefits: Reduce development time and cost with better design Activity 8:Activity 8: Focus resources based on operational profile
Benefits:Benefits: Speed up time to market by guiding development priorities, reduce development cost
SENG521 (Fall 2002) [email protected] 9
Implementation PhaseImplementation Phase Activity 8:Activity 8: Focus resources based on operational profiles
Develop more critical functions first in more detail
Activity 9:Activity 9: Manage fault introduction and propagation Practicing a development methodology; constructing modular
system; employing reuse; conducting inspection and review; controlling change
Activity 10:Activity 10: Measure reliability of acquired software, hardware and other systems
Certification test using reliability demonstration chart
SENG521 (Fall 2002) [email protected] 10
Implementation Phase : BenefitsImplementation Phase : Benefits Activity 8:Activity 8: Focus resources based on operational profile
Benefits:Benefits: Speed up time to market by guiding development priorities, reduce development cost
Activity 9:Activity 9: Manage fault introduction and propagation Benefits: Benefits: Maximize cost-effectiveness of reliability improvement
Activity 10:Activity 10: Measure reliability of acquired software, hardware and other systems
Benefits:Benefits: Reduce risks to reliability, schedule, cost from unknown software and systems
SENG521 (Fall 2002) [email protected] 11
System Test PhaseSystem Test Phase Activity 11:Activity 11: Determine operational profile used
for testing Decide upon critical operations Decide upon need of multiplicity of operational profile
Activity 12:Activity 12: Track testing progress and certify that reliability objectives are met Conduct feature test, regression test and performance and
load test Conduct reliability growth test
SENG521 (Fall 2002) [email protected] 12
System Test Phase : BenefitsSystem Test Phase : Benefits Activity 11:Activity 11: Determine operational profile used
for testing Benefits:Benefits: Reduce the chance of critical operations going
unattended, speed up time to market by saving test time, reduce test cost
Activity 12:Activity 12: Conduct reliability growth testing, track testing progress Benefits:Benefits: Know exactly what reliability the customer
would experience at different points in time if the software is released at those points
SENG521 (Fall 2002) [email protected] 13
Field Trial PhaseField Trial Phase Activity 13:Activity 13: Certify that reliability objectives and
release criteria are met Check accuracy of data collection Check whether test operational profile reflects field
operational profile Check customer’s definition of failure matches with what
was defined for testing the product
SENG521 (Fall 2002) [email protected] 14
Field Trial Phase : BenefitsField Trial Phase : Benefits Activity 13:Activity 13: Certify that reliability
objectives are met Benefits:Benefits: Release software at a time that meets
customer reliability needs but is as early and inexpensive as possible; verify that the customer reliability needs are actually met
SENG521 (Fall 2002) [email protected] 15
Post Delivery Phase /1Post Delivery Phase /1 Activity 14:Activity 14: Project post-release staff needs
Customer’s staff for system recovery; supplier’s staff to handle customer-reported failures and to remove faults
Activity 15:Activity 15: Monitor field reliability vs. objectives Collect post release failure data systematrically
Activity 16:Activity 16: Track customer satisfaction with reliability Survey product features with a sample customer set
SENG521 (Fall 2002) [email protected] 16
Post Delivery Phase /2Post Delivery Phase /2 Activity 17:Activity 17: Time new feature introduction by
monitoring reliability New features bring new defects. Add new features
desired by the customers if they can be managed without sacrificing reliability of the whole system
Activity 18:Activity 18: Guide product and process improvement with reliability measures Root-cause analysis for the faults Why the fault was not detected earlier in the development
phase and what should be done to reduce the probability of introducing similar faults
SENG521 (Fall 2002) [email protected] 17
Post Delivery Phase: Post Delivery Phase: BenefitsBenefits
Activity 14:Activity 14: Project post-release staff needs Benefits: Benefits: Reduce post-release costs with better planning
Activity 15-16:Activity 15-16: Monitor field reliability vs objectives, track customer satisfaction with reliability Benefits:Benefits: Maximize likelihood of pleasing customer with
reliability
SENG521 (Fall 2002) [email protected] 18
Post Delivery Phase: Post Delivery Phase: BenefitsBenefits
Activity 17:Activity 17: Time new feature introduction by monitoring reliability Benefits:Benefits: Ensure that software continues to meet
customer reliability needs in the field
Activity 18:Activity 18: Guide product and process improvement with reliability measures Benefits: Benefits: Maximize cost-effectiveness of product and
process improvements selected