Software Failure: ReasonsSoftware Failure: Reasons

• Incorrect, missing, impossible requirements* Requirement validation.

• Incorrect specification * Specification verification.

• Faulty system design * Design reviews.

• Faulty software code* Unit testing.

• Faulty hardware infrastructure

Human FactorHuman Factor

Developers, analysts and users make ‘simple’ mistakes. These mistakes can be uncovered and corrected with the help review meetings, verification, validation and testing process.

However, mistakes made due to miscommunication or misunderstanding are much harder to notice.

Fault TypesFault Types

• AlgorithmicIncorrect process steps

• ComputationIncorrect formulas

• PrecisionRounding errors, small data types

• DocumentationIncorrect interface/API docs leads to program errors

• Load / Capacity• Timing / Synchronization• Performance

Fault Distribution (HP)Fault Distribution (HP)

Test Process OrganizationTest Process Organization

Test TypesTest Types

Unit testing: verifies that the component functions property (in a controlled environment) with the expected input.

Integration testing: ensures that the system components work together as they are supposed.

Function testing: determines if the functions described in the requirement are performed. Successful function test yields validated system.

Acceptance testing: performed by customers, system is checked against the requirements.

Deployment testing: the software is tested again once it is deployed in a new (clean) environment.

Views of the Test ObjectsViews of the Test Objects

Black Box: internal structure is unknown

- Difficult to test comprehensively

White Box/Clear Box: internal structure is known

- Possible to devise specific & boundary tests

Code ExaminationCode Examination

• Code ReviewIndependent / external reviewers examine and critique the code

• Code WalkthroughYou walk the reviewers through your code

• Code Inspectionchecks the code against predetermined list of concerns

+ Can uncover up to 90% of faults!! It is best to use well prepared reviewers that

are very well familiar with the product.- Does not uncover requirement or design faults.

Formal Proof TechniquesFormal Proof Techniques

1. Write assertions that describes code’s input and output conditions. These should be logical statements.

2. Draw a flow chart diagram of the code.3. Using the flow chart and assertions devise

a series of theorems that trace logical transformations performed by the code for various paths through the flow chart.

4. Prove the theorems.+ Develops understanding of the code+ Ensures algorithm correctness- Difficult to set up and carry out the proofs- Impractical / impossible for complex algorithms

Choosing Test CasesChoosing Test Cases

Ideally we want to test every possible input. When the inputs are limitless or impractical to enumerate we need to separate data into the equivalence classes (when the object is white box). Then it would be sufficient to test data drawn from the equivalence class.

Equivalence classes need to be picked out such that they cover all branches and switches in the code in order to maximize the code coverage.

If the code under investigation maintains state we must devise a sequence of tests such that all states of the finite state machine are tried.

We also need to test boundary conditions as well as invalid input.

Test Thoroughness ApproachesTest Thoroughness Approaches

Statement testing: every statement has to be tested.

Branch testing: every brunch must be process in both directions.

Path testing: every distinct path through the code must be tested.

Definition-use testing: every path from every definition (variable) to every use of that definition must be tested.

Unit Testing ComparisonUnit Testing Comparison

ReadRead

Chapter 8 from the white book.