software testing techniques introduction prof. rimantas Šeinauskas [email protected]...

Software testing techniquesSoftware testing techniques

IntroductionIntroduction

Prof. Rimantas Š[email protected]

Kaunas University of Technology

Introduction to Software testing techniques 2

ObjectivesObjectives

1. To introduce software testing and to explain its importance

2. To introduce structure of teaching content

3. To explain software testing definition


SWEBOKSWEBOK projectproject

• Software Configuration Management • Software Construction • Software Design • Software Engineering Infrastructure • Software Engineering Management • Software Engineering Process • Software Evolution and Maintenance • Software Quality Analysis • Software Requirements Analysis • Software Testing !!!!!!

Curricula

• Module place on the curricula

ContentContent

1. Introduction2. Verification & Validation ( Repeated from software

engineering according I Sommerville textbook)

3. Software testing ( Repeated from software engineering according I. Sommerville textbook)

4. Written quiz Topics of seminars:

5. Unit testing

6. Integration testing

7. Interface testing


ContentContent

8. Drivers and stubs

9. Sandwich strategy

10.Testing objectives

11. Regression testing

12. Equivalence partitioning

13.Testing based on specifications

14.Testing based on error guessing

15. Input Output analyses


ContentContent

16. Testing criteria based on control flow

17. Testing criteria based on data flow

18. Cycle testing

19. Mutation testing

20. Object-oriented software testing

21.Graphical user interface testing

22. Testing maturity model

23.Testing tools


ContentContent

24. Automated software testing

25. Software testability

26. Statistical testing

27. State testing of automata


LabsLabs



The “software crisis”The “software crisis”

• By the 1980’s, “quality” in software became a goal; SEI was born

• “software engineering” became popular• the life cycle was studied• software developers and testers began to work

together• by the 1990’s, testing tools became available


First: a riddle about testingFirst: a riddle about testing

• A mathematician, a physicist, and an engineer are told: “All odd numbers are prime.”– The mathematician says, “That’s silly, nine is a non-

prime odd number.– The physicist says, “Let’s see, 3 is prime, 5 is prime,

7 is prime -- looks like it’s true.”– The engineer says, “let’s see, 3 is prime, 5 is prime, 7

is prime, 9 is prime, 11 is prime -- looks like it’s true.”


Testing: our experienceTesting: our experience

Software to be tested

Test Case

Output


When to stop?When to stop?


Test Case

Output

Enough?No

Verification

Yes

Test Coverage

Test Case

Generation


Testing exampleTesting example

Sorting Program

Sorting Program

SPECS:Takes a list of

numbers; returns a

sorted list.

{1,3,2}

{1,2,3}

{3,2,3}

{}

{-1, -2}

Just a list.A sorted list.Repeated entry.Empty list.

Negative numbers.

Test Cases

{1, 2, 3}

Output

{1, 2, 3}

Output

{2, 3, 3}

Output

{}

Output

{-2, -1}

Output


Manual testingManual testing


Test Case

Output

Enough?No

Verification

Test Coverage

Test Case

Generation

Yes


Automated testingAutomated testing


Test Case

Output

CoverageEvaluator

Test CaseGenerator

VerifierOR

Test Oracl

e

TestSpecs


Software testing?Software testing?

• “The process of executing computer software in order to determine whether the results it produces are correct”, Glass

• “The process of executing a program with the intent of finding errors”, Myers

• “Program testing can be used to show the presence of bugs, but never their absence”, Dijkstra


Software testingSoftware testing??

• “The aim is not to discover errors but to provide convincing evidence that there are none, or to show that particular classes of faults are not present”, Hennell

• “Testing is the measure of software quality”, Hetzel • The process of operating a system or component under

specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.”

IEEE/ANSI,


The point of viewThe point of view

• “If our goal is to show the absence of errors, we will find very few of them”

• “If our goal is to show the presence of errors, we will discover a large number of them”


Testing effortsTesting efforts

• 50% Brooks/Myers, • 80% Arthur Andersons’ Director of testing in

North America


Testers/programmersTesters/programmers

• 1:5-10 Mainframes

i.e.,1 tester for every 5 to 10 developers • 2:3 Microsoft• 2:1 Lotus (for 1-2-3 Windows)• 1:2 Average of 4 major companies,

Microsoft, Borland, WordPerfect, Novell


Software testing problemsSoftware testing problems

• poorly expressed requirements• informal design techniques• nothing executable until coding stage• Huge input set: consider testing software that

categorises an exam grade: 101 inputs


Exhaustive software testingExhaustive software testing

• Exhaustive software testing is intractable• Even if all possible inputs could be identified, the

problem of identifying expected results is undecidable

• Weyuker has shown that there is no algorithm that can determine if a given statement, branch or path will be exercised!


Software testingSoftware testing definition definition

• Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the specified expected behavior.


DynamicDynamic

• This term means testing always implies executing the program on valued inputs. Static analysis techniques, such as peer review and inspection are not considered; nor is program execution on symbolic inputs, or symbolic evaluation;


Finite

• Clearly the tests are not selected literally from an infinite set of possible tests, but a very large finite one (the set of all bit strings shorter than some length). Nonetheless, for even simple programs, so many test cases are theoretically possible that exhaustive testing could require even years to execute. This is why in practice the number of tests can generally be considered infinite. However, the number of executions which can realistically be observed in testing must obviously be finite, and -more than this- it must be manageable. Indeed, testing always implies a trade-off between limited resources and schedules, and inherently unlimited test requirements: this conflict points to well known problems of testing, both technical in nature (criteria for deciding test adequacy) and managerial in nature (estimating the effort to put in testing);


SelectedSelected

• The many proposed test techniques essentially differ in how they select the (finite) test set, and testers must be aware that different selection criteria may yield largely different effectiveness. The problem of identifying the most suitable selection criterion under given conditions is still under research.


ExpectedExpected

• It must be possible to decide whether the observed outcomes of program execution are acceptable or not, otherwise the testing effort would be useless. The observed behavior may be checked against user's expectations (commonly referred to as testing for validation) or against a functional specification (testing for verification). The test pass/fail decision is referred to as the oracle problem, which can be addressed with different approaches.


Static techniquesStatic techniques

• Software testing is a means for quality evaluation. In the SQA chapter of the SWEBOK, activities and techniques for quality analysis are categorized into: static techniques (no code execution), and dynamic techniques (code execution). Both categories are useful. We would like to emphasize that static techniques are equally important for the purpose of building quality in a software product. Static techniques are covered into the SQA KA description


Testing activityTesting activity

• In the years, the view of software testing has evolved towards a more constructive attitude. Testing is no longer seen as an activity that starts only after the coding phase is complete, with the limited purpose of finding mistakes. Software testing is nowadays seen as an activity that encompasses the whole development process, and is an important part itself of the actual construction. Indeed, planning for testing starts since the early stages of requirement analysis, and test plans and procedures are systematically and continuously refined as the development proceeds. These activities of planning and designing tests constitute themselves a useful input to designers for highlighting potential weaknesses.


Error preventionError prevention

• As more extensively discussed in the SQA chapter, the right attitude towards quality is one of prevention: it is better to avoid problems, rather than repairing them. Testing must be seen as a means primarily for demonstrating that the prevention has been effective, but also for identifying anomalies in those cases in which, for some reason, it has been not. Finally, it is worth recognizing that even after a good testing, the system could still contain mistakes. The remedy to system failures after delivery is provided by (corrective) maintenance actions. The maintenance issues of software engineering are covered into the Software Evolution and Maintenance chapter of the SWEBOK


General ContentGeneral Content

• A.Testing Basic Concepts and Definitions• B.Test Levels• C.Test Techniques• D.Test-related Measures• E.Managing the Test Process• F.Test Tools.


A.A. Testing Basic Concepts and DefinitionsTesting Basic Concepts and Definitions

• When the outcome of a test execution diverges from the expected, a failure is observed, whose cause is a fault. These and other basic definitions within software testing area, as well as an introduction to the terminology, will be provided in next two chapters


B.B. Testing levelsTesting levels

• Software testing is usually performed at different levels along the development process. That is to say, the object of the test can vary: a whole program, part of it (functionally or structurally related), a single module. The latter subject is expanded in Part B Test Levels. This consists of two (orthogonal) subsections: B.1 lists the phases in which the testing of large software systems is traditionally subdivided. In B.2 testing for specific conditions or properties is instead considered, and is referred to as "Objectives of testing". Clearly not all listed types of testing apply to every system, nor has every possible type been listed, but those most generally applied.


Test objectiveTest objective

• Testing is conducted in view of a specific purpose (test objective), which is stated more or less explicitly, and with varying degrees of precision. Stating the objective in precise, quantitative terms allows for establishing control over the test process.


Testing aimsTesting aims

• One of testing aims is to expose failures (as many as possible), and many popular test techniques have been developed for this objective. These techniques variously attempt to "break" the program, by running identified classes of (deemed equivalent) executions: the leading principle underlying such techniques is being as much systematic as possible in identifying a representative set of program behaviors (generally in the form of subclasses of the input domain). However, a comprehensive view of the KA of testing as a means for quality must include other as important objectives for testing, e.g., reliability measurement, usability evaluation, contractor’s acceptance, for which different approaches would be taken. Note that the test objective varies with the test object, i.e., in general different purposes are addressed at the different levels of testing.


Testing criteriaTesting criteria

• The test objective determines how the test set is identified, both with regard to its consistency -how much testing is enough for achieving the stated objective?- and its composition -which test cases should be selected for achieving the stated objective?-, although usually the "for achieving the stated objective" part is left implicit and only the first part of the two italicized questions above is posed. Criteria for addressing the first question are referred to as test adequacy criteria, while for the second as test selection criteria


C. Test Techniques C. Test Techniques

• Several Test Techniques have been developed in the last two decades according to various criteria, and new ones are still proposed. "Generally accepted" techniques will be described from chapter three.


D.D. Testing measure Testing measure

• Sometimes, it can happen that confusion is made between test objectives and techniques. For instance, branch coverage is a popular test technique. Achieving a specified branch coverage measure should not be considered per se as the objective of testing: it is a means to improve the chances of finding failures (by systematically exercising every program branch out of a decision point). To avoid such misunderstandings, a clear distinction should be made between test measures which evaluate the test set, like measures of coverage, and those which instead provide an evaluation of the program under test, based on the observed test outputs, like reliability


E. Managing the Test ProcessE. Managing the Test Process

• Testing concepts, strategies, techniques and metrics need to be integrated into a defined and controlled process, which is run by people. The test process should support testing activities and testing teams, from test planning to test outputs evaluation, in such a way as to provide justified assurance that the test objectives are met cost-effectively. Issues relative to Managing the Test Process.


F. F. Test ToolsTest Tools

• Software testing is a very expensive and labor-intensive part of development. For this reason, tools are instrumental to support test activities. Moreover, in order to enhance cost-effectiveness ratio, a key issue has always been pushing test automation as much as possible. Existing tools and concepts related to automating the constituent parts of test process are addressed in different chapters

RECOMMENDED REFERENCES FOR SOFTWARE TESTING

• K. Beck, Test-Driven Development by Example, Addison-Wesley, 2002.

• P. C. Jorgensen, Software Testing: A Craftsman's Approach, second edition, CRC Press, 2004

• C. Kaner, J. Bach, and B. Pettichord, Lessons Learned in Software Testing, Wiley Computer Publishing, 2001.

• S. L. Pfleeger, Software Engineering: Theory and Practice, second ed., Prentice Hall, 2001

software testing techniques introduction prof. rimantas Šeinauskas [email protected]...

Documents